373 Commits (35510ba4eb3615da78b15934bf6355b43378f499)

Author	SHA1	Message	Date
Ed Welch	35510ba4eb	Loki: only log "executing query" once per query in the frontend (#8337) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Irina	4cd1246b88	Logproto: Extract push.proto from logproto package to the separate module (#8259) ... Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Owen Diehl	db03ae2751	retry+backoff querier->qf response (#8285)	2 years ago
Danny Kopping	fd3d362e0c	Removing duplicated limits file and refactored references (#8210)	2 years ago
Owen Diehl	07487cd89d	fixes bug with queryIngesterWithin logic in asyncStore ingester stats… (#8145) ... Fixes a previous mistake in the logic calculating when to skip querying ingesters in the async store Statistics method. Notably `through.After` should be `through.Before` when skipping querying ingesters as that's when there's no overlap with the `query-ingesters-within` period: ```go // OLD CODE BELOW if a.queryIngestersWithin != 0 { // don't query ingesters if the query does not overlap with queryIngestersWithin. if !through.After(model.Now().Add(-a.queryIngestersWithin)) { // <----- should be through.Before return a.Store.Stats(ctx, userID, from, through, matchers...) } } ``` I discovered the problem while debugging querier OOMs during a boltdb-shipper -> tsdb migration and ultimately found this happened under the following circumstances: * Queries over high volumes of _recent_ data wouldn't query ingesters for index metadata * Without index metadata, we only checked storage metadata * There is a delay before we ship the index to storage, meaning we don't see it if ingesters are skipped * Calculating ideal shard factors without recent data for queries that only touch recent data _dramatically_ underestimates the desired shard factor * Queries aren't split enough and get scheduled onto too few querier replicas * They oom. We had some fun examples like a single replica trying to download 419,000 chunks/query To be clear, this is still a hypothesis, but a plausible one, especially after finding the boolean logic error this PR fixes. Instead of changing this one line, I took the opportunity to refactor this into a shared utility used by our other `GetChunkRefs` method which is already tested, ensuring the logic works as expected. I also added some more logging visibility into this code so we can understand what the difference is when querying statistics from storage vs ingesters. Finally, I added a helper to prepare our `Stats` objects to be logged which is now used in a few places.	2 years ago
Sandeep Sukhani	24deb6ed3b	fix bugs in logs results caching and its tests (#7925) ... **What this PR does / why we need it**: When a logs query results in an empty response, we cache it to avoid doing that query again and respond straight away with an empty response. However, we cache a single entry per time split interval with the query itself to keep things simple. For example, if the time split config for tenant `A` is `30m`, then queries for intervals `10m`-`20m` and `21m`-`25m` would have the same cache key. Here is roughly how cache hit is handled: * If the new query is within the cached query bounds, return empty results * If the start of new query is before the start time of the cached query, do a query from `newQuery.Start` to `cachedQuery.Start` * If the response of last query is also empty, set `cachedQuery.Start` = `newQuery.Start` * If the end of new query is after the end time of the cached query, do a query from `cachedQuery.End` to `newQuery.Start` * If the response of last query is also empty, set `cachedQuery.End` = `newQuery.End` * If we have changes in `cachedQuery.Start/End`, update it in the cache. The problem here is when we do queries to fill the gap, we sometimes do queries for the range outside of what the user requested and respond back without reducing the response to what the user requested. For example, if the cached query is from `21m`-`25m` and the user query is from `10m`-`15m`, we will query for the whole gap i.e `10m`-`21m`. If there are logs from `15m`-`21m` in the response, we will unexpectedly send it back to the user. This PR takes care of this issue by extracting the data and sending back only the user's requested logs. I have also found the tests for logs results cache were incorrect. They heavily use [mergeLokiResponse](e2842c69c5/pkg/querier/queryrange/codec.go (L1014)) for building the test data, but they were not setting the `limit` in test queries which was causing `mergeLokiResponse` to send back empty results. This means were were always comparing two empty results, which would always be the same. This PR also takes care of fixing it, and relevant changes to correct the test. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated	2 years ago
Christian Haudum	089ec1b05f	Fix various linter errors ... These changes have been generated using `make lint`. ```console $ golangci-lint --version golangci-lint has version 1.50.0 built from 704109c6 on 2022-10-04T10:25:07Z ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Danny Kopping	a4f306399a	Add store & cache download statistics (#7982)	2 years ago
Susana Ferreira	68a4d3271a	Fix documentation typos from #7916 (#7934) ... **What this PR does / why we need it**: Fix documentation typos from https://github.com/grafana/loki/pull/7916#pullrequestreview-1217192817 **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/pull/7916#pullrequestreview-1217192817 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916) ... **What this PR does / why we need it**: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](5550cd65ec/docs/sources/configuration/_index.md) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. **NOTE:** this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/83 **Special notes for your reviewer**: Files: * [docs/sources/configuration/index.template](5550cd65ec/docs/sources/configuration/index.template): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](c32e5d0acb/docs/sources/configuration/_index.md): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](5550cd65ec/tools/doc-generator) directory with documentation generation tool. **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Sandeep Sukhani	e9c93cd0f5	consider range and offset in queries while looking for schema config for query sharding (#7880) ... **What this PR does / why we need it**: We disable query sharding when a query touches multiple schema configs since they might have different sharding factors and might handle sharding differently. This check was done purely on the start and end time of the query without considering `range` and `offset`, which would change the length of the actual data being queried. Besides being incorrect, it also causes us to fail queries when migrating between tsdb and non-tsdb stores since both handle query sharding differently. For e.g., if the prev schema is `boltdb-shipper` and starting today, `tsdb` index is being used, so if we do a query `sum(rate({foo="bar"}[24h])` even with start and end within `tsdb` range, we will fail the query with error `incompatible index shard` if dynamic sharding goes with shard factor other than `32`(default shard factor in ingester for inverted index). The reason being the `range` here is `24h` which causes us to process data from previous schema as well. This PR takes care of the issue by factoring in `range` and `offset` in the queries while looking for schema config for query sharding. **Checklist** - [x] `CHANGELOG.md` updated	3 years ago
Robert Fratto	85392a9728	Update Prometheus dependency to latest release (v2.40.4) (#7826) ... Closes #7811, which is needed for Grafana Agent to update to v2.40 and add support for native histograms. I did not add support for native histograms to Loki, sorry :)	3 years ago
Christian Haudum	feaf9c3232	Log query string on retry alongside the error (#7834) ... **What this PR does / why we need it**: For better observability of query retries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Trevor Whitney	37b1c0fce0	guard against divide by 0 when splitting parallelism (#7831) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: We saw a spike in divide by zero panics in the code introduced in #7769. I was able to reproduce this error via a test that calculates `WeightedParallelism` with a start that's after the end. Not sure if this is possible, but we definitely saw this happening in our ops environment, so something is causing it, and the fix should guard against it in any case. **Special notes for your reviewer**: **Checklist** - [X] Tests updated Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
李国忠	63a57c7b5c	[bugfix] scheduler: Gracefully shutdown querier when using query-scheduler (#7735) ... **What this PR does / why we need it**: Gracefully shutdown querier when using query-scheduler This PR is an attempt to fix a bug that my loki cluster is unavailable for logql. The source code and ideas are from mimir of LGTM. Thanks mimir and pr author @pracucci **Which issue(s) this PR fixes**: Fixes #7722 **Special notes for your reviewer**: mimir PR https://github.com/grafana/mimir/pull/1756 https://github.com/grafana/mimir/pull/1767 **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	3 years ago
Sandeep Sukhani	89d81020ce	fix lint issues from PR 7804 (#7814) ... **What this PR does / why we need it**: I had enabled auto-merge in PR #7804, but somehow it still merged the PR without all the checks passing. This PR fixes the failing lint and tests.	3 years ago
Sandeep Sukhani	1410808ee9	use grpc for communicating with compactor for query time filtering of data requested for deletion (#7804) ... **What this PR does / why we need it**: Add grpc support to compactor for getting delete requests and gen number for query time filtering. Since these requests are internal to Loki, it would be good to use grpc instead of HTTP same as all the internal requests we do in Loki. I have added a new config for accepting the grpc address of the compactor. I tried having just the existing config and detecting if it is a grpc server, but it was hard to do it reliably, considering the different deployment modes we support. I think it is safe to keep it the same and eventually deprecate the existing config. **Checklist** - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated	3 years ago
Josef Johansson	b05f4fced3	Remove hardwired '127.0.0.1' values in default addresses (#7784) ... While running all-in-one Loki instance, it's not possible to listen on anything other than IPv4 since compactor_address is then set to http://127.0.0.1. This enables listening on only IPv6. Also if frontend_address, scheduler_ring and scheduler_address is not set, the default value is to connect to '127.0.0.1:<grpc_listen_port>'. Which means that grpc_listen_address is ignored. **What this PR does / why we need it**: It's not possible to use compactor while using an all-in-one instance and listening to IPv6. Let's not ignore grpc_listen_address to not confuse users. **Special notes for your reviewer**: As noted in the first commit, the default behavior is changed from defaulting to `127.0.0.1` to `localhost`. Not sure how major that change is though. My /etc/hosts says ``` $ grep localhost /etc/hosts 127.0.0.1 localhost ``` Signed-off-by: Josef Johansson <josef@oderland.se>	3 years ago
Danny Kopping	a63ad06509	Querier/Ruler: query blocker (#7785) ... Block malicious or expensive queries using a per-tenant runtime configuration.	3 years ago
Owen Diehl	22089415e8	Split parallelism across Period Configs (#7769) ... One of things we watch while updating non-TSDB period configs to TSDB period configs is the difference in query parallelism. TSDB dynamically shards queries into (potentially) much smaller units of work compared to the static shard factors uses prior. To account for this, we use much higher query parallelism configurations with TSDB period configs. This creates a potential problem when querying across `non-tsdb, tsdb` period boundaries: we may want a query parallelism of 512 for the tsdb portion but only 64 for the non-tsdb portion! However, we only had one limit to specify this per tenant, meaning this would be too high when querying non-tsdb periods or too low when querying tsdb ones. This PR * Introduces `tsdb_max_query_parallelism` (default `512`) to `limits_config` * Uses `tsdb_max_query_parallelism` and `max_query_parallelism` limits to find a better parallelism _per query_ by weighting the two respective configs by the proportion of each query spent on TSDB or non-TSDB period configurations. Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Dylan Guedes	ad2260aec2	Loki: Fix multitenant querying (#7708) ... **What this PR does / why we need it**: We recently broke multitenant querying because of recent changes to how timeouts working across Loki. This PR fixes this by: - Adapt the timeout wrapper to work with multitenant queries. It will take the shortest timeout across all given tenants - Adapt the query engine timeout assigning to work with multitenant queries. It will take the shortest timeout between all the tenants - Adapt query sharding to use smallest max query parallelism across given tenants - Add a functional test to ensure multitenant is behaving as expected Signed-off-by: DylanGuedes <djmgguedes@gmail.com> Signed-off-by: Mehmet Burak Devecí <mhmtbrkdvc@gmail.com> **Which issue(s) this PR fixes**: https://github.com/grafana/loki/issues/7696 **Special notes for your reviewer**: The regression was probably introduced by https://github.com/grafana/loki/pull/7555	3 years ago
Periklis Tsirakidis	e0a7b28a61	Add single compactor http client for delete and gennumber clients (#7453)	3 years ago
Sandeep Sukhani	020631ebac	add user-id transformer for logs results cache (#7581) ... **What this PR does / why we need it**: Adding transformer to results cache same as metrics cache done in PR #7542	3 years ago
Michel Hollands	16761723f4	Add way to override userId for caching (#7542) ... **What this PR does / why we need it**: Add a way to change the userId used in caching. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Ed Welch	45caba4459	Loki: Remove the bypass for "limited" queries (#7510) ... **What this PR does / why we need it**: Limited queries are queries which don't have a filter expression. Now all log queries will be handled by the LogFilterTripperware which will result in them being split by time (they previously were not) Signed-off-by: Edward Welch <edward.welch@grafana.com> **Which issue(s) this PR fixes**: We've found that very large timeframe `limited` queries will be sent to queriers and then ingesters and can stall out the read path because of the large time ranges. Splitting these by time avoids this problem by keeping any subquery limited in length to the `split_by_interval`. It's important to note that this will likely increase some extra work done by limited queries as more work for these will be parallelized and can result in extra data being processed. This will be the same as how filter queries are handled now, so it will be no worse than that. In fact this was an optimization based on the premise that it's advantageous not to split/shard limited queries but we are seeing this not be the case when limited queries are run over very large time ranges matching streams with huge volumes and combined with parsers like `| json` **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	3 years ago
Travis Patterson	c1bccac141	Results cache fix improvements (#7444) ... Move middleware to someplace more sensible and incorporate feedback missed in the review for this.	3 years ago
Travis Patterson	f2297d3d2a	Fix result cache misses on sharded queries (#7429) ... Headers weren't being propagated from query responses on sharded queries causing result cache misses. This PR: - Adds headers to `logproto.Result` so we can save response headers as they come back from queries - Metric-query results are turned into numbers well away from where responses are sent to the frontend: save all headers in the context so they can be pulled out when the final result is calculated - Propagate headers in the final responses	3 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340) ... I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	3 years ago
Owen Diehl	044d06015d	adds result cache key version comparison metrics (#7323) ... Followup to https://github.com/grafana/loki/pull/7300 which gives more metric visibility into the underlying errors when comparing cache key versions on the queriers vs query frontends.	3 years ago
Owen Diehl	2bd9c91564	Bugfix include cache generations on querier response (#7300) ... We check the cache number (incremented by deletes) after executing a query with the value returned by the queriers which executed it. If they’re different, we know we can’t safely cache. However, we’re never actually returning a number from the queriers. This means that as soon as this cache number is incremented by a delete, it’ll disable the results cache for that tenant completely. I suspect this has been a latent bug since we forked cortex (goes back to missing some code from this [PR](https://github.com/cortexproject/cortex/pull/2279) into cortex from @sandeepsukhani back in the day!). It just showed itself once we started using deletes.	3 years ago
Dylan Guedes	f9f1eddfc5	Loki: Improve query timeouts behavior (#7230) ... **What this PR does / why we need it**: Improves the different query-related timeouts behavior by fixing a bug and having better defaults. The changes are: - Fix the timeout middleware by creating a new request that will use the new WithDeadline context - `engine:timeout` was being ignored. This PR improves such scenario by using **the query timeout as a fallback** instead of the default `5min` for all cases	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212) ... The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Sandeep Sukhani	7f298ff72f	avoid using bloomfilters for chunks in stats calls by avoiding duplicates (#7209) ... **What this PR does / why we need it**: Avoid using bloomfilters for chunks deduplication in tsdb `Stats` calls by avoiding fetching duplicate entries. The idea is to split and align queries by [ObjectStorageIndexRequiredPeriod](61794710a7/pkg/storage/config/schema_config.go (L47)) and make each split process chunks with a start time >= start time of the table interval. In other terms, table interval that contains start time of the chunk, owns it. For e.g. if the table interval is 10s, and we have chunks 5-7, 8-12, 11-13. Query with range 6-15 would be split into 6-10, 10-15. query1 would process chunks 5-7, 8-12 and query2 would process chunks 11-13. This check is not applied for the first split so that we do not eliminate any chunks that overlaps the original query intervals but starts at the previous table. For e.g. if the table interval is 10s, and we have chunks 5-7, 8-13, 14-13. Query with range 11-12 should process chunk 8-13 even though its start time <= start time of table we will query for index. The caveat here is that we will overestimate the data we will be processing if the index is not compacted yet since it could have duplicate chunks when RF > 1. I think it is okay since the Stats call is just an estimation and need not be accurate. Removing all the extra processing saves us quite a bit of CPU and memory, as seen from the benchmark comparison between the two implementations: ``` name old time/op new time/op delta IndexClient_Stats-10 187µs ± 0% 34µs ± 1% -82.00% (p=0.008 n=5+5) name old alloc/op new alloc/op delta IndexClient_Stats-10 61.5kB ± 4% 12.5kB ± 2% -79.69% (p=0.008 n=5+5) name old allocs/op new allocs/op delta IndexClient_Stats-10 1.46k ± 0% 0.48k ± 0% -67.28% (p=0.008 n=5+5) ``` **Checklist** - [x] Tests updated	3 years ago
Ashwanth	3245625c40	ingester_querier: early exit queries on reaching quorum (#7172) ... Currently `forGivenIngesters` and its derivatives don't allow passing down context to the work function as a result of which we are not making use of the tracking functionality provided by `replicationSet.Do` to exit early and relieve pressure on the ingesters. This change ensures that Unary rpc calls receive ctx populated by the replication set tracker to exit early on reaching quorum or max failures. The same cannot be applied to stream rpc calls given how the code is structured at the moment where we are processing the streams outside of the work func	3 years ago
Travis Patterson	bc16450349	Revert "Hide duplicates caused by stream sharding (#7005)" (#7031) ... @owen-d had a suggestion that was dramatically dimpler. Revert this whole thing and the simplified PR will be forthcoming	3 years ago
李国忠	25cc9402b7	fix test increase timeout (#7022) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: increase timeout, fix test fail. ``` --- FAIL: TestInstanceFor (0.01s) downstreamer_test.go:205: Error Trace: downstreamer_test.go:205 downstreamer_test.go:256 Error: lock couldn't be acquired Test: TestInstanceFor FAIL ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Travis Patterson	a98b7ebaf4	Hide duplicates caused by stream sharding (#7005) ... As streams are sharded and resharded across different ingesters, the likelihood of duplicate logs being ingested increases dramatically. This removes duplicates at query time by dynamically removing the `__shard_stream__` label for both log and sample queries so the `stream hash` used by the `MergeEntryIterator` will match when doing comparisons for duplicates. The `stream hash` is created in either the `store` or `ingester instance` at query time so the labels are removed there. Additionally, the `MergeEntryIterator` has been modified to remove duplicates in a stream rather than just across streams. For completeness, `__shard_stream__` has also been removed from Label and Series Queries.	3 years ago
Owen Diehl	30db6c120e	[TSDB] 600MB max per shard in query planning (#7004) ... This PR gives a little better understanding into shard query planning and targets a range of `(300MB,600MB]` per shard. Doubling this also allows us to halve query parallelism configs and still schedule the same amount of work in parallel, which should make for easier schema upgrades as the parallelism differences between TSDB and prior period configs will be less. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Dylan Guedes	f6f49ec6bb	Return Querier/QueryTimeout YAML configuration. (#6972) ... What this PR does / why we need it: On PR #6835 I mistakenly removed the querier:query_timeout YAML configuration. This PR returns it and uses it if it is still configured while logging a warning message to alert users of the upcoming changes.	3 years ago
Dylan Guedes	15f8f42295	Adapt queryTimeout to be a per-tenant configuration (#6835) ... What this PR does / why we need it: * Add a new per-tenant query timeout * Adds new middleware to query calls. This new middleware will timeout requests based on the new per-tenant query timeout * Add span logs to query calls missing it * Deprecate `engine.timeout` configuration The motivation to change this configuration to be per-tenant instead of global is that it makes sense to alleviate the timeout for particular tenants or make it more strict for others. Especially useful for scenarios where the Loki client doesn't handle context cancellations correctly; in such scenarios, since Loki would still process expensive queries that were canceled, this allows one to have small timeouts for most tenants, which will help mitigate unnecessary work without having a timeout that is too short for important tenants.	3 years ago
Michel Hollands	25e8365d05	Improve error message (#6913) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Kaviraj Kanagaraj	693f9d7070	fix(stats_collector): Do not collect stats for failed requests. (#6878) ... * fix(stats_collector): Do not collect stats for failed requests. When any type of query fails, do not collect statistics. Rationale being when core http handler returns error, it's reponse will be nil and there won't be any statistics to collect and log. Fixes: 6231 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fix comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Kaviraj Kanagaraj	7e3d2433ed	embedded-cache: Bring `fifocache` and `groupcache` into single tent. (#6821) ... * chore(groupcache): Rename groupcache into memcache{distributed: true} 1. Introduced new config called memorycache config. 2. It has flag `distributed`. By enabling it, we use groupcache Going forward. We will bring fifocache into memcache config(with distributed:false) This PR is just a POC. To show how we can simplify in-built memory cache config and bring it into single config. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Memorycache -> EmbeddedCache * Bring fifocache into embeddedcache tent Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix edge case with enabling fifocache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix missing flags for embedded cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Warning message for fifocache config Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix listenport issue Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * idk. review remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks 1. update changelog and upgrade guide 2. s/Embeddedcache/EmbeddedCache/g 3. /groupcache/ handler -> /embedded-cache/ 4. some typos 5. Fix some cache prefixes Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fixing corresponding tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Support extra timeouts on embedded-cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unwanted single struct field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fix(stats-collector): Let cache.New() wrap stats collector for distributed cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Deprecate max_size_items from fifocache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks about the flag description Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/upgrading/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Owen Diehl	001a627f67	fix context propagation for child spans in shard resolver (#6853) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	57cec8e496	[TSDB] Ingester Index Sampling (#6852) ... * queryrangebase definitions protos & aliases Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexgatewaypb to logproto Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * updates queryrange.proto for new definitions file Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * s/indexgatewaypb/logproto/g Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * renames to indexseries for clobbering reasons Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more indexgatewaypb removal compatibility work Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto fmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * rebuild imports Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * goimports -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * query ingesters for index stats Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * compatibility during rollouts for idx-gw+ingester-querier post proto refactor Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto compatibility with old indexgateway pkg path Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * test compat Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * gofmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	f9d78e1748	[TSDB] Indexgateway Protobuf refactor (#6841) ... * queryrangebase definitions protos & aliases Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexgatewaypb to logproto Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * updates queryrange.proto for new definitions file Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * s/indexgatewaypb/logproto/g Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * renames to indexseries for clobbering reasons Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more indexgatewaypb removal compatibility work Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto fmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * rebuild imports Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * goimports -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704) ... * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	3 years ago
Travis Patterson	68141444f8	Introduce Groupcache (#6644) ... * implement groupcache * use mailgun groupcache * use memberlist * implement peer discovery with the ring * implement retrieval and storage * use a ticker * Correctly configuring chunk cache Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Enabling results cache Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing groupcache ring page Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * groupcache-poc: Add support for index cache. (#6535) * Hacked in prometheus metrics (#6544) Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * add http to addresses * always make a group for each cache * fix peer discovery * Fix groupcache config - Allow it to be enabled/disabled - Move all config to 'common' config - Always write to hot cache * Add http/2 support for groupcache (#6596) * Add groupcache latency metrics (#6619) * Add groupcache latency metrics * lint * lint * lint * Add groupcache test (#6620) * Add groupcache test * lint * Improve 'Fetch' allocs 11 -> 8 (#6623) * Move constants to the file where they're used * remove irrelevant comments Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Marco Pracucci	e7fb10f456	Loki: Update `grafana/dskit` and `weaveworks/common` (#6643) ... * Upgrade dskit and weaveworks/common Signed-off-by: Marco Pracucci <marco@pracucci.com> * 'go mod vendor' again Signed-off-by: Marco Pracucci <marco@pracucci.com>	3 years ago