143 Commits (6e2b7d25898f321bfd6f94de05d6ce934c785da9)

Author	SHA1	Message	Date
Owen Diehl	044d06015d	adds result cache key version comparison metrics (#7323) ... Followup to https://github.com/grafana/loki/pull/7300 which gives more metric visibility into the underlying errors when comparing cache key versions on the queriers vs query frontends.	3 years ago
Owen Diehl	2bd9c91564	Bugfix include cache generations on querier response (#7300) ... We check the cache number (incremented by deletes) after executing a query with the value returned by the queriers which executed it. If they’re different, we know we can’t safely cache. However, we’re never actually returning a number from the queriers. This means that as soon as this cache number is incremented by a delete, it’ll disable the results cache for that tenant completely. I suspect this has been a latent bug since we forked cortex (goes back to missing some code from this [PR](https://github.com/cortexproject/cortex/pull/2279) into cortex from @sandeepsukhani back in the day!). It just showed itself once we started using deletes.	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212) ... The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Sandeep Sukhani	7f298ff72f	avoid using bloomfilters for chunks in stats calls by avoiding duplicates (#7209) ... **What this PR does / why we need it**: Avoid using bloomfilters for chunks deduplication in tsdb `Stats` calls by avoiding fetching duplicate entries. The idea is to split and align queries by [ObjectStorageIndexRequiredPeriod](61794710a7/pkg/storage/config/schema_config.go (L47)) and make each split process chunks with a start time >= start time of the table interval. In other terms, table interval that contains start time of the chunk, owns it. For e.g. if the table interval is 10s, and we have chunks 5-7, 8-12, 11-13. Query with range 6-15 would be split into 6-10, 10-15. query1 would process chunks 5-7, 8-12 and query2 would process chunks 11-13. This check is not applied for the first split so that we do not eliminate any chunks that overlaps the original query intervals but starts at the previous table. For e.g. if the table interval is 10s, and we have chunks 5-7, 8-13, 14-13. Query with range 11-12 should process chunk 8-13 even though its start time <= start time of table we will query for index. The caveat here is that we will overestimate the data we will be processing if the index is not compacted yet since it could have duplicate chunks when RF > 1. I think it is okay since the Stats call is just an estimation and need not be accurate. Removing all the extra processing saves us quite a bit of CPU and memory, as seen from the benchmark comparison between the two implementations: ``` name old time/op new time/op delta IndexClient_Stats-10 187µs ± 0% 34µs ± 1% -82.00% (p=0.008 n=5+5) name old alloc/op new alloc/op delta IndexClient_Stats-10 61.5kB ± 4% 12.5kB ± 2% -79.69% (p=0.008 n=5+5) name old allocs/op new allocs/op delta IndexClient_Stats-10 1.46k ± 0% 0.48k ± 0% -67.28% (p=0.008 n=5+5) ``` **Checklist** - [x] Tests updated	3 years ago
李国忠	25cc9402b7	fix test increase timeout (#7022) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: increase timeout, fix test fail. ``` --- FAIL: TestInstanceFor (0.01s) downstreamer_test.go:205: Error Trace: downstreamer_test.go:205 downstreamer_test.go:256 Error: lock couldn't be acquired Test: TestInstanceFor FAIL ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Owen Diehl	30db6c120e	[TSDB] 600MB max per shard in query planning (#7004) ... This PR gives a little better understanding into shard query planning and targets a range of `(300MB,600MB]` per shard. Doubling this also allows us to halve query parallelism configs and still schedule the same amount of work in parallel, which should make for easier schema upgrades as the parallelism differences between TSDB and prior period configs will be less. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Dylan Guedes	15f8f42295	Adapt queryTimeout to be a per-tenant configuration (#6835) ... What this PR does / why we need it: * Add a new per-tenant query timeout * Adds new middleware to query calls. This new middleware will timeout requests based on the new per-tenant query timeout * Add span logs to query calls missing it * Deprecate `engine.timeout` configuration The motivation to change this configuration to be per-tenant instead of global is that it makes sense to alleviate the timeout for particular tenants or make it more strict for others. Especially useful for scenarios where the Loki client doesn't handle context cancellations correctly; in such scenarios, since Loki would still process expensive queries that were canceled, this allows one to have small timeouts for most tenants, which will help mitigate unnecessary work without having a timeout that is too short for important tenants.	3 years ago
Kaviraj Kanagaraj	693f9d7070	fix(stats_collector): Do not collect stats for failed requests. (#6878) ... * fix(stats_collector): Do not collect stats for failed requests. When any type of query fails, do not collect statistics. Rationale being when core http handler returns error, it's reponse will be nil and there won't be any statistics to collect and log. Fixes: 6231 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fix comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Kaviraj Kanagaraj	7e3d2433ed	embedded-cache: Bring `fifocache` and `groupcache` into single tent. (#6821) ... * chore(groupcache): Rename groupcache into memcache{distributed: true} 1. Introduced new config called memorycache config. 2. It has flag `distributed`. By enabling it, we use groupcache Going forward. We will bring fifocache into memcache config(with distributed:false) This PR is just a POC. To show how we can simplify in-built memory cache config and bring it into single config. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Memorycache -> EmbeddedCache * Bring fifocache into embeddedcache tent Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix edge case with enabling fifocache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix missing flags for embedded cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Warning message for fifocache config Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix listenport issue Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * idk. review remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks 1. update changelog and upgrade guide 2. s/Embeddedcache/EmbeddedCache/g 3. /groupcache/ handler -> /embedded-cache/ 4. some typos 5. Fix some cache prefixes Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fixing corresponding tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Support extra timeouts on embedded-cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unwanted single struct field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fix(stats-collector): Let cache.New() wrap stats collector for distributed cache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Deprecate max_size_items from fifocache Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks about the flag description Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/upgrading/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Owen Diehl	001a627f67	fix context propagation for child spans in shard resolver (#6853) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	f9d78e1748	[TSDB] Indexgateway Protobuf refactor (#6841) ... * queryrangebase definitions protos & aliases Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexgatewaypb to logproto Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * updates queryrange.proto for new definitions file Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * s/indexgatewaypb/logproto/g Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * renames to indexseries for clobbering reasons Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more indexgatewaypb removal compatibility work Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * proto fmt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * rebuild imports Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * goimports -s Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	094f8f3b20	[TSDB] fingerprint offsets is now correctly 16 byte aligned (#6630) ... * simpler shard factor calculations, disallows identity factor of 1 Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * log interval span as soon as it finishes Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * another shard bounds test Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexclient span logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more index span logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * correctly match single shard Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * shardedPostings test Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * shardedPostings maxOffset is properly exclusive Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * minor posting idiomatic-ness Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * store 16 byte aligned offsets for fingerprint table Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	2e348218e2	log addtl info about bytes/shard in planning (#6573) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	d6f50ca5e1	[TSDB] Use index sampling to determine shard factor (#6396) ... * use index sampling to determine shard factor * logging/tracing + better parallelism planning * queryrange downstreamer now checks max_query_parallelism * lint * handles interval, offset in tsdb planning and adds logging * fixes ns->ms confusion in index stats proto * handle zero shard value without panics * shardmapper will downstream a single unsharded query * uses concat expr with no shards to avoid query parsing errors * better logging * fixes wrong Size() method call and rounds to nearest KB when calculating chunk size * humanize bytes in log line * only adds defaultLookback to index sampling when interval is zero * removes comment * more logging * adjust through correctly * adds query length for index queries	4 years ago
Jesus Vazquez	db9dbe8e1d	Bump prometheus dependency (#6403) ... * Bump prometheus dependency We had to add a replace of prometheus in go mod because thanos is importing an older version and then replacing it. * Undo textparser.New variable mismatch * modtimevfs path update * Fix labels withlabels and withoutlabels usages WithoutLabels and WithLabels methods have dissappeared. Now labels.NewBuilder has to be used. * Fix RuleManager interface for Update() Update now accepts a new param called ruleGroupPostProcessFunc which is an optional function to be executed after rule group processing. In this case I've set it to nil in the usages i've found. * NewQueryRange now accepts QueryOpts The NewQueryRange function now accepts QueryOptions so I had to update all the calls. I've set the opts to nil since we're not using them. * gofmt	4 years ago
Owen Diehl	9218e4654b	TSDB/query-planning-groundwork (#6367) ... * begins building queryrange compatible types for index sampling * adds MatcherGroups() method to SampleExpr * MatcherGroups test + ShardResolver interface * index stats codec wiring * millisecond precision on index stats codec * lint * cleanMatchers in tsdb.IndexClient for consistency + adds catchall matcher if nil	4 years ago
Danny Kopping	36e0979cf5	Add cache usage statistics (#6317) ... * Adding cache statistics Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metrics to metrics.go Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Creating new stats context for use in metric queries middleware Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Clean up unnecessary log fields Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding stats tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * CHANGELOG entry Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Documenting function Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving CHANGELOG entry to appropriate section Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Implementing a stats collector cache wrapper to simplify stats collection If we keep the stats collection in pkg/storage/chunk/cache/instrumented.go, then any implementation that wraps it will cause the stats collected to be incomplete. For example: NewBackground(cacheName, cfg.Background, Instrument(cacheName, cache, reg), reg)) - the background cache requests are not collected Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Kaviraj Kanagaraj	1389857846	`metrics.go` support for metadata queries(labels and series) (#5971) ... * Add different functions to record labels and series queries * Add tests for logging series and labels queries * Add statistics to the Series and Labels proto response types * Plug in the statistics recorder on the query handler * Remove duplicate imports and cleanup comments * Remove the need to add usage statistics for metadata queries. * Add status based on returned error from the api * Fixing lint error by fixin return values order * Use explicit logger * Remove unused recorders * Add `total_entries` representing length of total result returned * Inject statscollector middleware for metadata queries * Fix query frontend stats middleware Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests with change in extra `totalEntriesReturned` field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * `make check-generated-files` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tripperware middleware for label values endpoint Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix bug with label extraction Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog entry Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix `total_entries` value for range query Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Kaviraj Kanagaraj	8b15632dd8	Fix(query-frontend): LokiCodec encode/decode works correctly with label values endpoint (#6084) ... When encode/decode `http.Request` into `querybase.Request` we re-create request.Path. Before this PR, both `/labels/` and `/labels/{name}/values` endpoint was decoding into same URL path `/labels/` before forwarding to the querier. So it always returns label names as response. This bug became visible after enabling all the middleware for `/labels/{name}/values` endpoint as well via #6072 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Kaviraj Kanagaraj	bb2e952d6f	Fix(query-frontend): `/label/<name>/values` endpoint to use right set of middlewares (#6072) ... * Fix(query-frontend): `/label/<name>/values` endpoint to use right middleware Previously, only `/label/` and `/labels` endpoint were using correct tripperware middleware on the query frontend(e.g: stats collector middleware) This PR fixes it for `/label/<name>/values` endpoint as well. This is useful to support `metrics.go` stats middleware for `label/{name}/values` endpoint as well. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove `unhandledpath` test Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unused test helper Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Susana Ferreira	6b6521f791	Add metrics to range mapper (#6030) ... * Refactor ShardingMetrics to MapperMetrics * Add mapper metrics to range mapper	4 years ago
Christian Simon	6c1a62d829	Ensure proto definitions are formatted [2/2] (#6000) ... * Update build image to 0.10.4 * Add a fmt target for proto defintions This uses buf to ensure the proto defintions are more consistently formatted. * Update formatted files	4 years ago
Travis Patterson	9cef86b162	Invalidate caches on delete (#5661) ... * Generate cache invalidation numbers in the delete store * Get cache generation numbers from the store on request * changlog * rename tombstones to something more meaningful * User invisible module * query frontend relies on a compactor to get the cache generation number * fix serialization * source -> name * lint errors * lint errors * log non-200 responses * add jsonnet changes * lint * review feedback * review feedback * client rename	4 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833) ... * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Susana Ferreira	0bce8d95a2	Split by range of instant queries (#5662) ... * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724)	4 years ago
Karsten Jeschkies	3d5a3936d1	Support limits in multi-tenant queries. (#5626) ... **What this PR does / why we need it**: This changes adds support for the the following limits in multi-tenant queries: - `MaxQuerySeries` - `QuerySplitDuration` - `MaxEntriesLimitPerQuery` - `MinShardingLookback` - `MaxQueryLength` - `MaxQueryParallelism` - `MaxQueriersPerUser` - `MaxCacheFreshness` It always uses the more restricted limit. That means the smallest `MaxQuerySeries` but biggest `QuerySplitDuration`. The split duration is special. Zero means disabled. Also, the global default resolver is the multi tenant resolver when multi-tenant queries are enabled. This will result HTTP 400 errors in case the user sends multiple tenants to endpoints that do not support it. **Which issue(s) this PR fixes**: **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [x] Tests updated - [x] Add an entry in the `CHANGELOG.md` about the changes. Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	4 years ago
Christian Haudum	95b885691d	Preserve interval parameter when splitting queries by time (#5622) ... * Preserve interval parameter when splitting queries by time Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add tests for interval param on Loki downstream requests Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add changelog entry for #5622 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Cyril Tovena	34f9649c6d	Remove more unused code (#5638) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	39101308f1	[Frontend] Log Result Cache (#5502) ... * Rough verion still require some fixes * with file saved * Finish off the code and logic Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactors metrics and hook the log cache middleware Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests, fixes and lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hash the key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/queryrange/log_result_cache.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/querier/queryrange/log_result_cache.go Co-authored-by: Christian Simon <simon@swine.de> * Fixes typo in variable namme Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use only one ok bool Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedbacks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * check array boundaries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Simon <simon@swine.de>	4 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539) ... * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	4 years ago
Susana Ferreira	ef49909430	Refactor sharding engine to downstream engine (#5546)	4 years ago
Cyril Tovena	9b6f17f168	Add the total number of subqueries to query statistics (#5397) ... * Add subqueries count Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	b3af2dbeef	Loki: include the component in metrics.go log line (#5379) ... * add a component key to our metrics.go line Signed-off-by: Edward Welch <edward.welch@grafana.com> * new computer... who dis Signed-off-by: Edward Welch <edward.welch@grafana.com> * one more lint Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Owen Diehl	2af3ca077e	Logql/parallel binop (#5317) ... * adds justification for keeping Downstreamer parallelism * loads binop legs in parallel * increases downstreamer default concurrency * astmapper spanlogger * always clone expr during mapping to prevent mutability bugs * Revert "astmapper spanlogger" This reverts commit 23f6b55c14. * cleanup + use errgroup	4 years ago
Dylan Guedes	cf419d1cb8	Implement `LegacySample`, `LegacyLabelPair` proto messages (#5301) ... * Add LegacySample and modify protos to use it. * [Skip this commit] Regenerate protos. * Modify codebase to use LegacySample instead of Sample. For reference, take a look at PR https://github.com/grafana/loki/pull/5151. * Implement LegacyLabelPair. * Use new Legacy implementations accordingly. * Use LegacySample in compat tests. - Since `compat.go` was forked from Cortex, we can reliably change all mentions from `Sample` to `LegacySample` there.	4 years ago
Owen Diehl	a88ee16534	better logic for when to shard wrt disabled lookback (#5303)	4 years ago
Christian Haudum	57b384df8f	Auto-expire old items from FIFO cache (#5148) ... * Remove validity from FIFO cache We can remove the validity setting of 1h in 2.4 for the chunks cache because it doesn't reduce memory usage and instead leads to valid chunks being ignore when querying the cache. Chunks are immutable so it doesn't make sense to not return them if they are present in the cache. Closes #4922 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add periodic task to prune old FIFO cache elements As described in #4921 this PR adds a periodic task that prunes expired items from the FIFO cache to free up memory. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Kaviraj Kanagaraj	f083aab09d	Remove `cortexpb` dependency from Loki packages. Use only `logproto` now. (#5264) ... * Getting rid of `cortexpb` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Get rid of `cortex/pkg/tenants` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * go mod vendor Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename some import alias Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257) ... * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Owen Diehl	91d837e79c	Refactor/remove global splitby (#5243) ... * cleans up internal limits struct, removes unnecessary logic * moves querier.split-queries-by-interval to limits, simplifies resulting code. * docs, changelog * flag supports model.duration instead of time.duration * lint and adds new error for deprecated field * reregisters queryrange flags * comment lint * ensures we validate queryrange config and simplifies embedded struct validate() * 0 still disables * simplify cache key generation and protect against divide by zero	4 years ago
Kaviraj Kanagaraj	7713ff78e2	Fix `cortexpb` -> `logproto` rename in some tests (#5231) ... * Fix `cortexpg` -> `logproto` rename in some tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix TimestampMs rename Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * queryrange test Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests on queryrangebase pacakage\ Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix some typos Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Callum Styan	f24b84c15a	[querier] Abort a query in limitedRoundTripper if parallelism is 0 (#5190) ... * Abort a query in limitedRoundTripper if parallelism is 0. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix failing tests related to fakeLimits change. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Change limitedRoundTripper parallelism error message. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Dylan Guedes	3f496e7336	Copy `cortex/pkg/cortexpb` package dependency into Loki (#5151) ... * Fork cortexpb into Cortex `pkg/logproto`. * Use `pkg/logproto` instead of Cortex `pkg/cortexpb`. * Rename `cortexpb` usage to `logproto`. * Fix encoding/decoding test. * Replace cortexpb usage in the ruler/base package. * Remove exemplars from logproto.	4 years ago
Susana Ferreira	50ca4d5186	Addition of cortex' queryrange tests (#5183)	4 years ago
Sandeep Sukhani	cd1537d06e	step align start and end time of the original query while splitting it (#5217) ... * step align start and end time of the original query while splitting it * also step align queries with step > split interval	4 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187) ... * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Sandeep Sukhani	dad7fca2fd	align metric queries by step and other queries by split interval (#5181)	4 years ago
Cyril Tovena	1a7614ff70	Avoid splitting large range vector aggregation. (#5172) ... * Reduce split by for large range vector Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/queryrange/split_by_interval.go Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * improve fn name. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
李国忠	2c20ce6857	[optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" (#4891) ... * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * Update pkg/storage/chunk/cache/instrumented.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * Update pkg/querier/queryrange/roundtrip.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update roundtrip.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago