287 Commits (a5921f791a1fac46dae1ce4c310599ae947e9c10)

Author	SHA1	Message	Date
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724)	3 years ago
Karsten Jeschkies	705baa39f4	Speed up re-labelling in multi-tenant queries. (#5663)	3 years ago
Karsten Jeschkies	3d5a3936d1	Support limits in multi-tenant queries. (#5626) ... **What this PR does / why we need it**: This changes adds support for the the following limits in multi-tenant queries: - `MaxQuerySeries` - `QuerySplitDuration` - `MaxEntriesLimitPerQuery` - `MinShardingLookback` - `MaxQueryLength` - `MaxQueryParallelism` - `MaxQueriersPerUser` - `MaxCacheFreshness` It always uses the more restricted limit. That means the smallest `MaxQuerySeries` but biggest `QuerySplitDuration`. The split duration is special. Zero means disabled. Also, the global default resolver is the multi tenant resolver when multi-tenant queries are enabled. This will result HTTP 400 errors in case the user sends multiple tenants to endpoints that do not support it. **Which issue(s) this PR fixes**: **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [x] Tests updated - [x] Add an entry in the `CHANGELOG.md` about the changes. Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	3 years ago
Christian Haudum	95b885691d	Preserve interval parameter when splitting queries by time (#5622) ... * Preserve interval parameter when splitting queries by time Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add tests for interval param on Loki downstream requests Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add changelog entry for #5622 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Cyril Tovena	59bd44a5df	Remove more chunkstore and schema version below v9 (#5650) ... * Remove more chunkstore and schema version below v9 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes remaining tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes remaining tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	34f9649c6d	Remove more unused code (#5638) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	df1a8e3632	Remove unused code from cortex (#5633) ... * Remove unused code from cortex Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the build Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flakky test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * update vendor Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Christian Simon	b6e6003724	Remove unused storage gateway references (#5627)	3 years ago
Cyril Tovena	39101308f1	[Frontend] Log Result Cache (#5502) ... * Rough verion still require some fixes * with file saved * Finish off the code and logic Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactors metrics and hook the log cache middleware Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests, fixes and lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hash the key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/queryrange/log_result_cache.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/querier/queryrange/log_result_cache.go Co-authored-by: Christian Simon <simon@swine.de> * Fixes typo in variable namme Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use only one ok bool Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedbacks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * check array boundaries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Simon <simon@swine.de>	3 years ago
Karsten Jeschkies	1deacceca9	Test multi-tenant context propagation. (#5620) ... **What this PR does / why we need it**: The original change injected the user id when the org id was required to fetch all data. **Which issue(s) this PR fixes**: Fixes grafana/loki-private#230 **Special notes for your reviewer**: The changes can be tested as follows 1. Change working directory to `tools/dev/loki-boltdb-storage-s3`. 2. Call `./compose-up.sh` to boot the cluster. 3. Query the logs with ``` curl -v -X GET -G "http://localhost:8007/loki/api/v1/query_range" -H "X-Scope-OrgID: 1|2" --data-urlencode 'query={compose_service=~"log-gen.*"}' | jq '.data.result[].stream' ``` It will should two streams with `__tenant_id: 1` and `__tenant_id_: 2` as part of the labels. **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Travis Patterson	88372c93cc	Query Filtering -- Querier (#5482) ... * Query Filtering -- Querier - Get deletes from the index gateway and attach them to store and ingester requests * lint * Fix test -- sometimes svc is nil * don't test for delete overlap on each selector * lint	3 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539) ... * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
Cyril Tovena	8f693e77b0	Update Prometheus to latest (#5545) ... * Update Prometheus to latest Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update promql.QueryTracker Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes querier test to use new grafana/regexp Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test with new textparse signature Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Kaviraj Kanagaraj	d6621f452f	Replace `PromQL` -> `LogQL` wordings in Querier (#5602)	3 years ago
Owen Diehl	d320873e82	Tsdb/sharding (#5600) ... * sharded support for GetChunkRefs and Series * sharded support for multi-index * linting	3 years ago
Susana Ferreira	ef49909430	Refactor sharding engine to downstream engine (#5546)	3 years ago
Owen Diehl	84c97ce43b	Revert "metrics: Add `user` label to `worker_inflight_queries` (#5485)" (#5534) ... This reverts commit 45d5f28e67.	3 years ago
Karsten Jeschkies	9e16ffbd3b	Support `SelectLogs` and `SelectSamples` in multi-tenant querier. (#5517) ... **What this PR does / why we need it**: This patch adds some functionality to the `MultiTenantQuerier` stub. `SelectLogs` and `SelectSamples` fan out into multiple calls, one for each tenant. The returned iterators are then merged back together and the proper tenant label is added. **Which issue(s) this PR fixes**: Fixes grafana/loki-private#232 **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [x] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Michel Hollands	d6d7de4edc	Remove extra param in call for inflightRequests metric (#5507) ... * Remove extra param Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add CHANGELOG.md entry Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove periods at end of line for older CHANGELOG.md entries Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Karsten Jeschkies	ae39676cce	Stub multi tenant querier. (#5490) ... **What this PR does / why we need it**: - Introduce Queries interface and Querier API. - Configure a multi-tenant querier that just passes requests through. **Which issue(s) this PR fixes**: Closes #231 **Special notes for your reviewer**: **Checklist** - [x] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Kaviraj Kanagaraj	45d5f28e67	metrics: Add `user` label to `worker_inflight_queries` (#5485) ... * metrics: Add `user` label to `loki_querier_worker_inflight_queries` * Add also `querier` label to it. * Remove `querier`	3 years ago
Ed Welch	4ca1fd0adb	set match_max_concurrent true by default (#5435) ... * set match_max_concurrent true by default Signed-off-by: Edward Welch <edward.welch@grafana.com> * add note to upgrade guide, fix up some of the existing notes Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Cyril Tovena	9b6f17f168	Add the total number of subqueries to query statistics (#5397) ... * Add subqueries count Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Kaviraj Kanagaraj	d970e737ab	Add more context to tailer-> ingester connect error. (#5394) ... NOTE: When client is tailing, Querier **has** to create grpc connections to **all** the ingesters. Sometimes when ingester is rolloing out and Querier unable to connect to ingester when client is tailing, then error message reported by Querier is confusing. This PR add some context to it. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Ed Welch	b3af2dbeef	Loki: include the component in metrics.go log line (#5379) ... * add a component key to our metrics.go line Signed-off-by: Edward Welch <edward.welch@grafana.com> * new computer... who dis Signed-off-by: Edward Welch <edward.welch@grafana.com> * one more lint Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Owen Diehl	2af3ca077e	Logql/parallel binop (#5317) ... * adds justification for keeping Downstreamer parallelism * loads binop legs in parallel * increases downstreamer default concurrency * astmapper spanlogger * always clone expr during mapping to prevent mutability bugs * Revert "astmapper spanlogger" This reverts commit 23f6b55c14. * cleanup + use errgroup	3 years ago
Dylan Guedes	cf419d1cb8	Implement `LegacySample`, `LegacyLabelPair` proto messages (#5301) ... * Add LegacySample and modify protos to use it. * [Skip this commit] Regenerate protos. * Modify codebase to use LegacySample instead of Sample. For reference, take a look at PR https://github.com/grafana/loki/pull/5151. * Implement LegacyLabelPair. * Use new Legacy implementations accordingly. * Use LegacySample in compat tests. - Since `compat.go` was forked from Cortex, we can reliably change all mentions from `Sample` to `LegacySample` there.	3 years ago
Owen Diehl	a88ee16534	better logic for when to shard wrt disabled lookback (#5303)	3 years ago
Cyril Tovena	3af7b7900e	Refactor HeapIterator into Merge and Sort Iterator. (#5281) ... * Refactor HeapIterator into merge and sort Iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Christian Haudum	57b384df8f	Auto-expire old items from FIFO cache (#5148) ... * Remove validity from FIFO cache We can remove the validity setting of 1h in 2.4 for the chunks cache because it doesn't reduce memory usage and instead leads to valid chunks being ignore when querying the cache. Chunks are immutable so it doesn't make sense to not return them if they are present in the cache. Closes #4922 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add periodic task to prune old FIFO cache elements As described in #4921 this PR adds a periodic task that prunes expired items from the FIFO cache to free up memory. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Kaviraj Kanagaraj	f083aab09d	Remove `cortexpb` dependency from Loki packages. Use only `logproto` now. (#5264) ... * Getting rid of `cortexpb` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Get rid of `cortex/pkg/tenants` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * go mod vendor Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename some import alias Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257) ... * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Owen Diehl	91d837e79c	Refactor/remove global splitby (#5243) ... * cleans up internal limits struct, removes unnecessary logic * moves querier.split-queries-by-interval to limits, simplifies resulting code. * docs, changelog * flag supports model.duration instead of time.duration * lint and adds new error for deprecated field * reregisters queryrange flags * comment lint * ensures we validate queryrange config and simplifies embedded struct validate() * 0 still disables * simplify cache key generation and protect against divide by zero	3 years ago
Dylan Guedes	4a70f8ec84	Copy `cortex/pkg/ingester/client` package dependency into Loki (#5164) ... * Fork ingester client. * Document magic numbers.	3 years ago
Eng Zer Jun	392fde1e27	refactor: use `T.TempDir` to create temporary test directory (#5253) ... * refactor: use `T.TempDir` to create temporary test directory The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: update CHANGELOG.md Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Kaviraj Kanagaraj	7713ff78e2	Fix `cortexpb` -> `logproto` rename in some tests (#5231) ... * Fix `cortexpg` -> `logproto` rename in some tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix TimestampMs rename Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * queryrange test Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests on queryrangebase pacakage\ Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix some typos Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Callum Styan	f24b84c15a	[querier] Abort a query in limitedRoundTripper if parallelism is 0 (#5190) ... * Abort a query in limitedRoundTripper if parallelism is 0. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix failing tests related to fakeLimits change. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Change limitedRoundTripper parallelism error message. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Dylan Guedes	3f496e7336	Copy `cortex/pkg/cortexpb` package dependency into Loki (#5151) ... * Fork cortexpb into Cortex `pkg/logproto`. * Use `pkg/logproto` instead of Cortex `pkg/cortexpb`. * Rename `cortexpb` usage to `logproto`. * Fix encoding/decoding test. * Replace cortexpb usage in the ruler/base package. * Remove exemplars from logproto.	3 years ago
Dylan Guedes	3f78d876ff	Copy `cortex/pkg/prom1` package dependency into Loki (#5196) ... * Fork cortex `prom1`. * Use forked `prom1` instead of Cortex one.	3 years ago
Susana Ferreira	50ca4d5186	Addition of cortex' queryrange tests (#5183)	3 years ago
Sandeep Sukhani	cd1537d06e	step align start and end time of the original query while splitting it (#5217) ... * step align start and end time of the original query while splitting it * also step align queries with step > split interval	3 years ago
Dylan Guedes	61ff29a574	Copy `cortex/pkg/chunk/purger` package dependency into Loki (#5169) ... * Fork cortex `pkg/chunk/purger`. * Replace `chunk/purger` usage. - Move away from Cortex implementation to instead use the new forked Loki implementation. * Remove unused code from `storage/chunk/purger`. * Remove unused vendored dependencies. * Remove `chunk/purger` implementation. - Loki already has the `compactor/deletion`, so the `chunk/purger` isn't really used.	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187) ... * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
李国忠	7fca88debd	[enhancement] querier : Add "query_memory_only" to make loki have option to rely only on memory availability like google monarch. (#5093) ... * [enhancement] querier : Add "query_memory_only" to make loki have option to rely only on memory availability like google monarch. * Update pkg/querier/querier.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * [enhancement] querier : Add "query_memory_only" to make loki have option to rely only on memory availability like google monarch. #5093 Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Sandeep Sukhani	dad7fca2fd	align metric queries by step and other queries by split interval (#5181)	3 years ago
Cyril Tovena	1a7614ff70	Avoid splitting large range vector aggregation. (#5172) ... * Reduce split by for large range vector Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/queryrange/split_by_interval.go Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * improve fn name. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
李国忠	2c20ce6857	[optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" (#4891) ... * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * Update pkg/storage/chunk/cache/instrumented.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * Update pkg/querier/queryrange/roundtrip.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update roundtrip.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	8c348614e3	Querier worker/inflight metrics (#5124) ... * refactors querier worker metrics to use metrics struct * metrics and logs for querier worker execution * loki_querier_worker_concurrency metric * alter log line * wires up prometheus.registerer args to prevent duplicate registration errors on tests	3 years ago
Dylan Guedes	fb457e5fc6	Copy `cortex/pkg/querier` package dependency into Loki (#5122) ... * Fork cortex querier and its dependencies. * Use forked querier instead of cortex implementation. * Fix files wrongly using cortex spanlogger. * Fix file wrongly using cortex gateway. * Remove storegateway from vendor. * Fix test. * Use seriesFunc const to fix lint. * Fix linting issues.	3 years ago
Sashank Agarwal	7ff1b45260	remove split_queries_by_interval validation (#5125)	3 years ago