apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Karsten Jeschkies	de2b0c6e9d	Rename per-request query limits. (#9011 ) What this PR does / why we need it: `maxQueryTime` is more in line with the other per-request limits we have. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Kaviraj Kanagaraj	cce3d3bed9	chore(format_query): Change response type to `application/json` (#9016 )	3 years ago
Owen Diehl	acb40ed40e	Eager stream merge (#8968 ) This PR introduces a specialized heap based datastructure to merge incoming log results in the frontend. Recently we've experienced an increase in OOMs on frontends due to logs queries which match lots of data. Sharded requests in loki split based on the amount of data we expect and some queries see thousands of sub requests. For log queries, we'll fetch up the `limit` from each shard, return them to the frontend, and merge. High shard counts * limit log lines, especially combined with large log lines (in byte terms) are accumulated on the frontend. Once they all are received, the frontend merges them. This creates opportunity for OOMs as it can hold up a lot of memory. This PR addresses one of these problems by eagerly accumulating responses as they're received and only retaining a total `limit` number of entries. There's still OOM potential due to race conditions between sub requests returning to the query-frontend and the query-frontend merging other sub requests, but this definitely improves the situation. I've been able to consistently run large limited queries that touch TBs of data (i.e. `{cluster=~".+"} \|= "a"`) that previously OOMed frontends. --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Periklis Tsirakidis	2ae80d7166	Align common instance_addr with memberlist advertise_addr (#8959 )	3 years ago
Owen Diehl	62403350a5	remove redundant splitby middleware (#8996 ) Found this double-copied line which a mistake. This PR removes one of them which won't change behavior (besides removing duplicate spans/etc).	3 years ago
Sandeep Sukhani	1fcefe8efa	index-gateway: fix failure in initializing index gateway when boltdb-shipper is not being used (#8755 ) What this PR does / why we need it: When only `tsdb` index is being used in the schema without configuring `boltdb-shipper` index store, we fail to initialize `index-gateway` since it also tries to build `boltdb-shipper` index client. This reference is used for serving `QueryIndex` calls which are for serving index from boltdb files. This PR fixes the issue by adding a check for the existence of `boltdb-shipper` in the schema config before initializing an index client. On the index gateway side, it adds a `failingIndexClient`, which errors when someone tries to query it. Checklist - [ ] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <masslessparticle@gmail.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Andrey Em	0f98b45d8d	Loki: Add UsageStatsURL to usagestats Config (#8779 ) What this PR does / why we need it: Reports used to be sent to a fixed URL usageStatsURL. Now this url can be configurable while using the old url as a default. Updated tests to use config for custom URLs instead of changing package variable. Which issue(s) this PR fixes: Fixes #6502 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Bill Franklin	b88aa08d89	Loki Canary: Enable insecure-skip-verify for all requests (#8869 ) What this PR does / why we need it: This supports cases where the cert cannot be trusted but we are not using mTLS, i.e. `loki-canary -tls=true -insecure=true` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Ed Welch	7264e0ef53	Loki: Remove global singleton of the tsdb.Store and keep it scoped to the storage.Store (#8928 ) What this PR does / why we need it: I made this change to fix an issue with the cmd/migrate tool which I'm also sneaking a few fixes on in this PR too. We had a global singleton of the tsdb.Store which was leading to the migrate tool reading from and writing to the `source` storage.Store because it created the singleton with an object store client only pointing to the source store. This PR changes the scoping of that singleton to not be global but instead local to each instance of storage.Store Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Ed Welch	b892cade6a	Loki: Fixes incorrect query result when querying with start time == end time (#8979 ) What this PR does / why we need it: In several places within Loki we need to determine if a query is a `range query` or `instant query`, this is done by checking to see if the start and end time are equal and the `step=0` The downstream handler was not checking for `step=0` and thus it incorrectly mapped a range query to an instant query when a query has a start time equal to and end time. There are a few other things at play here, mainly that we should really error anytime someone tries to run an instant query for logs which would have exposed this error much more easily. But that's something I'd like to handle in a different PR as it will be considered a breaking change depending on how we do it. This PR uses an existing function we have for testing the query type and addresses the issue found in #8885 Which issue(s) this PR fixes: Fixes #8885 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Ed Welch	edc6b0bff7	Loki: Add a limit for the [range] value on range queries (#8343 ) Signed-off-by: Edward Welch <edward.welch@grafana.com> What this PR does / why we need it: Loki does not currently split queries by time to a value smaller than what's in the [range] of a range query. Example ``` sum(rate({job="foo"}[2d])) ``` Imagine now this query being executed over a longer window of a few days with a step of something like 30m. Every step evaluation would query the last [2d] of data. There are use cases where this is desired, specifically if you force the step to match the value in the range, however what is more common is someone accidentally uses `[$__range]` in here instead of `[$__interval]` within Grafana and then sets the query time selector to a large value like 7 days. This PR adds a limit which will fail queries that set the [range] value higher than the configured limit. It's disabled by default. In the future it may be possible for Loki to perform splits within the [range] and remove the need for this limit, but until then this can be an important safeguard in clusters with a lot of data. Which issue(s) this PR fixes: Fixes #8746 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	3 years ago
Sandeep Sukhani	69d6ee293e	tsdb: use unique uploader name while building tsdb index filename, same as boltdb-shipper (#8975 ) What this PR does / why we need it: While building file names from `boltdb-shipper` and `tsdb`, we add the name of the uploader to it. To avoid collisions in filenames when running multiple ingesters with the same name, we also add a nanosecond precise timestamp to the ingester name, which is persistent as the uploader name in the active index directory. This is not being done for the tsdb index built from ingesters which can cause collisions in filenames when running multiple ingesters with the same name. I have moved the code to manage the uploader name to the index shipper config and used the same to share it with tsdb.	3 years ago
Dylan Guedes	9159c1dac3	Loki: Improve spans usage (#8927 ) What this PR does / why we need it: - At different places, inherit the span/spanlogger from the given context instead of instantiating a new one from scratch, which fix spans being orphaned on a read/write operation. - At different places, turn spans into events. Events are lighter than spans and by having fewer spans in the trace, trace visualization will be cleaner without losing any details. - Adds new spans/events to places that might be a bottleneck for our writes/reads.	3 years ago
Danny Kopping	de84737368	Fix fetched chunk from store size in metric (#8971 ) What this PR does / why we need it: The `Size()` function on `chunk.Chunk` calls out to `func (ChunkRef) Size() int` in the proto, which doesn't actually include the size of the chunk data! We need to call `chunk.Data.Size()` for that.	3 years ago
Suruthi-G-K	e7b0d4d26e	feat(cos): Support authentication with trusted profiles (#8939 ) What this PR does / why we need it: Add trusted profile authentication in COS client Which issue(s) this PR fixes: Fixes NA Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com>	3 years ago
Danny Kopping	6bb8a071ca	Querier: block query by hash (#8953 ) What this PR does / why we need it: Using the [query blocker](https://grafana.com/docs/loki/next/operations/blocking-queries/) can be unergonomic since queries can be long, require escaping, or hard to copy from logs. This change enables an operator to block queries by their hash.	3 years ago
Periklis Tsirakidis	1bcf683513	Expose optional label matcher for label values handler (#8824 )	3 years ago
Salva Corts	28a7733ede	Rename config for enforcing a minimum number of label matchers (#8940 ) What this PR does / why we need it: Followup PR for https://github.com/grafana/loki/pull/8918 renaming config. See https://github.com/grafana/loki/pull/8918/files#r1151820792. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki-private/issues/699 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Christian Haudum	abc0fd26d2	Enforce per tenant queue size (#8947 ) What this PR does / why we need it: Prior to the changes in https://github.com/grafana/loki/pull/8752 the max queue size per tenant was enforced by the size of the buffered channel to which a request was enqueued. However, since we have hierarchical queues, every sub-queue has the same channel capacity as the root (tenant) queue. Therefore the total queue size per tenant needs to be tracked separately, so requests can be rejected when the max queue size is reached. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	0adedfa689	Revert high cardinality metric in scheduler (#8946 ) What this PR does / why we need it: The metrics was useful for initial testing of the new scheduler queue implementation but yields high cardinality metrics, which is not desired. Also, the metric does not add additional value beyond the initial testing phase. Special notes for your reviewer: The metric was introduced with commit `cba31024d4` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Salva Corts	d421feafe6	Log when returning query-time limit (#8938 ) What this PR does / why we need it: At https://github.com/grafana/loki/pull/8727 we introduced various limits that can now be configured at query time. We always compare the value of the limit configured at query time with the value set on the overrides for the tenant or the default if not configured (aka original); applying the most restrictive one. If the most restrictive is the original value or the limit is not configured at query-time, we print the following debug message: `9e1846c47a/pkg/util/querylimits/limiter.go (L43)` It will be printed many times: every time the query is not configured at query-time, as well as every time the original value is more restrictive. Moreover, this log message lacks some useful information such as the original value, the query-time value, the tenant ID and the limit name. This PR fixes this by removing the debug message above and printing a new debug message only if we return the query-time limit. This new log also prints the tenant ID, the limit name, as well as the original and the query-time limit values. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/8932 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Travis Patterson	ee045312a9	Automatically Reorder Pipeline Filters (#8914 ) This PR moves Line Filters to be as early in a log pipeline as possible. For example: `{app="foo"} \| logfmt \|="some stuff"` becomes `{app="foo"} \|="some stuff" \| logfmt` Any LineFilter after a `LineFormat` stage will be moved to directly after the nearest `LineFormat` stage benchmarks: ``` goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/logql/syntax cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz │ reorder_old.txt │ reorder_new.txt │ │ sec/op │ sec/op vs base │ ReorderedPipeline-8 2104.5n ± 3% 173.9n ± 2% -91.74% (p=0.000 n=10) │ reorder_old.txt │ reorder_new.txt │ │ B/op │ B/op vs base │ ReorderedPipeline-8 336.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10) │ reorder_old.txt │ reorder_new.txt │ │ allocs/op │ allocs/op vs base │ ReorderedPipeline-8 16.00 ± 0% 0.00 ± 0% -100.00% (p=0.000 n=10) ```	3 years ago
Salva Corts	45775c82f7	Implement `RequiredNumberLabels` query limit (#8918 ) What this PR does / why we need it: As pointed out in https://github.com/grafana/loki/pull/8851, some queries can impose a great workload on a cluster by selecting too many streams. Similarly to the `RequiredLabels` limit introduced at https://github.com/grafana/loki/pull/8851, here we add a new limit `RequiredNumberLabels` to require queries to specify at least N label. For example, if the limit is set to 2, then the query should contain at least 2 label matchers. This limit can be configured per tenant and at query time. ![image](https://user-images.githubusercontent.com/8354290/228271398-4b9bcc49-f539-4e94-86c1-071e519a30a9.png) Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki-private/issues/699 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Salva Corts	ee69f2bd37	Split index request in 24h intervals (#8909 ) What this PR does / why we need it: At https://github.com/grafana/loki/pull/8670, we applied a time split of 24h intervals to all index stats requests to enforce the `max_query_bytes_read` and `max_querier_bytes_read` limits. When the limit is surpassed, the following message get's displayed: ![image](https://user-images.githubusercontent.com/8354290/227960400-b74a0397-13ef-4143-a1fc-48d885af55c0.png) As can be seen, the reported bytes read by the query are not the same as those reported by Grafana in the lower right corner of the query editor. This is because: 1. The index stats request for enforcing the limit is split in subqueries of 24h. The other index stats rquest is not time split. 2. When enforcing the limit, we are not displaying the bytes in powers of 2, but powers of 10 ([see here][2]). I.e. 1KB is 1000B vs 1KiB is 1024B. This PR adds the same logic to all index stats requests so we also time split by 24 intervals all requests that hit the Index Stats API endpoint. We also use powers of 2 instead of 10 on the message when enforcing `max_query_bytes_read` and `max_querier_bytes_read`. ![image](https://user-images.githubusercontent.com/8354290/227959491-f57cf7d2-de50-4ee6-8737-faeafb528f99.png) Note that the library we use under the hoot to print the bytes rounds up and down to the nearest integer ([see][3]); that's why we see 16GiB compared to the 15.5GB in the Grafana query editor. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/8910 Special notes for your reviewer: - I refactored the`newQuerySizeLimiter` function and the rest of the _Tripperwares_ in `rountrip.go` to reuse the new IndexStatsTripperware. So we configure the split-by-time middleware only once. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: https://grafana.com/docs/loki/latest/api/#index-stats [2]: https://github.com/grafana/loki/blob/main/pkg/querier/queryrange/limits.go#L367-L368 [3]: https://github.com/dustin/go-humanize/blob/master/bytes.go#L75-L78	3 years ago
Travis Patterson	163fd9d8af	Short circuit parsing when label matchers are present (#8890 ) This PR makes parsers aware of any downstream label-matcher stages at parse time. As labels are parsed, if one has a matcher, the matcher is checked at parse time. If the label does not match it's matcher, parsing is halted on that log line. ex 1: consider the log: `foo=1 bar=2 baz=3` And the query `{} \| logfmt \| bar=3` When `bar` is parsed it is immediately checked against it's matcher. The match fails so we the parser never spends time parsing the rest of the line. ex 2: consider the log: `foo=1 baz=3 bletch=4` And the query `{} \| logfmt \| bar=3` `bar` is never seen in the log so the whole line is parsed. Benchmarks: ``` │ parsers__old_2.txt │ parsers__new_3.txt │ │ sec/op │ sec/op vs base │ _Parser/json/inline_stages-8 3413.5n ± 5% 766.4n ± 4% -77.55% (p=0.000 n=10) _Parser/jsonParser-not_json_line/inline_stages-8 101.5n ± 6% 103.1n ± 8% ~ (p=0.645 n=10) _Parser/unpack/inline_stages-8 383.8n ± 4% 388.0n ± 9% ~ (p=0.954 n=10) _Parser/unpack-not_json_line/inline_stages-8 13.30n ± 2% 13.11n ± 1% ~ (p=0.247 n=10) _Parser/logfmt/inline_stages-8 2105.5n ± 16% 727.7n ± 4% -65.44% (p=0.000 n=10) _Parser/regex_greedy/inline_stages-8 4.220µ ± 4% 4.175µ ± 4% ~ (p=0.739 n=10) _Parser/regex_status_digits/inline_stages-8 319.8n ± 5% 326.4n ± 8% ~ (p=0.481 n=10) _Parser/pattern/inline_stages-8 185.2n ± 7% 154.2n ± 3% -16.74% (p=0.000 n=10) │ parsers__old_2.txt │ parsers__new_3.txt │ │ B/op │ B/op vs base │ _Parser/json/inline_stages-8 280.00 ± 0% 64.00 ± 0% -77.14% (p=0.000 n=10) _Parser/jsonParser-not_json_line/inline_stages-8 16.00 ± 0% 16.00 ± 0% ~ (p=1.000 n=10) _Parser/unpack/inline_stages-8 80.00 ± 0% 80.00 ± 0% ~ (p=1.000 n=10) _Parser/unpack-not_json_line/inline_stages-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) _Parser/logfmt/inline_stages-8 336.00 ± 0% 74.00 ± 0% -77.98% (p=0.000 n=10) _Parser/regex_greedy/inline_stages-8 193.0 ± 1% 192.0 ± 1% ~ (p=0.656 n=10) _Parser/regex_status_digits/inline_stages-8 51.00 ± 0% 51.00 ± 0% ~ (p=1.000 n=10) _Parser/pattern/inline_stages-8 35.000 ± 0% 3.000 ± 0% -91.43% (p=0.000 n=10) │ parsers__old_2.txt │ parsers__new_3.txt │ │ allocs/op │ allocs/op vs base │ _Parser/json/inline_stages-8 18.000 ± 0% 4.000 ± 0% -77.78% (p=0.000 n=10) _Parser/jsonParser-not_json_line/inline_stages-8 1.000 ± 0% 1.000 ± 0% ~ (p=1.000 n=10) _Parser/unpack/inline_stages-8 4.000 ± 0% 4.000 ± 0% ~ (p=1.000 n=10) _Parser/unpack-not_json_line/inline_stages-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) _Parser/logfmt/inline_stages-8 16.000 ± 0% 6.000 ± 0% -62.50% (p=0.000 n=10) _Parser/regex_greedy/inline_stages-8 2.000 ± 0% 2.000 ± 0% ~ (p=1.000 n=10) _Parser/regex_status_digits/inline_stages-8 2.000 ± 0% 2.000 ± 0% ~ (p=1.000 n=10) _Parser/pattern/inline_stages-8 2.000 ± 0% 1.000 ± 0% -50.00% (p=0.000 n=10) ``` --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Athira Varghese	ffb961c439	feat(storage): add support for IBM cloud object storage as storage client (#8826 ) What this PR does / why we need it: Add support for IBM cloud object storage as storage client Which issue(s) this PR fixes: Fixes NA Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated --------- Signed-off-by: Shahul <shahulsonhal@gmail.com> Co-authored-by: Aman Kumar Singh <amankrsingh2110@gmail.com> Co-authored-by: Suruthi-G-K <shruthi.suruthi@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com> Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Tareq Al-Maamari <tariq.mamari@gmail.com>	3 years ago
Salva Corts	336e08fc4b	Salvacorts/max querier size messaging (#8916 ) What this PR does / why we need it: In https://github.com/grafana/loki/pull/8670 we introduced a new limit `max_querier_bytes_read`. When the limit was surpassed the following error message is printed: ``` query too large to execute on a single querier, either because parallelization is not enabled, the query is unshardable, or a shard query is too big to execute: (query: %s, limit: %s). Consider adding more specific stream selectors or reduce the time range of the query ``` As pointed out in [this comment][1], a user would have a hard time figuring out whether the cause was `parallelization is not enabled`, `the query is unshardable` or `a shard query is too big to execute`. This PR improves the error messaging for the `max_querier_bytes_read` limit to raise a different error for each of the causes above. Which issue(s) this PR fixes: Followup for https://github.com/grafana/loki/pull/8670 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: https://github.com/grafana/loki/pull/8670#discussion_r1146008266 --------- Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
André Águas	44f1d8d7f6	azure: respect retry config before cancelling the context (#8732 ) What this PR does / why we need it: On GET blob operations a context timeout is set to `RequestTimeout`, however this value is not the full timeout. This is only the first timeout and afterwards there are retries. As a result, the retry configuration is never used because the context is immediately cancelled. To fix we should set the context timeout to a value larger than the Azure timeout after retries. Since the backoff is exponential and randomized we need to take the value of `MaxRetryDelay` to set an upper bound.	3 years ago
Danny Kopping	c9d5a91206	Ruler: Implement consistent rule evaluation jitter (#8896 ) What this PR does / why we need it: This PR replaces the previous random jitter with a consistent jitter. While both are random, having the random jitter be applied _consistently_ is essential for evaluating rules on a predictable cadence. If a rule is supposed to evaluate every minute, whether it evaluates at (e.g.) `01:00` or `01:03.234` is irrelevant because the evaluation _instant_ is not adjusted, so it will produce the same result whether run at `01:00` or `01:03.234`. However, if 1000 rules are set to evaluate at `01:00`, this will create a resource contention issue.	3 years ago
Danny Kopping	d38d481f35	Distributor: add detail to stream rates failure (#8900 ) What this PR does / why we need it: Currently we cannot see if a single ingester is the source of the `unable to get stream rates` failures. This change adds the client address to the log entry. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Christian Haudum	cba31024d4	Extend scheduler queue metrics with enqueue/dequeue counters (#8891 ) What this PR does / why we need it: Better o11y of the scheduler. This change yields new metrics with potentially high cardinality on the scheduler. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	3344d59fb5	Extract scheduler queue metrics into separate field This allows for easier passing of the metrics to the scheduler instantiation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Bryan Boreham	793a689d1f	Iterators: re-implement mergeEntryIterator using loser.Tree for performance (#8637 ) What this PR does / why we need it: Building on #8351, this re-implements `mergeEntryIterator` using `loser.Tree`; the benchmark says it goes much faster but uses a bit more memory (while building the tree). ``` name old time/op new time/op delta SortIterator/merge_sort-4 10.7ms ± 4% 2.9ms ± 2% -72.74% (p=0.008 n=5+5) name old alloc/op new alloc/op delta SortIterator/merge_sort-4 11.2kB ± 0% 21.7kB ± 0% +93.45% (p=0.008 n=5+5) name old allocs/op new allocs/op delta SortIterator/merge_sort-4 6.00 ± 0% 7.00 ± 0% +16.67% (p=0.008 n=5+5) ``` The implementation is very different: rather than relying on iterators supporting `Peek()`, `mergeEntryIterator` now pulls items into its buffer until it finds one with a different timestamp or stream, and always works off what is in the buffer. The comment `"[we] pop the ones whose common value occurs most often."` did not appear to match the previous implementation, and no attempt was made to match this comment. A `Push()` function was added to `loser.Tree` to support live-streaming. This works by finding or making an empty slot, then re-running the initialize function to find the new winner. A consequence is that the previous "winner" value is lost after calling `Push()`, and users must call `Next()` to see the next item. A couple of tests had to be amended to avoid assuming particular behaviour of the implementation; I recommend that reviewers consider these closely. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Bryan Boreham	3b57ad2a54	WAL: remove ePool that is unused (#8669 ) What this PR does / why we need it: Nothing is ever fetched from the pool - `GetEntries()` is never called. And the loop to call `PutEntries()` follows a call where the slice it ranges over is reset to empty, so that loop never added anything to the pool. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - NA Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Bill Franklin	3ed9f0c9ef	Canary: support filtering / parsing logs with LogQL (#8871 ) What this PR does / why we need it: The canary logs can be shipped to Loki in various formats. This allows users of the canary to parse out the canaries logs from any format e.g. nested json. The alternative is to do something different for the canary logs to prevent the logs being wrapped / manipulated (e.g. write to Loki directly, or use a different log pipeline) which limits the benefits of the canary. I've confirmed this works locally. Which issue(s) this PR fixes: Fixes #7775 Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Dylan Guedes	4f94b89fdb	Loki: Add more spans to write path (#8888 ) What this PR does / why we need it: Add new spans to our write path to better determine which operations are holding our write performance.	3 years ago
Salva Corts	d24fe3e68b	Max bytes read limit (#8670 ) What this PR does / why we need it: This PR implements two new per-tenant limits that are enforced on log and metric queries (both range and instant) when TSDB is used: - `max_query_bytes_read`: Refuse queries that would read more than the configured bytes here. Overall limit regardless of splitting/sharding. The goal is to refuse queries that would take too long. The default value of 0 disables this limit. - `max_querier_bytes_read`: Refuse queries in which any of their subqueries after splitting and sharding would read more than the configured bytes here. The goal is to avoid a querier from running a query that would load too much data in memory and can potentially get OOMed. The default value of 0 disables this limit. These new limits can be configured per tenant and per query (see https://github.com/grafana/loki/pull/8727). The bytes a query would read are estimated through TSDB's index stats. Even though they are not exact, they are good enough to have a rough estimation of whether a query is too big to run or not. For more details on this refer to this discussion in the PR: https://github.com/grafana/loki/pull/8670#discussion_r1124858508. Both limits are implemented in the frontend. Even though we considered implementing `max_querier_bytes_read` in the querier, this way, the limits for pre and post splitting/sharding queries are enforced close to each other on the same component. Moreover, this way we can reduce the number of index stats requests issued to the index gateways by reusing the stats gathered while sharding the query. With regard to how index stats requests are issued: - We parallelize index stats requests by splitting them into queries that span up to 24h since our indices are sharded by 24h periods. On top of that, this prevents a single index gateway from processing a single huge request like `{app=~".+"} for 30d`. - If sharding is enabled and the query is shardable, for `max_querier_bytes_read`, we re-use the stats requests issued by the sharding ware. Specifically, we look at the [bytesPerShard][1] to enforce this limit. Note that once we merge this PR and enable these limits, the load of index stats requests will increase substantially and we may discover bottlenecks in our index gateways and TSDB. After speaking with @owen-d, we think it should be fine as, if needed, we can scale up our index gateways and support caching index stats requests. Here's a demo of this working: <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918478-d4b6c2fd-de4d-478a-9c8b-e38fe148fa95.png"> <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918798-a71b1db8-ea68-4d00-933b-e5eb1524d240.png"> Which issue(s) this PR fixes: This PR addresses https://github.com/grafana/loki-private/issues/674. Special notes for your reviewer: - @jeschkies has reviewed the changes related to query-time limits. - I've done some refactoring in this PR: - Extracted logic to get stats for a set of matches into a new function [getStatsForMatchers][2]. - Extracted the _Handler_ interface implementation for [queryrangebase.roundTripper][3] into a new type [queryrangebase.roundTripperHandler][4]. This is used to create the handler that skips the rest of configured middlewares when sending an index stat quests ([example][5]). Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: `ff847305af/pkg/querier/queryrange/shard_resolver.go (L179-L186)` [2]: `ff847305af/pkg/querier/queryrange/shard_resolver.go (L72)` [3]: `3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L124)` [4]: `3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L163)` [5]: `f422e0a52b/pkg/querier/queryrange/roundtrip.go (L521)`	3 years ago
Karsten Jeschkies	94725e7908	Define `RequiredLabels` query limit. (#8851 ) What this PR does / why we need it: Some end-users can impose great workload on a cluster by selecting too many streams in their queries. We should be able to limit them. Therefore we introduce a new limit `RequiredLabelMatchers` which list label names that must be included in the stream selectors. The implementation follows the same approach as for max query limit. Which issue(s) this PR fixes: Fixes #8745 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Sandeep Sukhani	4e893a0a88	tsdb: sample chunk info from tsdb index to limit the amount of chunkrefs we read from index (#8742 ) What this PR does / why we need it: Previously we used to read the info of all the chunks from the index and then filter it out in a layer above within the tsdb code. This wastes a lot of resources when there are too many chunks in the index, but we just need a few of them based on the query range. Before jumping into how and why I went with chunk sampling, here are some points to consider: * Chunks in the index are sorted by the start time of the chunk. Since this does not tell us much about the end time of the chunks, we can only skip chunks that start after the end time of the query, which still would make us process lots of chunks when the query touches chunks that are near the end of the table boundary. * Data is written to tsdb with variable length encoding. This means we can't skip/jump chunks since each chunk info might vary in the number of bytes we write. Here is how I have implemented the sampling approach: * Chunks are sampled considering their end times from the index and stored in memory. * Here is how `chunkSample` is defined: ``` type chunkSample struct { largestMaxt int64 // holds largest chunk end time we have seen so far. In other words all the earlier chunks have maxt <= largestMaxt idx int // index of the chunk in the list which helps with determining position of sampled chunk offset int // offset is relative to beginning chunk info block i.e after series labels info and chunk count etc prevChunkMaxt int64 // chunk times are stored as deltas. This is used for calculating mint of sampled chunk } ``` * When a query comes in, we will find `chunkSample`, which has the largest "largestMaxt" that is less than the given query start time. In other words, find a chunk sample which skips all/most of the chunks that end before the query start time. * Once we have found a chunk sample which skips all/most of the chunks that end before the query start, we will sequentially go through chunks and consider only the once that overlap with the query range. We will stop processing chunks as soon as we see a chunk that starts after the end time of the query since the chunks are sorted by start time. * Sampling of chunks is done lazily for only the series that are queried, so we do not waste any resources on sampling series that are not queried. * To avoid sampling too many chunks, I am sampling chunks at `1h` steps i.e given a sampled chunk with chunk end time `t`, the next chunk would be sampled with end time >= `t + 1h`. This means typically, we should have ~28 chunks sampled for each series queried from each index file, considering 2h default chunk length and chunks overlapping multiple tables. Here are the benchmark results showing the difference it makes: ``` benchmark old ns/op new ns/op delta BenchmarkTSDBIndex_GetChunkRefs-10 12420741 4764309 -61.64% BenchmarkTSDBIndex_GetChunkRefs-10 12412014 4794156 -61.37% BenchmarkTSDBIndex_GetChunkRefs-10 12382716 4748571 -61.65% BenchmarkTSDBIndex_GetChunkRefs-10 12391397 4691054 -62.14% BenchmarkTSDBIndex_GetChunkRefs-10 12272200 5023567 -59.07% benchmark old allocs new allocs delta BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% benchmark old bytes new bytes delta BenchmarkTSDBIndex_GetChunkRefs-10 27286536 6398855 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286571 6399276 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286566 6400699 -76.54% BenchmarkTSDBIndex_GetChunkRefs-10 27286561 6399158 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286580 6399643 -76.55% ``` Checklist - [x] Tests updated	3 years ago
Christian Haudum	9844fad8b4	Rename LeafQueue to TreeQueue (#8856 ) What this PR does / why we need it: `TreeQueue` is the semantically more correct term for this type of queue. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Travis Patterson	9d20bed26a	Short circuit parsers (#8724 ) In many cases we only need a few labels from a line. Because most of our parsers parse lines incrementally, we can stop parsing a line after we have all the labels we want from it. This pr uses `ParserHints` to keep track of the number of extracted labels. It also provides a way for parsers to know when they should stop parsing. Notes: - parsers had inconsistent ordering between the `ShouldExtract` call and adding the `_extracted` label to duplicates. This PR makes appending `_extracted` always happen before `ShouldExtract` to keep counts of what is extracted compared to expected labels consistent. Next Steps: - When the user specifies a query with a grouping containing the `_extracted` label but there is no duplicate between the passed line and labels, short circuiting will not work. I'll address this in a follow up PR. Benchmarks: To try and show a balanced view of what this buys us, this pr picks a label our of the middle of a line. In a best case this might be much better. In the worst case, we have to parse the whole line. ``` benchstat short_circuit_old.txt short_circuit_new.txt name old time/op new time/op delta KeyExtraction/json-8 456ns ± 3% 256ns ± 2% -43.84% (p=0.000 n=9+10) KeyExtraction/logfmt-8 347ns ± 4% 171ns ± 2% -50.86% (p=0.000 n=10+10) KeyExtraction/logfmt-expression-8 552ns ± 2% 368ns ± 2% -33.23% (p=0.000 n=9+10) name old alloc/op new alloc/op delta KeyExtraction/json-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 16.0B ± 0% 16.0B ± 0% ~ (all equal) name old allocs/op new allocs/op delta KeyExtraction/json-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 2.00 ± 0% 2.00 ± 0% ~ (all equal) ``` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
W.T. Chang	183fe85f1c	Loki: Add route_randomly to Redis options (#8852 ) What this PR does / why we need it: Current Redis client only reads from master node, which makes master node under heavy pressure. Usually, one or more read replicas are expected on production environments. This PR helps to increase the utilization of read replicas.	3 years ago
Danny Kopping	528ed22dc2	Ruler: remote rule evaluation hardening (#8785 ) What this PR does / why we need it: This PR is part 2 of 2 implementing remote rule evaluation. See [part 1](https://github.com/grafana/loki/pull/8744) for more context.	3 years ago
Danny Kopping	2d93952929	(chore): remove incorrect cache config preample (#8859 ) With the addition of the [config generation tool](https://github.com/grafana/loki/pull/7916), a comment prefix was added for all cache types: `Cache config for index entry writing.` https://grafana.com/docs/loki/next/configuration/#cache_config This appears in the description of each cache config option, and is misleading.	3 years ago
Travis Patterson	5a08a6bcb9	Label selector optimizations (#8763 ) This PR introduces the following label selector optimizations: - any selector that is `=~<literal>` is converted to an `Equals/NotEquals` matcher - Use the greedy->non-greedy regex improvements that exist in line and label filters - Never run `.*` label matchers. Prometheus regex matchers anchor the match to the beginning and end of the line. That means matchers of the form `=~<literal_value>` are just testing equality. Benchmarks -- this is just a quick bench to demonstrate equality matchers are faster than their equivalent regex matchers: ``` BenchmarkMatcherTypes/regex-8 27694834 53.74 ns/op 0 B/op 0 allocs/op BenchmarkMatcherTypes/equals-8 627045850 2.153 ns/op 0 B/op 0 allocs/op ```	3 years ago
Ashwanth	7d2eeddfb8	LogQL: Fix panic when using `unpack` with empty line (#8853 ) What this PR does / why we need it: Fixes panic in unpack parser when processing empty lines ``` /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/logql/log/parser.go:583 github.com/grafana/loki/pkg/logql/log.(*UnpackParser).unpack(...) panic: runtime error: index out of range [0] with length 0 ``` Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Danny Kopping	5c3d204ebf	Ruler: rule evaluation jitter (#8848 ) What this PR does / why we need it: This PR introduces a configurable random sleep before each rule evaluation to prevent contention on the query engine should multiple runs execute concurrently. It's quite likely that rules will be evaluated ~concurrently if they have the same interval.	3 years ago
Karsten Jeschkies	f5f1753851	Print duration in error messages with more readable. (#8816 ) What this PR does / why we need it: The old error messages would print only up to hours. E.g. `169h30s`. This change will print it as `7d1h30s`. See [model.Duration](`66b493f42b/model/time.go (L259-L290)`) for details. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Christian Haudum	be8b4eece3	Scheduler: Add query fairness control across multiple actors within a tenant (#8752 ) What this PR does / why we need it: This PR wires up the scheduler with the hierarchical queues. It is the last PR to implement https://github.com/grafana/loki/pull/8585. When these changes are in place, the client performing query requests can control their QoS (query fairness) using the `X-Actor-Path` HTTP header. This header controls in which sub-queue of the tenant's scheduler queue the query request is enqueued. The place within the hierarchy where it is enqueued defines the probability with which the request gets dequeued. A common use-case for this QoS control is giving each Grafana user within a tenant their fair share of query execution time. Any documentation is still missing and will be provided by follow-up PRs. Special notes for your reviewer: ```console $ gotest -count=1 -v ./pkg/scheduler/queue/... -test.run=TestQueryFairness === RUN TestQueryFairness === RUN TestQueryFairness/use_hierarchical_queues_=_false dequeue_qos_test.go:109: duration actor a 2.007765568s dequeue_qos_test.go:109: duration actor b 2.209088331s dequeue_qos_test.go:112: total duration 2.209280772s === RUN TestQueryFairness/use_hierarchical_queues_=_true dequeue_qos_test.go:109: duration actor b 605.283144ms dequeue_qos_test.go:109: duration actor a 2.270931324s dequeue_qos_test.go:112: total duration 2.271108551s --- PASS: TestQueryFairness (4.48s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_false (2.21s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_true (2.27s) PASS ok github.com/grafana/loki/pkg/scheduler/queue 4.491s ``` ```console $ gotest -count=5 -v ./pkg/scheduler/queue/... -bench=Benchmark -test.run=^$ -benchtime=10000x -benchmem goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/scheduler/queue cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz BenchmarkGetNextRequest BenchmarkGetNextRequest/without_sub-queues BenchmarkGetNextRequest/without_sub-queues-8 10000 29337 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21348 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21595 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21189 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21602 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33770 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33596 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 34432 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33760 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33664 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 71405 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 59472 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 117163 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 106505 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 64374 ns/op 3200 B/op 300 allocs/op BenchmarkQueueRequest BenchmarkQueueRequest-8 10000 168391 ns/op 320588 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 166203 ns/op 320587 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 149518 ns/op 320584 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 219776 ns/op 320583 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 185198 ns/op 320597 B/op 1156 allocs/op PASS ok github.com/grafana/loki/pkg/scheduler/queue 64.648s ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Ashwanth	d08b87f799	json parser: fix panic when processing empty line (#8835 ) What this PR does / why we need it: Fixes panic when an empty line is processed by `JSONExpressionParser` ``` /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/logql/log/parser.go:537 github.com/grafana/loki/pkg/logql/log.isValidJSONStart(...) ... panic: runtime error: index out of range [0] with length 0 ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago

1 2 3 4 5 ...

2012 Commits (Alex3k-patch-2)