apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Karsten Jeschkies	125e168a91	Replace flaky server for log level handler test. (#9822 ) What this PR does / why we need it: I've noticed that the `TestLevelHandler` were flaky because the code wasn't waiting to the server to be running. Since the test is for a handler it's much simpler to call it directly with a response recorder. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Dylan Guedes	1d20fdbf57	LogPushFailures: Log the tenant id (#9819 ) What this PR does / why we need it: Modify the `writefailures.Manager` to - Log the tenant id. Otherwise, it is hard to identify which tenant is responsible for a problematic push - Log stream parsing failures Which issue(s) this PR fixes: N/A	3 years ago
Trevor Whitney	d4cfbaac4f	Implement series volume range queries (#9812 ) This PR adds a `series_volume_range` endpoint which allows for series volume queries over time with a specified step, returning timeseries data in the form of a Prometheus matrix response. The existing `series_volume` endpoint still returns Prometheus vector responses, and hardcodes the step to 0.	3 years ago
Salva Corts	6cc581bd26	Add back cache stats for index stats requests (#9816 ) What this PR does / why we need it: In https://github.com/grafana/loki/pull/9536, we added cache stats for index stats requests. That PR had a bug that inflated the query stats due to reusing the stats context in the query engine. Therefore, we had to revert the PR at https://github.com/grafana/loki/pull/9721. This PR brings back the changes from https://github.com/grafana/loki/pull/9536 but fixes the inflated starts by no longer reusing the same context in the query engine, but rather creating a new one for the shard resolver. I tested it on a dev cluster and seems to be working fine. here's the output for the same query: Stats with the bug from #9536: ``` ... Cache.StatsResult.Requests 980 Cache.StatsResult.EntriesRequested 490 Cache.StatsResult.EntriesFound 0 Cache.StatsResult.EntriesStored 490 Cache.StatsResult.BytesSent 0 B Cache.StatsResult.BytesReceived 0 B ... Summary.BytesProcessedPerSecond 43 GB Summary.LinesProcessedPerSecond 93305142 Summary.TotalBytesProcessed 945 GB Summary.TotalLinesProcessed 2059694183 ``` Stats from _main_ ``` ... Summary.BytesProcessedPerSecond 1.6 GB Summary.LinesProcessedPerSecond 3403718 Summary.TotalBytesProcessed 95 GB Summary.TotalLinesProcessed 207971404 ``` Stats with fix in this PR ``` .. Cache.StatsResult.Requests 132 Cache.StatsResult.EntriesRequested 66 Cache.StatsResult.EntriesFound 0 Cache.StatsResult.EntriesStored 66 Cache.StatsResult.BytesSent 0 B Cache.StatsResult.BytesReceived 0 B ... Summary.BytesProcessedPerSecond 4.3 GB Summary.LinesProcessedPerSecond 9468900 Summary.TotalBytesProcessed 95 GB Summary.TotalLinesProcessed 207793816 ``` As can be seen, with the changes in this PR, the summary stats are no longer inflated. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/pull/9536 Special notes for your reviewer: I think it's ok to skip reviewing the changes from the commit cherry-picking the changes from https://github.com/grafana/loki/pull/9536 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Ashwanth	f9fc1df4c0	logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default (#9626 ) What this PR does / why we need it: Introduces `--strict` flag for logfmt parser to enable strict parsing. With strict parsing enabled, logfmt parser stops scanning the log line and returns early with an error if it encounters any poorly formatted key=value pair. This pr updates the parser to perform non-strict parsing by default where it skips over the invalid tokens. This offers the flexibility to parse semi-structured log lines. Also adds `--keep-empty` flag to retain standalone keys(with empty value) as labels with value set to empty string Special notes for your reviewer: Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Dylan Guedes	dc5d458c14	Failures logging: avoid logging generic errors (#9796 ) What this PR does / why we need it: Modify our write path to not log generic messages with the `writefailures.Manager`. With generic messages I mean: contextCancel errors and contextDeadlineExceeded errors. Which issue(s) this PR fixes: N/A	3 years ago
Salva Corts	08e505dd50	Return all errors for a failed push request (#9807 ) What this PR does / why we need it: Distributors enforce limits and validate each push payload. before this PR, if a single push request yields multiple errors, only the last error is forwarded to the client. This PR modifies the distributor push handler so we forward all errors from a single push request to the client. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: - I wonder if the use of [MultiError][1] is well-suited for this. [This test case][2] will return 100 errors with the same content. I wonder if we can leverage the MultiError to group errors that have the same content. Printing for example: `1 error like: 'Error parsing labels...'; 100 errors like: 'Max entry size...'`. This would require extra processing but If we do this processing in the Error() method I think it should be ok-ish? - Edit: I ended up implementing this in [`4b0816f`](`4b0816f67f`) Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) [1]: `d4a4728933/pkg/util/errors.go (L32)` [2]: `f544b93c20/pkg/distributor/distributor_test.go (L75)`	3 years ago
Owen Diehl	e2a63e582c	adds tsdb-max-bytes-per-shard limit (#9811 ) Adds the per tenant limit `tsdb-max-bytes-per-shard` which is used in configuring the shard size for tsdb subqueries. This effectively gives control over how big subqueries should be (assuming they're shardable). The default is no different (`600MB`) than the previously hardcoded initial value. This should help us iterate to find optimal shard sizes to improve operations in the long term.	3 years ago
Christian Haudum	a65c99d9bf	Add per-tenant request counter metric to index gateway server (#9797 ) This commit add a counter metric `loki_index_gateway_requests_total` with labels `operation`, `tenant`, `status` for gRPC requests that are served by the index gateway. What for? The per-tenant RPS on the index gateway is used to derive the per-tenant shard factor. Why tracking on the server? Unlike tracking index gateway RPS on the client side, tracking on the server side does not yield that many series, even in multi-tenant installations with a lot of tenants, because the amount of index gateway instances is relatively small compared to the amount of queriers and frontends. Special notes for your reviewer: The previous approach of tracking requests on the client https://github.com/grafana/loki/pull/9781 has been abandoned. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Travis Patterson	8ca035ffbf	Log Volume: Do the prometheus-format conversion in dedicated middleware (#9776 ) This PR moves the Prometheus-format conversion to it's own middleware. This keeps it all in one place and consolidates logic that was in many places. It also sets us up to make range queries as a next step.	3 years ago
Bryan Boreham	6928aac776	loser tree: allow sequences to contain maxVal (#9057 ) What this PR does / why we need it: The tree stores the loser in each node, or maxVal if the sequence has ended. If sequences can contain maxVal, we need to take care not to let an ended sequence overtake an active one. Some test cases from Charles who pointed out the issue in #9053. Note this situation cannot happen for real in Loki, since we don't expect log lines with a timestamp that far into the future, but this is supposed to be a re-usable library. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Charles Korn <charles.korn@grafana.com>	3 years ago
Ryo Shindo	ed0d15a6bf	loki: enable PathPrefix for queryHandler endpoints (#8406 ) What this PR does / why we need it: Hi, loki developers. I would like to use the option `http_path_prefix` as described in the official documentation. This setting is valid for endpoints defined at `4d55a24ed3/pkg/loki/loki.go (L477-L494)` but not for the endpoints defined at `4d55a24ed3/pkg/loki/modules.go (L366-L392)` . The current latest version with `http_path_prefix` will return the expected response for the former endpoint, but will return a 404 not found error when accessing the latter endpoint. For example, if you define `http_path_prefix = /xxx`, requests for `/xxx/ready` will work as expected, but requests for `/xxx/loki/api/v1/query` will result in 404 errors. I've created a pull request for this issue because I've been experiencing some inconvenience when using loki. Please check it out. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/4756 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Tests updated --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Sandeep Sukhani	3e1f2fc273	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757 ) What this PR does / why we need it: Currently, when we find a relevant cached negative response for a logs query, we do the following: * If the cached query completely covers the new query: * return back an empty response. * else: * fill the gaps on either/both sides of the cached query. The problem with filling the gaps is that when the cached query does not overlap at all with the new query, we have to extend the query beyond what the query requests for. However, with the logs query, we have a limit on the number of lines we can send back in the response. So, this could result in the query response having logs which were not requested by the query, which then get filtered out by the [response extractor](`b78d3f0552/pkg/querier/queryrange/log_result_cache.go (L299)`), unexpectedly resulting in an empty response. For example, if the query was cached for start=15, end=20 and we get a `backwards` query for start=5, end=10. To fill the gap, the query would be executed for start=5, end=15. Now, if we have logs more than the query `limit` in the range 10-15, we would filter out all the data in the response extractor and send back an empty response to the user. This PR fixes the issue by doing the following changes when handling cache hit: * If the cached query completely covers the new query: * return back an empty response[_existing_]. * else if the cached query does not overlap with the new query: * do the new query as requested. * If the new query results in an empty response and has a higher interval than the cached query: * update the cache * else: * query the data for missing intervals on both/either side[_existing_] * update the cache with extended intervals if the new queries resulted in an empty response[_existing_] Special notes for your reviewer: We could do further improvements in the handling of queries not overlapping with cached query by selectively extending the queries based on query direction and cached query lying before/after the new query. For example, if the new query is doing `backwards` query and the `cachedQuery.End` < `newQuery.Start`, it should be okay to extend the query and do `cachedQuery.End` to `newQuery.End` to fill the cache since query would first fill the most relevant data before hitting the limits. I did not want to complicate the fix so went without implementing this approach. We can revisit later if we feel we need to improve our caching. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Ashwanth	10e55a6359	store: fix incorrect usage of period config in table range (#9754 ) What this PR does / why we need it: We are incorrectly taking reference to a loop variable in store init, I overlooked this part as the reference was being taken in the call to `GetIndexTableNumberRange` As a result of this, `TableRange` of each period (which is used by index-shipper) points to the latest `periodConfig`. This in itself is not a problem if the index prefix doesn't change as we use the `periodConfig` only to generate the `tableName`. But if the index prefix changes across periods, we end up losing access to the indexes using the old prefix. Special notes for your reviewer: Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Susana Ferreira	35465d0297	Fix instant query summary split stats (#9773 ) What this PR does / why we need it: Fix instant query summary statistic's `splits` corresponding to the number of subqueries a query is split into based on `split_queries_by_interval`. * Update rangemapper with a statistics structure to include the number of split queries a query is mapped into. * In the `split_by_range` middleware once the mapped query is returned update the middleware statistics with the number of split queries. This value will then be merged with the statistics of the Loki response. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Travis Patterson	4da0f63789	Remove unused Value field (#9774 ) We didn't end up needing the `Value` field because we can express everything we need to as selectors	3 years ago
Susana Ferreira	a894c3ccf0	Bugfix: Fix logic of offset operator in instant query splitting (#9763 ) What this PR does / why we need it: Fix the logic of the `offset` operator for downstream queries on instant query splitting of (range) vector aggregation expressions already containing an offset. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Susana Ferreira	b611641fea	Add comments to clarify Grouping struct in Aggregation expressions (#9761 ) What this PR does / why we need it: Add comments to clarify the Grouping struct in Vector Aggregation and Range Vector Aggregations expressions. Related with PR: https://github.com/grafana/loki/pull/9515 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Travis Patterson	806674fdaa	Add log-volume feature flag (#9762 ) Adds a feature flag for use with the new log-volume endpoints so associated features can be rolled out incrementally.	3 years ago
Trevor Whitney	dbc3040739	Convert SeriesVolume response to prometheus-style (#9703 ) Changes the response type of the label volume stats endpoint to return volumes as prometheus-style timeseries metrics. It currently only supports instant queries, but is a necessary step to eventually supporting range queries.	3 years ago
Trevor Whitney	d74c67a9d0	Add version to TSDB index builder struct (#9566 ) Add version as a property to the TSDB Index Builder, forcing users of the builder to specify a version for each builder instance. This is the first of many PRs that will add the TSDB index version to the schema version config.	3 years ago
Christian Haudum	ce81895241	Add shuffle sharding to index gateway (#9710 ) This PR introduces shuffle sharding of the index files across index gateways where the shuffle sharding key is the tenant ID. Instead of having a global replication factor on the index gateways which affects all tenants the same, whether they are small or large tenants, the shuffle sharding allows to specify a shard factor per tenant (on top of the replication factor). This a) reduces the amount of indexes that are downloaded on individual index gateways in case of a high replication factor (which is currently the de-factor standard to deal with high load) and b) allows to scale tenants individually. The shuffle sharding algorithm is part of dskit's ring implementation and is not part of this commit. If you have set a high RF for the index gateways, it is required to reduce the RF and instead set the shard factor for the tenants. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Dylan Guedes	24e9efb907	Loki: Report event for `GetChunkRef` instead of SpanLogging (#9748 ) What this PR does / why we need it: Modify our calls to `index_client.GetChunkRef` to report invocations through a span event instead of through span logging. This way we won't log a new log line for every GetChunkRef call. Which issue(s) this PR fixes: N/A	3 years ago
Dylan Guedes	c6e18a2efb	Loki: Humanize downloaded file size (bytes format) (#9746 ) What this PR does / why we need it: Humanize reported size of downloaded file. Which issue(s) this PR fixes: N/A	3 years ago
Dylan Guedes	c0d9a2b4ba	Log push failures per-entry instead of in batches (#9720 ) What this PR does / why we need it: Modify how we log push errors to do it per-entry instead of accumulating all errors and then logging them all in a single pass. Which issue(s) this PR fixes: N/A	3 years ago
Kaviraj Kanagaraj	74a4dca54e	fix(async_store): Fix the `storageconfig.EnableAsyncStore` config (#9725 ) What this PR does / why we need it: During reverting this [PR](https://github.com/grafana/loki/pull/9715). I missed one important flag switch. Which this PR fixes it. Which issue(s) this PR fixes: Fixes NA Special notes for your reviewer: We found a bug on querier due to async_store and we tracked down to[ this PR](https://github.com/grafana/loki/pull/9476). So I reverted the PR in [this PR ](https://github.com/grafana/loki/pull/9715)(manually. Auto revert was not possible as things changed a lot meanwhile) But during my revert PR I missed switching this async store config. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Bryan Boreham	695ea32226	stores/series tests: use abstractions over Labels (#9613 ) What this PR does / why we need it: Stop assuming `labels.Labels` is a slice. Special notes for your reviewer: This PR only touches tests. Upstream Prometheus is using a different implementation of `labels.Labels`, so I plan to create a series of PRs which make Loki compatible with that. #9532 was the first. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Bryan Boreham	904ea0586a	Remove some unused code (#9611 ) What this PR does / why we need it: Remove some functions I came across while making Loki compatible with different `Labels` structures. Move `MetricToLabels` to the only place it is now used, and make it call abstractions instead of assuming `Labels` is a slice. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - NA Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Salva Corts	61d407b4bb	Add keep stage to LogQL (#9693 ) What this PR does / why we need it: This PR adds a new `keep` stage to LogQL. It allows filtering specific labels and helps to avoid hitting the _max streams per query_ limit by keeping only specific labels in the label set. It works the opposite of [drop stage](https://grafana.com/docs/loki/latest/logql/log_queries/#drop-labels-expression). Special notes for your reviewer: We do not drop [special error labels](`edc6b0bff7/pkg/logqlmodel/error.go (L18-L20)`) added by Loki at query time. Users will have to explicitly drop them by using the drop stage. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Salva Corts	b7359c5d53	Revert "Add summary stats and metrics for stats cache (#9536 )" (#9721 ) This reverts commit `af287ac3eb`. There is a bug in this PR that inflates the stats returned for the query since we reuse the stats ctx in the query execution engine.	3 years ago
Kaviraj Kanagaraj	4c0e01b93f	revert(9476): Bringback async store like before for the queriers (#9715 ) This seems to cause some issue with `querier`, not getting chunks from ingesters correctly via `async_store`. I narrowed down to this PR. But hard `revert` seems not possible anymore. So created this manually. https://github.com/grafana/loki/pull/9476 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Travis Patterson	db97058a84	Series volume endpoint (#9704 ) This changes the `label_volume` endpoint to the `series_volume` endpoint. The new endpoint still returns volumes but now it does it for the requested streams defined by the selector names passed rather than individual labels. All relevant non-requested labels are aggregated into the returned results ex: Assume we have the following streams: ``` {cluster="prod", team="A", component="foo"} {cluster="prod", team="B", component="foo"} {cluster="dev", team="A", component="foo"} {cluster="dev", team="B", component="foo"} ``` - requesting `{cluster="prod"}` returns one result for all streams containing `{cluster="prod"}` - requesting `{cluster=~".+"}` returns two results for the streams containing `{cluster="prod"}` and `{cluster="dev"}` - requesting `{cluster=~".+", team=".+"}` returns four results for the streams containing: ``` {cluster="prod", team="A"} {cluster="prod", team="B"} {cluster="dev", team="A"} {cluster="dev", team="B"} ``` --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
Michel Hollands	79485cd30d	Add handling of timezones in unixToTime tests (#9707 ) What this PR does / why we need it: The unit tests for unixToTime do not take the current timezone into account. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Mohamed-Amine Bouqsimi	7069b897e5	Fix Ruler remote write panic (#9662 ) What this PR does / why we need it: This PR fixes the Ruler remote write panic, caused by the no-op scrape manager not being correctly configured.	3 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588 ) For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	3 years ago
Kaviraj Kanagaraj	57f9452231	Revert 9217 chaudum/tsdb chunkrefs pool (#9685 ) Revert #9217 (potential bug in query result) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Salva Corts	73ac208981	Improve docs for empty value in cache compression config (#9649 ) What this PR does / why we need it: Follow up PR for https://github.com/grafana/loki/pull/9535#discussion_r1218167670 Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Salva Corts	c6fbff26e1	Add config to avoid caching stats for recent data (#9537 ) What this PR does / why we need it: When we query the stats for recent data, we query both the ingesters and the index gateways for the stats. `ebdb2b1800/pkg/storage/async_store.go (L112-L114)` `ebdb2b1800/pkg/storage/async_store.go (L126-L127)` Then we merge all the responses, which means summing up all the stats `ebdb2b1800/pkg/storage/async_store.go (L157-L158)` `ebdb2b1800/pkg/storage/stores/index/stats/stats.go (L23-L26)` Because we have a replication factor of 3, this means that we will get the stats from the ingesters repeated up to 3 times, hence inflating the stats. In the stats cache, we store the stats for a given matcher set for the whole day, then we extract the stats from the cache by the factor of time from the request that is stored in the cache: `336283acad/pkg/querier/queryrange/index_stats_cache.go (L33)` `336283acad/pkg/querier/queryrange/index_stats_cache.go (L40)` Inflated stats for recent data will be cached, so subsequent stats extracted from the cache will be inflated regardless of the time. This PR adds a new per-tenant limit `max_stats_cache_freshness` to not cache requests with an end time that falls within Now minus this duration. Here's a scenario illustrating this. The graphs below show the bytes stats queried in the sharding middleware. We are running a log filter query that won't match any log, every 5 seconds with a length of 3h. ![image](https://github.com/grafana/loki/assets/8354290/45c2e6e9-185c-4a18-b290-47da27fc3e39) As can be seen, after enabling the stats cache and configuring`do_not_cache_request_within` to not cache stats for requests within 30m, the bytes stats used in the sharding middleware stopped increasing. In both cases the stats cache hit ration was 100%. ![image](https://github.com/grafana/loki/assets/8354290/cd35bcb8-0c77-4693-a06b-502741fd6e23) Special notes for your reviewer: - Blocked by https://github.com/grafana/loki/pull/9535 - Note that this PR doesn't fix the root issue of inflated stats form the ingesters, but rather buys us some time to work on that. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Michel Hollands	22779e13c5	Fix date template function with epoch times (#8886 ) What this PR does / why we need it: Adds new toUnixEpoch... functions to convert from a string with a Unix/Epoch time to an integer that can be used in the existing `toDate` function. Note that these are the opposites of some of the functions introduced in https://github.com/grafana/loki/pull/8774. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/8624. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Gregor Zeitlinger	b1917a6178	add "alignLeft" and "alignRight" functions (#9672 ) Fixes https://github.com/grafana/loki/issues/9667	3 years ago
Ashwanth	98d1307c7d	config: ensure storage config defaults apply to named stores (#9650 ) What this PR does / why we need it: Since named store config does not register any flags, storage configs defined under it do not get the defaults. For example [aws_storage_config](https://grafana.com/docs/loki/latest/configuration/#aws_storage_config) sets the default `storage_class` to `STANDARD`, but the same doesn't get applied by default when using named stores. This PR ensures that named storage configs are always assigned default values when they are unmarshalled by implementing `yaml.Unmarshaler` interface	3 years ago
Danny Kopping	1db560fdc1	Adding background cache (en\|de)queue counters (#9665 ) What this PR does / why we need it: The background writeback cache exposes gauge metric currently for the current queue size. Gauges can be useful, but they are susceptible to sample errors because they only represent the point in time as the time of the scrape. Exposing counters for the bytes (en\|de)queued to/from the cache will be more useful because they can be aggregated. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Dylan Guedes	609bc22933	Distributor: Make key configurable when logging failures (#9659 ) What this PR does / why we need it: Make appending `insight=true` key-value pair to log failures configurable. Which issue(s) this PR fixes: N/A	3 years ago
Ed Welch	79c5fdd9c0	Canary: Refactor how the direct Canary push works (#9569 ) What this PR does / why we need it: When doing some testing a while back I ran into several cases were we didn't handle errors/retries properly in the code we added for the canaries to send directly to Loki. The changes here effectively duplicate the way promtail's client does sends to Loki which is code we've been using/relying on for years now. I also had a hard time following the previous implementation which used the io.Writer interface for swapping out writing to stdout vs pushing to Loki and refactored this to use a new interface `EntryWriter` which has implementations that write to stdout and to Loki Unfortunately the changes in the diff don't read super well with this refactoring so hopefully the explanation helps. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	3 years ago
Ivana Huckova	eb7dae4583	Loki: Improve error message when step too low (#9641 ) What this PR does / why we need it: In https://github.com/grafana/grafana/pull/69648 we are in Grafana introducing a step editor in Loki. Unfortunately, the error message when user sets too low step parameter is hard to understand, so I am proposing following change to make it more understandable and actionable. Let me know what do you think. --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Salva Corts	af287ac3eb	Add summary stats and metrics for stats cache (#9536 ) What this PR does / why we need it: When a query finishes, we return (and log) the following stats: ```go Cache.Chunk.Requests 0 Cache.Chunk.EntriesRequested 0 Cache.Chunk.EntriesFound 0 Cache.Chunk.EntriesStored 0 Cache.Chunk.BytesSent 0 B Cache.Chunk.BytesReceived 0 B Cache.Chunk.DownloadTime 0s Cache.Index.Requests 0 Cache.Index.EntriesRequested 0 Cache.Index.EntriesFound 0 Cache.Index.EntriesStored 0 Cache.Index.BytesSent 0 B Cache.Index.BytesReceived 0 B Cache.Index.DownloadTime 0s Cache.Result.Requests 13 Cache.Result.EntriesRequested 13 Cache.Result.EntriesFound 13 Cache.Result.EntriesStored 0 Cache.Result.BytesSent 0 B Cache.Result.BytesReceived 2.5 kB Cache.Result.DownloadTime 4.600266ms ``` In addition to that, we log the following in metrics.go: ``` level=info ts=2023-05-29T09:17:10.93029945Z caller=metrics.go:152 component=frontend org_id=145265 traceID=52d59b78fe6b9221 sampled=true latency=fast query="{cluster=\"dev-us-central-0\", namespace=~\"loki.\", container=~\"distributor\|ingester \|promtail\|index-gateway\|compactor\"} \|= \"thislinewillnotexist\"" query_hash=1194136170 query_type=filter range_type=range length=3h0m0s start_delta=165h37m24.930289434s end_delta=162h37m24.930289612s step=43s duration=2.473055ms status=200 lim it=30 returned_lines=0 throughput=0B total_bytes=0B lines_per_second=0 total_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=0s splits=13 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes _fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_result_req=13 cache_result_hit=13 cache_result_download_time=4.600266ms ``` With the goal of being able to better monitor how the stats cache is performing; this PR adds stats for the index stats cache, similarly to how it's done for the results cache. Here's an example of the new stats being returned and printed: ```go ... Cache.StatsResult.Requests 180 Cache.StatsResult.EntriesRequested 129 Cache.StatsResult.EntriesFound 129 Cache.StatsResult.EntriesStored 51 Cache.StatsResult.BytesSent 0 B Cache.StatsResult.BytesReceived 75 kB ... ``` And the new stats from metrics.go ``` ... caller=metrics.go:155 ... cache_stats_results_req=129 cache_stats_results_hit=129 cache_stats_results_download_ti me=156.864429ms ... ``` Special notes for your reviewer: - Blocked by https://github.com/grafana/loki/pull/9535 - Note the new`stats.GetOrCreateContext` func. It's used inside the `query.Exec` method so we don't overwrite the stats added in the stats middleware. Checklist* - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Salva Corts	1694ad0f9b	Stats cache can be configured independently (#9535 ) What this PR does / why we need it: Before this PR, the index stats cache would use the same config as the query results cache. This was a limitation since: 1. We would not be able to point to a different cache for storing the index stats if needed. 2. We would not be able to add specific settings for this cache, without adding it to the results cache. In this PR, we refactor the index stats cache config to be independently configurable. Note that if it's not configured, it will try to use the results cache settings. Which issue(s) this PR fixes: This is needed for: - https://github.com/grafana/loki/pull/9537 - https://github.com/grafana/loki/pull/9536 Special notes for your reviewer: - This PR also refactors all the tripperwares in rountrip.go to reuse the same stats tripperware instead of each one creating their own. - Configuring a new cache in rountrip.go is a requirement for https://github.com/grafana/loki/pull/9536 so the stats summary can distinguish before the stats cache and the results cache. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Vladyslav Diachenko	3c52034d63	[logcli] fixed query command that is run against user's bucket with TSDB index enabled (#9597 ) What this PR does / why we need it: set TSDB shipper mode to ReadOnly and disabled indexGatewayClient during local query run. Also, I increase index downloading timeout to 1m because the users run the query from local machines that not always have more than 100Mbps Which issue(s) this PR fixes: Fixes #9555 Special notes for your reviewer: Unfortunately, we do not have similar tests and test architecture to cover it with unit tests properly. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Sandeep Sukhani	0a5e149ea5	query-scheduler: fix query distribution in SSD mode (#9471 ) What this PR does / why we need it: When we run the `query-scheduler` in `ring` mode, `queriers` and `query-frontend` discover the available `query-scheduler` instances using the ring. However, we have a problem when `query-schedulers` are not running in the same process as queriers and query-frontend since [we try to get the ring client interface from the scheduler instance](`abd6131bba/pkg/loki/modules.go (L358)`). This causes queries not to be spread across all the available queriers when running in SSD mode because [we point querier workers to query frontend when there is no ring client and scheduler address configured](`b05f4fced3/pkg/querier/worker_service.go (L115)`). I have fixed this issue by adding a new hidden target to initialize the ring client in `reader`/`member` mode based on which service is initializing it. `reader` mode will be used by `queriers` and `query-frontend` for discovering `query-scheduler` instances from the ring. `member` mode will be used by `query-schedulers` for registering themselves in the ring. I have also made a couple of changes not directly related to the issue but it fixes some problems: * [reset metric registry for each integration test](`18c4fe5907`) - Previously we were reusing the same registry for all the tests and just [ignored the attempts to register same metrics](`01f0ded7fc/integration/cluster/cluster.go (L113)`). This causes the registry to have metrics registered only from the first test so any updates from subsequent tests won't reflect in the metrics. metrics was the only reliable way for me to verify that `query-schedulers` were connected to `queriers` and `query-frontend` when running in ring mode in the integration test that I added to test my changes. This should also help with other tests where earlier it was hard to reliably check the metrics. * [load config from cli as well before applying dynamic config](`f9e2448fc7`) - Previously we were applying dynamic config considering just the config from config file. This results in unexpected config changes, for example, [this config change](`4148dd2c51/integration/loki_micro_services_test.go (L66)`) was getting ignored and [dynamic config tuning was unexpectedly turning on ring mode](`52cd0a39b8/pkg/loki/config_wrapper.go (L94)`) in the config. It is better to do any config tuning based on both file and cli args configs. Which issue(s) this PR fixes: Fixes #9195	3 years ago

1 2 3 4 5 ...

2151 Commits (bcdc807dd34d7a4e2e1dcfdffecd869da66e6efe)