370 Commits (8abe080fd6d04dc5efb074ee0764a9bfd39c82d2)

Author	SHA1	Message	Date
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601) ... **What this PR does / why we need it**: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588) ... For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	3 years ago
Periklis Tsirakidis	69392de8b6	Fix duplicate label values from ingester streams (#9629)	3 years ago
Travis Patterson	6d8a184334	Revert "revert count unflused chunks (#9504)" (#9615) ... Now that the bug where huge values is fixed and known not to be this. Let's readd it. Original: https://github.com/grafana/loki/pull/9479	3 years ago
Salva Corts	87a659a6db	Add span events for index stats and result cache (#9552) ... **What this PR does / why we need it**: This PR adds events to the traces to have some extra observability for how we compute the index stats. We also add some trace events to the results cache.      **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Travis Patterson	a4ef06890b	revert count unflused chunks (#9504) ... revert this commit: aeba51ab0f We seem to be dramatically overcounting bytes now which is causing oversharding This is the same query before and after: Before: ~20 GB  After: ~8 EB 	3 years ago
Travis Patterson	aeba51ab0f	Count unflushed chunks in instance stats (#9479) ... The instance wasn't counting unflushed chunks in the stats. This change checks that the chunk's `flushed time` is 0 which means it hasn't been flushed yet.	3 years ago
Owen Diehl	64c6f6b8cc	Better combines stats in ingesters (#9474) ... * Reduces double counting streams when possible * proportionally add stats based on chunk overlap range * code reuse	3 years ago
Dylan Guedes	358b8965c4	Distributor: Add org_id to log error (#9475) ... **What this PR does / why we need it**: Add `org_id` to push error when the error is "max active streams limit exceeded". Apparently all other push errors correctly report the `org_id`, but not this one. **Which issue(s) this PR fixes**: N/A	3 years ago
Dylan Guedes	20567f9ed8	Ingester: Avoid repeating `streamName` on error message (#9419) ... **What this PR does / why we need it**: Modify how our ingesters build the final error message to not append the streamName for every single entry. All entries are already part of the same stream, so repeating it is unnecessary.	3 years ago
Travis Patterson	fdc337692d	Stream Sharding amortize push payloads across the observed rate. (#9414) ... This should account for large payloads that come infrequently	3 years ago
Trevor Whitney	50bc1cee71	Rename usagestats to analytics (#9212) ... **What this PR does / why we need it**: Rename usagestats to analytics. The config section in the yaml is already called analytics, this makes things more consistent. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Michel Hollands	475eb6a190	Remove shutdown marker file when shutting down (#9314) ... **What this PR does / why we need it**: This removes the shutdown marker file when actually shutting down. If the PVC is re-used then the marker file shouldn't be there. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205)	3 years ago
Dylan Guedes	46f044fdb7	Loki: Fix wrong association of PSRL when sharding is enabled (#9176) ... Obs: PSRL stands for `per-stream rate limit` **What this PR does / why we need it**: - Fixes the scenario where all entries errors are accounted as PSRL if sharding is enabled - Fixes invalid/rejected entries being accounted as candidates to be inserted	3 years ago
Michel Hollands	abd6131bba	Update prepare shutdown (#9175) ... **What this PR does / why we need it**: This updates the PrepareShutdown method so it supports GET and DELETE methods as well. This makes it similar to Mimir: https://github.com/grafana/mimir/pull/4718. The status is now stored in a local file. A new config setting had to be added for this file as there is no obvious place to store it. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Dylan Guedes	9c2e11d186	Ingester: Reuse allocations on `GetStreamRates` (#9180) ... **What this PR does / why we need it**: - Reuse rates mapping returned by the calculator to reduce the number of allocations	3 years ago
Dylan Guedes	bdfc0f92a0	Loki: Add debug event to ingester `GetStreamRates` (#9157) ... **What this PR does / why we need it**: - Add a new span event to our ingesters `GetStreamRates` function. - Add how many streams were updated on a `updateRates` call.	3 years ago
Dylan Guedes	9159c1dac3	Loki: Improve spans usage (#8927) ... **What this PR does / why we need it**: - At different places, inherit the span/spanlogger from the given context instead of instantiating a new one from scratch, which fix spans being orphaned on a read/write operation. - At different places, turn spans into events. Events are lighter than spans and by having fewer spans in the trace, trace visualization will be cleaner without losing any details. - Adds new spans/events to places that might be a bottleneck for our writes/reads.	3 years ago
Periklis Tsirakidis	1bcf683513	Expose optional label matcher for label values handler (#8824)	3 years ago
Bryan Boreham	3b57ad2a54	WAL: remove ePool that is unused (#8669) ... **What this PR does / why we need it**: Nothing is ever fetched from the pool - `GetEntries()` is never called. And the loop to call `PutEntries()` follows a call where the slice it ranges over is reset to empty, so that loop never added anything to the pool. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - NA Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Dylan Guedes	4f94b89fdb	Loki: Add more spans to write path (#8888) ... **What this PR does / why we need it**: Add new spans to our write path to better determine which operations are holding our write performance.	3 years ago
Dylan Guedes	53b9a351e5	Loki: Add a `prepare-shutdown` endpoint (#8786) ... **What this PR does / why we need it**: Add a new `/ingester/prepare_shutdown` endpoint. This new endpoint configures the ingester to release resources on the next SIGTERM. --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Callum Styan	9a2a038f43	Allow passing of context to query related limits functions (#8689) ... In this PR we're allowing for passing of a `context.Context` via the Limits interfaces (some of which are new, to clean up hardcoding/embedding of `validation.Overrides`) This is based on work/ideas by @jeschkies . Fixes #8694 --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Bryan Boreham	08ac5336d5	WAL: store pointers in record pool (#8667) ... **What this PR does / why we need it**: Storing a slice in `sync.Pool` wastes 24 bytes every time it is returned to the heap; see https://staticcheck.io/docs/checks#SA6002 Storing a pointer to a slice allows the Pool to work without waste. This came up in discussion about the benchmark in #8489 - with this change the memory usage comes down: ``` name old time/op new time/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 7.37ms ±13% 7.45ms ± 9% ~ (p=0.841 n=5+5) name old alloc/op new alloc/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 727kB ± 0% 679kB ± 0% -6.63% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 12.0k ± 0% 10.0k ± 0% -16.68% (p=0.000 n=5+4) ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	322783e3d8	LogQL: [optimization] syntax: Replace "panic" in "/pkg/logql/syntax" with "error" (#7208)	3 years ago
Pablo	fda14ca156	Re-license and export `pkg/ingester` WAL code to be used in Promtail's WAL (#8315)	3 years ago
Kaviraj Kanagaraj	e18b4fca6f	ingester: Add `ingester_memory_streams_labels_bytes` metric (#8212) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> This is to give visibility on `labels` size of in-memory streams on the ingesters. It caculate the size of labels via `len(labels.String())`. Meaning if you have ```go lbs := labels.Labels{ {Name: "foo", Value: "bar"}, {Name: "instance", Value: "loki"}, } ``` size would be number of bytes in `{foo="bar", instance="loki"}` which is `28`. NOTE: `len(string)` gives [bytes length in Go](https://pkg.go.dev/builtin#len)	3 years ago
Ashwanth	0723678573	limits: add tenant to log lines for per-stream rate-limited and out-of-order errors (#8177)	3 years ago
Christian Haudum	089ec1b05f	Fix various linter errors ... These changes have been generated using `make lint`. ```console $ golangci-lint --version golangci-lint has version 1.50.0 built from 704109c6 on 2022-10-04T10:25:07Z ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Travis Patterson	92fca94bb5	Revert "Update Ingester Rate Calculations (#7652)" (#7991) ... This change was supposed to smooth out the per-second rate of streams so distributors could react more uniformly to streams with variable rates. In practice, it resulted in drastically more resource usage on distributors because `GetStreamRates` now responds with all streams on an ingester rather than only the streams seen in the last second. This change also didn't seem to affect sharding precision one way or another. If we'd still like to implement exponential smoothing for stream rates, it should be implemented in distributors to keep communication overhead to a minimum.	3 years ago
Susana Ferreira	68a4d3271a	Fix documentation typos from #7916 (#7934) ... **What this PR does / why we need it**: Fix documentation typos from https://github.com/grafana/loki/pull/7916#pullrequestreview-1217192817 **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/pull/7916#pullrequestreview-1217192817 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916) ... **What this PR does / why we need it**: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](5550cd65ec/docs/sources/configuration/_index.md) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. **NOTE:** this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/83 **Special notes for your reviewer**: Files: * [docs/sources/configuration/index.template](5550cd65ec/docs/sources/configuration/index.template): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](c32e5d0acb/docs/sources/configuration/_index.md): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](5550cd65ec/tools/doc-generator) directory with documentation generation tool. **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Robert Fratto	85392a9728	Update Prometheus dependency to latest release (v2.40.4) (#7826) ... Closes #7811, which is needed for Grafana Agent to update to v2.40 and add support for native histograms. I did not add support for native histograms to Loki, sorry :)	3 years ago
Travis Patterson	343e39f67b	Update Ingester Rate Calculations (#7652) ... - Use weighted moving average for rates - Keep Streams around until they are removed Previously, the rate calculator only kept the most recent second. This led to issues where bursty streams that didn't get pushes every second resulted in undersharding those streams	3 years ago
洪阿南	776ee95a00	modify usage stat metric unit to match definition (#7588) ... Output the flushedChunksLifespanStats metrics in seconds instead of hours	3 years ago
李国忠	6848ea37dd	fix ingester: grpc method=/logproto.Querier/Query fail. (#5970) ... When ingester's ctx times out, don't try to use grpc to return data to the querier again. This PR fixes the ingester error rate > 0 for our loki cluster. We rely on this error rate indicator for alarming Fixes #5964	3 years ago
Travis Patterson	198c81fc7b	Reduce number of locks and partition by tenant and stream hash (#7583) ... This PR reduces the number of locks in the rate calculator and allows it to hold an arbitrary number of rates by partitioning by tenant and stream hash Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Travis Patterson	c75b822fc6	Track rates per stream and per tenant (#7564) ... It is possible for two tenants to have streams with the same labels. This PR changes the stream tracking in the `shard_tracker`, `stream_rate_calculator`, and `rate_store` to account for tenant ID when tracking streams.	3 years ago
Travis Patterson	45d59ba571	Improve rate store refresh latency (#7492) ... Rather than iterating streams and getting rates per stream, keep all necessary information in the rate calculator and just hand back the current rates when asked. This change took p99 latencies from ~1.8s to ~400ms and p50 latences from ~1s to 6ms.	3 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	3 years ago
Travis Patterson	3e7adb2c34	Measure Per-Stream rates (#7370) ... This PR implements the ability for ingesters to measure per-stream rates at 1/s resolution. It also fixes bugs in the kind of Limits used in the RateStore and the Ingester API	3 years ago
Travis Patterson	7979cfbe07	Implement Distributor Rate Store (#7349) ... The `RateStore` concurrently queries all ingester `GetStreamRate` APIs and stores the results. `RateStore` deduplicates replicas and combines sharded streams so a caller of `RateFor` gets an unsharded view of a stream's rate.	3 years ago
Dylan Guedes	b1d4efab12	Loki: Per-tenant stream sharding (#7311) ... **What this PR does / why we need it**: - Move stream sharding configuration to its own package to avoid cyclic imports - Change stream sharding to be a per-tenant configuration - Change ingesters to reject whole streams due to rate-limit based on per-tenant stream sharding - Change stream sharding flags prefix from `distributor.shard-stream` to `shard-stream`	3 years ago
Travis Patterson	e3e4e27c8c	Implement ingester API to return stream rates (#7176) ... This PR introduces an API on the Ingester that reports all current streams in memory and their current rate in bytes/second. The reponse from the API contains a slice of `StreamRate` structs: ```go type StreamRate struct { StreamHash uint64 `protobuf:"varint,1,opt,name=streamHash,proto3" json:"streamHash,omitempty"` StreamHashNoShard uint64 `protobuf:"varint,2,opt,name=streamHashNoShard,proto3" json:"streamHashNoShard,omitempty"` Rate int64 `protobuf:"varint,3,opt,name=rate,proto3" json:"rate,omitempty"` } ``` In the interest of space on the caller and in transmission, only hashes and rates are sent. - `StreamHash` will be used to deduplicate replicas - `StreamHashNoShard` will be used to aggregate a sharded stream's total rate. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340) ... I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212) ... The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Travis Patterson	071118a090	revert stream push error response (#7160) ... We've changed approaches to how we want to do per-stream rate limiting. The new approach will rely on creating an api on the ingester so the distributors can decide how streams should be sharded. As a result, the work to return rate-limited streams in the error body is no longer needed.	3 years ago
Travis Patterson	3f0ecd3c7b	Implement all-or-nothing stream ingestion (#7094) ... Per-stream sharding will introduce automatic stream sharding on rate-limited streams. The ingester's current line-by-line behavior may result in duplicates when a partially-accepted stream is resharded and sent to other ingesters. This PR makes the ingester either rate limit the entire stream before attempting to write lines to a chunk. The PR 1. Separates validation, storage, and wal replay into their own functions 2. Implements a flag to rate limit the whole stream or not 3. Checks rate limits for the whole stream rather than line-by-line 4. Returns an error with information on the whole stream when rate limited	3 years ago
Travis Patterson	bc16450349	Revert "Hide duplicates caused by stream sharding (#7005)" (#7031) ... @owen-d had a suggestion that was dramatically dimpler. Revert this whole thing and the simplified PR will be forthcoming	3 years ago