510 Commits (8976709f24dc9baa603d030fa23484382f7ba767)

Author	SHA1	Message	Date
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	11 months ago
renovate[bot]	b37eefe24f	fix(deps): update github.com/prometheus/prometheus (main) (#15950) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	11 months ago
Trevor Whitney	5b16c0be7d	refactor: allow sample expressions to have multiple extractors (#16224)	11 months ago
Salva Corts	0be39a0ed1	feat(policies): Log when multiple policies match the same stream (#16321)	11 months ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	12 months ago
Dylan Guedes	0bc23db95d	refactor: Add retention hours to `discarded` metrics (#15875) ... **What this PR does / why we need it**: We have no distinction between the type of data that gets discarded. On this PR I'm adding `retention_hours` to our discarded metrics so that we can better look at what data was discarded.	12 months ago
Owen Diehl	2a2b52808c	perf(ingester): refactor lock acquisitions related to `not_owned` series limit functionality (#15839)	12 months ago
Periklis Tsirakidis	f63c16f77d	fix(partition-ingester): Add stream creation error logging (#15640)	1 year ago
Ned Andreev	04f621ec86	fix: data race in ingester test (#15465)	1 year ago
Cyril Tovena	6dfe90a12d	chore: Introduce back phase for kafka delay metrics (#15407)	1 year ago
Paul Rogers	f2c2a22bdc	chore: Preparation for incoming static code analysis CI check (#15164) ... Co-authored-by: Danny Cooper <danny.cooper@grafana.com>	1 year ago
George Robinson	0a12cabf52	feat: add phase to log lines in kafka_consumer.go (#15176)	1 year ago
Eng Zer Jun	062d1e9ca4	refactor: replace experimental `maps` and `slices` with stdlib (#15051) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	1 year ago
George Robinson	0c388c30ad	chore: add comments to help explain durability guarantees of the consumer (#15105)	1 year ago
Owen Diehl	62e7d61943	refactor(kafka-reader): Separates kafka constructs for better separation of concerns & reuse. (#14982)	1 year ago
benclive	ec95ed1d05	fix: Ensure successive WAL replays don't overwrite each other (#14848)	1 year ago
Cyril Tovena	90c5dbf0d3	chore: Revert syntax module (#14831)	1 year ago
Cyril Tovena	28c9645d10	chore: Introduce logql/syntax go module (#14790)	1 year ago
Vladyslav Diachenko	a962edba33	feat(loki): include structured_metadata size while asserting rate limit (#14571) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
benclive	524ed81395	feat(kafka): Start ingester flush loop before trying to catch up from Kafka (#14505)	1 year ago
benclive	83a8893a3f	feat(kafka): Remove rate limits for kafka ingestion (#14460)	1 year ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	1 year ago
benclive	5cbb23994b	feat(kafka): Implement limiter using partition ring for Kafka (#14359)	1 year ago
benclive	39b57ec4ea	feat(kafka): Replay kafka from last commit before allowing ingesters to become ready (#14330)	1 year ago
Trevor Whitney	7983f94b15	feat: detected field values (#14350)	1 year ago
Cyril Tovena	1a4436c417	feat(ingester): implement partition shuffle sharding for ingester (#14304)	1 year ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
benclive	3c36ba949d	feat: Implement owned streams calculation using Partition Ring (#14282)	1 year ago
Cyril Tovena	beca6f3366	feat(kafka): enqueue commit offset only once per batch process (#14278) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	1 year ago
Cyril Tovena	b6e9945f83	feat(kafka): Add Ingestion from Kafka in Ingesters (#14192) ... Co-authored-by: Joao Marcal <joao.marcal12@gmail.com> Co-authored-by: benclive <ben.clive@grafana.com>	1 year ago
Cyril Tovena	a4aee4f4ff	fix: Wait for OwnedStreams service in Ingester startup (#14208)	1 year ago
Dylan Guedes	a47515300a	fix: Report PSRL error message correctly (#14187) ... **What this PR does / why we need it**: Modify our `validateEntries` code to report a log failure when a PSRL is identified.	1 year ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	1 year ago
Cyril Tovena	44958c4b8b	chore(kafka): Refactor configurations and remove old modules (#14161)	1 year ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	1 year ago
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
Paul Rogers	8fb7b488bd	chore(deps): update dskit 20240819 (#13924)	1 year ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Ed Welch	646c75409e	chore: log stats around chunks being flushed (#13699) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Trevor Whitney	5a87ccb648	feat: instrument failed chunk encoding/decoding (#13684)	1 year ago
George Robinson	71cb767727	chore: replace pkg/errors with errors (#13583)	2 years ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	2 years ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	2 years ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
JordanRushing	e28c15f56c	fix: support multi-zone ingesters when converting global to local limits for streams in limiter.go (#13321) ... Signed-off-by: JordanRushing <rushing.jordan@gmail.com>	2 years ago
Vladyslav Diachenko	5ae5b31b1f	fix: stream ownership check (#13314) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	2 years ago
Christian Haudum	82fbb2fd96	feat(ingester): Smooth out chunk flush operations (#9994) ... In order to prevent spikes in chunk flush operations every flush check period, we smooth out the flush operations over the period of the interval. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ed Welch	f897758747	fix: remove tenant label tagging from profiles to reduce cardinality (#13270) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Vladyslav Diachenko	2ca1ac66a3	feat: flush not owned streams (#13254) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago