484 Commits (d7c0f2bfc136155bbdde79fe481ebddee4e84023)

Author	SHA1	Message	Date
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
benclive	3c36ba949d	feat: Implement owned streams calculation using Partition Ring (#14282)	2 years ago
Cyril Tovena	beca6f3366	feat(kafka): enqueue commit offset only once per batch process (#14278) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	2 years ago
Cyril Tovena	b6e9945f83	feat(kafka): Add Ingestion from Kafka in Ingesters (#14192) ... Co-authored-by: Joao Marcal <joao.marcal12@gmail.com> Co-authored-by: benclive <ben.clive@grafana.com>	2 years ago
Cyril Tovena	a4aee4f4ff	fix: Wait for OwnedStreams service in Ingester startup (#14208)	2 years ago
Dylan Guedes	a47515300a	fix: Report PSRL error message correctly (#14187) ... **What this PR does / why we need it**: Modify our `validateEntries` code to report a log failure when a PSRL is identified.	2 years ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	2 years ago
Cyril Tovena	44958c4b8b	chore(kafka): Refactor configurations and remove old modules (#14161)	2 years ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	2 years ago
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Paul Rogers	8fb7b488bd	chore(deps): update dskit 20240819 (#13924)	2 years ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Ed Welch	646c75409e	chore: log stats around chunks being flushed (#13699) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Trevor Whitney	5a87ccb648	feat: instrument failed chunk encoding/decoding (#13684)	2 years ago
George Robinson	71cb767727	chore: replace pkg/errors with errors (#13583)	2 years ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	2 years ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	2 years ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
JordanRushing	e28c15f56c	fix: support multi-zone ingesters when converting global to local limits for streams in limiter.go (#13321) ... Signed-off-by: JordanRushing <rushing.jordan@gmail.com>	2 years ago
Vladyslav Diachenko	5ae5b31b1f	fix: stream ownership check (#13314) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	2 years ago
Christian Haudum	82fbb2fd96	feat(ingester): Smooth out chunk flush operations (#9994) ... In order to prevent spikes in chunk flush operations every flush check period, we smooth out the flush operations over the period of the interval. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ed Welch	f897758747	fix: remove tenant label tagging from profiles to reduce cardinality (#13270) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Vladyslav Diachenko	2ca1ac66a3	feat: flush not owned streams (#13254) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Vladyslav Diachenko	ce8645900e	feat: improve Owned Streams feature observability (#13232) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Vladyslav Diachenko	7ac19f00b4	fix: update fixed limit once streams ownership re-checked (#13231) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Ed Welch	32097c8462	fix: Redo ingester profile tagging (#13239) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
JordanRushing	e7689b248d	feat: add recalculateOwnedStreams to check stream ownership if the ring is changed (#13103) ... Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	2 years ago
George Robinson	9767807680	feat: Add backoff to flush op (#13140)	2 years ago
Callum Styan	691b174138	fix: properly return http status codes from ingester to querier for RPC function calls (#13134) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Yuri Kotov	808425953f	feat: API: Expose optional label matcher for label names API (#11982) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	2 years ago
Ed Welch	00d3c7a52d	feat: add profile tagging to ingester (#13068) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: poyzannur <poyzannur.taneli@grafana.com> Co-authored-by: Poyzan <31743851+poyzannur@users.noreply.github.com>	2 years ago
George Robinson	cc3694eecd	feat: Add ingester_chunks_flush_failures_total (#12925) ... Signed-off-by: George Robinson <george.robinson@grafana.com>	2 years ago
Vladyslav Diachenko	4901a5c452	fix: not owned stream count (#13030) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Vladyslav Diachenko	1111595179	feat: new stream count limiter (#13006) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	2 years ago
Ashwanth	bf8a278879	chore: remove duplicate imports (#13001)	2 years ago
Ashwanth	1f5291a4a3	fix(indexstats): do not collect stats from "IndexStats" lookups for other query types (#12978)	2 years ago
Karsten Jeschkies	88c671162f	fix: Track bytes discarded by ingester. (#12981) ... **What this PR does / why we need it**: Only the distributor was tracking discarded bytes. The ingester was missing the tracker and calls. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] Title matches the required conventional commits format, see [here](https://www.conventionalcommits.org/en/v1.0.0/) - **Note** that Promtail is considered to be feature complete, and future development for logs collection will be in [Grafana Alloy](https://github.com/grafana/alloy). As such, `feat` PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
benclive	4a5edf1a2a	perf: Replace channel check with atomic bool in tailer.send() (#12976)	2 years ago
Shantanu Alshi	e7fdeb974a	perf: Improve Detected labels API (#12816)	2 years ago
Kaviraj Kanagaraj	b05c4f7288	chore(logging): Add entry's timestamp when rejected with `too far behind` (#12933) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Dylan Guedes	afd9e36306	chore: Call `shardstreams.Config` by value instead of by reference (#12915) ... Use `shardstreams.Config` by value instead of by reference to fix docs generation. Our `docs-generator` tool relies on the struct address/references to assume that flags are present. Using this config by value fixes it.	2 years ago
Owen Diehl	bc78d13d9b	feat(blooms): compute chunks once (#12664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Shantanu Alshi	587a6d20e9	feat: Detected labels from store (#12441)	2 years ago
Trevor Whitney	6c33809015	feat: split detected fields queries (#12491)	2 years ago
Callum Styan	ec81991f4d	feat: include the stream we failed to create in the stream limit error message (#12437) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Ed Welch	670041368a	chore: add more validation at startup to make sure we have a proper schema, and necessary directories are created (#12496) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Cyril Tovena	491d2513e7	feat: add log patterns ingestion and query API. (#12403) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Anton Kolesnikov <anton.e.kolesnikov@gmail.com> Co-authored-by: Sven Grossmann <svennergr@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Shantanu Alshi	2c878c8301	fix: Bugfixes for detected labels API (#12466)	2 years ago
Ed Welch	24f4e45165	feat: Enable automatic stream sharding by default (#12451) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago