56 Commits (36ace66b73e9f9ad2a2d367fbc20803c0d9779c2)

Author	SHA1	Message	Date
Cyril Tovena	b6e9945f83	feat(kafka): Add Ingestion from Kafka in Ingesters (#14192) ... Co-authored-by: Joao Marcal <joao.marcal12@gmail.com> Co-authored-by: benclive <ben.clive@grafana.com>	8 months ago
benclive	7c78232ad3	feat: Optionally require writes to kafka on Push requests (#14186)	8 months ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183) ... This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Vladyslav Diachenko	31368806a9	feat: add backoff mechanism to the retention process (#14182) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	8 months ago
Cyril Tovena	44958c4b8b	chore(kafka): Refactor configurations and remove old modules (#14161)	8 months ago
Cyril Tovena	d178f4c7e2	feat: Implement WAL segment ingestion via Kafka with partition ring (#14043) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	9 months ago
Sven Grossmann	0780456662	feat(max-allowed-line-length): add config to set `max-allowed-line-length` in pattern ingester (#14070) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Paul Rogers	73cbbb0f22	feat: add retries for s3 ObjectExists calls (#14062) ... **What this PR does / why we need it**: It was determined via careful examination that Loki would issue retries for S3 `GetObject` calls, but would not retry calls to `ObjectExists`. For transient issues (e.g. rate-limiting), it makes sense to retry the command to see if an object does exist. If the object is not found (meaning, a 404 return code, and a successful query of the S3 storage to show the object does not exist), the retries are not leveraged, as it is not needed.	9 months ago
Trevor Whitney	4f962ef7af	fix: enable service detection for otlp endoint (#14036) ... Co-authored-by: Sven Grossmann <Svennergr@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	9 months ago
benclive	33c26f4dd1	feat: Add initial support for a kafka-based ingest path (#13992) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	9 months ago
Ned Andreev	f9824b7895	chore: update dskit and thanos/objstore (#13998)	9 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Salva Corts	b5ac6a0258	feat: Limit to block ingestion until configured date (#13958)	9 months ago
Paul Rogers	8fb7b488bd	chore(deps): update dskit 20240819 (#13924)	9 months ago
Trevor Whitney	913e9f9347	feat: aggregate byte and count metrics (#13731)	9 months ago
Trevor Whitney	c6ab6b31e8	chore: remove initial metric aggregation experiment (#13729)	9 months ago
Vladyslav Diachenko	bb257f54b3	feat(loki): add ability to disable AWS S3 dualstack endpoints usage (#13785) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	10 months ago
George Robinson	62c5c5c641	fix: add missing flush op timeout (#13679)	10 months ago
benclive	7553775d72	chore: Remove client header interceptors from metastore client (#13659)	10 months ago
benclive	3a99b69ae5	feat: Add baseline rf1 querier implementation (#13639) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	10 months ago
Cyril Tovena	0b474981dc	feat: Bootstrap metastore for wal segments (#13550)	10 months ago
benclive	c0f5fb6e4d	chore: Clean up old streams periodically in RF-1 ingester (#13511)	10 months ago
George Robinson	c4405fe1c4	feat: WAL Manager configuration options (#13531)	10 months ago
Trevor Whitney	2c053ee00c	feat: downsample aggregated metrics (#13449)	11 months ago
Salva Corts	2cc901ad69	refactor(blooms): Apply retention in planner (#13484)	11 months ago
Cyril Tovena	e08b4a7f88	feat: Pattern ingesters add a limiter for high eviction rate (#13464)	11 months ago
Salva Corts	47f6ea53fc	feat(loki): Do not enforce max-query-bytes-read and max-querier-bytes-read in limited tripperware (#13406)	11 months ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	11 months ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	11 months ago
Salva Corts	8ebce009f3	refactor(blooms): Implement retry in builder (#13306)	11 months ago
Christian Haudum	f06eabbf0e	feat: Add settings for cpu/mutex/block profiling options (#13278) ... The HTTP endpoints for scraping CPU, mutex and block profiles are already exposed by the server, however, block and mutex profiling must be explicitly enabled by setting a non-zero value for `SetBlockProfileRate` and `SetMutexProfileFraction` respectively. For more information about the available options consult the Go documentation: * https://pkg.go.dev/runtime#SetBlockProfileRate * https://pkg.go.dev/runtime#SetMutexProfileFraction * https://pkg.go.dev/runtime#SetCPUProfileRate Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Trevor Whitney	467eb1bb1b	feat: collect and serve pre-aggregated bytes and counts (#13020) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	11 months ago
George Robinson	384f451193	chore: Change default version to Alertmanager APIv2 (#13263)	11 months ago
JordanRushing	e7689b248d	feat: add recalculateOwnedStreams to check stream ownership if the ring is changed (#13103) ... Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	11 months ago
George Robinson	9767807680	feat: Add backoff to flush op (#13140)	12 months ago
Salva Corts	f7887269f8	refactor(blooms): Limit task retries in bloom planner (#13139)	12 months ago
Salva Corts	cd64d6ddbd	refactor(blooms): Implement retry mechanisms in planner (#13064)	12 months ago
Karsten Jeschkies	6e4555010e	fix: Introduce feature flag for [last|first]_over_time sharding. (#13067)	12 months ago
Salva Corts	ab7af0575d	refactor(blooms): Builder retrieves tasks from planner (#13046)	12 months ago
Vladyslav Diachenko	1111595179	feat: new stream count limiter (#13006) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	1 year ago
Salva Corts	efd8f5dc1b	refactor(blooms): Add queue to bloom planner and enqueue tasks (#13005)	1 year ago
Salva Corts	3195036435	refactor(bloom planner): Compute gaps and build tasks from metas and TSDBs (#12994)	1 year ago
Salva Corts	8978ecf0c8	feat: Boilerplate for new bloom build planner and worker components. (#12989)	1 year ago
Dylan Guedes	18a558c7fd	docs: Don't allow running new and old querier worker grpc clients (#12916)	1 year ago
Dylan Guedes	afd9e36306	chore: Call `shardstreams.Config` by value instead of by reference (#12915) ... Use `shardstreams.Config` by value instead of by reference to fix docs generation. Our `docs-generator` tool relies on the struct address/references to assume that flags are present. Using this config by value fixes it.	1 year ago
Dylan Guedes	7b6f0577c3	feat: Querier: Split gRPC client into two. (#12726) ... **What this PR does / why we need it**: Split the gRPC client used by the querier into two, one for the communication with the scheduler, the other for communicating with the query-frontend. - This change is retrocompatible: you don't have to change anything to keep existing behavior. - To configure the custom scheduler grpc client, you can use the new `query_scheduler_grpc_client` config or the new CLI flag `querier.scheduler-grpc-client` - If you'd like to configure your frontend grpc client using a better named section, you can use the new `query_frontend_grpc_client` instead of the old `grpc_client_config`. Just make sure you don't use both at the same time, it will result in an error. This work is necessary for configuring custom behavior between `querier<->scheduler` vs `querier<->frontend`. A use case is configuring mTLS when a different certificate is used by queriers, schedulers and frontends. You can only configure a single `server_name` with our current setup, making it impossible.	1 year ago
Shantanu Alshi	6904a6520d	fix: Fix log level detection (#12651)	1 year ago
Christian Haudum	fcd544c2d9	feat(blooms): Add in-memory LRU cache for meta files (#12862) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	bc78d13d9b	feat(blooms): compute chunks once (#12664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Owen Diehl	eac56224b8	feat(blooms): limit bloom size during creation (#12796)	1 year ago