571 Commits (867ce3d711f3b9e823e8eb9b61c076acca339096)

Author	SHA1	Message	Date
Christian Haudum	022847b607	chore: Update targets list in documentation (#14567) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Trevor Whitney	80aec25482	feat: Improve pattern ingester tracing (#14707)	7 months ago
Trevor Whitney	c1fde26730	feat: move metric aggregation to a per-tenant config (#14709)	7 months ago
Salva Corts	31eea9042a	fix: Add tenant limits as dependency to pattern ingester (#14665)	7 months ago
Ashwanth	8bca2e7608	feat(ruler): enables ruler store that uses clients from thanos-io/objstore pkg (#11713)	7 months ago
Salva Corts	70508975fd	feat: Configurable list of json fields to mine patterns (#14528)	7 months ago
Ashwanth	b405ae8557	chore: hide docs for thanos clients + restore original cli prefix (#14545)	7 months ago
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132) ... Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
benclive	633bb5eb7e	feat(kafka): Enable querier to optionally query partition ingesters (#14418)	8 months ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	8 months ago
Cyril Tovena	8438d41593	fix(distributor): validate partition ring is kafka is enabled (#14303)	8 months ago
Trevor Whitney	7983f94b15	feat: detected field values (#14350)	8 months ago
Robert Jacob	59ff1ece1d	fix(config): Copy Alibaba and IBM object storage configuration from common (#14297)	8 months ago
George Robinson	6354deda90	fix: missing dep PartitionRing for Ingester (#14292)	8 months ago
benclive	3c36ba949d	feat: Implement owned streams calculation using Partition Ring (#14282)	8 months ago
Cyril Tovena	b6e9945f83	feat(kafka): Add Ingestion from Kafka in Ingesters (#14192) ... Co-authored-by: Joao Marcal <joao.marcal12@gmail.com> Co-authored-by: benclive <ben.clive@grafana.com>	8 months ago
benclive	7c78232ad3	feat: Optionally require writes to kafka on Push requests (#14186)	8 months ago
Cyril Tovena	44958c4b8b	chore(kafka): Refactor configurations and remove old modules (#14161)	8 months ago
Christian Haudum	8307c42c54	fix(blooms): Fix panic in initialisation of the bloom planner and builder (#14110) ... The targets `bloom-planner` and `bloom-builder` panic at startup on branch `main`. ```console $ make loki && ./cmd/loki/loki -target=bloom-builder CGO_ENABLED=0 go build -ldflags "-extldflags \"-static\" -s -w -X github.com/grafana/loki/v3/pkg/util/build.Branch=main -X github.com/grafana/loki/v3/pkg/util/build.Version=main-b29b4b4 -X github.com/grafana/loki/v3/pkg/util/build.Revision=b29b4b4bb -X github.com/grafana/loki/v3/pkg/util/build.BuildUser=christian@grafana1210 -X github.com/grafana/loki/v3/pkg/util/build.BuildDate=2024-09-11T08:30:11Z" -tags netgo -o cmd/loki/loki ./cmd/loki panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x287b590] goroutine 1 [running]: github.com/grafana/dskit/ring.(*BasicLifecycler).GetInstanceID(...) /home/christian/sandbox/grafana/loki/vendor/github.com/grafana/dskit/ring/basic_lifecycler.go:132 github.com/grafana/loki/v3/pkg/bloombuild/builder.New({{0x6400000, 0x6400000, {0x0, 0x0}, 0x0, 0x0, 0x0, {0x5f5e100, 0x2540be400, 0xa}, ...}, ...}, ...) /home/christian/sandbox/grafana/loki/pkg/bloombuild/builder/builder.go:92 +0x470 github.com/grafana/loki/v3/pkg/loki.(*Loki).initBloomBuilder(0xc002984008) /home/christian/sandbox/grafana/loki/pkg/loki/modules.go:1750 +0x4b6 github.com/grafana/dskit/modules.(*Manager).initModule(0xc000d2e6f0, {0x7ffe36edcb45, 0xd}, 0xc0027a22f8, 0xc0013ce960) /home/christian/sandbox/grafana/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x1ea github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc000d2e6f0, {0xc000dbeed0, 0x1, 0x7510c18f88e2e5ce?}) /home/christian/sandbox/grafana/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 +0xe8 github.com/grafana/loki/v3/pkg/loki.(*Loki).Run(0xc002984008, {0x0?, {0x4?, 0x2?, 0x6457fc0?}}) /home/christian/sandbox/grafana/loki/pkg/loki/loki.go:497 +0x97 main.main() /home/christian/sandbox/grafana/loki/cmd/loki/main.go:129 +0x131e ```` This is because `t.indexGatewayRingManager` is not nil and therefore incorrectly tries to initialize the ring watcher, which fails. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Cyril Tovena	d178f4c7e2	feat: Implement WAL segment ingestion via Kafka with partition ring (#14043) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	9 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Christian Haudum	bf60455c8e	feat(blooms): Add bloom planner and bloom builder to `backend` target (#13997) ... Previously, the bloom compactor component was part of the `backend` target in the Simple Scalable Deployment (SSD) mode. However, the bloom compactor was removed (https://github.com/grafana/loki/pull/13969) in favour of planner and builder, and therefore also removed from the backend target. This PR adds the planner and builder components to the backend target so it can continue building blooms if enabled. The planner needs to be run as singleton, therefore there must only be one instance that creates tasks for the builders, even if multiple replicas of the backend target are deployed. This is achieved by leader election through the already existing index gateway ring in the backend target. The planner leader is determined by the ownership of the leader key. Builders connect to the planner leader to pull tasks. ---- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
benclive	33c26f4dd1	feat: Add initial support for a kafka-based ingest path (#13992) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	9 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Trevor Whitney	913e9f9347	feat: aggregate byte and count metrics (#13731)	9 months ago
Trevor Whitney	c6ab6b31e8	chore: remove initial metric aggregation experiment (#13729)	10 months ago
benclive	00e686ddce	chore: Add metastore client as dep for rf1 querier & ignore auth for ListBlocks (#13786)	10 months ago
Vladyslav Diachenko	bb257f54b3	feat(loki): add ability to disable AWS S3 dualstack endpoints usage (#13785) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	10 months ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	10 months ago
Cyril Tovena	917053a730	feat: Introduce wal segment read path. (#13695) ... Co-authored-by: Ben Clive <ben.clive@grafana.com>	10 months ago
benclive	7553775d72	chore: Remove client header interceptors from metastore client (#13659)	10 months ago
benclive	3a99b69ae5	feat: Add baseline rf1 querier implementation (#13639) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	10 months ago
Cyril Tovena	04613b492f	feat: Instrument metastore GRPC calls (#13598)	10 months ago
Cyril Tovena	0b474981dc	feat: Bootstrap metastore for wal segments (#13550)	10 months ago
George Robinson	47d057854f	chore: Fix not ready error message in RF-1 Ingester (#13546)	10 months ago
Christian Haudum	5f4b8fc9e4	fix: Fix panic in BloomStore initialisation (#13457) ... The initialisation of the `BloomStore` service failed with a panic if the index gateways were configured to use `ring` mode. This happened, because the bloom store requires to wire up the configuration for the shipper store, see https://github.com/grafana/loki/blob/1a508b0aa/pkg/loki/modules.go#L864-L867 At this time, the index gateway ring manager wasn't initialized yet, leading to a `nil` pointer dereference. This PR fixes the issue by adding the index gateway ring as dependency of the bloom store so it gets initialised first. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
benclive	6da364e94f	chore: Set memberlist func for ingester-rf1 init (#13424)	11 months ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	11 months ago
benclive	0f9088bc89	chore: use read-only index store for ingester RF1 (#13419)	11 months ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	11 months ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	11 months ago
Salva Corts	0ddf1e6113	refactor(blooms): Bloom building integration test (#13296)	11 months ago
Christian Haudum	f06eabbf0e	feat: Add settings for cpu/mutex/block profiling options (#13278) ... The HTTP endpoints for scraping CPU, mutex and block profiles are already exposed by the server, however, block and mutex profiling must be explicitly enabled by setting a non-zero value for `SetBlockProfileRate` and `SetMutexProfileFraction` respectively. For more information about the available options consult the Go documentation: * https://pkg.go.dev/runtime#SetBlockProfileRate * https://pkg.go.dev/runtime#SetMutexProfileFraction * https://pkg.go.dev/runtime#SetCPUProfileRate Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Trevor Whitney	467eb1bb1b	feat: collect and serve pre-aggregated bytes and counts (#13020) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	11 months ago
JordanRushing	e7689b248d	feat: add recalculateOwnedStreams to check stream ownership if the ring is changed (#13103) ... Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	12 months ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162)	12 months ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039) ... This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Salva Corts	fa2c78936f	refactor(blooms): Build new metas and blocks (#13074)	12 months ago
Salva Corts	3195036435	refactor(bloom planner): Compute gaps and build tasks from metas and TSDBs (#12994)	1 year ago