369 Commits (241f5aa6490b09b299c1136e70e5b67f85b8632d)

Author	SHA1	Message	Date
renovate[bot]	7b35a41c17	fix(deps): update module golang.org/x/oauth2 to v0.27.0 [security] (release-3.3.x) (#16591) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	2 months ago
renovate[bot]	0546ca5908	chore(deps): update module golang.org/x/net to v0.33.0 [security] (release-3.3.x) (#15892) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	4 months ago
renovate[bot]	c9407cba74	chore(deps): update module golang.org/x/crypto to v0.31.0 [security] (release-3.3.x) (#15891) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	4 months ago
loki-gh-app[bot]	02eb02458e	feat: Introduce shardable probabilistic topk for instant queries. (backport k227) (#14765) ... Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	7 months ago
renovate[bot]	53a1ab7625	fix(deps): update aws-sdk-go-v2 monorepo (#14742) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	7 months ago
Paul Rogers	2f41584c4c	chore: Update docs, dockerfiles to use latest versions of promtail&loki (#14728)	7 months ago
renovate[bot]	1f9beae0cc	chore(deps): update terraform google to v6.9.0 (#14718) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	7 months ago
renovate[bot]	8c38f67b04	chore(deps): update dependency pytz to v2019.3 (#14669) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	7 months ago
Ashwanth	8bca2e7608	feat(ruler): enables ruler store that uses clients from thanos-io/objstore pkg (#11713)	7 months ago
mwimpelberg28	db0889e274	fix: Add s3 principal to iam policy attached to sqs in lambda-promtail terraform code (#14619)	7 months ago
jinjiadu	99c423a1f5	chore: fix some function names (#14628) ... Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	7 months ago
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132) ... Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Paul Rogers	71d1c6bcd4	docs: Pre-format example for named_stores_config (#14524)	7 months ago
renovate[bot]	c2f38e18c6	fix(deps): update github.com/grafana/dskit digest to 687ec48 (#14395) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	8 months ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	8 months ago
renovate[bot]	9d7a6ea680	fix(deps): update github.com/grafana/dskit digest to b69ac1b (#14355) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	8 months ago
renovate[bot]	ae09f4fed2	chore(deps): update terraform google to v6.5.0 (#14344) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	8 months ago
renovate[bot]	a4f3edfb52	fix(deps): update github.com/grafana/dskit digest to f52de24 (#14319) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	8 months ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Paul Rogers	6875c872a6	chore: Move to Go 1.23.1, along with golangci update (#14294)	8 months ago
renovate[bot]	f39cdbd541	fix(deps): update github.com/grafana/dskit digest to 7c41a40 (#14277) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	8 months ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Christian Haudum	e64124ec1a	chore(blooms): Expose series/stream metadata when iterating over blooms (#14097) ... The bloom block format stores metadata from indexing in the index part of the block (series file), which contains information, such as the fields (metadata names) that have been indexed in the bloom filter. This will be needed in the read path (bloom gateway) later on, to check whether a provided label name from the label filter expression was actually indexed or not. In this PR, the newly available metadata is used by the bloom inspect tool (located in ./tools/bloom/inspect/) to list the fields along the series, as well as the individual pages of the series that contain the actual scalable bloom filters. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	a2fbaa8e09	feat(blooms)!: Index structured metadata into blooms (#14061) ... Instead of indexing n-grams of the log line content, we index the plain values of the structured metadata keys and values. Resulting tokens are: * `name` * `chunkPrefix + name` * `value` * `chunkPrefix + value` * `name + '=' value` * `chunkPrefix + name + '=' + value` Indexed fields (metadata name) are also extracted into the series metadata. Indexed metadata values cannot be queried with the bloom gateways yet. This PR does not cleanup unused code used for ngram-tokenization, it is scope for a follow up. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
renovate[bot]	7c18642521	fix(deps): update github.com/grafana/dskit digest to 931a021 (#14032) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	8 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
renovate[bot]	24ad3bac21	chore(deps): update terraform google to v6 (#14040) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	9 months ago
renovate[bot]	6f491233ca	fix(deps): update aws-sdk-go-v2 monorepo (#13986) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	9 months ago
Ned Andreev	f9824b7895	chore: update dskit and thanos/objstore (#13998)	9 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
renovate[bot]	ba87d5227c	chore(deps): update terraform google to v5 (#13962) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	9 months ago
Paul Rogers	07f916936b	chore(deps): lamba-promtail, move back to al2, update krb5-libs (#13939)	9 months ago
Paul Rogers	1527eadfb5	chore(deps): LambdaPromtail - Bump to AmazonLinux 2023 (#13938)	9 months ago
Paul Rogers	8319d1cb81	chore(deps): Update dependencies, build version, and go version for lambda-promtail (#13904)	9 months ago
Max Forasteiro	9bf08f7cc0	feat(lambda-promtail): build lambda with zip file (#13787) ... Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	9 months ago
renovate[bot]	fd69813bbb	chore(deps): update alpine docker tag to v3.20.2 (#13882) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	9 months ago
renovate[bot]	ea6395a2eb	chore(deps): update golang docker tag to v1.22.6 (#13847) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	9 months ago
samuelebstein	2d92fff2aa	feat(lambda-promtail): Adding S3 log parser support for AWS GuardDuty (#13148)	10 months ago
renovate[bot]	c9c50885cb	chore(deps): update confluentinc/cp-kafka docker tag to v6.2.15 (main) (#11283) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	10 months ago
Alexandre Steppé	dbfb19be49	feat: lambda-promtail; ensure messages to Kinesis are usable by refactoring parsing of KinesisEvent to match parsing of CWEvents + code cleanup (#13098)	10 months ago
Paul Rogers	ca2f11d93e	chore: Update Docker images to Go 1.22.5 (#13398)	11 months ago
Christian Haudum	f06eabbf0e	feat: Add settings for cpu/mutex/block profiling options (#13278) ... The HTTP endpoints for scraping CPU, mutex and block profiles are already exposed by the server, however, block and mutex profiling must be explicitly enabled by setting a non-zero value for `SetBlockProfileRate` and `SetMutexProfileFraction` respectively. For more information about the available options consult the Go documentation: * https://pkg.go.dev/runtime#SetBlockProfileRate * https://pkg.go.dev/runtime#SetMutexProfileFraction * https://pkg.go.dev/runtime#SetCPUProfileRate Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093)	12 months ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162)	12 months ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039) ... This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Paul Rogers	d16a3bf139	test: First pass at unregistering metrics so that we can run multiple tests (#12927)	1 year ago
Dylan Guedes	7b6f0577c3	feat: Querier: Split gRPC client into two. (#12726) ... **What this PR does / why we need it**: Split the gRPC client used by the querier into two, one for the communication with the scheduler, the other for communicating with the query-frontend. - This change is retrocompatible: you don't have to change anything to keep existing behavior. - To configure the custom scheduler grpc client, you can use the new `query_scheduler_grpc_client` config or the new CLI flag `querier.scheduler-grpc-client` - If you'd like to configure your frontend grpc client using a better named section, you can use the new `query_frontend_grpc_client` instead of the old `grpc_client_config`. Just make sure you don't use both at the same time, it will result in an error. This work is necessary for configuring custom behavior between `querier<->scheduler` vs `querier<->frontend`. A use case is configuring mTLS when a different certificate is used by queriers, schedulers and frontends. You can only configure a single `server_name` with our current setup, making it impossible.	1 year ago
Christian Haudum	57f78b574a	fix(packaging): Require online network in systemd unit file for Loki and Promtail (#12741) ... From https://systemd.io/NETWORK_ONLINE/: **How do I make sure that my service starts after the network is really online?** That depends on your setup and the services you plan to run after it (see above). If you need to delay you service after network connectivity has been established, include ```systemd After=network-online.target Wants=network-online.target ``` in the `.service` file. This will delay boot until the network management software says the network is “up”. For details, see the next question. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Tanat Lokejaroenlarb	8892dc8923	feat: parameterise the MaximumEventAgeInSeconds, LogGroupName, and IAMRoleName for lambda-promtail CloudFormation template (#12728)	1 year ago