166 Commits (69089eff41a6dda2d4ab22aa5524bcfbbc44b00d)

Author	SHA1	Message	Date
Dylan Guedes	69089eff41	feat: Add support for blocking a policy to be ingested (#16203) ... **What this PR does / why we need it**: Add support for blocking a policy from being ingested. The policy is defined by stream mapping.	10 months ago
Salva Corts	5fda84bbf0	feat(policies): Per policy enforced labels (#16182)	10 months ago
Dylan Guedes	41d9c8b5ec	fix: Make policy test non-flaky (#16158)	10 months ago
Dylan Guedes	eda27ddebd	docs: Fix config for stream policies mapping (#16104)	10 months ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	10 months ago
Ashwanth	86dc47b3db	feat: update log level discovery from json to detect nested fields (#16026)	11 months ago
Dylan Guedes	0bc23db95d	refactor: Add retention hours to `discarded` metrics (#15875) ... **What this PR does / why we need it**: We have no distinction between the type of data that gets discarded. On this PR I'm adding `retention_hours` to our discarded metrics so that we can better look at what data was discarded.	11 months ago
Emma Neiss	878727d348	docs: Fix typos in limits.go (#15840) ... Signed-off-by: Emma Neiss <33004979+eneiss@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	11 months ago
Christian Haudum	1a9f382e6f	chore: Remove experimental bloom settings that have never been used or are not used any more (#15787) ... Remove experimental results cache for bloom gateway client This feature never made it into production. Since it is only an experimental feature, it can be removed without prior deprecation. Remove unused `bloom_gateway_shard_size` runtime setting The setting was once used when bloom gateways still used a ring for sharding. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Dylan Guedes	a2c8ec7dc1	feat: Add a new `enforced_labels` limit (#15704) ... **What this PR does / why we need it**: Introduce the new experimental `enforced_labels` limit. By default it is empty but when configured, it configures Loki to only accept push requests which streams have all enforced labels.	11 months ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Ravishankar	17bf32bc50	fix: Return error 422 (Unprocessable Content) when push request does not contain any streams (#13706)	1 year ago
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Shantanu Alshi	867ce3d711	chore(level_detection): Make log level detection configurable (#14784)	1 year ago
Karsten Jeschkies	7b53f20f70	feat: Introduce shardable probabilistic topk for instant queries. (#14243) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Trevor Whitney	c1fde26730	feat: move metric aggregation to a per-tenant config (#14709)	1 year ago
Trevor Whitney	f2a16f43b2	feat: add app_name as a service label (#13660)	1 year ago
Ashwanth	8bca2e7608	feat(ruler): enables ruler store that uses clients from thanos-io/objstore pkg (#11713)	1 year ago
Vladyslav Diachenko	a962edba33	feat(loki): include structured_metadata size while asserting rate limit (#14571) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Salva Corts	673ede16a5	feat: New bloom planning using chunk size TSDB stats (#14547)	1 year ago
Salva Corts	70508975fd	feat: Configurable list of json fields to mine patterns (#14528)	1 year ago
Salva Corts	ab5e6eaaee	feat: Extract task computing into a strategy interface (#13690)	1 year ago
Cyril Tovena	1a4436c417	feat(ingester): implement partition shuffle sharding for ingester (#14304)	1 year ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183) ... This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	1 year ago
Trevor Whitney	4f962ef7af	fix: enable service detection for otlp endoint (#14036) ... Co-authored-by: Sven Grossmann <Svennergr@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	1 year ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	b5ac6a0258	feat: Limit to block ingestion until configured date (#13958)	1 year ago
Owen Diehl	4220737a52	feat(exporter): include boolean values in limit exporter (#13466)	1 year ago
Salva Corts	47f6ea53fc	feat(loki): Do not enforce max-query-bytes-read and max-querier-bytes-read in limited tripperware (#13406)	1 year ago
Salva Corts	f7887269f8	refactor(blooms): Limit task retries in bloom planner (#13139)	2 years ago
Salva Corts	cd64d6ddbd	refactor(blooms): Implement retry mechanisms in planner (#13064)	2 years ago
Vladyslav Diachenko	1111595179	feat: new stream count limiter (#13006) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	2 years ago
Salva Corts	efd8f5dc1b	refactor(blooms): Add queue to bloom planner and enqueue tasks (#13005)	2 years ago
Salva Corts	3195036435	refactor(bloom planner): Compute gaps and build tasks from metas and TSDBs (#12994)	2 years ago
Sandeep Sukhani	7cc9a9386a	fix: fix parsing of default per tenant otlp config (#12836)	2 years ago
Dylan Guedes	afd9e36306	chore: Call `shardstreams.Config` by value instead of by reference (#12915) ... Use `shardstreams.Config` by value instead of by reference to fix docs generation. Our `docs-generator` tool relies on the struct address/references to assume that flags are present. Using this config by value fixes it.	2 years ago
Shantanu Alshi	6904a6520d	fix: Fix log level detection (#12651)	2 years ago
Owen Diehl	7bbd8b5087	feat(blooms): ingester aware bounded impl (#12840)	2 years ago
Owen Diehl	bc78d13d9b	feat(blooms): compute chunks once (#12664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Owen Diehl	eac56224b8	feat(blooms): limit bloom size during creation (#12796)	2 years ago
Func	4a05964d55	fix: Use to the proper config names in warning messages (#12114) ... Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Cyril Tovena	397aa56e15	feat: Enable log volume endpoint by default (#12628)	2 years ago
Callum Styan	ec81991f4d	feat: include the stream we failed to create in the stream limit error message (#12437) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Christian Haudum	a9345d0fb6	chore(blooms): Clean up settings and make reasonable defaults (#12483) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Cyril Tovena	9e6e6d22e0	feat: Detect level as structured metadata by default (#12479)	2 years ago
Ed Welch	7607f04484	feat!: enable structure metadata by default. (#12453) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Sandeep Sukhani	b66c343383	feat: add support for discovering and adding log levels as structured metadata (#12428)	2 years ago