197 Commits (main)

Author	SHA1	Message	Date
Christian Haudum	326c7d1ea0	feat: Client side index gateway shuffle sharding (#20124) ... This PR adds the functionality to add a client-side shuffle sharding for index gateway requests. The per-tenant configuration index_gateway_max_capacity defines a fraction (0.0 to 1.0) of index gateways that become available for each tenant. The client uses the jumphash of the tenant ID to achieve greatest possible consitant hashing without using a server-side, statefule hash ring. This will allow to distribute workloads from different tenants across all index gateway instances and prevent individual tenants from overloading all index gateways. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
Francisco Robles Martín	129ebd696d	docs: Fix max_line_size default behavior comment (#20067) ... Signed-off-by: Francisco Robles Martín <f.robles.martin@pm.me> Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 month ago
Trevor Whitney	b8b9648ae4	docs: document the json pattern tokenizable field limits (#19964)	2 months ago
Robert Fratto	0f0846adec	chore(engine): make stream/task debug logging configurable (#19842) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Salva Corts	48d13d15db	feat: Add resolved policy to blocked and enforced label error (#19826)	2 months ago
Christian Haudum	4b118fb729	chore: Admission lanes for workflow tasks (#19584) ... This PR introduces "admission lanes" for dispatching tasks to the runner (scheduler), that are controlled by token buckets, to avoid overflowing the runner with a large amount of tasks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Salva Corts	eacd943646	fix: Forward ingestion policy set through header on kafka (#19632)	2 months ago
Salva Corts	2bbdfa54e7	fix: Pass ingestion policy set through header to ingesters via grpc (#19630)	2 months ago
Salva Corts	8486acb981	feat: support per policy overriden stream limits on limits service (#19403)	3 months ago
Salva Corts	987840b5d3	feat: Resolve ingestion policy via a header (#19548)	3 months ago
Trevor Whitney	26f8b992ee	fix: correctly register allowed log level fields (#19261)	3 months ago
Trevor Whitney	8b38d8e418	Revert "fix: correctly register allowed log level fields" ... This reverts commit d8c4997120.	4 months ago
Trevor Whitney	d8c4997120	fix: correctly register allowed log level fields	4 months ago
Sandeep Sukhani	8057c5c424	fix: apply global otlp config to tenant config only when it is updated in the overrides (#19213)	4 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206)	4 months ago
Salva Corts	af257fd058	feat: per policy overridable stream limits (#18994)	4 months ago
Salva Corts	3b2498b896	feat: Default ingestion policy mappings merged with per-tenant mappings (#18926)	5 months ago
Trevor Whitney	c690827c6a	fix: limit volume and frequency of persisted patterns (#18362) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Shantanu Alshi	61e81c5a4c	chore(otlp): OTLP Metrics (#18281)	6 months ago
Christian Haudum	0ee8e76ba2	feat: Mark truncated log lines with identifier (#18262) ... While truncated JSON log lines are usually easy to identify (because the JSON becomes invalid), it is not always the case with other plain text log lines. This PR adds the possibility to add an identifier as suffix for truncated log lines, such as `[...]` so that truncated log lines are easier to spot and identify when querying them. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Trevor Whitney	888e4012bc	chore: clean up pattern persistence configs (#18017)	7 months ago
Trevor Whitney	fd7321c88b	feat: persist patterns as aggregated metrics (#17737) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	7 months ago
Periklis Tsirakidis	1d99f4d86d	feat(distributor): Add MaxRecvMsgSize config for uncompressed message size limits (#16915)	9 months ago
Trevor Whitney	eda3ba865b	feat(ruler): add per-tenant configuration to disable WAL replay (#16717)	10 months ago
Shantanu Alshi	4d1f5a774d	chore(otel): Allow log attributes to be promoted to labels (#16673)	10 months ago
Trevor Whitney	23800f6100	chore: convert multi-variant queries from global to per-tenant config (#16718)	10 months ago
Cyril Tovena	7119f6e480	feat(distributor): Add simulated latency (#16733)	10 months ago
Dylan Guedes	ea13730e63	feat: Snapshopt stream overrides values on each request (#16523) ... **What this PR does / why we need it**: Caches a snapshot of resolved policies and retention temporarily to guarantee consistent results on the same request. We evaluate retention and policies in a map when emitting some metrics so having policies or retention inconsistent along the chain for the same push request can cause us to miscalculate things or worse, interact with a nil map.	10 months ago
Salva Corts	837b70ac78	feat(policies): Support global policy (#16439)	10 months ago
Dylan Guedes	3d6163abc7	fix: Fix blocked ingestion returned error when 260 (#16387) ... **What this PR does / why we need it**: Modify how we handle blocked ingestion errors: * If it is 200, don't expose any error * Otherwise, expose the error as the configured statusCode, but only if no other validation error occur. I'm calling this a secondTierErr Also modifying the enforced labels error message to log present labels.	11 months ago
Salva Corts	0be39a0ed1	feat(policies): Log when multiple policies match the same stream (#16321)	11 months ago
Dylan Guedes	69089eff41	feat: Add support for blocking a policy to be ingested (#16203) ... **What this PR does / why we need it**: Add support for blocking a policy from being ingested. The policy is defined by stream mapping.	11 months ago
Salva Corts	5fda84bbf0	feat(policies): Per policy enforced labels (#16182)	11 months ago
Dylan Guedes	41d9c8b5ec	fix: Make policy test non-flaky (#16158)	11 months ago
Dylan Guedes	eda27ddebd	docs: Fix config for stream policies mapping (#16104)	11 months ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	11 months ago
Ashwanth	86dc47b3db	feat: update log level discovery from json to detect nested fields (#16026)	11 months ago
Dylan Guedes	0bc23db95d	refactor: Add retention hours to `discarded` metrics (#15875) ... **What this PR does / why we need it**: We have no distinction between the type of data that gets discarded. On this PR I'm adding `retention_hours` to our discarded metrics so that we can better look at what data was discarded.	12 months ago
Emma Neiss	878727d348	docs: Fix typos in limits.go (#15840) ... Signed-off-by: Emma Neiss <33004979+eneiss@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	12 months ago
Christian Haudum	1a9f382e6f	chore: Remove experimental bloom settings that have never been used or are not used any more (#15787) ... Remove experimental results cache for bloom gateway client This feature never made it into production. Since it is only an experimental feature, it can be removed without prior deprecation. Remove unused `bloom_gateway_shard_size` runtime setting The setting was once used when bloom gateways still used a ring for sharding. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Dylan Guedes	a2c8ec7dc1	feat: Add a new `enforced_labels` limit (#15704) ... **What this PR does / why we need it**: Introduce the new experimental `enforced_labels` limit. By default it is empty but when configured, it configures Loki to only accept push requests which streams have all enforced labels.	1 year ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ravishankar	17bf32bc50	fix: Return error 422 (Unprocessable Content) when push request does not contain any streams (#13706)	1 year ago
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Shantanu Alshi	867ce3d711	chore(level_detection): Make log level detection configurable (#14784)	1 year ago
Karsten Jeschkies	7b53f20f70	feat: Introduce shardable probabilistic topk for instant queries. (#14243) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Trevor Whitney	c1fde26730	feat: move metric aggregation to a per-tenant config (#14709)	1 year ago
Trevor Whitney	f2a16f43b2	feat: add app_name as a service label (#13660)	1 year ago
Ashwanth	8bca2e7608	feat(ruler): enables ruler store that uses clients from thanos-io/objstore pkg (#11713)	1 year ago
Vladyslav Diachenko	a962edba33	feat(loki): include structured_metadata size while asserting rate limit (#14571) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago