164 Commits (9d0d1084e39fdfd4a58d701b13ff755f4e5f36eb)

Author	SHA1	Message	Date
Robert Fratto	0f0846adec	chore(engine): make stream/task debug logging configurable (#19842) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Christian Haudum	4b118fb729	chore: Admission lanes for workflow tasks (#19584) ... This PR introduces "admission lanes" for dispatching tasks to the runner (scheduler), that are controlled by token buckets, to avoid overflowing the runner with a large amount of tasks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Salva Corts	8486acb981	feat: support per policy overriden stream limits on limits service (#19403)	2 months ago
Trevor Whitney	26f8b992ee	fix: correctly register allowed log level fields (#19261)	2 months ago
Trevor Whitney	8b38d8e418	Revert "fix: correctly register allowed log level fields" ... This reverts commit d8c4997120.	3 months ago
Trevor Whitney	d8c4997120	fix: correctly register allowed log level fields	3 months ago
Sandeep Sukhani	8057c5c424	fix: apply global otlp config to tenant config only when it is updated in the overrides (#19213)	4 months ago
Salva Corts	af257fd058	feat: per policy overridable stream limits (#18994)	4 months ago
Salva Corts	3b2498b896	feat: Default ingestion policy mappings merged with per-tenant mappings (#18926)	5 months ago
Trevor Whitney	c690827c6a	fix: limit volume and frequency of persisted patterns (#18362) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Christian Haudum	0ee8e76ba2	feat: Mark truncated log lines with identifier (#18262) ... While truncated JSON log lines are usually easy to identify (because the JSON becomes invalid), it is not always the case with other plain text log lines. This PR adds the possibility to add an identifier as suffix for truncated log lines, such as `[...]` so that truncated log lines are easier to spot and identify when querying them. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Trevor Whitney	888e4012bc	chore: clean up pattern persistence configs (#18017)	7 months ago
Trevor Whitney	fd7321c88b	feat: persist patterns as aggregated metrics (#17737) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	7 months ago
Trevor Whitney	eda3ba865b	feat(ruler): add per-tenant configuration to disable WAL replay (#16717)	10 months ago
Shantanu Alshi	4d1f5a774d	chore(otel): Allow log attributes to be promoted to labels (#16673)	10 months ago
Trevor Whitney	23800f6100	chore: convert multi-variant queries from global to per-tenant config (#16718)	10 months ago
Cyril Tovena	7119f6e480	feat(distributor): Add simulated latency (#16733)	10 months ago
Salva Corts	837b70ac78	feat(policies): Support global policy (#16439)	10 months ago
Salva Corts	0be39a0ed1	feat(policies): Log when multiple policies match the same stream (#16321)	11 months ago
Dylan Guedes	69089eff41	feat: Add support for blocking a policy to be ingested (#16203) ... **What this PR does / why we need it**: Add support for blocking a policy from being ingested. The policy is defined by stream mapping.	11 months ago
Salva Corts	5fda84bbf0	feat(policies): Per policy enforced labels (#16182)	11 months ago
Dylan Guedes	eda27ddebd	docs: Fix config for stream policies mapping (#16104)	11 months ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	11 months ago
Ashwanth	86dc47b3db	feat: update log level discovery from json to detect nested fields (#16026)	11 months ago
Emma Neiss	878727d348	docs: Fix typos in limits.go (#15840) ... Signed-off-by: Emma Neiss <33004979+eneiss@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	12 months ago
Christian Haudum	1a9f382e6f	chore: Remove experimental bloom settings that have never been used or are not used any more (#15787) ... Remove experimental results cache for bloom gateway client This feature never made it into production. Since it is only an experimental feature, it can be removed without prior deprecation. Remove unused `bloom_gateway_shard_size` runtime setting The setting was once used when bloom gateways still used a ring for sharding. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Dylan Guedes	a2c8ec7dc1	feat: Add a new `enforced_labels` limit (#15704) ... **What this PR does / why we need it**: Introduce the new experimental `enforced_labels` limit. By default it is empty but when configured, it configures Loki to only accept push requests which streams have all enforced labels.	12 months ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Shantanu Alshi	867ce3d711	chore(level_detection): Make log level detection configurable (#14784)	1 year ago
Karsten Jeschkies	7b53f20f70	feat: Introduce shardable probabilistic topk for instant queries. (#14243) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Trevor Whitney	c1fde26730	feat: move metric aggregation to a per-tenant config (#14709)	1 year ago
Trevor Whitney	f2a16f43b2	feat: add app_name as a service label (#13660)	1 year ago
Ashwanth	8bca2e7608	feat(ruler): enables ruler store that uses clients from thanos-io/objstore pkg (#11713)	1 year ago
Vladyslav Diachenko	a962edba33	feat(loki): include structured_metadata size while asserting rate limit (#14571) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Salva Corts	673ede16a5	feat: New bloom planning using chunk size TSDB stats (#14547)	1 year ago
Salva Corts	70508975fd	feat: Configurable list of json fields to mine patterns (#14528)	1 year ago
Salva Corts	ab5e6eaaee	feat: Extract task computing into a strategy interface (#13690)	1 year ago
Cyril Tovena	1a4436c417	feat(ingester): implement partition shuffle sharding for ingester (#14304)	1 year ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183) ... This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	1 year ago
Trevor Whitney	4f962ef7af	fix: enable service detection for otlp endoint (#14036) ... Co-authored-by: Sven Grossmann <Svennergr@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	1 year ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	b5ac6a0258	feat: Limit to block ingestion until configured date (#13958)	1 year ago
Salva Corts	47f6ea53fc	feat(loki): Do not enforce max-query-bytes-read and max-querier-bytes-read in limited tripperware (#13406)	2 years ago
Salva Corts	f7887269f8	refactor(blooms): Limit task retries in bloom planner (#13139)	2 years ago
Salva Corts	cd64d6ddbd	refactor(blooms): Implement retry mechanisms in planner (#13064)	2 years ago
Vladyslav Diachenko	1111595179	feat: new stream count limiter (#13006) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	2 years ago
Salva Corts	efd8f5dc1b	refactor(blooms): Add queue to bloom planner and enqueue tasks (#13005)	2 years ago