155 Commits (5aa5bbecaf142e6ae98066fcea809fbd0372ea46)

Author	SHA1	Message	Date
Owen Diehl	1228e713f6	chore(tenant-topics): configurable throughput per partition (#17457)	10 months ago
Christian Haudum	6b4430afd0	chore(engine): Collect pipeline records into Loki result (#17447) ... This PR adds the missing piece between existing HTTP handler and the new execution engine, which converts the Apache Arrow records from the vectorized, columnar, pull-based iterator pipeline into the existing (row-based) Loki response format. Since the engine only supports logs queries at the moment, the transformation logic does not take possible metric query results into account. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Footur	ba8830b080	docs: Document the default value for the working_directory for compactor (#17075) ... Signed-off-by: Footur <3769085+Footur@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	10 months ago
George Robinson	1d68047aee	feat: support disabling the partition consumers cache (#17318)	11 months ago
Trevor Whitney	2ba8626b1d	chore: disable ui by default (#17289)	11 months ago
Christian Haudum	32745ba006	chore(deps): Update grafana/dskit to latest commit (#17235) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
aarogoss	e962398462	feat: Optionally enable kafka histograms for read/write latency (#17089)	11 months ago
Periklis Tsirakidis	54ff5dce52	feat(ingest-limits): Add partition id caching (#16964)	11 months ago
Christian Haudum	1705f98a9f	chore(engine): Wire up new execution engine (#17032) ... This PR connects the already existing components of the new query engine and wires them up in the querier API. The feature flag `-querier.engine.enable-v2-engine` controls whether supported queries should be executed using the new engine. Note that this implementation currently does not execute any queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Periklis Tsirakidis	5229a2ab65	fix(ingest-limits): Get usage from owing instances only (#16937)	11 months ago
Periklis Tsirakidis	1d99f4d86d	feat(distributor): Add MaxRecvMsgSize config for uncompressed message size limits (#16915)	11 months ago
matthewhudsonedb	c01d9c7512	fix: Allow dashes in storage_prefix config (#16934)	11 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833)	11 months ago
Trevor Whitney	eda3ba865b	feat(ruler): add per-tenant configuration to disable WAL replay (#16717)	12 months ago
Shantanu Alshi	4d1f5a774d	chore(otel): Allow log attributes to be promoted to labels (#16673)	12 months ago
Paul Rogers	24c260e3e4	chore(deps): Update dskit (#16782)	12 months ago
Trevor Whitney	23800f6100	chore: convert multi-variant queries from global to per-tenant config (#16718)	12 months ago
George Robinson	d6c499bbc3	feat: add dry-run mode for checking limits in distributors (#16754)	12 months ago
Cyril Tovena	7119f6e480	feat(distributor): Add simulated latency (#16733)	12 months ago
Periklis Tsirakidis	3a3df62c59	feat(ingest-limits): Implement global tenant rate limiting (#16727)	12 months ago
Notedop	8429b0fbd9	feat: add S3 chunk delimiter config to support MinIO running on Windows (#16319) ... Signed-off-by: Notedop <raoul2405@hotmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	12 months ago
George Robinson	bc2111e8f1	feat: enforce limits in distributors (#16705)	12 months ago
George Robinson	18d143befe	feat: wire up ingest-limits service (#16660)	12 months ago
Salva Corts	837b70ac78	feat(policies): Support global policy (#16439)	1 year ago
Sandeep Sukhani	5b33e65bcd	feat: add support for using sqlite for storing delete requests (#16437)	1 year ago
Cyril Tovena	7b3f829592	chore(ui): Only rejoin new nodes and add back instrumentation (#16445)	1 year ago
Ned Andreev	f1ebd970e6	feat: add a config option for custom GCS endpoints (#16419)	1 year ago
Trevor Whitney	5b16c0be7d	refactor: allow sample expressions to have multiple extractors (#16224)	1 year ago
Jackson Coelho	a51e5a21e2	chore: flush idle dataobjects after X seconds (#16348)	1 year ago
Dylan Guedes	69089eff41	feat: Add support for blocking a policy to be ingested (#16203) ... **What this PR does / why we need it**: Add support for blocking a policy from being ingested. The policy is defined by stream mapping.	1 year ago
Cyril Tovena	dbf2befc1d	feat: Distributed Operational UI (#16097)	1 year ago
Cyril Tovena	78a141c6a9	feat(dataobj): Ensure constant sharding for the dataobj querier (#16273)	1 year ago
Salva Corts	5fda84bbf0	feat(policies): Per policy enforced labels (#16182)	1 year ago
J Stickler	b8b3ed1784	docs: adding Promtail deprecation banners (#16227)	1 year ago
Cyril Tovena	1fa952dbf2	refactor(dataobj): Consolidate configuration and add querier support (#16144)	1 year ago
Dylan Guedes	eda27ddebd	docs: Fix config for stream policies mapping (#16104)	1 year ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	1 year ago
benclive	7f32de6cf2	chore(dataobj): Refactor processor & builder to separate concerns (#16055)	1 year ago
Owen Diehl	439977573d	feat(dataobj): tenant partition consumer (#16065)	1 year ago
Ashwanth	1cd95f5212	docs(thanos): unhide configuration docs for thanos storage clients (#16051)	1 year ago
benclive	ca4c025ad0	chore(dataobj): Create initial dataobj builder (#16011) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	1 year ago
Ashwanth	86dc47b3db	feat: update log level discovery from json to detect nested fields (#16026)	1 year ago
Owen Diehl	c25841901c	feat(kafka): tenant topics (#15977)	1 year ago
Emma Neiss	878727d348	docs: Fix typos in limits.go (#15840) ... Signed-off-by: Emma Neiss <33004979+eneiss@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 year ago
Cyril Tovena	9b1407e736	feat(dataobj): Add a Explorer UI for dataobj (#15908)	1 year ago
Christian Haudum	1a9f382e6f	chore: Remove experimental bloom settings that have never been used or are not used any more (#15787) ... Remove experimental results cache for bloom gateway client This feature never made it into production. Since it is only an experimental feature, it can be removed without prior deprecation. Remove unused `bloom_gateway_shard_size` runtime setting The setting was once used when bloom gateways still used a ring for sharding. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Dylan Guedes	a2c8ec7dc1	feat: Add a new `enforced_labels` limit (#15704) ... **What this PR does / why we need it**: Introduce the new experimental `enforced_labels` limit. By default it is empty but when configured, it configures Loki to only accept push requests which streams have all enforced labels.	1 year ago
Avanish Vaghela	a96a695882	docs: Add clarifications for ruler remote write config options (#15167) ... The update provides better understanding of the ruler remote write config behaviour. If a user specifies an 'X-Scope-OrgID' header under the 'headers' section of RemoteWriteConfig, the config parser silently drops this. This is not currently clear to the user and can lead to confusion. This update aims to clarify this. It also adds clarification about what tenant ID value will be added to the 'X-Scope-OrgID' header in the remote write request if 'add_org_id_header' config parameter is set to 'true'.	1 year ago
Ashwanth	b5c627b813	chore(blockbuilder): mark a job as complete only after stopping sync updates (#15670)	1 year ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago