130 Commits (8976709f24dc9baa603d030fa23484382f7ba767)

Author	SHA1	Message	Date
Cyril Tovena	7b3f829592	chore(ui): Only rejoin new nodes and add back instrumentation (#16445)	11 months ago
Ned Andreev	f1ebd970e6	feat: add a config option for custom GCS endpoints (#16419)	11 months ago
Trevor Whitney	5b16c0be7d	refactor: allow sample expressions to have multiple extractors (#16224)	11 months ago
Jackson Coelho	a51e5a21e2	chore: flush idle dataobjects after X seconds (#16348)	11 months ago
Dylan Guedes	69089eff41	feat: Add support for blocking a policy to be ingested (#16203) ... **What this PR does / why we need it**: Add support for blocking a policy from being ingested. The policy is defined by stream mapping.	11 months ago
Cyril Tovena	dbf2befc1d	feat: Distributed Operational UI (#16097)	11 months ago
Cyril Tovena	78a141c6a9	feat(dataobj): Ensure constant sharding for the dataobj querier (#16273)	11 months ago
Salva Corts	5fda84bbf0	feat(policies): Per policy enforced labels (#16182)	11 months ago
J Stickler	b8b3ed1784	docs: adding Promtail deprecation banners (#16227)	11 months ago
Cyril Tovena	1fa952dbf2	refactor(dataobj): Consolidate configuration and add querier support (#16144)	11 months ago
Dylan Guedes	eda27ddebd	docs: Fix config for stream policies mapping (#16104)	11 months ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	12 months ago
benclive	7f32de6cf2	chore(dataobj): Refactor processor & builder to separate concerns (#16055)	12 months ago
Owen Diehl	439977573d	feat(dataobj): tenant partition consumer (#16065)	12 months ago
Ashwanth	1cd95f5212	docs(thanos): unhide configuration docs for thanos storage clients (#16051)	12 months ago
benclive	ca4c025ad0	chore(dataobj): Create initial dataobj builder (#16011) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	12 months ago
Ashwanth	86dc47b3db	feat: update log level discovery from json to detect nested fields (#16026)	12 months ago
Owen Diehl	c25841901c	feat(kafka): tenant topics (#15977)	12 months ago
Emma Neiss	878727d348	docs: Fix typos in limits.go (#15840) ... Signed-off-by: Emma Neiss <33004979+eneiss@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	12 months ago
Cyril Tovena	9b1407e736	feat(dataobj): Add a Explorer UI for dataobj (#15908)	12 months ago
Christian Haudum	1a9f382e6f	chore: Remove experimental bloom settings that have never been used or are not used any more (#15787) ... Remove experimental results cache for bloom gateway client This feature never made it into production. Since it is only an experimental feature, it can be removed without prior deprecation. Remove unused `bloom_gateway_shard_size` runtime setting The setting was once used when bloom gateways still used a ring for sharding. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Dylan Guedes	a2c8ec7dc1	feat: Add a new `enforced_labels` limit (#15704) ... **What this PR does / why we need it**: Introduce the new experimental `enforced_labels` limit. By default it is empty but when configured, it configures Loki to only accept push requests which streams have all enforced labels.	1 year ago
Avanish Vaghela	a96a695882	docs: Add clarifications for ruler remote write config options (#15167) ... The update provides better understanding of the ruler remote write config behaviour. If a user specifies an 'X-Scope-OrgID' header under the 'headers' section of RemoteWriteConfig, the config parser silently drops this. This is not currently clear to the user and can lead to confusion. This update aims to clarify this. It also adds clarification about what tenant ID value will be added to the 'X-Scope-OrgID' header in the remote write request if 'add_org_id_header' config parameter is set to 'true'.	1 year ago
Ashwanth	b5c627b813	chore(blockbuilder): mark a job as complete only after stopping sync updates (#15670)	1 year ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	c43aa56089	fix(blockbuilder): min job size (#15617) ... Co-authored-by: Ashwanth <iamashwanth@gmail.com>	1 year ago
Matt Veitas	3b8d993500	feat: Skip writeback for chunks fetched by queriers older than a duration (#15605)	1 year ago
Owen Diehl	b2f46de418	fix(block-scheduler): one job per partition (local branch copy) (#15579)	1 year ago
Ashwanth	558080cf78	feat(block-scheduler): introduce job lease and requeue expired jobs (#15560)	1 year ago
Karsten Jeschkies	d0c11a689e	feat(api): Support Parquet as a query response format. (#15408) ... **What this PR does / why we need it**: This changes adds support for the header `Accept: application/vnd.apacher.parquet`. If this header is set the response for metric and stream queries will response with a Parquet file. A metrics response has the columns `timestamp`, `labels` and `value`. A stream response has the columns `timestamp`, `labels` and `line`. Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
J Stickler	d75698cee7	docs: adding exceptions to Vale rule (#15431)	1 year ago
Ashwanth	fe50c72da0	chore(block-scheduler): planner improvements to limit jobs and consider retention when planning (#15432)	1 year ago
Dylan Guedes	2c5eabdb72	feat: Add TLS config to the analytics client (#15227) ... **What this PR does / why we need it**: Add a `tls_config` block and a `proxy_url` config to the analytics client. In conjunction they can be useful in scenarios where Loki is running behind a proxy.	1 year ago
Owen Diehl	71264415f7	refactor(kafka): partition lib refactor (#15314)	1 year ago
Ashwanth	c519ab6f18	feat(scheduler): implement and register block builder rpc service (#15248)	1 year ago
Owen Diehl	532bdbc156	feat(blockbuilder): consolidate on record counting planner (#15247)	1 year ago
Ashwanth	0d678311de	feat: update blockbuilder to use scheduler for fetching jobs (#15224)	1 year ago
Ashwanth	ad322c0fc2	feat(block-scheduler): adds service and basic planner support for scheduler (#15200)	1 year ago
George Robinson	12386a21b0	feat: remove target lag and keep just maximum lag (#15120)	1 year ago
Bayan Taani	44523e085d	feat: Add objstore support for Swift using thanos.io/objstore (#11672) ... Co-authored-by: Joao Marcal <jmarcal@redhat.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	1 year ago
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	cbdd36a412	feat: blockbuilder component (#14621) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	1 year ago
Matt Veitas	2d407bd308	feat: Add otlp attribute deployment.environment.name to list of default labels (#14747)	1 year ago
Salva Corts	fd9d33241d	fix: Improve docs for min and max table offsets (#14890) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	b646861e77	feat(blooms): disk-backed queue for the bloom-planner (#14874) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
洪阿南	ac2e21fb42	feat: Support config timeout for AlibabaCloud OSS requests (#14856)	1 year ago
Salva Corts	94a8572718	refactor(blooms): Extract planner queue into pkg (#14861)	1 year ago
benclive	f2da621f72	fix: Add flags for path & configure kafka for non-memberlist kv store (#14850)	1 year ago
George Robinson	307603829c	chore: remove initialization of metastore (#14822)	1 year ago
George Robinson	9a1fe95c11	chore: remove initialization of RF-1 client (#14810)	1 year ago