278 Commits (8976709f24dc9baa603d030fa23484382f7ba767)

Author	SHA1	Message	Date
Dylan Guedes	3d6163abc7	fix: Fix blocked ingestion returned error when 260 (#16387) ... **What this PR does / why we need it**: Modify how we handle blocked ingestion errors: * If it is 200, don't expose any error * Otherwise, expose the error as the configured statusCode, but only if no other validation error occur. I'm calling this a secondTierErr Also modifying the enforced labels error message to log present labels.	11 months ago
Salva Corts	0be39a0ed1	feat(policies): Log when multiple policies match the same stream (#16321)	12 months ago
Dylan Guedes	69089eff41	feat: Add support for blocking a policy to be ingested (#16203) ... **What this PR does / why we need it**: Add support for blocking a policy from being ingested. The policy is defined by stream mapping.	12 months ago
Salva Corts	5fda84bbf0	feat(policies): Per policy enforced labels (#16182)	12 months ago
Salva Corts	c2e1e88e85	feat(policies): Add PoliciesStreamMapping to loghttp limits interface (#16105)	12 months ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	12 months ago
Travis Patterson	f2bff201d7	fix: Determine when all logs have been filtered (#16073)	12 months ago
Ashwanth	86dc47b3db	feat: update log level discovery from json to detect nested fields (#16026)	1 year ago
Owen Diehl	ed3a807c36	chore(kafka): tenant prefix codec (#16020)	1 year ago
Owen Diehl	c25841901c	feat(kafka): tenant topics (#15977)	1 year ago
Dylan Guedes	0bc23db95d	refactor: Add retention hours to `discarded` metrics (#15875) ... **What this PR does / why we need it**: We have no distinction between the type of data that gets discarded. On this PR I'm adding `retention_hours` to our discarded metrics so that we can better look at what data was discarded.	1 year ago
Dylan Guedes	a2c8ec7dc1	feat: Add a new `enforced_labels` limit (#15704) ... **What this PR does / why we need it**: Introduce the new experimental `enforced_labels` limit. By default it is empty but when configured, it configures Loki to only accept push requests which streams have all enforced labels.	1 year ago
Sandeep Sukhani	1130542950	chore: clone the context instead of using background context while pushing logs to ingesters from distributor (#15735)	1 year ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ned Andreev	b8168a87f5	fix: data race in distributor tests (#15423)	1 year ago
Christian Haudum	db2b6dbfed	fix(ci): Fix test compile error (#15404) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ravishankar	17bf32bc50	fix: Return error 422 (Unprocessable Content) when push request does not contain any streams (#13706)	1 year ago
Callum Styan	e9d0c3ee52	feat: Add a metric ingestion time SM sanitization (#15222) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Callum Styan	be4f17eefe	feat: sanitize structured metadata during ingestion in the distributor (#15141) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Salva Corts	63a2442191	fix(otlp): Write protobuf status on error (#15097)	1 year ago
renovate[bot]	3abb3b116d	fix(deps): update module github.com/hashicorp/golang-lru to v2 (#14979) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	1 year ago
Cyril Tovena	b2df31d9ed	chore: Fix kafka producer metrics names (#14936)	1 year ago
Ned Andreev	0d6d68d9bf	feat: add per-tenant time sharding for long out-of-order ingestion (#14711)	1 year ago
Shantanu Alshi	867ce3d711	chore(level_detection): Make log level detection configurable (#14784)	1 year ago
Vladyslav Diachenko	a962edba33	feat(loki): include structured_metadata size while asserting rate limit (#14571) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132) ... Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	1 year ago
Trevor Whitney	41c6b6c2c2	fix: always write detected_level when enabled, even if unknown (#14464) ... Co-authored-by: shantanualshi <shantanu.alshi@grafana.com>	1 year ago
benclive	e2a209c076	feat(kafka): Add support for SASL auth to Kafka (#14487)	1 year ago
Trevor Whitney	242a852d7d	fix: level detection for warning level (#14444)	1 year ago
benclive	8aa8a2bb0e	fix(kafka): Set namespace for Loki kafka metrics (#14426)	1 year ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	1 year ago
Trevor Whitney	d7ff426646	feat: ability to log stream selectors before service name detection (#14154)	1 year ago
Cyril Tovena	1a4436c417	feat(ingester): implement partition shuffle sharding for ingester (#14304)	1 year ago
Cyril Tovena	f80d68a1ed	feat(distributors): Use a pool of worker to push to ingesters. (#14245)	1 year ago
Cyril Tovena	3f47233079	fix(kafka): Fixes partition selection in distributors (#14242)	1 year ago
benclive	7c78232ad3	feat: Optionally require writes to kafka on Push requests (#14186)	1 year ago
Vladyslav Diachenko	c65721e7ad	feat: track discarded data by usageTracker (#14081) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	1 year ago
Salva Corts	b5ac6a0258	feat: Limit to block ingestion until configured date (#13958)	1 year ago
Paul Rogers	8fb7b488bd	chore(deps): update dskit 20240819 (#13924)	1 year ago
Trevor Whitney	913e9f9347	feat: aggregate byte and count metrics (#13731)	1 year ago
Travis Patterson	573a1848d1	chore: add service_name label earlier in the ingestion pipeline (#13702)	2 years ago
benclive	6c4b0622aa	feat: Ignore empty streams in distributor if all entries fail validation (#13674)	2 years ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	2 years ago
Cyril Tovena	9f31b25253	feat: Optimize log parsing performance by using unsafe package (#13223)	2 years ago
Yahima Duarte	c9bfa3ebbf	fix: add detected_level info when the info word appears on log message (#13218)	2 years ago
Salva Corts	b31e04e3f1	fix(otel): Map 500 errors to 503 (#13173)	2 years ago
Dylan Guedes	afd9e36306	chore: Call `shardstreams.Config` by value instead of by reference (#12915) ... Use `shardstreams.Config` by value instead of by reference to fix docs generation. Our `docs-generator` tool relies on the struct address/references to assume that flags are present. Using this config by value fixes it.	2 years ago
Shantanu Alshi	6904a6520d	fix: Fix log level detection (#12651)	2 years ago
Sandeep Sukhani	0831802a99	fix: fix setting of info log level when trying to detect level from log lines (#12635)	2 years ago