3430 Commits (a2c8ec7dc11546e2876e159e76eabd727ebc1929)

Author	SHA1	Message	Date
Dylan Guedes	a2c8ec7dc1	feat: Add a new `enforced_labels` limit (#15704) ... **What this PR does / why we need it**: Introduce the new experimental `enforced_labels` limit. By default it is empty but when configured, it configures Loki to only accept push requests which streams have all enforced labels.	12 months ago
Robert Fratto	aec8e968da	chore(dataobj): track non-NULL values in pages/columns (#15758)	12 months ago
renovate[bot]	6f7bbae1a6	fix(deps): update module google.golang.org/grpc to v1.69.4 (#15739) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	12 months ago
Cyril Tovena	889404791d	feat(kafka): Add cooperative active sticky balancer (#15706)	12 months ago
Robert Fratto	d1fd0b9ffc	chore(dataobj): add streams tracking and encoding (#15713)	12 months ago
Sandeep Sukhani	1130542950	chore: clone the context instead of using background context while pushing logs to ingesters from distributor (#15735)	12 months ago
Robert Fratto	bcec63d9c2	chore(dataobj): logs section metadata and encoding/decoding (#15720)	12 months ago
Robert Fratto	e838ee3181	chore(dataobj): add skeleton for public API to build objects (#15719)	12 months ago
Robert Fratto	896e138425	chore(dataobj): data object encoding and decoding (#15676)	12 months ago
Avanish Vaghela	a96a695882	docs: Add clarifications for ruler remote write config options (#15167) ... The update provides better understanding of the ruler remote write config behaviour. If a user specifies an 'X-Scope-OrgID' header under the 'headers' section of RemoteWriteConfig, the config parser silently drops this. This is not currently clear to the user and can lead to confusion. This update aims to clarify this. It also adds clarification about what tenant ID value will be added to the 'X-Scope-OrgID' header in the remote write request if 'add_org_id_header' config parameter is set to 'true'.	12 months ago
Aaron	b272573efc	feat: Add Rule Name and Type to Query Tags (#15055) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Ashwanth	732cc8b387	chore(block-scheduler): pin fallback offset during start-up (#15642)	1 year ago
Robert Fratto	c581218c4f	chore(dataobj): add Dataset with iteration and sorting support (#15652)	1 year ago
Ashwanth	b5c627b813	chore(blockbuilder): mark a job as complete only after stopping sync updates (#15670)	1 year ago
Karsten Jeschkies	bef20431cb	perf(approx_topk): Reduce memory usage of HyperLogLog in approx_topk. (#15559) ... The count min sketch data structure backing the new approx_topk aggregation uses HyperLogLog (HLL) to track the actual cardinality of the aggregated vector. We were using the sparse version of the HLL. However, that resulted in memory and allocation overhead.	1 year ago
jackyin	5f476a39e1	fix: Fix goroutine leak in queryrange downstreamer (#15665)	1 year ago
matthewhudsonedb	e347eb7c80	fix: Fix loki ruler generator url left parameter url encoding (#15601) ... The generate URL `left` parameter is not URL encoded. The double quotes in str variable around "queries" break hyperlinking.	1 year ago
Robert Fratto	f2fc0c26c2	chore(dataobj): column building (#15634) ... This commit adds the ability to accumulate sequences of dataset.Value into a column, which is split up across multiple pages. Each page is broken down into two parts: * A bitmap-encoded sequence of booleans, where 1 indicates a row has a value and 0 indicates the row is NULL, and * the encoded sequence of non-NULL values, whose encoding is determined by the column options. The sequence of non-NULL values is then optionally compressed. This commit also includes initial support for reading these columns, starting with internal-only helper utilities for unit tests.	1 year ago
Jonathan Lange	639ac74c48	feat(logcli): Include common labels (#15611) ... Adds a new `--include-common-labels` flag to logcli query, which causes the output to include all common labels. This is mostly useful with `--quiet` and `--output=jsonl` as it allows users to get structured information from Loki that has all the information that Loki can provide. --- Signed-off-by: Jonathan Lange <jml@mumak.net> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Periklis Tsirakidis	f63c16f77d	fix(partition-ingester): Add stream creation error logging (#15640)	1 year ago
Robert Fratto	6042d0712e	chore(dataobj): add bitmap encoding (#15629)	1 year ago
Robert Fratto	9a21590238	chore(dataobj): initial commit of value encoding (#15606)	1 year ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	c43aa56089	fix(blockbuilder): min job size (#15617) ... Co-authored-by: Ashwanth <iamashwanth@gmail.com>	1 year ago
Owen Diehl	55f2dd8ad9	fix(blockbuilder): use index path prefix in objectclient for tsdb creation (#15614)	1 year ago
Matt Veitas	3b8d993500	feat: Skip writeback for chunks fetched by queriers older than a duration (#15605)	1 year ago
Owen Diehl	b2f46de418	fix(block-scheduler): one job per partition (local branch copy) (#15579)	1 year ago
Trevor Whitney	8e260fef83	fix: bytes unit detection in detected fields (#15525)	1 year ago
Owen Diehl	10194f7e0f	feat(block-scheduler): status page shows completed jobs (#15580) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	1 year ago
Sandeep Sukhani	ff199557d2	fix: do not retain copy of chunk while indexing a new chunk in tsdb while processing delete requests (#15541)	1 year ago
Ashwanth	558080cf78	feat(block-scheduler): introduce job lease and requeue expired jobs (#15560)	1 year ago
Ashwanth	61df08572a	chore(block-builder): update block builder to use kafka clients directly (#15433) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Ashwanth	a3a5acd07a	chore: add status page for block scheduler (#15553)	1 year ago
Ashwanth	52d745fc65	fix(blockbuilder): copy entries returned by record decoder (#15549)	1 year ago
Ed Welch	a3808b6b1d	fix: structured metadata not always passed down for processing deletes (#15526) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Christian Haudum	5846ea2ea0	fix(blooms): Match series to newest block only (#15481) ... **What this PR does / why we need it**: While running bloom filters in production we noticed some Loki clusters that showed a very high percentage of missing chunks when querying blooms, thus resulting in lower filter rate. The reason is that old, superseded blocks are still considered up-to-date, because they cover a keyspace that is not covered by newer blocks with smaller keyspaces (due to larger individual series). ``` | series fingerprint keyspace ------------+---------------------------------------------------------------- | o o o o o o o o o o o o o ------------+---------------------------------------------------------------- iteration 1 | 111111111111111111111111111111111111111111111111111111111 iteration 2 | 22222222222222 3333333333333333 444444 iteration 3 | 5555555 6666666 77777777 888888888 9999999999 ... up-to-date | 555555522266666661111177777777333388888888811119999999999 ------------+---------------------------------------------------------------- | x ``` The chart shows the different blocks marked with the numbers 1 to 9 for a subset of the full series fingerprint keyspace. The blocks are generated in multiple successive bloom building iterations. The first block covers a larger keyspace (more series), because the individual blooms in the blocks are smaller in the beginning of the day. Later, the blooms get larger and therefore the block fingerprint ranges gets smaller. However, since we are dealing with fingerprint ranges, not individual fingerprints, the newer blocks cause "gaps" in the range of the previously larger keyspace. In the case above, every block except block 4, are considered up-to-date, since each of them covers a keyspace that is otherwise not covered. When resolving blocks for a series at query time, we consider looking at all up-to-date blocks, which are referenced by the meta files. The series `x` in the chart shows, that it is within the range of 3 up-to-date blocks: 1, 2, 5. However, only the newest block (5) may contain the requested series. This PR changes the block resolver on the index-gateway to only match the newest block to a series, based on the timestamp of the TSDB from with the blocks were generated. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	606f2f1a4d	test: Increase buffer size to prevent timeouts in tests (#15498)	1 year ago
Karsten Jeschkies	04994ca821	perf: Improve `approx_topk` performance by reducing allocations. (#15450) ... **What this PR does / why we need it**: The metrics slice should keep a constant amount of memory by removing the smallest item when the maximum of labels is reached. ``` › benchstat before.log after.log goos: linux goarch: amd64 pkg: github.com/grafana/loki/v3/pkg/logql cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.log │ after.log │ │ sec/op │ sec/op vs base │ _HeapCountMinSketchVectorAdd-16 839.0m ± 3% 418.9m ± 2% -50.07% (p=0.000 n=10) │ before.log │ after.log │ │ B/op │ B/op vs base │ _HeapCountMinSketchVectorAdd-16 72.58Mi ± 0% 12.10Mi ± 0% -83.33% (p=0.000 n=10) │ before.log │ after.log │ │ allocs/op │ allocs/op vs base │ _HeapCountMinSketchVectorAdd-16 4073.9k ± 0% 116.9k ± 0% -97.13% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] Title matches the required conventional commits format, see [here](https://www.conventionalcommits.org/en/v1.0.0/) - **Note** that Promtail is considered to be feature complete, and future development for logs collection will be in [Grafana Alloy](https://github.com/grafana/alloy). As such, `feat` PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	1 year ago
renovate[bot]	0ad17ebcf4	fix(deps): update module github.com/influxdata/telegraf to v1.33.0 (#15456) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
renovate[bot]	70570285d3	fix(deps): update module golang.org/x/net to v0.33.0 [security] (#15490) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
Ned Andreev	fad06ee692	fix: data race in chunk client hedging tests (#15466)	1 year ago
Christian Haudum	c8a4afe2ac	chore: Update ASCII diagram of chunk binary encoding (#15386) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	2a23cc836e	chore(deps): Update Thanos library (#15483)	1 year ago
Karsten Jeschkies	d0c11a689e	feat(api): Support Parquet as a query response format. (#15408) ... **What this PR does / why we need it**: This changes adds support for the header `Accept: application/vnd.apacher.parquet`. If this header is set the response for metric and stream queries will response with a Parquet file. A metrics response has the columns `timestamp`, `labels` and `value`. A stream response has the columns `timestamp`, `labels` and `line`. Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ned Andreev	04f621ec86	fix: data race in ingester test (#15465)	1 year ago
Ashwanth	fe50c72da0	chore(block-scheduler): planner improvements to limit jobs and consider retention when planning (#15432)	1 year ago
Ned Andreev	e6d82b9253	fix: remove unsafe pkg usage from util.mempool (#15428)	1 year ago
Cyril Tovena	2150fbcf52	feat: line_format bypass template execution if possible (#15411) ... Co-authored-by: Ed Welch <ed@edjusted.com>	1 year ago
Ned Andreev	b8168a87f5	fix: data race in distributor tests (#15423)	1 year ago
Ned Andreev	5d8220c3e9	fix: data race / nil channel read in pattern aggregation push (#15410)	1 year ago