apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Salva Corts	2bbdfa54e7	fix: Pass ingestion policy set through header to ingesters via grpc (#19630 )	2 months ago
Salva Corts	987840b5d3	feat: Resolve ingestion policy via a header (#19548 )	2 months ago
Salva Corts	af257fd058	feat: per policy overridable stream limits (#18994 )	4 months ago
Robert Fratto	86c57c8616	chore: consistently use new labels.Labels API (#18490 )	5 months ago
Shantanu Alshi	61e81c5a4c	chore(otlp): OTLP Metrics (#18281 )	6 months ago
Oleg Zaytsev	c8a15f451c	refactor: Use OTel tracing library (#17859 ) Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	7 months ago
Karsten Jeschkies	845822b124	refactor(stringlabels): Support stringlabels in ingester (#17243 )	9 months ago
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438 ) Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	10 months ago
Trevor Whitney	5b16c0be7d	refactor: allow sample expressions to have multiple extractors (#16224 )	10 months ago
Salva Corts	0be39a0ed1	feat(policies): Log when multiple policies match the same stream (#16321 )	10 months ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982 ) What this PR does / why we need it: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod\|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod\|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	11 months ago
Dylan Guedes	0bc23db95d	refactor: Add retention hours to `discarded` metrics (#15875 ) What this PR does / why we need it: We have no distinction between the type of data that gets discarded. On this PR I'm adding `retention_hours` to our discarded metrics so that we can better look at what data was discarded.	11 months ago
Owen Diehl	2a2b52808c	perf(ingester): refactor lock acquisitions related to `not_owned` series limit functionality (#15839 )	11 months ago
Periklis Tsirakidis	f63c16f77d	fix(partition-ingester): Add stream creation error logging (#15640 )	12 months ago
Cyril Tovena	90c5dbf0d3	chore: Revert syntax module (#14831 )	1 year ago
Cyril Tovena	28c9645d10	chore: Introduce logql/syntax go module (#14790 )	1 year ago
Vladyslav Diachenko	a962edba33	feat(loki): include structured_metadata size while asserting rate limit (#14571 ) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
benclive	83a8893a3f	feat(kafka): Remove rate limits for kafka ingestion (#14460 )	1 year ago
benclive	3c36ba949d	feat: Implement owned streams calculation using Partition Ring (#14282 )	1 year ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071 )	1 year ago
George Robinson	71cb767727	chore: replace pkg/errors with errors (#13583 )	1 year ago
Vladyslav Diachenko	5ae5b31b1f	fix: stream ownership check (#13314 ) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084 )	2 years ago
Vladyslav Diachenko	2ca1ac66a3	feat: flush not owned streams (#13254 ) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
JordanRushing	e7689b248d	feat: add recalculateOwnedStreams to check stream ownership if the ring is changed (#13103 ) Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	2 years ago
Callum Styan	691b174138	fix: properly return http status codes from ingester to querier for RPC function calls (#13134 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Vladyslav Diachenko	1111595179	feat: new stream count limiter (#13006 ) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	2 years ago
Karsten Jeschkies	88c671162f	fix: Track bytes discarded by ingester. (#12981 ) What this PR does / why we need it: Only the distributor was tracking discarded bytes. The ingester was missing the tracker and calls. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] Title matches the required conventional commits format, see [here](https://www.conventionalcommits.org/en/v1.0.0/) - Note that Promtail is considered to be feature complete, and future development for logs collection will be in [Grafana Alloy](https://github.com/grafana/alloy). As such, `feat` PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Shantanu Alshi	587a6d20e9	feat: Detected labels from store (#12441 )	2 years ago
Callum Styan	ec81991f4d	feat: include the stream we failed to create in the stream limit error message (#12437 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Cyril Tovena	491d2513e7	feat: add log patterns ingestion and query API. (#12403 ) Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Anton Kolesnikov <anton.e.kolesnikov@gmail.com> Co-authored-by: Sven Grossmann <svennergr@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Shantanu Alshi	2c878c8301	fix: Bugfixes for detected labels API (#12466 )	2 years ago
Travis Patterson	cf04ec1320	feat: Enable warnings in Loki query responses (#12425 )	2 years ago
Shantanu Alshi	643f353cef	feat: Detected labels - get and filter labels from ingesters (#12432 )	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Travis Patterson	c8d2c32002	feat: Introduce special header that tells Loki not to modify query results (#12327 )	2 years ago
Travis Patterson	d175184fe5	chore: remove shards from wrapper signature (#12322 )	2 years ago
Travis Patterson	fed7566ba9	fix: Pass query to pipeline wrappers from the AST (#12305 )	2 years ago
Ashwanth	3b5ed8c3ec	chore: pass shard count to injected pipelines and extractors (#12213 )	2 years ago
Karsten Jeschkies	243e6f813a	refactor: Pass context to usage tracker. (#12272 )	2 years ago
Karsten Jeschkies	6578a00481	feat: Support usage trackers for received and discarded bytes. (#11840 )	2 years ago
Dylan Guedes	9b7e48aca9	Ingesters: Revert "Add metrics.go log line when reading from an ingester (#11571 )" (#11818 ) What this PR does / why we need it: This reverts commit `f04d0db434` Although it is useful, the amount of logging it generates is too massive. In the future we might revisit it but with a different approach.	2 years ago
Michel Hollands	f04d0db434	Add metrics.go log line when reading from an ingester (#11571 ) What this PR does / why we need it: Add a metrics.go log line when a querier reads from an ingester. The output is like this: ``` level=info ts=2024-01-03T13:59:06.919965546Z caller=metrics.go:275 component=ingester org_id=fake traceID=0b216e7d014c5f87 latency=fast query_type=ingester_series start=2024-01-03T13:00:00Z end=2024-01-03T13:59:06.834Z start_delta=59m6.91996413s end_delta=85.964255ms length=59m6.834s duration=110.792µs status=200 query=<omitted> query_hash=2166136261 total_entries=9 ``` Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [X] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Travis Patterson	f668a1b80b	Add context and tenant to pipeline/extractor wrappers (#11528 ) Adding the context here means the wrapping pipeline can know when the query has stopped or been cancelled.	2 years ago
Karsten Jeschkies	9aae25008d	Optimize series response format by using repeated fileds. (#11498 ) What this PR does / why we need it: The Protobuf map type is encodied as a repeated field of map entries. Decoding them to a slice is much faster than decoding them into a map. Since Loki is not using the fast key check for a map we can use the slice decoding. This change also allows us to decode the JSON directly into the right protobuf struct. This doulbes the JSON decoding speed and reduces the memory pressure by ~40%. ``` › go test -bench=. -run=^$ -count=10 ./pkg/querier/queryrange > before.log › go test -bench=. -run=^$ -count=10 ./pkg/querier/queryrange > after.log › benchstat before.log after.log goos: darwin goarch: arm64 pkg: github.com/grafana/loki/pkg/querier/queryrange │ before.log │ after.log │ │ sec/op │ sec/op vs base │ ResponseMerge/mergeStreams_unlimited-10 32.36m ± 0% 32.63m ± 2% ~ (p=0.393 n=10) ResponseMerge/mergeOrderedNonOverlappingStreams_unlimited-10 1.050m ± 1% 1.080m ± 3% +2.84% (p=0.005 n=10) ResponseMerge/mergeStreams_limited-10 33.02m ± 0% 32.60m ± 1% -1.29% (p=0.004 n=10) ResponseMerge/mergeOrderedNonOverlappingStreams_limited-10 15.11m ± 0% 15.07m ± 0% ~ (p=0.075 n=10) _CodecDecodeLogs-10 4.395m ± 1% 4.364m ± 0% -0.72% (p=0.005 n=10) _CodecDecodeSamples-10 16.97m ± 0% 16.84m ± 2% -0.77% (p=0.023 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 745.8µ ± 8% 736.8µ ± 12% ~ (p=0.739 n=10) _CodecDecodeSeries/application/json;_charset=utf-8-10 15.37m ± 1% 10.60m ± 0% -31.03% (p=0.000 n=10) _MergeResponses-10 1186.9m ± 2% 149.8m ± 1% -87.38% (p=0.000 n=10) _UnwrapSeries-10 9.399m ± 1% 4.049m ± 0% -56.92% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 666.0m ± 3% 194.6m ± 0% -70.79% (p=0.000 n=10) geomean 18.87m 12.51m -33.70% │ before.log │ after.log │ │ B/op │ B/op vs base │ _CodecDecodeLogs-10 3.649Mi ± 0% 3.649Mi ± 0% ~ (p=0.364 n=10) _CodecDecodeSamples-10 18.12Mi ± 0% 18.12Mi ± 0% ~ (p=0.926 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 7.647Mi ± 0% 7.647Mi ± 0% ~ (p=0.587 n=10) _CodecDecodeSeries/application/json;_charset=utf-8-10 27.94Mi ± 0% 16.99Mi ± 0% -39.18% (p=0.000 n=10) _MergeResponses-10 2.362Mi ± 0% 2.408Mi ± 0% +1.98% (p=0.000 n=10) _UnwrapSeries-10 19.495Mi ± 0% 8.595Mi ± 0% -55.91% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 772.3Mi ± 0% 772.3Mi ± 0% ~ (p=0.912 n=10) geomean 17.50Mi 14.54Mi -16.91% │ before.log │ after.log │ │ allocs/op │ allocs/op vs base │ _CodecDecodeLogs-10 41.10k ± 0% 41.10k ± 0% ~ (p=1.000 n=10) ¹ _CodecDecodeSamples-10 411.9k ± 0% 411.9k ± 0% ~ (p=1.000 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 32.00 ± 0% 32.00 ± 0% ~ (p=1.000 n=10) ¹ _CodecDecodeSeries/application/json;_charset=utf-8-10 304.2k ± 0% 298.1k ± 0% -2.01% (p=0.000 n=10) _MergeResponses-10 100.1k ± 0% 100.1k ± 0% -0.00% (p=0.002 n=10) _UnwrapSeries-10 201.1k ± 0% 198.0k ± 0% -1.54% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 203.1k ± 0% 203.1k ± 0% ~ (p=0.621 n=10) geomean 48.95k 48.70k -0.51% ¹ all samples are equal ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Travis Patterson	be71a80b15	Implement hooks to instrument query pipelines (#11493 ) This PR implements interfaces to inject new pipelines/extractors into the query path of queriers and ingesters.	2 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025 ) What this PR does / why we need it: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Kaviraj Kanagaraj	bbfb13c3e4	Pin `chunk` and `index` format to `schema` version. (#10213 ) We pin all three `Chunk`, `HeadBlock` and `TSDB` Version to `schema` version in period config. This is the following mapping (after being discussed with @owen-d and @sandeepsukhani ) `v12` (current existing schema) - ChunkFormatV3 (UnorderedHeadBlock) + TSDBv2 `v13` (introducing new schema) - ChunkFormatV4 (UnorderedWithNonIndexedLabelsHeadBlockFmt) + TSDBv3 Note the new schema `v13` supports the latest chunk and index format. NOTES for Reviewer 1. General approach is we removed the idea of `index.LiveFormat`, `chunkenc.DefaultChunkFormat` and `chunkenc.DefaultHeadBlockFmt` and made following two changes. These variables were used before to tie chunk and tsdb formats specific to Loki versions. This PR remove that coupling and pin these formats to `schema` version instead. 1. These variables were replaced with explicit chunk and index formats within those packages (and it's tests) 2. If these variables were used outside it's own packages say by ingester, compactor, etc. Then we extract correct chunk and index versions from the `schema` config. 2. Add two methods to `periodConfig`. (1) `ChunkFormat()` returning chunk and head format tied to schema (2) `TSDBFormat()` returning tsdb format tied to schema. 2. Other ideas I thought of doing but didn't end up doing is make `ChunkFormat` and `IndexFormat` as separate type (rather than `byte` and `int` currently. Similar to `HeadBlockFmt` type). But didnt' do it eventually to keep the PR small and don't want to complicate with lots of changes. 4. Moved couple of test cases from `chunkenc` to `config` package, because the test case was actually testing methods on `schemaconfig` and it was creating cycling dependencies. --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Bryan Boreham	f89a51fde8	Tracing: elide small traces for Stats call (#10308 ) What this PR does / why we need it: At work I see a lot of traces getting data dropped due to being tens of megabytes in size. Looking at what was captured, I see many many repeats of these four spans: <img width="1070" alt="image" src="https://github.com/grafana/loki/assets/8125524/4268e664-6910-49bb-8f68-69b139a5a66b"> Events on the parent trace are cheaper and, for small operations like this, just as useful. Where there was an event line I added a `function` field matching the previous span name. For `storeEntry.Stats()` I didn't add an event line as it doesn't seem to do very much. But please correct me; I'm not hugely familiar with the Loki codebase. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA If the change is worth mentioning in the release notes, add `add-to-release-notes` label - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago

1 2 3 4 5

201 Commits (main)