apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ed Welch	705379cc81	chore: remove experimental flags from features and configs which are no longer so. (#12481 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Cyril Tovena	9e6e6d22e0	feat: Detect level as structured metadata by default (#12479 )	1 year ago
Ed Welch	24f4e45165	feat: Enable automatic stream sharding by default (#12451 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Sandeep Sukhani	1ae5b3e683	chore: Log level detection improvements (#12440 )	1 year ago
Ed Welch	824f5aa20a	chore: fix submodule for v3 (#12438 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Sandeep Sukhani	b66c343383	feat: add support for discovering and adding log levels as structured metadata (#12428 )	1 year ago
Sandeep Sukhani	288c0061bf	feat: discovery of name of services emitting the logs (#12392 )	1 year ago
Cyril Tovena	ca190a3a7d	feat: Allows to wrap distributor Tee (#12391 )	1 year ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292 ) A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Karsten Jeschkies	243e6f813a	refactor: Pass context to usage tracker. (#12272 )	1 year ago
Karsten Jeschkies	9a2e00bc17	refactor: Replace TenantConfig method with interface. (#12260 ) What this PR does / why we need it: This change allows the retrieval of the `TenantConfig` or the `DefaultConfig`. It also defines an interface instead of a tenant config function which makes it a little easier to grok the code. Eventually this will enable us to introduce GEL specific configurations. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Sandeep Sukhani	8b9d8b0226	feat: Make list of otel blessed attributes configurable (#12180 )	1 year ago
Travis Patterson	b0478e61c8	feat: add the ability to inject log parsers to the distributor (#12029 )	1 year ago
Karsten Jeschkies	6578a00481	feat: Support usage trackers for received and discarded bytes. (#11840 )	1 year ago
Paul Rogers	edc9a44f66	test: Finish sync work for mockIngester.pushed var (#12028 )	1 year ago
JordanRushing	cad4b8e749	Reduce Distributor auto-forget unhealthy cycles from 10->2 (#11935 ) Signed-off-by: JordanRushing <rushing.jordan@gmail.com>	1 year ago
Sandeep Sukhani	e625d7f1ca	otel: add support for per tenant configuration for mapping otlp data to loki format (#11143 ) What this PR does / why we need it: In OTEL, we pick select Resource Attributes to identify the streams and store all the other attributes as Structured Metadata, as explained here. The problem however is that the list of Resource Attributes that are picked as Stream labels are hardcoded, and there is no way to drop unwanted data. This PR adds support for configuring how data is mapped from OTEL to Loki format per tenant. It also adds support for dropping unwanted data. We decided to make the config look similar to Prometheus's relabling config to make it familiar. Special notes for your reviewer: Opening a draft PR to get some initial feedback. I will add documentation in a separate PR. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated	1 year ago
Travis Patterson	d2f43787b6	Add tenant to tee (#11573 ) It is useful for implementations of Tee to also know the tenant who sent the logs. This PR adds tenant to the tee interface.	1 year ago
na--	724a841f0f	Add log duplication (tee-ing) in the Distributor (#11496 ) This allows us to inject a Tee in the Distributor that can duplicate all of the incoming log stream, before they are replicated.	1 year ago
na--	71e9fcaab7	Refactor Distributor.Push() to be able to Tee log streams (#11488 ) This allows us to duplicate (i.e. "tee") some of the log streams from the `Distributor` by introducing an optional interface in a follow-up PR.	1 year ago
Angus Dippenaar	258a8b3108	Update nix configuration (#8452 ) Fix nix configuration (mainly failing test), and break up various binaries (ie. logcli, promtail, loki, etc.) into their own pakcages. --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Ashwanth	766f27645d	config: remove flags & YAML settings deprecated in 2.9.x (#11151 ) What this PR does / why we need it: - remove `-ruler.evaluation-delay-duration` - remove `boltdb.shipper.compactor.deletion-mode` - remove compactor flags with prefix`-boltdb.shipper.compactor.` - remove `validation.enforce-metric-name` - remove `log.use-buffered`, `log.use-sync` Special notes for your reviewer: `log.use-buffered`, `log.use-sync` are not mentioned in the upgrade guide since they were hidden fields Checklist* - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [X] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Periklis Tsirakidis	6385b19573	fix: All lifecycler cfgs ref a valid IPv6 addr and port combination (#11121 ) What this PR does / why we need it: Ensure that all constructors of LifeCyclerConfig are using `net.JoinHostPort` for instance addr and port combinations. This enables IPv6 usage compatible across the code based. Currently only distributors provide IPv6 compatibility because they use `Lifecycler` (that pulls the addr/port in the `NewLifeCycler`) while everything else uses `BasicLifeCycler` delegate this to the user defining an addr/port combination in `LifecyclerConfig`.	2 years ago
Michel Hollands	e93f5bfd93	Add dashboard changes for migrating from cortex_ metrics to loki_ metrics (#11082 ) What this PR does / why we need it: - add missing namespace param for ruler manager metrics - update the loki mixin to use the renamed metrics so both are supported - created the compiled mixin - make the prefix of the cortex_distributor_ingester_clients metrics configurable Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Shantanu Alshi	fb6f6d2e2f	Stop using global logger in modules (#11051 ) What this PR does / why we need it: Refactors Loki to inject logger from the constructors instead of using global logger variable. It is generally considered a good practice. Which issue(s) this PR fixes: [Fixes #<issue number>](https://github.com/grafana/loki-private/issues/810) Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR -->	2 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025 ) What this PR does / why we need it: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Sandeep Sukhani	6069df8f7d	ingestion: native otlp ingestion support (#10727 ) What this PR does / why we need it: Add support for natively supporting logs ingestion in OTLP format. `/otlp/v1/logs` is the new endpoint where users can push logs in OTLP format. It accepts logs serialized in JSON or proto format. Since OTEL format is very different than what Loki storage model, here is how data in OTEL format will be mapped to Loki data model: * Index labels: The Resource Attributes map quite well to Index labels in Loki since both usually identify the source of the logs. The problem however is that Resource attributes in OTLP can have an unbounded number of values while Loki has a default limit of having up to 30 labels. Since Index labels in Loki can largely drive the kind of querying experience the users are going to have, we have chosen select attributes which would be picked as Index Labels. The ones that are not picked up as Index labels would be stored as Structured Metadata with each log entry. * Timestamp: LogRecord.TimeUnixNano * LogLine: LogRecord.Body holds the body of the log. However, since Loki only supports Log body in string format, we will stringify non-string values using [AsString method from OTEL collector lib](`ab3d6c5b64/pdata/pcommon/value.go (L353)`). * Structured Metadata: Anything which can’t be stored in Index labels and LogLine. Here is a non-exhaustive list of what will be stored in Structured Metadata to give a sense of what it will hold: * Resource Attributes not stored as Index labels is replicated and stored with each log entry. * Everything under InstrumentationScope is replicated and stored with each log entry. * Everything under LogRecord except LogRecord.Body, LogRecord.TimeUnixNano and sometimes LogRecord.ObservedTimestamp. NOTES: * Since Loki does not support `.` or any other special characters other than `_` in label names, we replace all non-supported characters with `_`. * Since Loki only supports string in values of Index Labels and Structured Metadata, all the complex types are converted as follows: * Map would be flattened into label keys using `_` as separator, same as how we do it in [json parser in LogQL](https://grafana.com/docs/loki/latest/query/log_queries/#json). * Everything else is stringified using [AsString method from OTEL collector lib](`ab3d6c5b64/pdata/pcommon/value.go (L353)`) Special notes for your reviewer: I will open follow-up PRs for: * Documentation * Make blessed attributes list configurable per tenant. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label	2 years ago
Christian Haudum	2abb472e7e	Unify ring managers across components (#10931 ) What this PR does / why we need it: This PR creates a re-usable `RingManager` implementation that is then used by * Query Scheduler ring * Index Gateway ring * Bloom Gateway ring * Bloom Compactor ring Which issue(s) this PR fixes: Every component had its own implementation of the ring manager which only really differed in the log messages containing the component name. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	b49b3ce6e1	Add basic structure of bloom gateways (#10782 ) ### Summary This pull requests adds the basic structure for the new bloom gateway component. - Adds new `bloom-gateway` target that runs with multiple instances joined by a ring - Adds a querier and client component on the index gateway to filter chunk refs - Adds the gRPC protobuf definitions for commication between index gateways and bloom gateways - Adds a store component used on the bloom gateways to query binary bloom files ``` Querier Query Frontend \| \| ................................... service boundary \| \| +----+------+ \| indexgateway.Gateway \| bloomgateway.BloomQuerier \| bloomgateway.GatewayClient \| logproto.BloomGatewayClient \| ................................... service boundary \| bloomgateway.Gateway \| bloomshipper.Store \| bloomshipper.Shipper \| bloomshipper.BloomFileClient \| ObjectClient \| ................................... service boundary \| object storage not part of this PR ``` This PR still contains a lot of TODOs and possibilities for optimisations, which will be addressed in subsequent pull requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Shantanu Alshi	183d6f2c39	Upgrade prometheus to 0.47.1 and dskit (#10814 )	2 years ago
Ashwanth	91cae03fe1	chore: restructure compactor package (#10775 ) What this PR does / why we need it: This pr moves the compactor pkg from `pkg/storage/stores/shipper/indexshipper/compactor` to `pkg/compactor` - same level as other high level components (querier, distributor...) table compaction code however resides in the respective store pkg closer to other store specific code. `pkg/storage/stores/shipper/indexshipper/boltdb/compactor/table_compactor.go` `pkg/storage/stores/shipper/indexshipper/tsdb/compactor.go` Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Christian Haudum	2e775f3f44	chore(storage): Restructure shipper packages (#10729 ) What this PR does / why we need it: This PR restructures the code for the shipper component of the Loki storage layer. New package layout: ```console $ tree -d pkg/storage/stores/shipper pkg/storage/stores/shipper └── indexshipper ├── boltdb │ └── compactor ├── compactor │ ├── client │ │ └── grpc │ ├── deletion │ ├── deletionmode │ ├── generationnumber │ └── retention ├── downloads ├── gatewayclient ├── index ├── indexgateway ├── storage ├── testutil ├── tsdb │ ├── index │ ├── testdata │ └── testutil ├── uploads └── util 23 directories ``` * TSDB and BoltDB specific code is under `./pkg/storage/stores/shipper/indexshipper/tsdb` and `./pkg/storage/stores/shipper/indexshipper/boltdb` respectively. * Common code for both TSDB and BoltDB is directly under `./pkg/storage/stores/shipper/indexshipper` and subdirectories, such as `uploads/`, `downloads/`, `compactor/`, ... Special notes for your reviewer: This PR is identical to https://github.com/grafana/loki/pull/10724 except of the package `pkg/storage/stores/indexshipper` the package is `pkg/storage/stores/shipper/indexshipper` (one level deeper). Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	5f7bde74ac	Add limits (#10552 ) What this PR does / why we need it: This PR introduces two new limits for structured metadata: - `max_structured_metadata_size`: Defaults to 64 KB. Limits the maximum size of the structured metadata attached to a log line. - `max_structured_metadata_entries_count`: Default to 128. Limits the maximum number of structured metadata entries attached to a log line. We discussed adding other limits like `max_structured_metadata_label_name_length` and `max_structured_metadata_label_value_length` but discarded this idea for now.	2 years ago
Danny Kopping	2c84959901	Distributor: add ingester append timeouts error (#10456 ) What this PR does / why we need it: Failing to send samples to ingesters because the request exceeded its timeout is a very clear signal that ingesters are unable to keep up with demand. In an incident today we saw that ingesters' push latencies were increased sharply by an expensive regex query which was starving other goroutines of time on CPU. This new alert `loki_distributor_ingester_append_timeouts_total` will give us a high-signal metric which we can use for alerting.	2 years ago
Owen Diehl	2cc80c59c4	per tenant gate for structured metadata feature (#10402 ) Adds a per tenant configuration to block access to the upcoming structured-metadata push API extension. --------- Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
akhilanarayanan	3303844cb0	Refactored forGivenIngestors to use DoUntilQuorum instead of Do (#9984 ) This changes the forGivenIngestors function in pkg/querier/ingester_querier.go to use `DoUntilQuorum` (from [PR #293](https://github.com/grafana/dskit/pull/293)) instead of `Do`. `DoUntilQuorum` is much more efficient than `Do`. The exact differences between them can be seen [here](https://github.com/grafana/dskit/pull/293). --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Danny Kopping	8fa4134ef9	Update dskit (#10091 )	2 years ago
Dylan Guedes	d1be9d54ed	Change "tenant" to "org_id" (#10017 ) What this PR does / why we need it: Change our writefailures module to use "org_id" instead of "tenant" to be more consistent with the rest of our system.	2 years ago
Vladyslav Diachenko	be1a9c4ec5	Revert "updated to the latest version of dskit" (#9953 ) Reverts grafana/loki#9920	2 years ago
Dylan Guedes	9ed7fa9df4	WriteFailures: Add metrics to increase monitoring (#9904 ) What this PR does / why we need it: Add metrics for our write failures manager. This way we can easily monitor how much logging we're doing and how much we're discarding. Which issue(s) this PR fixes: N/A	2 years ago
Vladyslav Diachenko	3243676f4c	updated to the latest version of dskit (#9920 ) What this PR does / why we need it: Updated Loki with the latest version of dskit Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Hamish Forbes	d0baa40df3	Only modify identical timestamps when increment-duplicate-timestamps enabled (#9437 ) When `-validation.increment-duplicate-timestamps` is enabled out of order log entries within the same push request can have their timestamps incorrectly mutated. The code is currently just looking for 2 consecutive entries where the timestamp is not increasing. It should only affect entries where the timestamp is a duplicate Test push request payload: ```json { "streams": [ { "stream": { "increment_duplicate_timestamp": "fixed\|true\|false" }, "values": [ [ "1683665111628000000", "1a: 1683665111628000000" ], [ "1683665111628000000", "1b: 1683665111628000000" ], [ "1683665111628000003", "3: 1683665111628000003" ], [ "1683665111628000002", "2: 1683665111628000002" ] ] } ] } ```` As only the first 2 of these timestamps are the same I would expect all but `1b` to have the timestamps recorded as-is and the logs returned by a query in the order: 1a,1b,2,3. Actual Results: ``` > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="fixed"}' \| jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628000001Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000002Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="true"}' \| jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628000001Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000004Z" } > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="false"}' \| jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000002Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } ``` `true` and `false` were written to Loki 2.8.2 with the `-validation.increment-duplicate-timestamps` enabled or disabled `fixed` with this patch applied and the flag enabled Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Dylan Guedes	1d20fdbf57	LogPushFailures: Log the tenant id (#9819 ) What this PR does / why we need it: Modify the `writefailures.Manager` to - Log the tenant id. Otherwise, it is hard to identify which tenant is responsible for a problematic push - Log stream parsing failures Which issue(s) this PR fixes: N/A	2 years ago
Dylan Guedes	dc5d458c14	Failures logging: avoid logging generic errors (#9796 ) What this PR does / why we need it: Modify our write path to not log generic messages with the `writefailures.Manager`. With generic messages I mean: contextCancel errors and contextDeadlineExceeded errors. Which issue(s) this PR fixes: N/A	2 years ago
Salva Corts	08e505dd50	Return all errors for a failed push request (#9807 ) What this PR does / why we need it: Distributors enforce limits and validate each push payload. before this PR, if a single push request yields multiple errors, only the last error is forwarded to the client. This PR modifies the distributor push handler so we forward all errors from a single push request to the client. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: - I wonder if the use of [MultiError][1] is well-suited for this. [This test case][2] will return 100 errors with the same content. I wonder if we can leverage the MultiError to group errors that have the same content. Printing for example: `1 error like: 'Error parsing labels...'; 100 errors like: 'Max entry size...'`. This would require extra processing but If we do this processing in the Error() method I think it should be ok-ish? - Edit: I ended up implementing this in [`4b0816f`](`4b0816f67f`) Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) [1]: `d4a4728933/pkg/util/errors.go (L32)` [2]: `f544b93c20/pkg/distributor/distributor_test.go (L75)`	2 years ago
Dylan Guedes	c0d9a2b4ba	Log push failures per-entry instead of in batches (#9720 ) What this PR does / why we need it: Modify how we log push errors to do it per-entry instead of accumulating all errors and then logging them all in a single pass. Which issue(s) this PR fixes: N/A	2 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	2 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588 ) For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	2 years ago

1 2 3 4 5

228 Commits (c9142722684ad6aca2bb09ae5487d5a2af087d0f)