249 Commits (8aa8a2bb0e766da4d64313d17337fa54ab84f8a4)

Author	SHA1	Message	Date
benclive	8aa8a2bb0e	fix(kafka): Set namespace for Loki kafka metrics (#14426)	8 months ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	8 months ago
Trevor Whitney	d7ff426646	feat: ability to log stream selectors before service name detection (#14154)	8 months ago
Cyril Tovena	1a4436c417	feat(ingester): implement partition shuffle sharding for ingester (#14304)	8 months ago
Cyril Tovena	f80d68a1ed	feat(distributors): Use a pool of worker to push to ingesters. (#14245)	8 months ago
Cyril Tovena	3f47233079	fix(kafka): Fixes partition selection in distributors (#14242)	8 months ago
benclive	7c78232ad3	feat: Optionally require writes to kafka on Push requests (#14186)	8 months ago
Vladyslav Diachenko	c65721e7ad	feat: track discarded data by usageTracker (#14081) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	9 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Salva Corts	b5ac6a0258	feat: Limit to block ingestion until configured date (#13958)	9 months ago
Paul Rogers	8fb7b488bd	chore(deps): update dskit 20240819 (#13924)	9 months ago
Trevor Whitney	913e9f9347	feat: aggregate byte and count metrics (#13731)	9 months ago
Travis Patterson	573a1848d1	chore: add service_name label earlier in the ingestion pipeline (#13702)	10 months ago
benclive	6c4b0622aa	feat: Ignore empty streams in distributor if all entries fail validation (#13674)	10 months ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	11 months ago
Cyril Tovena	9f31b25253	feat: Optimize log parsing performance by using unsafe package (#13223)	12 months ago
Yahima Duarte	c9bfa3ebbf	fix: add detected_level info when the info word appears on log message (#13218)	12 months ago
Salva Corts	b31e04e3f1	fix(otel): Map 500 errors to 503 (#13173)	12 months ago
Dylan Guedes	afd9e36306	chore: Call `shardstreams.Config` by value instead of by reference (#12915) ... Use `shardstreams.Config` by value instead of by reference to fix docs generation. Our `docs-generator` tool relies on the struct address/references to assume that flags are present. Using this config by value fixes it.	1 year ago
Shantanu Alshi	6904a6520d	fix: Fix log level detection (#12651)	1 year ago
Sandeep Sukhani	0831802a99	fix: fix setting of info log level when trying to detect level from log lines (#12635)	1 year ago
Ed Welch	705379cc81	chore: remove experimental flags from features and configs which are no longer so. (#12481) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Cyril Tovena	9e6e6d22e0	feat: Detect level as structured metadata by default (#12479)	1 year ago
Ed Welch	24f4e45165	feat: Enable automatic stream sharding by default (#12451) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Sandeep Sukhani	1ae5b3e683	chore: Log level detection improvements (#12440)	1 year ago
Ed Welch	824f5aa20a	chore: fix submodule for v3 (#12438) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Sandeep Sukhani	b66c343383	feat: add support for discovering and adding log levels as structured metadata (#12428)	1 year ago
Sandeep Sukhani	288c0061bf	feat: discovery of name of services emitting the logs (#12392)	1 year ago
Cyril Tovena	ca190a3a7d	feat: Allows to wrap distributor Tee (#12391)	1 year ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292) ... A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Karsten Jeschkies	243e6f813a	refactor: Pass context to usage tracker. (#12272)	1 year ago
Karsten Jeschkies	9a2e00bc17	refactor: Replace TenantConfig method with interface. (#12260) ... **What this PR does / why we need it**: This change allows the retrieval of the `TenantConfig` or the `DefaultConfig`. It also defines an interface instead of a tenant config function which makes it a little easier to grok the code. Eventually this will enable us to introduce GEL specific configurations. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	1 year ago
Sandeep Sukhani	8b9d8b0226	feat: Make list of otel blessed attributes configurable (#12180)	1 year ago
Travis Patterson	b0478e61c8	feat: add the ability to inject log parsers to the distributor (#12029)	1 year ago
Karsten Jeschkies	6578a00481	feat: Support usage trackers for received and discarded bytes. (#11840)	1 year ago
Paul Rogers	edc9a44f66	test: Finish sync work for mockIngester.pushed var (#12028)	1 year ago
JordanRushing	cad4b8e749	Reduce Distributor auto-forget unhealthy cycles from 10->2 (#11935) ... Signed-off-by: JordanRushing <rushing.jordan@gmail.com>	1 year ago
Sandeep Sukhani	e625d7f1ca	otel: add support for per tenant configuration for mapping otlp data to loki format (#11143) ... **What this PR does / why we need it**: In OTEL, we pick select Resource Attributes to identify the streams and store all the other attributes as Structured Metadata, as explained here. The problem however is that the list of Resource Attributes that are picked as Stream labels are hardcoded, and there is no way to drop unwanted data. This PR adds support for configuring how data is mapped from OTEL to Loki format per tenant. It also adds support for dropping unwanted data. We decided to make the config look similar to Prometheus's relabling config to make it familiar. **Special notes for your reviewer**: Opening a draft PR to get some initial feedback. I will add documentation in a separate PR. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated	1 year ago
Travis Patterson	d2f43787b6	Add tenant to tee (#11573) ... It is useful for implementations of Tee to also know the tenant who sent the logs. This PR adds tenant to the tee interface.	1 year ago
na--	724a841f0f	Add log duplication (tee-ing) in the Distributor (#11496) ... This allows us to inject a Tee in the Distributor that can duplicate all of the incoming log stream, before they are replicated.	1 year ago
na--	71e9fcaab7	Refactor Distributor.Push() to be able to Tee log streams (#11488) ... This allows us to duplicate (i.e. "tee") some of the log streams from the `Distributor` by introducing an optional interface in a follow-up PR.	1 year ago
Angus Dippenaar	258a8b3108	Update nix configuration (#8452) ... Fix nix configuration (mainly failing test), and break up various binaries (ie. logcli, promtail, loki, etc.) into their own pakcages. --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Ashwanth	766f27645d	config: remove flags & YAML settings deprecated in 2.9.x (#11151) ... **What this PR does / why we need it**: - remove `-ruler.evaluation-delay-duration` - remove `boltdb.shipper.compactor.deletion-mode` - remove compactor flags with prefix`-boltdb.shipper.compactor.*` - remove `validation.enforce-metric-name` - remove `log.use-buffered`, `log.use-sync` **Special notes for your reviewer**: `log.use-buffered`, `log.use-sync` are not mentioned in the upgrade guide since they were hidden fields **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [X] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Periklis Tsirakidis	6385b19573	fix: All lifecycler cfgs ref a valid IPv6 addr and port combination (#11121) ... **What this PR does / why we need it**: Ensure that all constructors of LifeCyclerConfig are using `net.JoinHostPort` for instance addr and port combinations. This enables IPv6 usage compatible across the code based. Currently only distributors provide IPv6 compatibility because they use `Lifecycler` (that pulls the addr/port in the `NewLifeCycler`) while everything else uses `BasicLifeCycler` delegate this to the user defining an addr/port combination in `LifecyclerConfig`.	2 years ago
Michel Hollands	e93f5bfd93	Add dashboard changes for migrating from cortex_ metrics to loki_ metrics (#11082) ... **What this PR does / why we need it**: - add missing namespace param for ruler manager metrics - update the loki mixin to use the renamed metrics so both are supported - created the compiled mixin - make the prefix of the cortex_distributor_ingester_clients metrics configurable **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Shantanu Alshi	fb6f6d2e2f	Stop using global logger in modules (#11051) ... **What this PR does / why we need it**: Refactors Loki to inject logger from the constructors instead of using global logger variable. It is generally considered a good practice. **Which issue(s) this PR fixes**: [Fixes #<issue number>](https://github.com/grafana/loki-private/issues/810) **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR -->	2 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025) ... **What this PR does / why we need it**: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Sandeep Sukhani	6069df8f7d	ingestion: native otlp ingestion support (#10727) ... **What this PR does / why we need it**: Add support for natively supporting logs ingestion in OTLP format. `/otlp/v1/logs` is the new endpoint where users can push logs in OTLP format. It accepts logs serialized in JSON or proto format. Since OTEL format is very different than what Loki storage model, here is how data in OTEL format will be mapped to Loki data model: * Index labels: The Resource Attributes map quite well to Index labels in Loki since both usually identify the source of the logs. The problem however is that Resource attributes in OTLP can have an unbounded number of values while Loki has a default limit of having up to 30 labels. Since Index labels in Loki can largely drive the kind of querying experience the users are going to have, we have chosen select attributes which would be picked as Index Labels. The ones that are not picked up as Index labels would be stored as Structured Metadata with each log entry. * Timestamp: LogRecord.TimeUnixNano * LogLine: LogRecord.Body holds the body of the log. However, since Loki only supports Log body in string format, we will stringify non-string values using [AsString method from OTEL collector lib](ab3d6c5b64/pdata/pcommon/value.go (L353)). * Structured Metadata: Anything which can’t be stored in Index labels and LogLine. Here is a non-exhaustive list of what will be stored in Structured Metadata to give a sense of what it will hold: * Resource Attributes not stored as Index labels is replicated and stored with each log entry. * Everything under InstrumentationScope is replicated and stored with each log entry. * Everything under LogRecord except LogRecord.Body, LogRecord.TimeUnixNano and sometimes LogRecord.ObservedTimestamp. *NOTES*: * Since Loki does not support `.` or any other special characters other than `_` in label names, we replace all non-supported characters with `_`. * Since Loki only supports string in values of Index Labels and Structured Metadata, all the complex types are converted as follows: * Map would be flattened into label keys using `_` as separator, same as how we do it in [json parser in LogQL](https://grafana.com/docs/loki/latest/query/log_queries/#json). * Everything else is stringified using [AsString method from OTEL collector lib](ab3d6c5b64/pdata/pcommon/value.go (L353)) **Special notes for your reviewer**: I will open follow-up PRs for: * Documentation * Make blessed attributes list configurable per tenant. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label	2 years ago
Christian Haudum	2abb472e7e	Unify ring managers across components (#10931) ... **What this PR does / why we need it**: This PR creates a re-usable `RingManager` implementation that is then used by * Query Scheduler ring * Index Gateway ring * Bloom Gateway ring * Bloom Compactor ring **Which issue(s) this PR fixes**: Every component had its own implementation of the ring manager which only really differed in the log messages containing the component name. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago