30 Commits (3a68bc46369fa066c669cbca967e4621ea1f0e58)

Author	SHA1	Message	Date
Paul Rogers	f2c2a22bdc	chore: Preparation for incoming static code analysis CI check (#15164) ... Co-authored-by: Danny Cooper <danny.cooper@grafana.com>	6 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Sandeep Sukhani	7cc9a9386a	fix: fix parsing of default per tenant otlp config (#12836)	1 year ago
Cyril Tovena	9e6e6d22e0	feat: Detect level as structured metadata by default (#12479)	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Christian Haudum	71602eb10f	chore(blooms): Clean up bloom component configuration (#12387) ... * Remove unused setting and move download parallelism configuration one level up. * Update description of bloom_shipper config block Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Sandeep Sukhani	288c0061bf	feat: discovery of name of services emitting the logs (#12392)	1 year ago
Trevor Whitney	4cf47da08e	ci: track main of loki-release on main branch (#12264)	1 year ago
Owen Diehl	7560067439	Bloom/running (#11918) ... Includes a few small fixes in order to get the compactor running, including * Wires up i/o from `v1` pkg and the `bloomshipper` pkg in order to archive+write blocks remotely after building * config error handling * fix an error iterating chunks * handles gzipping when loading tsdbs from storage * bypasses internal cache in `BloomTSDBStore`'s internal `indexshipper/storage.Client` to get up to date results, which is easier for prototyping * `ClosableReadSeekerAdapter` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	6bcac00c70	Add e2e tests for bloom filtering (#11645) ... **What?** This PR adds an end-to-end integration test for creating bloom filters and using them in the gateway to filter chunks. **Why?** This test helped to identify bugs in the bloom shipper code that would have taken a very long time to discover in other ways, such as deploying to a dev environment or running a long-running docker compose setup. **Notes** The following commits of this PR are actual changes to the compactor/gateway code to make e2e test work: * 2dcc761ba0 * 84052ddb34 * 3ba44f83c8 The bloom gateway code path for processing blocks was cleaned up, because it still contained the unused "sequention processing" path of blocks, which was initially kept to verify the callback based processing works alike: * f55d79b84f --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Karsten Jeschkies	4ed3b117d2	Use newest build image 0.31.2 with golangci-lint update. (#11118) ... **What this PR does / why we need it**: This just updates the Loki build imaage changed in #11114. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
Ashwanth	a118e994d8	Remove shared_store and shared_store_key_prefix from shipper and compactor (#10840) ... **What this PR does / why we need it**: #### Removes `shared_store` and `shared_store_key_prefix` from index shipper and compactor configs and their corresponding CLI flags. - `-tsdb.shipper.shared-store` - `-boltdb.shipper.shared-store` - `-tsdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.compactor.shared-store` - `-boltdb.shipper.compactor.shared-store.key-prefix` `shared_store` has been a confusing option allowing users to easily misconfigure Loki. Going forward `object_store` setting in the [period_config](https://grafana.com/docs/loki/latest/configure/#period_config) (which already configured the store for chunks) will be used to configure store for the index. And the newly added `path_prefix` option under the `index` key in `period_config` will configure the path under which index tables are stored. This change enforces chunks and index files for a given period reside together in the same storage bucket. More details in the upgrade guide. --- `-compactor.delete-request-store` has to be **explicitly configured** going forward. Without setting this, loki wouldn't know which object store to use for storing delete requests. Path prefix for storing deletes is decided by `-compactor.delete-request-store.key-prefix` which defaults to `index/`. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Sandeep Sukhani	6069df8f7d	ingestion: native otlp ingestion support (#10727) ... **What this PR does / why we need it**: Add support for natively supporting logs ingestion in OTLP format. `/otlp/v1/logs` is the new endpoint where users can push logs in OTLP format. It accepts logs serialized in JSON or proto format. Since OTEL format is very different than what Loki storage model, here is how data in OTEL format will be mapped to Loki data model: * Index labels: The Resource Attributes map quite well to Index labels in Loki since both usually identify the source of the logs. The problem however is that Resource attributes in OTLP can have an unbounded number of values while Loki has a default limit of having up to 30 labels. Since Index labels in Loki can largely drive the kind of querying experience the users are going to have, we have chosen select attributes which would be picked as Index Labels. The ones that are not picked up as Index labels would be stored as Structured Metadata with each log entry. * Timestamp: LogRecord.TimeUnixNano * LogLine: LogRecord.Body holds the body of the log. However, since Loki only supports Log body in string format, we will stringify non-string values using [AsString method from OTEL collector lib](ab3d6c5b64/pdata/pcommon/value.go (L353)). * Structured Metadata: Anything which can’t be stored in Index labels and LogLine. Here is a non-exhaustive list of what will be stored in Structured Metadata to give a sense of what it will hold: * Resource Attributes not stored as Index labels is replicated and stored with each log entry. * Everything under InstrumentationScope is replicated and stored with each log entry. * Everything under LogRecord except LogRecord.Body, LogRecord.TimeUnixNano and sometimes LogRecord.ObservedTimestamp. *NOTES*: * Since Loki does not support `.` or any other special characters other than `_` in label names, we replace all non-supported characters with `_`. * Since Loki only supports string in values of Index Labels and Structured Metadata, all the complex types are converted as follows: * Map would be flattened into label keys using `_` as separator, same as how we do it in [json parser in LogQL](https://grafana.com/docs/loki/latest/query/log_queries/#json). * Everything else is stringified using [AsString method from OTEL collector lib](ab3d6c5b64/pdata/pcommon/value.go (L353)) **Special notes for your reviewer**: I will open follow-up PRs for: * Documentation * Make blessed attributes list configurable per tenant. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label	2 years ago
Owen Diehl	2cc80c59c4	per tenant gate for structured metadata feature (#10402) ... Adds a per tenant configuration to block access to the upcoming structured-metadata push API extension. --------- Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Sandeep Sukhani	962e03932d	non-indexed labels: support for deletion of logs with non-indexed labels (#10337) ... **What this PR does / why we need it**: We recently added support for storing and querying non-indexed labels with each log line. This PR wires up the code for supporting deletion using non-indexed labels. **Checklist** - [x] Tests updated	2 years ago
Ashwanth	10e55a6359	store: fix incorrect usage of period config in table range (#9754) ... **What this PR does / why we need it**: We are incorrectly taking reference to a loop variable in store init, I overlooked this part as the reference was being taken in the call to `GetIndexTableNumberRange` As a result of this, `TableRange` of each period (which is used by index-shipper) points to the latest `periodConfig`. This in itself is not a problem if the index prefix doesn't change as we use the `periodConfig` only to generate the `tableName`. But if the index prefix changes across periods, we end up losing access to the indexes using the old prefix. **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Sandeep Sukhani	0a5e149ea5	query-scheduler: fix query distribution in SSD mode (#9471) ... **What this PR does / why we need it**: When we run the `query-scheduler` in `ring` mode, `queriers` and `query-frontend` discover the available `query-scheduler` instances using the ring. However, we have a problem when `query-schedulers` are not running in the same process as queriers and query-frontend since [we try to get the ring client interface from the scheduler instance](abd6131bba/pkg/loki/modules.go (L358)). This causes queries not to be spread across all the available queriers when running in SSD mode because [we point querier workers to query frontend when there is no ring client and scheduler address configured](b05f4fced3/pkg/querier/worker_service.go (L115)). I have fixed this issue by adding a new hidden target to initialize the ring client in `reader`/`member` mode based on which service is initializing it. `reader` mode will be used by `queriers` and `query-frontend` for discovering `query-scheduler` instances from the ring. `member` mode will be used by `query-schedulers` for registering themselves in the ring. I have also made a couple of changes not directly related to the issue but it fixes some problems: * [reset metric registry for each integration test](18c4fe5907) - Previously we were reusing the same registry for all the tests and just [ignored the attempts to register same metrics](01f0ded7fc/integration/cluster/cluster.go (L113)). This causes the registry to have metrics registered only from the first test so any updates from subsequent tests won't reflect in the metrics. metrics was the only reliable way for me to verify that `query-schedulers` were connected to `queriers` and `query-frontend` when running in ring mode in the integration test that I added to test my changes. This should also help with other tests where earlier it was hard to reliably check the metrics. * [load config from cli as well before applying dynamic config](f9e2448fc7) - Previously we were applying dynamic config considering just the config from config file. This results in unexpected config changes, for example, [this config change](4148dd2c51/integration/loki_micro_services_test.go (L66)) was getting ignored and [dynamic config tuning was unexpectedly turning on ring mode](52cd0a39b8/pkg/loki/config_wrapper.go (L94)) in the config. It is better to do any config tuning based on both file and cli args configs. **Which issue(s) this PR fixes**: Fixes #9195	2 years ago
Ashwanth	52cd0a39b8	compactor: multi-store support (#7447) ... **What this PR does / why we need it**: This PR adds multi-store support for compactors. Since loki allows users to configure mutiple stores using schema_config, compactor should be able to operate on multiple object stores that contain index. Currently, it can perform compaction on indexes in a single store. To maintain backward compatibility: if `boltdb.shipper.compactor.shared-store` is set, compactor will only operate on that store, else compactor will be initialized to operate on all the object store indexes (`boltdb, tsdb`) defined in the schema config. This PR also adds a new config option to define where delete requests are to be stored - `boltdb.shipper.compactor.delete-request-store`. If it's not set, `boltdb.shipper.compactor.shared-store` is used for storing them, this is to ensure no config changes are required by the users when upgrading. Refer to [docs/sources/upgrading/_index.md](a85d01c9cd/docs/sources/upgrading/_index.md) for more details. **Which issue(s) this PR fixes:** Fixes https://github.com/grafana/loki/issues/7276 **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [X] Documentation added - [X] Tests updated - [X] `CHANGELOG.md` updated - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Ashwanth	5cef03d0ea	index-shipper: add support for multiple stores (#7754) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: Currently loki initializes a single instance of index-shipper to [handle all the table ranges](ff7b462973/pkg/storage/factory.go (L188)) (from across periods) for a given index type `boltdb-shipper, tsdb`. Since index-shipper only has the object client handle to the store defined by `shared_store_type`, it limits the index uploads to a single store. Setting `shared_store_type` to a different store at a later point in time would mean losing access to the indexes stored in the previously configured store. With this PR, we initialize a separate index-shipper & table manager for each period if `shared_store_type` is not explicity configured. This offers the flexibility to store index in multiple stores (across providers). **Note**: - usage of `shared_store_type` in this commit text refers to one of these config options depending on the index in use: `-boltdb.shipper.shared-store`, `-tsdb.shipper.shared-store` - `shared_store_type` used to default to the `object_store` from the latest `period_config` if not explicitly configured. This PR removes these defaults in favor of supporting index uploads to multiple stores. **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: All the instances of downloads table manager operate on the same cacheDir. But it shouldn't be a problem as the tableRanges do not overlap across periods. **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Danny Kopping	61311957a3	Integration Tests: allow loki logs to be optionally output in integration tests (#8793) ... **What this PR does / why we need it**: Debugging integration tests can be difficult without Loki's logs being output. Changing the call from `cluster.New()` to `cluster.New(true, level.WarnValue())` now allows the author the option to temporarily or permanently enable logging for their test, as well as configure a level filter to control the noise. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Danny Kopping	33e44ed39d	Ruler: remote rule evaluation (#8744) ... **What this PR does / why we need it**: Adds the ability to evaluate recording & alerting rules against a given `query-frontend`, allowing these queries to be executed with all the parallelisation & optimisation that regular adhoc queries have. This is important because with `local` evaluation all queries are single-threaded, and rules that evaluate a large range/volume of data may timeout or OOM the `ruler` itself, leading to missed metrics or alerts. When `remote` evaluation mode is enabled, the `ruler` effectively just becomes a gRPC client for the `query-frontend`, which will dramatically improve the reliability of the `ruler` and also drastically reduce its resource requirements. **Which issue(s) this PR fixes**: This PR implements the feature discussed in https://github.com/grafana/loki/pull/8129 (**LID 0002: Remote Rule Evaluation**).	2 years ago
Sandeep Sukhani	b2d0481cbe	deletion: fix log deletion with line filters (#8151)	2 years ago
Dylan Guedes	ad2260aec2	Loki: Fix multitenant querying (#7708) ... **What this PR does / why we need it**: We recently broke multitenant querying because of recent changes to how timeouts working across Loki. This PR fixes this by: - Adapt the timeout wrapper to work with multitenant queries. It will take the shortest timeout across all given tenants - Adapt the query engine timeout assigning to work with multitenant queries. It will take the shortest timeout between all the tenants - Adapt query sharding to use smallest max query parallelism across given tenants - Add a functional test to ensure multitenant is behaving as expected Signed-off-by: DylanGuedes <djmgguedes@gmail.com> Signed-off-by: Mehmet Burak Devecí <mhmtbrkdvc@gmail.com> **Which issue(s) this PR fixes**: https://github.com/grafana/loki/issues/7696 **Special notes for your reviewer**: The regression was probably introduced by https://github.com/grafana/loki/pull/7555	3 years ago
Sandeep Sukhani	97a7ed0b61	fix logs deletion integration test to properly validate deleted data (#7751) ... **What this PR does / why we need it**: Currently, the integration test for deletion does not validate whether the delete request properly deleted the requested data or not. It just currently checks if the delete request got processed. This was due to the singleton `boltdb-shipper` client causing problems in flushing chunks. I have used `ResetBoltDBIndexClientWithShipper` to reset the boltdb-shipper client to get the chunk flush from the ingester working. The test now does the checks to ensure that the right data got deleted by the compactor. This should help us catch any bugs earlier for any future changes in deletion code. **Checklist** - [x] Tests updated	3 years ago
Travis Patterson	e89c30ce6b	Fix Flaky Integration Tests (#7360) ... Allow the OS kernel to decide what's an open port instead of trying to guess	3 years ago
Dylan Guedes	2dc5a71a67	Tests: Reduce flakiness of GCPLog and integration tests (#7273) ... **What this PR does / why we need it**: - Reduce flakiness of GCPLog tests by reusing context and stopping GCP topic - Reduce flakiness of integration tests by: - using port as they're assigned - reusing context - adding timeout to calls (hence, if something get stuck, tests fail earlier)	3 years ago
Mohamed-Amine Bouqsimi	3d4788fb51	Add ability for Ruler to remote write to multiple clients (#6823)	3 years ago
Christian Simon	d63add09c5	Fix free port allocation flake (#6898) ... Occassionaly we can see test flakes like that: > server: listen tcp :34473: bind: address already in use This is because we allocated ports by checking if they are free and then using them. Altough there is a chance that already allocated ports are used twice. This changes prevents this from happening by keeping track of the ports that are about to be used.	3 years ago
Michel Hollands	65e3148bc9	Update delete integration test (#6192) ... * Add wait until processed Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add query lines Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove Cortex reference Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add deleted lines metric and check in test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Set negative cancel period so default 1 minute delay is compensated for Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Reduce timeout Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add helper function Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Comment out assertions that only work with flush Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove unused metric type param Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Check counter in unit test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename ClientOption to Option as per the linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove redundant check Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * fill in MsgAndArgs field of test assertion Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Christian Simon	213bb86802	Add integration tests which test happy path (#5968) ... * Add integration tests which test happy path This PR adds simple go based integration tests, which can run Loki in two modes: - single-binary - micro-services with index-gateway After standing up it performs the most basic duties like: - ingest logs - run a range query - query label names and label values This is meant to catch more significant problem as early as possible in the PR. We should aim to keep this as simple as possible. * Replace config generation with template Makes configuration much more readable and avoids easy mistakes * Apply suggestions from code review Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago