716 Commits (54edb2143f9dbf8280e1cfa1c8efbfef1bb3e5bb)

Author	SHA1	Message	Date
Ashwanth	766f27645d	config: remove flags & YAML settings deprecated in 2.9.x (#11151) ... **What this PR does / why we need it**: - remove `-ruler.evaluation-delay-duration` - remove `boltdb.shipper.compactor.deletion-mode` - remove compactor flags with prefix`-boltdb.shipper.compactor.*` - remove `validation.enforce-metric-name` - remove `log.use-buffered`, `log.use-sync` **Special notes for your reviewer**: `log.use-buffered`, `log.use-sync` are not mentioned in the upgrade guide since they were hidden fields **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [X] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Paul Rogers	c4ed0d0b7f	Tokenizer tests and TokenizeLine updates (#11133) ... **What this PR does / why we need it**: The thrust of this PR is to ensure we have tests for each major function of the Bloom Tokenizer. In addition, there was some cleanup, in that constants are used to set some common parameters. Lastly, the TokenizeLine() call was updated to correctly tokenize a line when a "skip tokenizer" is utilized. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	3 years ago
Poyzan	cc6f0eca98	Return checksum from BuildFrom function (#11178) ... **What this PR does / why we need it**: Follow up from https://github.com/grafana/loki/pull/11115 We want to know the checksum of the bloom blocks before uploading them to storage. Instead of opening the file and reading it, this PR returns the checksum from where it's created.	3 years ago
Poyzan	fa378ac0ec	Add first step for creating a single bloom from series (#11115) ... **What this PR does / why we need it**: This is the first step to create a bloom from a series. The logic to get tenants and tables is a temporary hack and will be replaced by the ring logic in https://github.com/grafana/loki/pull/11154 **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: It does not handle appending multiple blooms into blocks. It does not handle creating and uploading of meta files. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	3 years ago
Dylan Guedes	30fbe38341	Revert "Introduces cache to TSDB postings (#9621)" (#11157) ... This reverts commit 1221658fc5. **What this PR does / why we need it**: We decided to discontinue the postings cache a few releases ago for not finding any evidences that postings was the culprit for the bad igw performance. Plus, we noticed a major issue with it: it doesn't bubble down shards correctly, so every query end up processing every shard.	3 years ago
Michel Hollands	e93f5bfd93	Add dashboard changes for migrating from cortex_ metrics to loki_ metrics (#11082) ... **What this PR does / why we need it**: - add missing namespace param for ruler manager metrics - update the loki mixin to use the renamed metrics so both are supported - created the compiled mixin - make the prefix of the cortex_distributor_ingester_clients metrics configurable **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	3 years ago
Karsten Jeschkies	4ed3b117d2	Use newest build image 0.31.2 with golangci-lint update. (#11118) ... **What this PR does / why we need it**: This just updates the Loki build imaage changed in #11114. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	3 years ago
Owen Diehl	6bfd2baec4	Bloom/fuse queries (#11088) ... Introduces a "Fuse" operation on a single block querier which multiplexes queries against a single block. This is advantageous for a few reasons: * A "query" contains one of more series, with each series needing to test one or more chunks for some search string(s). * Blocks are sorted by fingerprint, which are the result of a hash function. This means series are ~uniformly distributed across the hash keyspace (`uint64` in our case). Because of this, a query for some number of series are likely to be spread out across the block itself. * Blocks are organized into individually compressed "pages". In essence, querying them requires Seeking to the correct offset, decompressing the page, then seeking within the page to the actual offset for a particular series. * Since the series for any query are likely spread out across the block, performing `n` queries decompresses each page `n` times. * By multiplexing many queries into a single pass iteration of the block, we amortize the cost of loading & decompressing Misc: Also introduces a bunch of generic-powered iterators. These can probably be optimized (I bet they use dynamic dispatch under the hood), but they're very convenient to prototype with and we can improve them in the future if it proves warranted.	3 years ago
Poyzan	df9c6fcac4	Add series population to tokenizer (#11070) ... **What this PR does / why we need it**: I started to use tokenizer module for bloom creation and populating the SBF without the series with chunk ref is weird, as we populate only half of a data structure. I modified it to populate entire `SeriesWithBloom` as it has all the information. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	3 years ago
Vladyslav Diachenko	45b10effac	embedded cache refactoring (#10989) ... **What this PR does / why we need it**: refactored embedded cache to allow using it for any types, not only for `string -> []byte`. Also, the new implementation allows passing the callback that will be called for the removed entry. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
Shantanu Alshi	fb6f6d2e2f	Stop using global logger in modules (#11051) ... **What this PR does / why we need it**: Refactors Loki to inject logger from the constructors instead of using global logger variable. It is generally considered a good practice. **Which issue(s) this PR fixes**: [Fixes #<issue number>](https://github.com/grafana/loki-private/issues/810) **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR -->	3 years ago
April Schleck	4b750cc0cc	Storage: Allow setting a constant prefix for all created keys (#10096) ... **What this PR does / why we need it**: Adds a new option under the aws stanza named key_prefix. This is useful if you have many Loki installations and do not want to create a million buckets. This is different than `compactor.shared_store_key_prefix` because it also affects eg the `loki_cluster_seed.json` file. **Which issue(s) this PR fixes**: Fixes #5889 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Ed Welch <edward.welch@grafana.com>	3 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025) ... **What this PR does / why we need it**: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	3 years ago
Ashwanth	a118e994d8	Remove shared_store and shared_store_key_prefix from shipper and compactor (#10840) ... **What this PR does / why we need it**: #### Removes `shared_store` and `shared_store_key_prefix` from index shipper and compactor configs and their corresponding CLI flags. - `-tsdb.shipper.shared-store` - `-boltdb.shipper.shared-store` - `-tsdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.compactor.shared-store` - `-boltdb.shipper.compactor.shared-store.key-prefix` `shared_store` has been a confusing option allowing users to easily misconfigure Loki. Going forward `object_store` setting in the [period_config](https://grafana.com/docs/loki/latest/configure/#period_config) (which already configured the store for chunks) will be used to configure store for the index. And the newly added `path_prefix` option under the `index` key in `period_config` will configure the path under which index tables are stored. This change enforces chunks and index files for a given period reside together in the same storage bucket. More details in the upgrade guide. --- `-compactor.delete-request-store` has to be **explicitly configured** going forward. Without setting this, loki wouldn't know which object store to use for storing delete requests. Path prefix for storing deletes is decided by `-compactor.delete-request-store.key-prefix` which defaults to `index/`. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Paul Rogers	c7eb757ed8	Initial loki bloom creation library (#10957) ... **What this PR does / why we need it**: Optimization and benchmarking of various LRUs that sit in front of the bloom filters. Initial creation of a library that can be used to create blooms from chunks **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Kaviraj Kanagaraj	a550b767d3	config: Remove already deprecated `store.max-look-back-period`. (#11038) ... This is already been deprecated longback. Decided to remove in next versions of Loki Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Salva Corts	52a3f16039	Flag categorize labels on streams response (#10419) ... We recently introduced support for ingesting and querying structured metadata in Loki. This adds a new dimension to Loki's labels since now we arguably have three categories of labels: _stream_, _structured metadata_, and _parsed_ labels. Depending on the origin of the labels, they should be used in LogQL expressions differently to achieve optimal performance. _stream_ labels should be added to stream matchers, _structured metadata_ labels should be used in a filter expression before any parsing expression, and _parsed_ labels should be placed after the parser expression extracting them. The Grafana UI has a hard time dealing with this same problem. Before https://github.com/grafana/grafana/pull/73955, the filtering functionality in Grafana was broken since it was not able to distinguish between _stream_ and _structured metadata_ labels. Also, as soon as a parser expression was added to the query, filters added by Grafana would be appended to the end of the query regardless of the label category. The PR above implements a workaround for this problem but needs a better API on Loki's end to mitigate all corner cases. Loki currently returns the following JSON for log queries: ```json ... { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", "level": "info", "traceID": "68810cf0c94bfcca" }, "values": [ [ "1693996529000222496", "1693996529000222496 aaaaaaaaa.....\n" ], ... }, { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", "level": "debug", "traceID": "a7116cj54c4bjz8s" }, "values": [ [ "1693996529000222497", "1693996529000222497 bbbbbbbbb.....\n" ], ... }, ... ``` As can be seen, there is no way we can distinguish the category of each label. This PR introduces a new flag `X-Loki-Response-Encoding-Flags: categorize-labels` that makes Loki return categorized labels as follows: ```json ... { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", }, "values": [ [ "1693996529000222496", "1693996529000222496 aaaaaaaaa.....\n", { "structuredMetadata": { "traceID": "68810cf0c94bfcca" }, "parsed": { "level": "info" } } ], [ "1693996529000222497", "1693996529000222497 bbbbbbbbb.....\n", { "structuredMetadata": { "traceID": "a7116cj54c4bjz8s" }, "parsed": { "level": "debug" } } ], ... }, ... ``` Note that this PR only supports log queries, not metric queries. From a UX perspective, being able to categorize labels in metric queries doesn't have any benefit yet. Having said that, supporting this for metric queries would require some minor refactoring on top of what has been implemented here. If we decide to do that, I think we should do it on a separate PR to avoid making this PR even larger. I also decided to leave out support for Tail queries to avoid making this PR even larger. Once this one gets merged, we can work to support tailing. --- **Note to reviewers** This PR is huge since we need to forward categorized all over the codebase (from parsing logs all the way to marshaling), fortunately, many of the changes come from updating tests and refactoring iterators. Tested out in a dev cell with query `'{stream="stdout"} | label_format new="text"`. - Without the new flag: ``` $ http http://127.0.0.1:3100/loki/api/v1/query_range\?direction\=BACKWARD\&end\=1693996529322486000\&limit\=30\&query\=%7Bstream%3D%22stdout%22%7D+%7C+label_format+new%3D%22text%22\&start\=1693992929322486000 X-Scope-Orgid:REDACTED { "data": { "result": [ { "stream": { "new": "text", "pod": "loki-canary-986bd6f4b-xqmb7", "stream": "stdout" }, "values": [ [ "1693996529000222496", "1693996529000222496 pppppppppppp...\n" ], [ "1693996528499160852", "1693996528499160852 pppppppppppp...\n" ], ... ``` - With the new flag ``` $ http http://127.0.0.1:3100/loki/api/v1/query_range\?direction\=BACKWARD\&end\=1693996529322486000\&limit\=30\&query\=%7Bstream%3D%22stdout%22%7D+%7C+label_format+new%3D%22text%22\&start\=1693992929322486000 X-Scope-Orgid:REDACTED X-Loki-Response-Encoding-Flags:categorize-labels { "data": { "encodingFlags": [ "categorize-labels" ], "result": [ { "stream": { "pod": "loki-canary-986bd6f4b-xqmb7", "stream": "stdout" }, "values": [ [ "1693996529000222496", "1693996529000222496 pppppppppppp...\n", { "parsed": { "new": "text" } } ], [ "1693996528499160852", "1693996528499160852 pppppppppppp...\n", { "parsed": { "new": "text" } } ], ... ```	3 years ago
Owen Diehl	0a7737e7c7	adds a merge-builder for building blocks from other blocks+storage (#10974) ... Some minor refactoring + a (simple) `MergeBuilder` struct to build bloom blocks from other blocks and storage, depending on what's already been indexed into the blocks.	3 years ago
Poyzan	cbc0df3241	Add NewBlockOptions constructor (#10955) ... Adds a `NewBlockOptions()` constructor plus a quick fix for a missed err check.	3 years ago
Christian Haudum	2abb472e7e	Unify ring managers across components (#10931) ... **What this PR does / why we need it**: This PR creates a re-usable `RingManager` implementation that is then used by * Query Scheduler ring * Index Gateway ring * Bloom Gateway ring * Bloom Compactor ring **Which issue(s) this PR fixes**: Every component had its own implementation of the ring manager which only really differed in the log messages containing the component name. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Salva Corts	89829065e7	Use lazy bloom filter (#10896) ... **What this PR does / why we need it**: When querying bloom filters, use lazy decoders from https://github.com/owen-d/BoomFilters/pull/1 so we don't make a copy of bloom filters.	3 years ago
Christian Haudum	d84026cab6	Adjust bloom gateway request structure (#10926) ... **What this PR does / why we need it**: Since we already group chunkRefs by fingerprint on the client side to determine the correct bloom gateway to send the request to, we can keep the grouping for the gRPC request. This will not only optimize the request size, but also helps to reduce complexity of querying blooms on the bloom gateway itself. There is still conversion between `[]*logproto.ShortRef` and `v1.ChunkRefs` required, which could be removed by sending `v1.ChunkRefs` over the wire. **Special notes for your reviewer**: Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Karsten Jeschkies	9fcc42dc48	Support protobuf `QueryRequest` in querier. (#10858) ... **What this PR does / why we need it**: This is one step to support pure protobuf encoding without `httpgrpc`. It is only on the scheduler and is fully backwards compatible. **Which issue(s) this PR fixes**: This is a sub change of https://github.com/grafana/loki/pull/10688 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Owen Diehl	ad0f7ab845	Blooms/merge-iters (#10923) ... Adds a bunch of bloom filterutilities in preparation for merging & extending bloom blocks with new chunks/series	3 years ago
Poyzan	0832256d7b	Add basic structure of bloom compactor (#10748) ... This pull request adds the basic structure for the new bloom compactor component. - Adds new `bloom-compactor` target that runs with with multiple instances joined by a ring - Adds boilerplate functions to index blooms and compact blocks The main goal of this PR is to provide a basic functionality on which future smaller PRs can build upon. Since the code path is completely separate in a new component, it is not used anywhere at the moment.	3 years ago
Owen Diehl	f861068df4	moves boomfilters lib into loki (#10898) ... In an effort to speed up development, this moves a subset of [my fork](https://github.com/owen-d/BoomFilters) of the https://github.com/tylertreat/BoomFilters library into loki itself. I've done my best to preserve original license headers/etc.	3 years ago
Christian Haudum	b49b3ce6e1	Add basic structure of bloom gateways (#10782) ... ### Summary This pull requests adds the basic structure for the new bloom gateway component. - Adds new `bloom-gateway` target that runs with multiple instances joined by a ring - Adds a querier and client component on the index gateway to filter chunk refs - Adds the gRPC protobuf definitions for commication between index gateways and bloom gateways - Adds a store component used on the bloom gateways to query binary bloom files ``` Querier Query Frontend | | ................................... service boundary | | +----+------+ | indexgateway.Gateway** | bloomgateway.BloomQuerier | bloomgateway.GatewayClient | logproto.BloomGatewayClient | ................................... service boundary | bloomgateway.Gateway | bloomshipper.Store | bloomshipper.Shipper | bloomshipper.BloomFileClient** | ObjectClient** | ................................... service boundary | object storage ** not part of this PR ``` This PR still contains a lot of TODOs and possibilities for optimisations, which will be addressed in subsequent pull requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Owen Diehl	b52d836867	Blooms/chunk check (#10886) ... [WIP] Adds ability to return the list of chunks needed to be downloaded by a query based on bloom membership results	3 years ago
Vladyslav Diachenko	127a8e27e7	[BloomShipper] updated bloom shipper to extract the block using the library (#10876) ... Initial implementation was done based on the assumption that we would use ZIP to archive the blocks. However, we will use `.tar.gz` archives, so, the implementation has been adjusted to use helpers from the library that allow to archive/unarchive `.tar.gz` files. re: https://github.com/grafana/loki/pull/10849 Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
Owen Diehl	7f02ef3fa7	helpers for tar+gz over bloom blocks (#10849) ... Simple start, but helpful for building & extracting tar+gz archives over bloom block directories.	3 years ago
Ashwanth	8bb615c2cc	config: loki better defaults (#10793) ... **What this PR does / why we need it**: Updates some of the configuration defaults to provide a better experience for users out of the box. I tried to add notes for non-trivial changes explaining the reason. | configuration | new default | old default | notes | | ------------------------------------------------------ | ----------- | ----------- | -------- | `compactor.delete-max-interval` | 24h | 0 | splits the delete requests into intervals no longer than `delete_max_interval` | | `distributor.max-line-size` | 256KB | 0 | - | | `ingester.sync-period` | 1h | 0 | ensures that the chunk cuts for a given stream are synchronized across the ingesters in the replication set. Helps with deduplicating chunks. | | `ingester.sync-min-utilization` | 0.1 | 0 | ^ | | `frontend.max-querier-bytes-read` | 150GB | 0 | - | | `frontend.max-cache-freshness` | 10m | 1m | avoid caching results for data that is still in churn | | `frontend.max-stats-cache-freshness` | 10m | 0 | avoid caching results for data that is still in churn | | `memcached.batchsize` | 256 | 1024 | - | | `memcached.parallelism` | 10 | 100 | - | | `querier.tsdb-max-query-parallelism` | 128 | 512 | avoid over parallelising queries for smaller installations | | `querier.split-queries-by-interval` | 1h | 30m | avoid over parallelising queries for smaller installations | | `querier.compress-http-responses` | true | false | compress response if the request accepts gzip encoding | | `query-scheduler.max-outstanding-requests-per-tenant` | 32000 | 100 | current default is too small, opening a big dashboard could hit this limit | | `validation.max-label-names-per-series` | 15 | 30 | reducing this to avoid blowing up the series count and index size. Also enforces users to think about what labels are being added. | **Special notes for your reviewer**: regarding `TestQueryTSDB_WithCachedPostings`: removed the assert on get calls as this is hard to get right (splits, querySize mw making stats calls) validating additions to the cache and misses is more important here I think. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Arve Knudsen	fc13c9843b	Fix some typos (#10845) ... **What this PR does / why we need it**: Fix some typos, no behavioral changes. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Arve Knudsen	489ca25bab	Upgrade to Prometheus@4b9c19fe5510 (#10843) ... **What this PR does / why we need it**: Upgrade to Prometheus@4b9c19fe5510. Since also golang.org/x/exp gets upgraded, calls to e.g. `slices.SortFunc` need updating (due to changed comparison function signature). I'm not aware of any functional changes from Prometheus (or otherwise), but please review carefully. **Special notes for your reviewer**: The Mimir team needs for Loki to be compatible with the latest Prometheus, so we can vendor mimir-prometheus@main into GEM. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [na] Documentation added - [x] Tests updated - [na] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [na] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [na] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Shantanu Alshi	183d6f2c39	Upgrade prometheus to 0.47.1 and dskit (#10814)	3 years ago
Vladyslav Diachenko	8b94439434	Bloom Shipper (#10806) ... implemented bloom shipper that allows to iterates over all active blocks This new component will be used by the bloom-compactor and bloom-gateway. #### How BloomShipper works 1. looks up all Metas 2. from metas gets list of all blocks 3. from metas gets list of tombstones 4. removes tombstoned blocks from the list of all blocks 5. passes all active blockReferences to bloomClient 6. once bloomClient returns the downloaded block, it extracts all files from bloom block archive into working directory 7. creates `*BlockQuerier` and run a callback function with it **Note:** `BlockQuerier` will be created using `BloomDirectoryReader` that will be created using the past from step #6 --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
Ed Welch	7ccec83926	Index gateway simple mode connection pool (#10783) ... **What this PR does / why we need it**: The current implementation of "simple mode" (which does not use the ring for gateway discovery) only makes a single connection at startup to a single query. This doesn't lend to the best load balancing when running with many gateways. This PR introduces the same pool we use for ring mode with a DNS based discovery mechanism. Note: outside of the scope of this PR, we create a connection pool per index schema period. This isn't really a problem but I think it would be better to revisit this some day to only have a single connection pool per querier. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Poyzan <31743851+poyzannur@users.noreply.github.com>	3 years ago
Vladyslav Diachenko	b31ed050ae	Bloom Client (#10697) ... Implemented BloomClient that allows : 1. fetch all meta.json files that satisfy the search criteria 2. upload meta.json 3. delete meta.json 4. fetch bloom blocks in bulk 5. upload bloom blocks in bulk 6. delete bloom blocks in bulk Bloom shipper was extracted to a separate PR https://github.com/grafana/loki/pull/10806 --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
Owen Diehl	c405db4317	Format + Library for building bloom filter "blocks" from chunks (#10780) ... This PR introduces both a binary format for bloom filter "blocks" and a library for building/querying them. Each "block" consists of two files: 1) `Index`. This is a list of individually compressed pages of stream+chunk information sorted in ascending fingerprint order. Each series lists both the chunks indexed in the bloom filters as well as the relevant offsets for said series in the bloom file. 2) `Blooms`. This is a list of individually compressed pages of bloom filters for each series, again sorted in ascending fingerprint order. There is still more work to do, but I'm opening this PR as an initial pass.	3 years ago
Ed Welch	0c58c78828	Index Gateway: Set some concurrency limits around concurrent index table processing (#10768) ... **What this PR does / why we need it**: This addresses an issue we've seen with index gateways when handling queries that touch a large number of series, although these are improvements not necessarily solutions. We've seen a tremendous amount of goroutines being spawned while processing GetChunkRefs requests, specifically when the gateways have recently found a lot of uncompacted recent ingester tables. This PR puts some limits on the parallelism of processing tables, the implementation is a bit arbitrary, we limit the concurrent iterations to GOMAXPROCS/2 In practice this is potentially still too much as an index gateway will be serving many requests in parallel witch each able to do parallel table iteration, however it's a more reasonable line in the sand than having no limits and eases some pressure on the Go scheduler. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Paschalis Tsilias	07cbef9226	Update Prometheus dependency to v2.46.0 (#10777) ... This PR updates the Prometheus dependency to v2.46.0. We need this to update the Agent to use Prometheus' Native Histograms after some bug fixes in the latest version. I bumped the version in go.mod, ran `go mod tidy && go mod vendor` and fixed compilation issues. **Special notes for your reviewer**: The only changes needed to have `make all` successfully compile was to use the new `wlog.NewSize` to use wlog.CompressionNone (instead of false), and wlog.CompressionSnappy (the default, instead of true), as well as add a new argument to the textparse.New constructor to work as it did without scraping native histograms via protobuf. I don't think there are any other user-facing changes or changes related to Loki. --------- Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>	3 years ago
Ashwanth	91cae03fe1	chore: restructure compactor package (#10775) ... **What this PR does / why we need it**: This pr moves the compactor pkg from `pkg/storage/stores/shipper/indexshipper/compactor` to `pkg/compactor` - same level as other high level components (querier, distributor...) table compaction code however resides in the respective store pkg closer to other store specific code. `pkg/storage/stores/shipper/indexshipper/boltdb/compactor/table_compactor.go` `pkg/storage/stores/shipper/indexshipper/tsdb/compactor.go` **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Ashwanth	3038170aee	Add deprecation node for write dedupe cache (#10736) ... **What this PR does / why we need it**: Adds a deprecation node for write dedupe cache. This cannot be fully removed yet as legacy indexes rely on this. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Ashwanth	fcdd55b6c9	embedded cache: update metric namespace and remove duplicate metrics (#10693) ... **What this PR does / why we need it**: Removes the following embedded cache metrics since `loki_cache_request_duration_seconds` (which instruments requests made to all the caches) [already covers them](6a07988491/pkg/storage/chunk/cache/cache.go (L110)): - `querier_cache_added_total` - `querier_cache_gets_total` - `querier_cache_misses_total` Updates the namespace and subsystem for the following metrics - `querier_cache_added_new_total` is renamed to `loki_embeddedcache_added_new_total` - `querier_cache_evicted_total` is renamed to `loki_embeddedcache_evicted_total` - `querier_cache_entries` is renamed to `loki_embeddedcache_entries` - `querier_cache_memory_bytes` is renamed to `loki_embeddedcache_memory_bytes` Removes already deprecated metric `querier_cache_stale_gets_total` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Christian Haudum	2e775f3f44	chore(storage): Restructure shipper packages (#10729) ... **What this PR does / why we need it**: This PR restructures the code for the shipper component of the Loki storage layer. New package layout: ```console $ tree -d pkg/storage/stores/shipper pkg/storage/stores/shipper └── indexshipper ├── boltdb │   └── compactor ├── compactor │   ├── client │   │   └── grpc │   ├── deletion │   ├── deletionmode │   ├── generationnumber │   └── retention ├── downloads ├── gatewayclient ├── index ├── indexgateway ├── storage ├── testutil ├── tsdb │   ├── index │   ├── testdata │   └── testutil ├── uploads └── util 23 directories ``` * TSDB and BoltDB specific code is under `./pkg/storage/stores/shipper/indexshipper/tsdb` and `./pkg/storage/stores/shipper/indexshipper/boltdb` respectively. * Common code for both TSDB and BoltDB is directly under `./pkg/storage/stores/shipper/indexshipper` and subdirectories, such as `uploads/`, `downloads/`, `compactor/`, ... **Special notes for your reviewer**: This PR is identical to https://github.com/grafana/loki/pull/10724 except of the package `pkg/storage/stores/indexshipper` the package is `pkg/storage/stores/shipper/indexshipper` (one level deeper). Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	76842262c7	Rename NewShipper to NewIndexClient (#10700) ... The constructor function name is misleading as it suggests that the returned object is a [shipper](https://github.com/grafana/loki/blob/main/pkg/storage/stores/shipper/index/table_manager.go#L48), rather than an index client. Also, change the return type to be a struct pointer, rather than an interface. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Ashwanth	2902f986a4	cache: Enable embedded cache if no other cache is configured. (#10620) ... **What this PR does / why we need it**: FIFO cache is replaced by embedded-cache in https://github.com/grafana/loki/pull/6821 This PR enables embedded cache for query range results, chunks cache, index_stats_results and volume_results if no other cache is explicitly configured (redis, memcache). **Special notes for your reviewer**: Also removes a deprecated flag `cache-split-interval` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Roel Arents	5fb1cdb5dd	azure: add connection-string (#10187) ... **What this PR does / why we need it**: Adds a connection-string option to the azure blob storage config. So the [Azurite](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite) emulator can be used in local test environments (my use case). But also to support the use of a SAS token. See the (future) addition of this option to the Thanos object storage client: https://github.com/thanos-io/objstore/pull/51 **Which issue(s) this PR fixes**: n/a **Special notes for your reviewer**: https://github.com/thanos-io/objstore/pull/51 isn't merged yet. But I'm already creating this PR to see what others think. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Travis Patterson	641c9ee48a	Don't allow unbounded parallelism when downloading indices (#10634) ... Index Gateways can stall when certain queries try to download tons of indices because. `ErrGroup` needs an explicit limit set on it. This PR sets the limit provisionally until a more configurable solution can be introduced. --------- Co-authored-by: Dylan Guedes <dylan.guedes@grafana.com>	3 years ago
Christian Haudum	3a8a5e54dd	Fix object store type for table manager (#10632) ... **What this PR does / why we need it**: This commit fixes a regression in the object store client creation introduced with #10518 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	769979c9f6	chore(storage): Refactor store factory (#10518) ... **What this PR does / why we need it**: This PR makes a clearer distinction between supported and deprecated index/chunk storage types. There are individual commits for the different factory functions: * Refactor table client factory * Refactor index client factory * Refactor chunk client factory * Split initStore function --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago