258 Commits (bbbd82bc73322d662ba81efeda3884efcdc09708)

Author	SHA1	Message	Date
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
renovate[bot]	246a1dfbe2	fix(deps): update module github.com/ncw/swift to v2 (#13951) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
renovate[bot]	d43b2de1b4	fix(deps): update module github.com/fsouza/fake-gcs-server to v1.47.7 (#13935) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
Dylan Guedes	3c9c647922	fix: Read "404" as object not exist (#13901) ... **What this PR does / why we need it**: Modify the s3 client to recognize "NotFound" errors when checking if an object error is "NotFound". I did some tests locally and `Head` calls to objects that doesn't exist gives a "NotFound" instead of "NoSuchKey".	1 year ago
Vladyslav Diachenko	bb257f54b3	feat(loki): add ability to disable AWS S3 dualstack endpoints usage (#13785) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Trevor Whitney	5a87ccb648	feat: instrument failed chunk encoding/decoding (#13684)	2 years ago
Cyril Tovena	d42476aa58	fix: Handle block offset exceeding chunk length in memchunk.go (#13661)	2 years ago
benclive	d9c441efcc	feat: Implement GetObjectRange for all storage providers (#13650)	2 years ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	2 years ago
Christian Haudum	69b805d8ff	test: Fix flaky congestion test (#13410) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
benclive	973aa2d7ba	feat: Use prefixed WAL storage path in Object Storage (#13377)	2 years ago
Vladyslav Diachenko	33bd390f19	chore: adjusted SegmentWriter to the reader conversion (#13342) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: George Robinson <george.robinson@grafana.com> Co-authored-by: Ben Clive <ben.clive@grafana.com>	2 years ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	2 years ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	2 years ago
Dylan Guedes	c1fada9af0	refactor: Remove unnecessary spanlogger usage (#13255) ... Historically, we've been using `spanlogger` to enrich our spans. That isn't the right usage for it (as of now), as they'll be emitting log lines. Our current usage is causing: - Noisy/important log lines less visible - Unnecessary logging volume - Spread of the bad practice. With this PR I'm moving away from `spanlogger` all places where the log message is too bad or nonexistent. That's because these cases are indicating we don't care about that log line at all, and only about the data injected in the span.	2 years ago
Owen Diehl	1086783a1d	fix: separates directory creation from permission checks (#13248)	2 years ago
Kaviraj Kanagaraj	88e545fc95	feat(cache): Add `Cache-Control: no-cache` support for Loki instant queries. (#12896) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
jackyin	a46d14fb05	fix: Optimize regular initialization (#12926)	2 years ago
Paul Rogers	d16a3bf139	test: First pass at unregistering metrics so that we can run multiple tests (#12927)	2 years ago
MarkDaveny	3a46d3717d	style: fix function names (#12817) ... Signed-off-by: MarkDaveny <peicuiping@aliyun.com>	2 years ago
Travis Patterson	a772ed705c	fix: Invalidate caches when pipeline wrappers are disabled (#12903)	2 years ago
Christian Haudum	667076d935	fix(blooms): Do not fail requests when fetching metas from cache fails (#12838) ... The bloom shipper uses metas to resolve available blocks. Metas are fetched from cache, and if not available from object storage. If fetching metas from cache fails, e.g. timeout, the request should not fail, but proceed as if no metas were available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Trevor Whitney	af09f53a33	ci: release workflows get build image from makefile (#12778)	2 years ago
Paul Rogers	d34f1627cc	test: 2 data race fixes (#12584)	2 years ago
Christian Haudum	58c8c61905	perf(blooms): Use jumpmap for client side sharding of the filter requests (#12470) ... **What this PR does / why we need it**: Use the same client-side sharding mechanism for bloom gateway filter requests that we use for client side sharding in the memcached client. This uses DNS discovery to resolve a list of addresses uses as buckets for the jumphash algorithm to shard GroupedChunkRefs across bloom gateway servers. The ring component is still part of the bloom gateway, but it is not used any more. **Special notes for your reviewer**: https://arxiv.org/pdf/1406.2294.pdf Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Ed Welch	3ece2ea6c4	refactor!: remove async cache writeback on chunk fetcher (#12456) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	a509871c55	chore: remove experimental flags for l2 cache and memcached "addresses" config (#12410)	2 years ago
Paul Rogers	aca2a38ad6	test: gcs object client test data race (#12324)	2 years ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Dylan Guedes	51c54adc4f	feat: Memcached: Add TLS support (#12318)	2 years ago
BiKangNing	75e031d8e8	chore: fix function names in doc strings (#12313) ... Signed-off-by: depthlending <bikangning@outlook.com>	2 years ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292) ... A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	0d1a3694f5	test: Don't start server until all configured (#12226)	2 years ago
Owen Diehl	63e0bcb84b	fix(blooms): embeddedcache size panic (#12130)	2 years ago
Paul Rogers	97ae790f36	test: Update chunk/cache/mock to have a GetKeys() function (#12122)	2 years ago
Danny Kopping	b5f488caec	fix(query/query-frontend): correct congestion control stats aggregation (#12072) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Danny Kopping	609fdb3214	feat(querier/query-frontend): track and log congestion control latency (#12058) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Salva Corts	4fa5148eb5	refactor: Pass query plan down to bloom gateway (#12037)	2 years ago
Christian Haudum	b9ce005ec1	fix: Ensure working dir for bloomstore exists (#12019) ... Fail startup if directory does not exist or there are not enough permissions. This prevents the bloomstore to fail later in the process once it tries to download and extract blocks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	9e7725b31b	feat(metadata): introduce a separate split interval for recent query window (#11897)	2 years ago
Christian Haudum	3b122e4be1	Bloom gateway: Wire up cache config with implementation (#11898) ... Until now, the cache configuration in the bloom gateway did not initialise the cache. This PR wires the config with the actual implementation and passes them to the bloom store. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	73edf7a943	(chore) Bloomshipper: Separate store and client (#11865) ... **What this PR does / why we need it**: This PR removes the `StoreAndClient` interface that was accepted by the `BloomShipper`. Since the `BloomStore` had to not only implement the `Store` interface, but also the `Client` interface, it caused re-implementation of the same methods in different ways. Now the shipper solely relies on the `Store` interface. See individual commit messages for more context. Tests have been rewritten from scratch and placed in their own respective test files for store and client. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Michel Hollands	2a63d24729	Fix the ObjectExists method on FSObjectClient (#11747) ... **What this PR does / why we need it**: This method wasn't working. A test was added as well. **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Ashwanth	5c8fd520d3	fix(background cache): increment queue size exactly once (#11776) ... **What this PR does / why we need it**: While looking at the metrics from one of our cells, we noticed that the queue size is set to a very high value even when the length is pretty much 0. Background cache is incrementing the queue size twice for each enqueued key. This pr removes the additional increment call. `TestBackgroundSizeLimit` might be flaky because of the [writeBackLoop](8eb09c78c8/pkg/storage/chunk/cache/background.go (L193)) which dequeues from the channel and reduces the queue size concurrently. To make the test predictable, I have set the `WriteBackGoroutines` to 0. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
Danny Kopping	e7b782b186	Query-frontend: use the same query split interval for generated cache keys (#11679) ... **What this PR does / why we need it**: Follow up from #11535 (specifically from [this thread](https://github.com/grafana/loki/pull/11535#issuecomment-1887151515)), this PR modifies the results cache implementation to use the same interval for generating cache keys.	2 years ago
Danny Kopping	88aaa7dc5e	Cache: correctly check background cache size (#11654) ... **What this PR does / why we need it**: The size check was not being performed atomically, which led to flakiness in the `TestBackgroundSizeLimit` test. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	852becf227	results cache: add new stat `query_length_served` to measure cache effectiveness (#11589) ... **What this PR does / why we need it**: cache hit rate that is currently being measured using metrics and stats does not account for the fact that a cache hit could return partial results. When we query the cache for a key, we get back a list of extents and these need no cover the entire (split) range of the cache key. This pr adds a new cache stat called `query_length_served` to better measure the cache effectiveness. ``` query_length - sum(length of downstream queries) ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
Christian Haudum	62176339c4	Bloom querying: Propagate line filters from querier to bloom filter (#11492) ... In order to query the bloom filters with the search strings from the user query, we need to propagate the line filters through the different implementations of the store, namely through the GetChunks function. In order to avoid converting data structures, the line filter from the syntax tree is used as custom protobuf type, because it is passed from querier to index gateway and from index gateway to bloom gateway. So far, the line filters request have not been tokenized to n-grams. This PR adds the ability to read the n-gram length from the block schema at query time and tokenizes the line filters from the input just ahead of searching the block. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago