apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438 ) Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	11 months ago
Matt Veitas	3b8d993500	feat: Skip writeback for chunks fetched by queriers older than a duration (#15605 )	1 year ago
Paul Rogers	f2c2a22bdc	chore: Preparation for incoming static code analysis CI check (#15164 ) Co-authored-by: Danny Cooper <danny.cooper@grafana.com>	1 year ago
Eng Zer Jun	062d1e9ca4	refactor: replace experimental `maps` and `slices` with stdlib (#15051 ) Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	1 year ago
renovate[bot]	9decb65714	fix(deps): update module github.com/sony/gobreaker to v2 (#15036 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
renovate[bot]	33a214880b	fix(deps): update module github.com/go-redis/redis/v8 to v9 (#14978 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
Robert Fratto	bf54cf1496	chore(bloom): remove unused code from blooms (#14539 )	1 year ago
Christian Haudum	f568dda136	chore: Improve logging of jumphash server selector (#14306 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257 ) This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167 ) Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	1 year ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071 )	1 year ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084 )	2 years ago
Dylan Guedes	c1fada9af0	refactor: Remove unnecessary spanlogger usage (#13255 ) Historically, we've been using `spanlogger` to enrich our spans. That isn't the right usage for it (as of now), as they'll be emitting log lines. Our current usage is causing: - Noisy/important log lines less visible - Unnecessary logging volume - Spread of the bad practice. With this PR I'm moving away from `spanlogger` all places where the log message is too bad or nonexistent. That's because these cases are indicating we don't care about that log line at all, and only about the data injected in the span.	2 years ago
Kaviraj Kanagaraj	88e545fc95	feat(cache): Add `Cache-Control: no-cache` support for Loki instant queries. (#12896 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
jackyin	a46d14fb05	fix: Optimize regular initialization (#12926 )	2 years ago
Travis Patterson	a772ed705c	fix: Invalidate caches when pipeline wrappers are disabled (#12903 )	2 years ago
Christian Haudum	667076d935	fix(blooms): Do not fail requests when fetching metas from cache fails (#12838 ) The bloom shipper uses metas to resolve available blocks. Metas are fetched from cache, and if not available from object storage. If fetching metas from cache fails, e.g. timeout, the request should not fail, but proceed as if no metas were available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	d34f1627cc	test: 2 data race fixes (#12584 )	2 years ago
Christian Haudum	58c8c61905	perf(blooms): Use jumpmap for client side sharding of the filter requests (#12470 ) What this PR does / why we need it: Use the same client-side sharding mechanism for bloom gateway filter requests that we use for client side sharding in the memcached client. This uses DNS discovery to resolve a list of addresses uses as buckets for the jumphash algorithm to shard GroupedChunkRefs across bloom gateway servers. The ring component is still part of the bloom gateway, but it is not used any more. Special notes for your reviewer: https://arxiv.org/pdf/1406.2294.pdf Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Ed Welch	3ece2ea6c4	refactor!: remove async cache writeback on chunk fetcher (#12456 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	a509871c55	chore: remove experimental flags for l2 cache and memcached "addresses" config (#12410 )	2 years ago
Dylan Guedes	51c54adc4f	feat: Memcached: Add TLS support (#12318 )	2 years ago
BiKangNing	75e031d8e8	chore: fix function names in doc strings (#12313 ) Signed-off-by: depthlending <bikangning@outlook.com>	2 years ago
Owen Diehl	63e0bcb84b	fix(blooms): embeddedcache size panic (#12130 )	2 years ago
Paul Rogers	97ae790f36	test: Update chunk/cache/mock to have a GetKeys() function (#12122 )	2 years ago
Ashwanth	9e7725b31b	feat(metadata): introduce a separate split interval for recent query window (#11897 )	2 years ago
Christian Haudum	3b122e4be1	Bloom gateway: Wire up cache config with implementation (#11898 ) Until now, the cache configuration in the bloom gateway did not initialise the cache. This PR wires the config with the actual implementation and passes them to the bloom store. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	5c8fd520d3	fix(background cache): increment queue size exactly once (#11776 ) What this PR does / why we need it: While looking at the metrics from one of our cells, we noticed that the queue size is set to a very high value even when the length is pretty much 0. Background cache is incrementing the queue size twice for each enqueued key. This pr removes the additional increment call. `TestBackgroundSizeLimit` might be flaky because of the [writeBackLoop](`8eb09c78c8/pkg/storage/chunk/cache/background.go (L193)`) which dequeues from the channel and reduces the queue size concurrently. To make the test predictable, I have set the `WriteBackGoroutines` to 0. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Danny Kopping	e7b782b186	Query-frontend: use the same query split interval for generated cache keys (#11679 ) What this PR does / why we need it: Follow up from #11535 (specifically from [this thread](https://github.com/grafana/loki/pull/11535#issuecomment-1887151515)), this PR modifies the results cache implementation to use the same interval for generating cache keys.	2 years ago
Danny Kopping	88aaa7dc5e	Cache: correctly check background cache size (#11654 ) What this PR does / why we need it: The size check was not being performed atomically, which led to flakiness in the `TestBackgroundSizeLimit` test. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	852becf227	results cache: add new stat `query_length_served` to measure cache effectiveness (#11589 ) What this PR does / why we need it: cache hit rate that is currently being measured using metrics and stats does not account for the fact that a cache hit could return partial results. When we query the cache for a key, we get back a list of extents and these need no cover the entire (split) range of the cache key. This pr adds a new cache stat called `query_length_served` to better measure the cache effectiveness. ``` query_length - sum(length of downstream queries) ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Matthew Penner	a91f3f11b3	frontend: Use `net.JoinHostPort` to support IPv6 addresses (#10650 ) Signed-off-by: Matthew Penner <me@matthewp.io> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Kaviraj Kanagaraj	ed1fccfa12	Revert "bugfix(memcached): Make `memcached` batch fetch truely context aware" (#11479 ) Reverts grafana/loki#11363 This PR was causing some issue. Spend some time testing on some internal cell. Still couldn't track down the issue yet. Reverting to make `main` stable for now.	2 years ago
Vladyslav Diachenko	9b691904ef	BloomShipper: add cache for downloaded blocks (#11394 ) adapted embeddedcache.go to store downloaded bloom blocks on local filesystem. What this PR does / why we need it: This cache will be used by bloom-gateway to not download the blocks each time. In the cache we store the objects(`cachedBlock` struct) that contain: 1. `blockDirectory`: the `path` to the directory where the block was extracted 2. `activeQueriers`: thread-safe counter of active block users. This field is important because we do not want the directory to be removed while it's in use. 3. rest fields are needed for service needs When the downloadingWorker receives this entity, it increases the counter `activeQueriers` and creates blockQuerier wrapper that has `Close` function that will decrease the counter once blockQuerier is not needed anymore. When the cache entry is being removed from the cache, the cache calls the function `removeDirectoryAsync` which asynchronously removes the block's folder. This function checks every `Xms` if there are still active block queriers and once `activeQueriers` count is `0` the folder will be removed. Also, there is a timeout in this function, and once the timeout is reached, the folder will be force removed. Special notes for your reviewer: * If the cache is disabled, then the blocks will be downloaded each time the block is requested. * If the cache is used, the folder will be deleted by embeddedCache when it reaches memory size limit or items count limit. Otherwise, the block's folder will be deleted when `Close` function is called. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Kaviraj Kanagaraj	3555001aee	bugfix(memcached): Make `memcached` batch fetch truely context aware (#11363 ) What this PR does / why we need it: Currently, `Fetch` method of memcached is not truely respecting the `ctx` passed to it. It get lost in `fetchKeysBatched` method https://github.com/grafana/loki/blob/489ac8d529ec39352/pkg/storage/chunk/cache/memcached.go#L171-L226 So even if client cancels the context, memcached would still be fetching keys. This PR make it respect context cancelation. Changes: 1. Add select case to check for `ctx.Done()` before submitting work to `inputCh`. 2. Given, now we have two exit point to stop the mecached actor one via old `Stop()` method and other via context cancel, I Added aditional flag `stopped` to make sure we are not closing channels that are already closed. 3. Updated tests to lock this behaviour. Which issue(s) this PR fixes: Fixes NA Special notes for your reviewer: This bug played a role in some incident recently on our Loki SaaS. cc @slim-bean Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Salva Corts	a0b462d366	Bloom-Gateway cache (#11380 ) What this PR does / why we need it: This PR adds caching to the bloom-gateway client. It uses the result cache from https://github.com/grafana/loki/pull/11343. Here's how we: - Merge responses: group all chunks by FP and remove duplicated chunk checksums. - Extract responses based on time span: For all chunks in each FP, add to the extracted response only the chunks that overlaps with the desired start and end time.	2 years ago
Salva Corts	489ac8d529	Extract results cache into new pkg (#11343 ) What this PR does / why we need it: This extracts the results cache from `queryrangebase` into its own pkg so we can reuse it in other components such as the bloom-gateway without having to import `queryrangebase`. - Most of the logic inside `pkg/querier/queryrange/queryrangebase/results_cache.go` now lives in `pkg/storage/chunk/cache/results_cache/cache.go`. - Some of the tests in `pkg/querier/queryrange/queryrangebase/results_cache.go` are moved into pkg/storage/chunk/cache/results_cache/cache_test.go. - Note that here we don't have access to the types we use in `queryrangebase` so we created a new set of mock request/response types to test with.	2 years ago
Vladyslav Diachenko	45b10effac	embedded cache refactoring (#10989 ) What this PR does / why we need it: refactored embedded cache to allow using it for any types, not only for `string -> []byte`. Also, the new implementation allows passing the callback that will be called for the removed entry. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025 ) What this PR does / why we need it: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Ashwanth	8bb615c2cc	config: loki better defaults (#10793 ) What this PR does / why we need it: Updates some of the configuration defaults to provide a better experience for users out of the box. I tried to add notes for non-trivial changes explaining the reason. \| configuration \| new default \| old default \| notes \| \| ------------------------------------------------------ \| ----------- \| ----------- \| -------- \| `compactor.delete-max-interval` \| 24h \| 0 \| splits the delete requests into intervals no longer than `delete_max_interval` \| \| `distributor.max-line-size` \| 256KB \| 0 \| - \| \| `ingester.sync-period` \| 1h \| 0 \| ensures that the chunk cuts for a given stream are synchronized across the ingesters in the replication set. Helps with deduplicating chunks. \| \| `ingester.sync-min-utilization` \| 0.1 \| 0 \| ^ \| \| `frontend.max-querier-bytes-read` \| 150GB \| 0 \| - \| \| `frontend.max-cache-freshness` \| 10m \| 1m \| avoid caching results for data that is still in churn \| \| `frontend.max-stats-cache-freshness` \| 10m \| 0 \| avoid caching results for data that is still in churn \| \| `memcached.batchsize` \| 256 \| 1024 \| - \| \| `memcached.parallelism` \| 10 \| 100 \| - \| \| `querier.tsdb-max-query-parallelism` \| 128 \| 512 \| avoid over parallelising queries for smaller installations \| \| `querier.split-queries-by-interval` \| 1h \| 30m \| avoid over parallelising queries for smaller installations \| \| `querier.compress-http-responses` \| true \| false \| compress response if the request accepts gzip encoding \| \| `query-scheduler.max-outstanding-requests-per-tenant` \| 32000 \| 100 \| current default is too small, opening a big dashboard could hit this limit \| \| `validation.max-label-names-per-series` \| 15 \| 30 \| reducing this to avoid blowing up the series count and index size. Also enforces users to think about what labels are being added. \| Special notes for your reviewer: regarding `TestQueryTSDB_WithCachedPostings`: removed the assert on get calls as this is hard to get right (splits, querySize mw making stats calls) validating additions to the cache and misses is more important here I think. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Ashwanth	fcdd55b6c9	embedded cache: update metric namespace and remove duplicate metrics (#10693 ) What this PR does / why we need it: Removes the following embedded cache metrics since `loki_cache_request_duration_seconds` (which instruments requests made to all the caches) [already covers them](`6a07988491/pkg/storage/chunk/cache/cache.go (L110)`): - `querier_cache_added_total` - `querier_cache_gets_total` - `querier_cache_misses_total` Updates the namespace and subsystem for the following metrics - `querier_cache_added_new_total` is renamed to `loki_embeddedcache_added_new_total` - `querier_cache_evicted_total` is renamed to `loki_embeddedcache_evicted_total` - `querier_cache_entries` is renamed to `loki_embeddedcache_entries` - `querier_cache_memory_bytes` is renamed to `loki_embeddedcache_memory_bytes` Removes already deprecated metric `querier_cache_stale_gets_total` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Ashwanth	2902f986a4	cache: Enable embedded cache if no other cache is configured. (#10620 ) What this PR does / why we need it: FIFO cache is replaced by embedded-cache in https://github.com/grafana/loki/pull/6821 This PR enables embedded cache for query range results, chunks cache, index_stats_results and volume_results if no other cache is explicitly configured (redis, memcache). Special notes for your reviewer: Also removes a deprecated flag `cache-split-interval` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Vladyslav Diachenko	00a3c5b026	renamed `non indexed labels` feature to `structured metadata` (#10432 ) Loki team has made a decision to rename the feature because the new name is more precise. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Christian Haudum	31981bd4af	Remove dependency on Bigchunk, which is a leftover of Cortex and is only used in tests (#10354 ) What this PR does / why we need it: This PR removes the dependency on "Bigchunk", which is a "default" implementation of the `chunk.Data` interface and used only in tests. Special notes for your reviewer: The PR replaces the usage of this chunk type with the Loki implementation (`chunkenc.MemChunk` wrapped into `chunkenc.Facade`) where possible. In case where it was not possible (tests in `pkg/storage/chunk` package) due to otherwise circular import, it was replaces by the `chunk.dummyChunk` implementation, which is just a no-op that implements `chunk.Data` interface) The PR also removes the protos for the ingester client (`pkg/ingester/client/ingester.proto`) which were only referenced in the removed `pkg/util/chunkcompat/compat.go` file. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ed Welch	8ca1d1e0a4	Loki: simplify the FetchChunks method and optimize the experimental l2 cache a little more (#10160 ) What this PR does / why we need it: This started as a simple PR to optimize the new L2 chunks cache code such that we don't look up chunks in L1 that we know won't be there, however, the complexity of the FetchChunks method taking both a slice of requested Chunks as well as a slice of requested Keys made this really challenging. Looking at how we use this method, I saw no reason for there to be a requirement to pass both a list of chunks and keys, we only ever want the Chunks and everywhere we used this we were generating the list of keys external to this function from the list of chunks being passed in. The second change was to the implementation of how we process the cache results to look for found vs missing keys. The current implementation required the chunks to be sorted by external key so that it could iterate through them, I tested out a different version of this which uses a map instead of sorting and iterating. The map performance was slightly better in terms of reduced allocations, not enough to make a meaningful difference, but I think this code is easier to understand and reason about, and it removes any requirement for sorting of the passed in Chunks by External key. Benchmarks: Previous: ``` 100 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3445 334642 ns/op 478074 B/op 7536 allocs/op BenchmarkFetch-32 2947 344439 ns/op 478534 B/op 7537 allocs/op BenchmarkFetch-32 3489 339239 ns/op 478092 B/op 7536 allocs/op BenchmarkFetch-32 3523 340550 ns/op 478003 B/op 7536 allocs/op BenchmarkFetch-32 2935 346810 ns/op 478590 B/op 7537 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.948s 1000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 363 3225519 ns/op 5305481 B/op 78133 allocs/op BenchmarkFetch-32 358 3308456 ns/op 5308299 B/op 78143 allocs/op BenchmarkFetch-32 336 3379067 ns/op 5322534 B/op 78191 allocs/op BenchmarkFetch-32 357 3330118 ns/op 5308854 B/op 78145 allocs/op BenchmarkFetch-32 349 3390415 ns/op 5313813 B/op 78162 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 25.607s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3 334471200 ns/op 639282104 B/op 2630192 allocs/op BenchmarkFetch-32 2 503434741 ns/op 922833480 B/op 3412973 allocs/op BenchmarkFetch-32 1 1064668769 ns/op 1771502096 B/op 5741173 allocs/op BenchmarkFetch-32 1 1027755910 ns/op 1771419128 B/op 5749049 allocs/op BenchmarkFetch-32 1 1027270578 ns/op 1774711872 B/op 5752726 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.251s ``` Now ``` 100 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3856 300569 ns/op 462932 B/op 6214 allocs/op BenchmarkFetch-32 4016 305387 ns/op 462803 B/op 6214 allocs/op BenchmarkFetch-32 3382 303078 ns/op 463201 B/op 6215 allocs/op BenchmarkFetch-32 3319 303367 ns/op 463308 B/op 6215 allocs/op BenchmarkFetch-32 3289 356986 ns/op 463341 B/op 6215 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 22.879s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 393 2641470 ns/op 5193282 B/op 65956 allocs/op BenchmarkFetch-32 433 2684287 ns/op 5174832 B/op 65894 allocs/op BenchmarkFetch-32 427 2778092 ns/op 5177334 B/op 65902 allocs/op BenchmarkFetch-32 418 2714370 ns/op 5181193 B/op 65915 allocs/op BenchmarkFetch-32 415 2822163 ns/op 5183436 B/op 65920 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 25.053s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3 362233430 ns/op 638097202 B/op 2510070 allocs/op BenchmarkFetch-32 2 611639304 ns/op 921128060 B/op 3292828 allocs/op BenchmarkFetch-32 1 1022180867 ns/op 1770465496 B/op 5621431 allocs/op BenchmarkFetch-32 1 1253448115 ns/op 1771278456 B/op 5630224 allocs/op BenchmarkFetch-32 1 1189564672 ns/op 1774053064 B/op 5628113 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.960s ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Ed Welch	e97e53f1b3	Loki: add an expirmental l2 chunks cache (#10061 ) What this PR does / why we need it: As we play around with our caching strategies, there has surfaced a potential benefit to having 2 layers of chunks cache with a `duration from now` handoff. i.e. chunks older than _x_ days get sent to an L2 chunks cache. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Sandeep Sukhani	3e1f2fc273	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757 ) What this PR does / why we need it: Currently, when we find a relevant cached negative response for a logs query, we do the following: * If the cached query completely covers the new query: * return back an empty response. * else: * fill the gaps on either/both sides of the cached query. The problem with filling the gaps is that when the cached query does not overlap at all with the new query, we have to extend the query beyond what the query requests for. However, with the logs query, we have a limit on the number of lines we can send back in the response. So, this could result in the query response having logs which were not requested by the query, which then get filtered out by the [response extractor](`b78d3f0552/pkg/querier/queryrange/log_result_cache.go (L299)`), unexpectedly resulting in an empty response. For example, if the query was cached for start=15, end=20 and we get a `backwards` query for start=5, end=10. To fill the gap, the query would be executed for start=5, end=15. Now, if we have logs more than the query `limit` in the range 10-15, we would filter out all the data in the response extractor and send back an empty response to the user. This PR fixes the issue by doing the following changes when handling cache hit: * If the cached query completely covers the new query: * return back an empty response[_existing_]. * else if the cached query does not overlap with the new query: * do the new query as requested. * If the new query results in an empty response and has a higher interval than the cached query: * update the cache * else: * query the data for missing intervals on both/either side[_existing_] * update the cache with extended intervals if the new queries resulted in an empty response[_existing_] Special notes for your reviewer: We could do further improvements in the handling of queries not overlapping with cached query by selectively extending the queries based on query direction and cached query lying before/after the new query. For example, if the new query is doing `backwards` query and the `cachedQuery.End` < `newQuery.Start`, it should be okay to extend the query and do `cachedQuery.End` to `newQuery.End` to fill the cache since query would first fill the most relevant data before hitting the limits. I did not want to complicate the fix so went without implementing this approach. We can revisit later if we feel we need to improve our caching. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago

1 2 3

105 Commits (8976709f24dc9baa603d030fa23484382f7ba767)