599 Commits (8958eb91eda56a5334d0bdb683a9fb592cd5d17e)

Author	SHA1	Message	Date
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Owen Diehl	cbdd36a412	feat: blockbuilder component (#14621) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	6 months ago
renovate[bot]	33590be53e	fix(deps): update module k8s.io/apimachinery to v0.31.2 (#14959) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	7 months ago
Christian Haudum	ff111dce05	fix: Correctly propagate index stats to metrics.go log line (#14941) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Cyril Tovena	90c5dbf0d3	chore: Revert syntax module (#14831)	7 months ago
Cyril Tovena	28c9645d10	chore: Introduce logql/syntax go module (#14790)	7 months ago
Owen Diehl	a90b74d6b1	refactor(indexshipper): removes unnecessary seeker impl from index storage client (#14802)	7 months ago
Christian Haudum	57f16a0a26	perf(blooms): Replace JSON lib for encoding/decoding metas (#14767) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Owen Diehl	1c993f9bb0	chore(TSDB): add hooks for in-memory only tsdb creation (#14734) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	7 months ago
Owen Diehl	831c0d56b8	feat(TSDB): allow tsdb index creation in memory only (#14732)	7 months ago
Christian Haudum	baaaa83c78	chore: Add chunk count from `GetChunkRef` call to stats context (#14636) ... This PR improves the observability of the GetChunkRef call, especially for when bloom filters are enabled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Salva Corts	51c42e8645	feat: Do not add empty blooms to offsets (#14577)	7 months ago
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132) ... Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Christian Haudum	fdeec618f0	chore: Log errors when processing a download task fails (#14436) ... At the moment, errors are silently ignored when asynchronous blocks downloading fails. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	8 months ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Ashwanth	a584fb713d	chore: sync tables without acquiring read lock the whole time (#14179)	8 months ago
Christian Haudum	cdf084fdae	perf(blooms): Remove compression of `.tar` archived bloom blocks (#14159) ... Decompression is a CPU intensive task, especially un-gzipping. The gain of compressing a tar archive of storage optimized binary blocks is neglectable (question: is it?). In this example, the block of ~170MiB is ~3.3MiB bigger when not compressed, which is a ratio of ~2% ```bash $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar 177048 -rw-rw-r-- 1 christian christian 181293056 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz 173732 -rw-rw-r-- 1 christian christian 177897689 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz $ qalc '(181293056 - 177897689) / 1024/ 1024' ((181293056 − 177897689) / 1024) / 1024 ≈ 3.238074303 $ qalc '181293056 / 177897689' 181293056 / 177897689 ≈ 1.019086066 ``` After some consideration, we decided to store the encoding of the bloom block in the `BlockRef`. This means, that the changes in this PR do not break compatibility with existing blocks compressed with gzip, although new blocks will not be compressed any more. However, the PR adds support for different compression algorithms, such as gzip, snappy, lz4, flate, and zstd. Compression is not configurable yet. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Sandeep Sukhani	89c4282654	chore: avoid some allocations in tsdb while reading posting offset table (#14111)	9 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Sandeep Sukhani	4e4174400f	fix: do not retain span logger created with index set initialized at query time (#14027)	9 months ago
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Cyril Tovena	917053a730	feat: Introduce wal segment read path. (#13695) ... Co-authored-by: Ben Clive <ben.clive@grafana.com>	10 months ago
洪阿南	bd20171975	fix(blooms): Improve error wrap to make ignoreNotFound work when fetching blocks (#13656) ... Fix an issue where the `options.ignoreNotFound` of the fetcher is not effective. The root cause of this issue is that the error returned by `fmt.Errorf()` in `BloomClient` loses the underlying cause. As a result, `S3ObjectClient.IsObjectNotFoundErr()` fails to determine if the underlying cause of the returned error is of type `awserr.Error`.	10 months ago
Christian Haudum	2624a4bdd4	fix(blooms): Use correct key to populate blockscache at startup (#13624) ... The cache key for block directories in the blocks cache are the block's address without the directory prefix. This is how the directory is put to the LRU cache after downloading and extracting the block tarball. This PR fixes the incorrect cache key used to populate the cache from disk on startup, which contained the file system directory prefix. Since the cached item from startup is never used, it gets evicted first in case of a full cache, or due to its TTL. This causes also the underlying directory on disk to be deleted, which can however still be referenced from the correct cache key for that directory from a later download. That caused the error `getting index reader: opening series file: open /path/to/block/series: no such file or directory` when trying to query the block, because the correct cache key was still present. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Christian Haudum	c9950e394d	fix(blooms): Fix eviction of multiple blockcache items (#13573) ... This commit fixes a bug in the eviction of least recently used items. When an element from the doubly-linked-list of the cache is removed, then the element's prev and next items become nil. Therefore the call elem.Prev() always broke the loop's condition for checking entries to be evicted. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	e0ca67dd45	fix(query engine): Include lines with ts equal to end timestamp of the query range when executing range aggregations (#13448) ... **Background** When performing range vector aggregations, such as `count_over_time({env="dev"}[1h])`, the query range is divided into multiple steps at which the aggregation operation (e.g. counting the log lines) is evaluated. Each step starts at `current step - step interval` and ends at `current step`, as depicted in the following chart. The select range for the logs is extended by the `step interval` into the past, in order to select logs for calculating the first step.  However, the select range for logs is `start` inclusive and `end` exclusive (written as `[start, end)`), but the evaluation of the steps for the range aggregation is `start` exclusive and `end` inclusive (written as `(start, end]`). This leads to the problem that the very first timestamp at the beginning of the select range and the very last timestamp at the end of the select range are not included in the range aggregation. The "missing" last timestamp is not a problem, because a) in an instant query it is not supposed to be included anyway because of the `[start, end)` inclusivity of the query range and b) in a range query the last point of the previous step will be part of the next step evaluation. **Issue** The missing first timestamp, however, gets problematic when executing an instant query and the log timestamps are exactly at the start of the query range. This can happen when the query is split in the query frontend into multiple smaller time ranges, e.g. `1h`, `30m`, ... Since the sub queries are executed independently on the queriers, all logs that have a timestamp exactly a multiple of the split interval, e.g. 00:00, 01:00, 02:00, ... for a 1h interval, are dismissed and therefore missing in the query result over the full time range of the original query. **Fix** In order to avoid the missing logs that have a timestamp a multiple of the split interval in instant queries, we need to adjust the query range for logs to also include the `end` timestamp (written as `[start, end]`). This is done by adding a "leap nanosecond" to the `end` timestamp of the log select range. This ensures that the included `end` timestamp of the step evaluation is also included in the log selection. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	11 months ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	11 months ago
Salva Corts	0c289a83df	refactor(blooms): Do not use sleeps on integration test (#13308)	11 months ago
Owen Diehl	0cb3ff1830	perf(blooms): always return bloom pages to allocator (#13288)	11 months ago
Dylan Guedes	c1fada9af0	refactor: Remove unnecessary spanlogger usage (#13255) ... Historically, we've been using `spanlogger` to enrich our spans. That isn't the right usage for it (as of now), as they'll be emitting log lines. Our current usage is causing: - Noisy/important log lines less visible - Unnecessary logging volume - Spread of the bad practice. With this PR I'm moving away from `spanlogger` all places where the log message is too bad or nonexistent. That's because these cases are indicating we don't care about that log line at all, and only about the data injected in the span.	11 months ago
Christian Haudum	dbc0e2489c	chore(index-gateway): Improve instrumentation of index download/sync (#13194) ... This PR contains changes to improve the instrumentation and obervability of the download/sync operations that happen on the index gateways (index shipper). This includes fixes to the download latency and wait time metrics. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Ed Welch	28642b6d32	chore(index-gateway): Download then extract in separate steps (#13202) ... This provides additional timing information about what step takes how long. Additionally, the `awaitReady()` function is gated with a timeout context to avoid overly long waiting times. Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	1d6f8d51fc	feat: Introduce a new Object Storage WAL format. (#13253)	12 months ago
Owen Diehl	1086783a1d	fix: separates directory creation from permission checks (#13248)	12 months ago
Karsten Jeschkies	ca917b3ec0	chore: Update go.etcd.io/bbolt to v1.3.8 (#13227)	12 months ago
洪阿南	f98ff7f584	fix: Fix duplicate enqueue item problem in bloom download queue when do sync download (#13114) ... A missing return will cause item enqueued twice in sync download scenario. That cause the response not like expected. Suppose we want to fetch a blocks array: [0, 1, 2]. Enqueued items are [0, 0, 1, 1, 2, 2]，If we want response sorted we wait 3 times response will get [0, 0, 1]	12 months ago
Salva Corts	9c96d26895	refactor(blooms): Delete outdated metas (#13153)	12 months ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093)	12 months ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162)	12 months ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039) ... This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Salva Corts	6e119aaef5	refactor(blooms): Forward created metas from builder to planner (#13133)	12 months ago
Owen Diehl	ad279e518c	fix(blooms): bloomshipper no longer returns empty metas on fetch (#13130) ... Traced some bloom compactor problems to this store interface returning empty `Metas` (with the exception of the reference key) on fetch when the object does not exist in storage. I originally found this due to a panic in the outdated meta removal code because empty `Metas` don't have any source TSDBs. Fixes the impl to only return metas which existed which have their contents populated.	12 months ago
Christian Haudum	d9cc513fd2	fix: Do not filter out chunks for store when `From==Through` and `From==start` (#13117) ... This PR fixes a bug where chunks are incorrectly filtered out when their `From` timestamp is equal to their `Through` timestamp (which happens when there is a single log line) and the `From` timestamp is equal to the `from` time of the of the request. How to reproduce: 1. Insert a single log line with a timestamp exactly at point of an hour 2. Flush ingester 3. Query log line with a split interval of 1h Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ed Welch <edward.welch@grafana.com>	1 year ago
Yuri Kotov	808425953f	feat: API: Expose optional label matcher for label names API (#11982) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	1 year ago