607 Commits (8976709f24dc9baa603d030fa23484382f7ba767)

Author	SHA1	Message	Date
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	1 year ago
renovate[bot]	b37eefe24f	fix(deps): update github.com/prometheus/prometheus (main) (#15950) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
honganan	99d9f1cffd	fix: Export `exportTSInSecs` field in TSDB identifier to make sure correct gap calculation (#13116) ... In summary, the Meta(now in loki/pkg/storage/stores/shipper/bloomshipper/client.go) structure contains a Sources []tsdb.SingleTenantTSDBIdentifier field, and within tsdb.SingleTenantTSDBIdentifier, there is an exportTSInSecs field. When Meta is serialized and written to OSS storage, the exportTSInSecs field is being set to its default value, false, which is not the expected behavior. It should retain its actual value; otherwise, this will lead to issues during subsequent gap calculations. Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Matt Veitas	3b8d993500	feat: Skip writeback for chunks fetched by queriers older than a duration (#15605)	1 year ago
Sandeep Sukhani	ff199557d2	fix: do not retain copy of chunk while indexing a new chunk in tsdb while processing delete requests (#15541)	1 year ago
Paul Rogers	f2c2a22bdc	chore: Preparation for incoming static code analysis CI check (#15164) ... Co-authored-by: Danny Cooper <danny.cooper@grafana.com>	1 year ago
Eng Zer Jun	062d1e9ca4	refactor: replace experimental `maps` and `slices` with stdlib (#15051) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	1 year ago
hongmengning	c27d116c6f	chore: fix some function names in comment (#15096) ... Signed-off-by: hongmengning <go@before.tech>	1 year ago
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	cbdd36a412	feat: blockbuilder component (#14621) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	1 year ago
renovate[bot]	33590be53e	fix(deps): update module k8s.io/apimachinery to v0.31.2 (#14959) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
Christian Haudum	ff111dce05	fix: Correctly propagate index stats to metrics.go log line (#14941) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Cyril Tovena	90c5dbf0d3	chore: Revert syntax module (#14831)	1 year ago
Cyril Tovena	28c9645d10	chore: Introduce logql/syntax go module (#14790)	1 year ago
Owen Diehl	a90b74d6b1	refactor(indexshipper): removes unnecessary seeker impl from index storage client (#14802)	1 year ago
Christian Haudum	57f16a0a26	perf(blooms): Replace JSON lib for encoding/decoding metas (#14767) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1c993f9bb0	chore(TSDB): add hooks for in-memory only tsdb creation (#14734) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Owen Diehl	831c0d56b8	feat(TSDB): allow tsdb index creation in memory only (#14732)	1 year ago
Christian Haudum	baaaa83c78	chore: Add chunk count from `GetChunkRef` call to stats context (#14636) ... This PR improves the observability of the GetChunkRef call, especially for when bloom filters are enabled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	51c42e8645	feat: Do not add empty blooms to offsets (#14577)	2 years ago
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132) ... Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Christian Haudum	fdeec618f0	chore: Log errors when processing a download task fails (#14436) ... At the moment, errors are silently ignored when asynchronous blocks downloading fails. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	2 years ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	a584fb713d	chore: sync tables without acquiring read lock the whole time (#14179)	2 years ago
Christian Haudum	cdf084fdae	perf(blooms): Remove compression of `.tar` archived bloom blocks (#14159) ... Decompression is a CPU intensive task, especially un-gzipping. The gain of compressing a tar archive of storage optimized binary blocks is neglectable (question: is it?). In this example, the block of ~170MiB is ~3.3MiB bigger when not compressed, which is a ratio of ~2% ```bash $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar 177048 -rw-rw-r-- 1 christian christian 181293056 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz 173732 -rw-rw-r-- 1 christian christian 177897689 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz $ qalc '(181293056 - 177897689) / 1024/ 1024' ((181293056 − 177897689) / 1024) / 1024 ≈ 3.238074303 $ qalc '181293056 / 177897689' 181293056 / 177897689 ≈ 1.019086066 ``` After some consideration, we decided to store the encoding of the bloom block in the `BlockRef`. This means, that the changes in this PR do not break compatibility with existing blocks compressed with gzip, although new blocks will not be compressed any more. However, the PR adds support for different compression algorithms, such as gzip, snappy, lz4, flate, and zstd. Compression is not configurable yet. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	2 years ago
Sandeep Sukhani	89c4282654	chore: avoid some allocations in tsdb while reading posting offset table (#14111)	2 years ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	2 years ago
Sandeep Sukhani	4e4174400f	fix: do not retain span logger created with index set initialized at query time (#14027)	2 years ago
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Cyril Tovena	917053a730	feat: Introduce wal segment read path. (#13695) ... Co-authored-by: Ben Clive <ben.clive@grafana.com>	2 years ago
洪阿南	bd20171975	fix(blooms): Improve error wrap to make ignoreNotFound work when fetching blocks (#13656) ... Fix an issue where the `options.ignoreNotFound` of the fetcher is not effective. The root cause of this issue is that the error returned by `fmt.Errorf()` in `BloomClient` loses the underlying cause. As a result, `S3ObjectClient.IsObjectNotFoundErr()` fails to determine if the underlying cause of the returned error is of type `awserr.Error`.	2 years ago
Christian Haudum	2624a4bdd4	fix(blooms): Use correct key to populate blockscache at startup (#13624) ... The cache key for block directories in the blocks cache are the block's address without the directory prefix. This is how the directory is put to the LRU cache after downloading and extracting the block tarball. This PR fixes the incorrect cache key used to populate the cache from disk on startup, which contained the file system directory prefix. Since the cached item from startup is never used, it gets evicted first in case of a full cache, or due to its TTL. This causes also the underlying directory on disk to be deleted, which can however still be referenced from the correct cache key for that directory from a later download. That caused the error `getting index reader: opening series file: open /path/to/block/series: no such file or directory` when trying to query the block, because the correct cache key was still present. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	c9950e394d	fix(blooms): Fix eviction of multiple blockcache items (#13573) ... This commit fixes a bug in the eviction of least recently used items. When an element from the doubly-linked-list of the cache is removed, then the element's prev and next items become nil. Therefore the call elem.Prev() always broke the loop's condition for checking entries to be evicted. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	e0ca67dd45	fix(query engine): Include lines with ts equal to end timestamp of the query range when executing range aggregations (#13448) ... **Background** When performing range vector aggregations, such as `count_over_time({env="dev"}[1h])`, the query range is divided into multiple steps at which the aggregation operation (e.g. counting the log lines) is evaluated. Each step starts at `current step - step interval` and ends at `current step`, as depicted in the following chart. The select range for the logs is extended by the `step interval` into the past, in order to select logs for calculating the first step.  However, the select range for logs is `start` inclusive and `end` exclusive (written as `[start, end)`), but the evaluation of the steps for the range aggregation is `start` exclusive and `end` inclusive (written as `(start, end]`). This leads to the problem that the very first timestamp at the beginning of the select range and the very last timestamp at the end of the select range are not included in the range aggregation. The "missing" last timestamp is not a problem, because a) in an instant query it is not supposed to be included anyway because of the `[start, end)` inclusivity of the query range and b) in a range query the last point of the previous step will be part of the next step evaluation. **Issue** The missing first timestamp, however, gets problematic when executing an instant query and the log timestamps are exactly at the start of the query range. This can happen when the query is split in the query frontend into multiple smaller time ranges, e.g. `1h`, `30m`, ... Since the sub queries are executed independently on the queriers, all logs that have a timestamp exactly a multiple of the split interval, e.g. 00:00, 01:00, 02:00, ... for a 1h interval, are dismissed and therefore missing in the query result over the full time range of the original query. **Fix** In order to avoid the missing logs that have a timestamp a multiple of the split interval in instant queries, we need to adjust the query range for logs to also include the `end` timestamp (written as `[start, end]`). This is done by adding a "leap nanosecond" to the `end` timestamp of the log select range. This ensures that the included `end` timestamp of the step evaluation is also included in the log selection. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	2 years ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	2 years ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	2 years ago
Salva Corts	0c289a83df	refactor(blooms): Do not use sleeps on integration test (#13308)	2 years ago
Owen Diehl	0cb3ff1830	perf(blooms): always return bloom pages to allocator (#13288)	2 years ago
Dylan Guedes	c1fada9af0	refactor: Remove unnecessary spanlogger usage (#13255) ... Historically, we've been using `spanlogger` to enrich our spans. That isn't the right usage for it (as of now), as they'll be emitting log lines. Our current usage is causing: - Noisy/important log lines less visible - Unnecessary logging volume - Spread of the bad practice. With this PR I'm moving away from `spanlogger` all places where the log message is too bad or nonexistent. That's because these cases are indicating we don't care about that log line at all, and only about the data injected in the span.	2 years ago
Christian Haudum	dbc0e2489c	chore(index-gateway): Improve instrumentation of index download/sync (#13194) ... This PR contains changes to improve the instrumentation and obervability of the download/sync operations that happen on the index gateways (index shipper). This includes fixes to the download latency and wait time metrics. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ed Welch	28642b6d32	chore(index-gateway): Download then extract in separate steps (#13202) ... This provides additional timing information about what step takes how long. Additionally, the `awaitReady()` function is gated with a timeout context to avoid overly long waiting times. Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Cyril Tovena	1d6f8d51fc	feat: Introduce a new Object Storage WAL format. (#13253)	2 years ago
Owen Diehl	1086783a1d	fix: separates directory creation from permission checks (#13248)	2 years ago
Karsten Jeschkies	ca917b3ec0	chore: Update go.etcd.io/bbolt to v1.3.8 (#13227)	2 years ago
洪阿南	f98ff7f584	fix: Fix duplicate enqueue item problem in bloom download queue when do sync download (#13114) ... A missing return will cause item enqueued twice in sync download scenario. That cause the response not like expected. Suppose we want to fetch a blocks array: [0, 1, 2]. Enqueued items are [0, 0, 1, 1, 2, 2]，If we want response sorted we wait 3 times response will get [0, 0, 1]	2 years ago