apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Salva Corts	0ee2a6126a	fix(blooms): Fix findGaps when ownership goes to MaxUInt64 and that is covered by existing meta (#12558 )	2 years ago
Ed Welch	670041368a	chore: add more validation at startup to make sure we have a proper schema, and necessary directories are created (#12496 ) Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Christian Haudum	a9345d0fb6	chore(blooms): Clean up settings and make reasonable defaults (#12483 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Christian Haudum	71602eb10f	chore(blooms): Clean up bloom component configuration (#12387 ) * Remove unused setting and move download parallelism configuration one level up. * Update description of bloom_shipper config block Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	86c768c3cc	feat: Blooms retention (#12258 )	2 years ago
Christian Haudum	19c046f14b	chore(blooms): Make max bloom page size for querying configurable (#12337 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Christian Haudum	eb03974086	feat(blooms): Make bloom block compression configurable (#12293 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	63c88489d0	refactor: Add RF and Tokens to Loki ring Cfg and allow overwriting docs. (#12142 ) Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Salva Corts	4b28f8221b	fix: Read all series in NewTSDBSeriesIter and close index file (#12211 )	2 years ago
Owen Diehl	8a8756aa7a	chore(blooms): honors bloom_compactor_enabled for tenants (#12190 )	2 years ago
Christian Haudum	f6cfff5356	refactor(bloomstore): Introduce fetch option for blocks fetcher (#12160 ) Requests to the bloom gateway should not wait for blocks to be downloaded into cache first, but should operate on readily available blocks. This PR introduces fetch options for the bloom store that specify how the store should behave when requesting blocks from block refs. `WithIgnoreMissing`: Ignore errors from blocks that could not be found in object storage, and instead return a `nil` value in the response. `WithFetchAsync`: Return only the blocks that are available locally in cache and dispatch downloading of blocks to the blocks downloader, that operates in the background, and make them available for subsequent requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	49f8f7b3ee	refactor: Chunkref as alias to logproto.ShortRef (#12159 )	2 years ago
Owen Diehl	30a240b470	fix(blooms): filter out nil blocks (#12150 )	2 years ago
Owen Diehl	102e3ea07f	chore(blooms): use the right iter reference in blockLoadingIter.loadNext (#12146 )	2 years ago
Salva Corts	f7710ad7aa	feat: (blooms) Configurable number of tokens (#12125 )	2 years ago
Owen Diehl	17988d5965	fix(blooms): measurable compaction progress (#12101 )	2 years ago
Karsten Jeschkies	6578a00481	feat: Support usage trackers for received and discarded bytes. (#11840 )	2 years ago
Christian Haudum	2f54f8d5be	chore(blooms): Implement BloomStore as a service (#12044 ) The bloomstore service is instantiated once at startup and used as dependency for both the bloom compactor and the bloom gateway. This will prevent instantiating the store in different ways. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	1f56da206d	refactor: Simplify fingerprint partitioning in bloom gateway client (#12018 ) Use the recently introduced `tokenRangesForInstance()` function to determine the the ranges for each replication set. For this purpose, the function moves into the `bloomutils` packages. This PR eliminates the structs `addrsWithBounds` and `instanceWithFingerprints` (which had somewhat similar usage) and unifies them into `rsWithRanges` that holds both the fingerprint ranges and the assigned GroupedChunkRefs. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	e835b6192f	refactor(blooms): refactors `blockLoadingIter`, exposing more error cases + additional metrics (#12015 )	2 years ago
Owen Diehl	09aa0f80b5	Blooms/contiguous bounds union (#12009 )	2 years ago
Owen Diehl	166229818d	[Blooms] Consistent hashing via tokens for bloomcompactor (#12002 )	2 years ago
Owen Diehl	85f7baaeda	Blooms/integration fixes (#11979 )	2 years ago
Salva Corts	ef40136715	Use TSDB index prefix on blooms directory (#11977 )	2 years ago
Owen Diehl	5d1798f16d	Blooms/misc fixes (#11974 )	2 years ago
Owen Diehl	f158b5bc3e	correctly set block iter when no overlapping blocks are found (#11973 )	2 years ago
Owen Diehl	87ae2efe41	Pull/11950 ammendments (#11972 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	543aaab055	(Blooms) Add metrics to compactor (#11966 )	2 years ago
Christian Haudum	5967ceeb43	Replace min/max token with `TokenRange` in bloom ring utilities (#11960 ) This PR replaces min/max token fields with a `TokenRange` field that uses the `Range[uint32]` type. The `Range[uint32]` uses similar semantics as the `FingerprintBounds` we use for fingerprint ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	6434df7345	Bloom compactor shuffle-sharding (#11947 )	2 years ago
Owen Diehl	bd12e16339	[Blooms] Use correct table address function (#11955 )	2 years ago
Owen Diehl	3c06b360ec	meta & removal support for bloom compaction (#11941 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Christian Haudum	1c43991ddc	Remove `bloomcompactor.DayTable` in favour of `config.DayTime` (#11917 ) Both structs shared the same semantics. This PR moves additional functionality from the `DayTable` to the `DayTime` struct. To get the table name of a day (ordinal number of day since unix epoch), call `DayTime.Table()`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	eb8464a641	Bloom compactor: Load blocks lazily in batches (#11919 ) To avoid loading possibly lots of blocks upfront, this PR introduces lazy loading of blocks in batches using an iterator that loads blocks on demand. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	0bb2574040	makes batchedLoader generic + removes unnecessary interfaces & adapters (#11924 ) While reviewing https://github.com/grafana/loki/pull/11919, I figured it'd be nice to make `batchedLoader` generic so we can reuse it's logic. This let me test it easier and remove a lot of now-unnecessary adapter code (interfaces, types)	2 years ago
Owen Diehl	7560067439	Bloom/running (#11918 ) Includes a few small fixes in order to get the compactor running, including * Wires up i/o from `v1` pkg and the `bloomshipper` pkg in order to archive+write blocks remotely after building * config error handling * fix an error iterating chunks * handles gzipping when loading tsdbs from storage * bypasses internal cache in `BloomTSDBStore`'s internal `indexshipper/storage.Client` to get up to date results, which is easier for prototyping * `ClosableReadSeekerAdapter` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	7a95cb8663	Configurable bloom tokenizer and block settings (#11889 ) What this PR does / why we need it: This PR makes the max block size configurable and wires up the already existing nGramLen and nGramSkip settings. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Salva Corts	681bb57971	Limit bloom block size (#11878 ) What this PR does / why we need it: This PR limits the size of the blocks created by the compactor. The block builder keeps adding series' blooms to a block until the limit is exceeded, meaning that blocks may grow beyond the configured maximum. This is needed so we avoid having tiny blocks which had space for small blooms but later a bigger blooms didn't fit. Blocks are built lazily: the generator returns an iterator that builds one block at a time. Special notes for your reviewer: The maximum size is currently set to 50 MBs. We will make this configurable on a followup PR. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	d58031e853	Blooms/bloomcompactor wiring (#11914 ) * Moves `router` struct into the `Compactor` struct * Implements period aware `TSDBStore` * Integrates everything together, hooks into `modules.go` * Removes some now unused compactor code * Light refactoring to make types align where appropriate	2 years ago
Owen Diehl	3a1bd41ef5	[Blooms] Routing & wiring for bloom building (#11894 ) A few changes wiring up bloom building logic: * A `Router` as the controlling mechanism for iterating tenants+periods, queueing work for parallel computation, determining which fingerprint ranges and tenants are owned by a replica, and waiting for it's completion * Updates the `SimplerBloomController` to accept ownership ranges, periods, and tenants as arguments for use by the `Router` * Builds a `BloomTSDBStore` struct around the `indexshipper/storage.Client` interface in order to iterate relevant TSDBs/etc	2 years ago
Owen Diehl	ba0538c2a5	[Blooms] tsdb iterator implementation for bloom construction (#11886 ) Builds a lazy iterator over a tsdb file for bloom construction. This is just the library utility, not the integration.	2 years ago
Christian Haudum	c4ac168a8c	BloomStore: Change signature of `FetchBlocks()` to return `[]CloseableBlockQuerier` (#11884 ) What this PR does / why we need it: In order to encapsulate the reference counting on the `BlockDirectory`, the `BloomStore` now returns a slice of `CloseableBlockQuerier` instead of the directory itself. The caller is responsible for closing the returned querier in order to release the reader resource and decrement the ref counter. The PR also renames `ClosableBlockQuerier` to `CloseableBlockQuerier`. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	af66ecec6a	[Blooms] bloomshipper meta integration (#11874 ) Builds on top of https://github.com/grafana/loki/pull/11872, standardizing on the `bloomshipper` types `Meta` and `MetaRef`.	2 years ago
Christian Haudum	73edf7a943	(chore) Bloomshipper: Separate store and client (#11865 ) What this PR does / why we need it: This PR removes the `StoreAndClient` interface that was accepted by the `BloomShipper`. Since the `BloomStore` had to not only implement the `Store` interface, but also the `Client` interface, it caused re-implementation of the same methods in different ways. Now the shipper solely relies on the `Store` interface. See individual commit messages for more context. Tests have been rewritten from scratch and placed in their own respective test files for store and client. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	d413c28fa3	Blooms: Align compactor to shipper (#11855 ) Does some alignment work between the `bloomcompactor` and the `bloomshipper` pkgs. Notably: * Uses `bloomshipper.BlockRef` everywhere (removes old bloomshipper struct * Integrates `v1.FingerprintBounds` in `Ref` struct * `Location` interface to distinguish local paths vs paths in object storage for certain types (`{Meta,Bloom}Ref`s) * Introduces `KeyResolver` interface to generate locations from these structs * Integrates `KeyResolver` into our bloom store. In the future, this will allow us to change key structures across schema boundaries when we want to change\|improve them. * Removes `BlockPath` from `BlockRef` in favor of the new resolving functionality. This is also beneficial because it lets us _calculate_ locations from the pure Ref objects, rather than tie some arbitrary state to them which can change (or not be populated by accident).	2 years ago
Owen Diehl	928e3f34f8	Blooms: Prepare compactor refactor (#11847 ) Prepares `bloomcompactor` pkg for integration with new logic, primarily removes/thins out existing code.	2 years ago
Christian Haudum	4411649a0e	(chore) Bloom shipper: Replace `Keyspace` struct with `v1.FingerprintBounds` (#11839 ) The latter struct has more utility functions to compare and operate on bounds. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	509f6f6b66	Bloom/controller wiring (#11831 ) Functionality & testware for building bloom block plans.	2 years ago

1 2

84 Commits (0d8ff9ee7929b8facbdb469abe344c320d3bd5ce)