17 Commits (ca19e8d7941bfac86355b2853ef2ea110f32075e)

Author	SHA1	Message	Date
Dylan Guedes	1221658fc5	Introduces cache to TSDB postings (#9621) ... **What this PR does / why we need it**: Introduces a new interface for fetching TSDB postings and implements a cached TSDB postings. The new postings interface is named `PostingsReader` and contains two implementations: a simple one, which is identical to the former Postings behavior, and a new one, that caches postings using the existing `index_queries_cache_config`. By default, the simple/former implementation is used. The new cached implementation can be configured in the following way: ```yaml storage_config: tsdb_shipper: cache_postings: true ``` In the future, we'll remove the `cache_postings` flag and change the behavior to cache postings by default.	3 years ago
Trevor Whitney	d74c67a9d0	Add version to TSDB index builder struct (#9566) ... Add version as a property to the TSDB Index Builder, forcing users of the builder to specify a version for each builder instance. This is the first of many PRs that will add the TSDB index version to the schema version config.	3 years ago
Travis Patterson	3170edf3e3	Add timeouts to deletions (#6879) ... * Add context to chunk iteration * Configure timeouts * add expiration timeout * update test * update test * Clarify timeout implications, log more specifically	4 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704) ... * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	4 years ago
Sandeep Sukhani	25912ea02d	generic index compactor (#6537) ... * make compactor generic to support compacting various index types * implement and register compaction for boltdb-shipper index type * lint and fix broken test * add more comments * fix some code and tests * remove duplicate user-id kv in logger * some improvements in code and test cases * lint	4 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049) ... * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Cyril Tovena	8c1095a554	Moves TSDB to the stores package. (#5852) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833) ... * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	0c826b834a	Tsdb/compaction (#5783) ... * ChunkMetas.finalize uses pool * slight refactoring. Uses pools during chunkmetas.finalize and adds an InclusiveBounds helper * multi idx explicitly wraps tsdb files * TSDB Checksum helper + requires at least 2 indices for MultiIndex constructor * tsdb compaction impl * skips compacting a single file * compaction + testing * tsdb compaction tests * passes reference to &index.ChunkMetasPool for avoiding nocopy * linting * gitignore .vscode * s/Combine/FilePath/g	4 years ago
Owen Diehl	51fbc95bbd	fixes existing test case and adds more (#5687)	4 years ago
Owen Diehl	3f28a33d93	TSDB Index reuses slices, adds pools (#5630) ... * adds a pool for ChunkMetas * index takes slice pointers for allocation reasons and adds pools for all index types * reuse slices in tsdb benchmarks * adds count to bench script * properly returns series to pool * more ergonomic index signatures while still supporting slice reuse * tsdb index documentation * aligns tsdb-map tooling with new index signatures	4 years ago
Owen Diehl	d084dc7f4b	Tsdb/hash order (#5631) ... * encode fingerprint into series in tsdb * uses series fingerprint in index reads * order tsdb series by fprint * tsdb writes fingerprint offsets table * shardannotation is part of tsdb/index, fixes testware * loads fingerprint offsets from tsdb * removes unnecessary sortedpostings * adds prefix sharding logic * sharded tsdb postings * updates benchmark to pass nil hash * runs tsdb benches with variable count	4 years ago
Owen Diehl	ec809e665f	Tsdb/sharding-benchmarks (#5628) ... * adds shard to tsdb/index pkg * add sharded tsdb benchmark	4 years ago
Owen Diehl	d320873e82	Tsdb/sharding (#5600) ... * sharded support for GetChunkRefs and Series * sharded support for multi-index * linting	4 years ago
Owen Diehl	67f961e4dc	more tsdb benchmarks + updates script to work with incompatible branches (#5590) ... * more tsdb benchmarks + updates script to work with incompatible branches * shellcheck	4 years ago
Owen Diehl	0a5e01ba01	Rounds chunk bytes to kb in tsdb and includes benchmarking script (#5479) ... * rounds chunk bytes to kb in tsdb and includes benchmarking script * shellcheck * fix querier test	4 years ago
Owen Diehl	a8397161dd	Adds a tool for converting boltdb->tsdb and benchmarking queries (#5430) ... * exports NewChunkIndexIterator * starts building tsdb-bench * adds query benchmark for tsdb prototyping * lint	4 years ago