17 Commits (094f8f3b20d9bb5485ccd52b2daf9b31abc8802a)

Author	SHA1	Message	Date
Sandeep Sukhani	f18edcf9d8	use the configured index prefixes while building table names for tsdb index (#6436) ... * use the configured index prefixes while building table names for tsdb index * do not fail compaction when schema for a table can't be found	3 years ago
Sandeep Sukhani	9590e5ca4e	Tsdb tenant index querying (#6598) ... * update per tenant index identifier and relevant changes * add a flag to ForEach index callback to tell whether it is multitenant index or not	3 years ago
Dylan Guedes	bcb57f100c	During query readiness, log duration of all object-store operations (#6489) ... * During query readiness, log duration of all object-store operations. * Fix duration assignment.	3 years ago
Dylan Guedes	c69605f2af	Loki: Fix flaky IndexGatewayClient test (#6481) ... * Stop gRPC clients in tests.	3 years ago
Dylan Guedes	9686e4ba30	Add total number of unique users involved in query readiness. (#6035)	3 years ago
Owen Diehl	0011c2d7b7	TSDB/index-sampling-wiring (#6338) ... * wires up index sampling * regens protos * mockstore stats impl * addtl linting * renames to avoid duplicate via linter	3 years ago
Sandeep Sukhani	086dbf7270	always retry syncing index files from indexshipper when the index list cache is stale (#6327)	3 years ago
Sandeep Sukhani	82da01ad43	make index shipper read path handle range of tables by type of index (#6304)	3 years ago
Sandeep Sukhani	9a2df5a06e	copy boltdb-shipper cache changes from PR #6054 to generic index shipper (#6316)	3 years ago
Sandeep Sukhani	2d4ff73cc2	add metrics for tracking index shipper operations (#6278)	3 years ago
Sandeep Sukhani	6ac955cc7d	make boltdb-shipper use the generic indexshipper for managing index in object storage (#6226)	3 years ago
Sandeep Sukhani	ef4bc8f1ed	copy all the fixes/improvements that went into boltdb-shipper that are missing from generic index-shipper (#6200)	3 years ago
Sandeep Sukhani	6e3bf3f7d0	tsdb query fixes (#6142) ... * query both downloads and uploads manager when non-nil * fix reading of index file with nodename having a `-` * remove __name__ matcher at a layer above to avoid passing it during tsdb head queries	3 years ago
Owen Diehl	729bf3fc91	loads previously created TSDBs into shipper on startup (#6117) ... * loads previously created TSDBs into shipper on startup * noopTSDBManager impl * info logs for initial tsdb loading	3 years ago
Owen Diehl	b45efd4c2d	TSDB shipper + WAL (#6049) ... * begins speccing out TSDB Head * auto incrementing series ref + mempostings * mintime/maxtime methods * tsdb head IndexReader impl * head correctly populates ref lookup * tsdb head tests * adds prometheus license to tsdb head * linting * [WIP] speccing out tsdb head wal * fix length check and adds tsdb wal encoding tests * exposes wal structs & removes closed semantics * logs start time in the tsdb wal * wal interface + testing * exports walrecord + returns ref when appending * specs out head manager * tsdb head manager wal initialization * tsdb wal rotation * wals dont use node name, but tsdb files do * cleans up fn signature * multi tsdb idx now just wraps Index interfaces * no longer sorts indices when creating multi-idx * tenantHeads & HeadManger index impls * head mgr tests * bugfixes & head manager tests * tsdb dir selection now helper fns * period utility * pulls out more code to helpers, fixes some var races * head recovery is more generic * tsdb manager builds from wals * pulls more helpers out of headmanager * lockedIdx, Close() on idx, tsdbManager update * removes mmap from index reader implementation * tsdb file * adds tsdb shipper config and refactors initStore * removes unused tsdbManager code * implements stores.Index and stores.ChunkWriter for tsdb * chunk.Data now supports an Entries() method * moves walreader to new util/wal pkg to avoid circular dep + tsdb storage alignment * tsdb store * passes indexWriter to chunkWriter * build a tsdb per index bucket in according with shipper conventions * dont open tsdb files until necessary for indexshipper * tsdbManager Index impl * tsdb defaults + initStore fix for invalid looping * fixes UsingTSDB helper * disables deleteRequestStore when using TSDB * pass limits to tsdb store * always start headmanager for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes copy bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * fixes duplicate tenant label bug Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * debug logs, uses label builder, removes __name__=logs for tsdb Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * tsdb fixes labels at earlier pt Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * account for setting tenant label in head manager test * changing tsdb dir names * identifier interface, builder to tsdb pkg * tsdb version path prefix * fixes buildfromwals identifier * fixes tsdb shipper paths * split buckets once per user set * refactors combining single and multi tenant tsdb indices on shipper reads * indexshipper ignores old gzip logic * method name refactor * remove unused record type * removes v1 prefix in tsdb paths and refactores indices method * ignores double optimization in tsdb looking for multitenant idx, shipper handles this * removes 5-ln requirement on shipper tablename regexp * groups identifiers, begins removing multitenant prefix in shipped files * passses open fn to indexshipper * exposes RealByteSlice * TSDBFile no longer needs a file descriptor, parses gzip extensions * method signature fixing * stop masquerading as compressed indices post-download in indexshipper * variable bucket regexp * removes accidental configs committed * label matcher handling for multitenancy and metricname in tsdb * explicitly require fingerprint when creating tsdb index * only add tenant label when creating multitenant tsdb write fingerprints without synthetic tenant label strip out tenant labels from queries * linting + unused removal * more linting :( * goimports * removes uploadername from indexshipper * maxuint32 for arm32 builds * tsdb chunk filterer support * always set ingester name when using object storage index Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Sandeep Sukhani	2758dc659a	boltdb shipper index list cache improvements (#6054) ... * bypass index list cache when doing query time downloading of index * detect and refresh stale index list cache during sync	3 years ago
Sandeep Sukhani	b3c818e132	build generic index shipper to be shared between boltdb-shipper and tsdb-shipper (#5705) ... * build generic index shipper to be shared between boltdb-shipper and tsdb-shipper * changes suggested from pr review * Update pkg/storage/stores/indexshipper/downloads/index_set.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * update comments on trying to open corrupt files * fix references broken due to storage refactoring * lint Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago