400 Commits (47b17b590c354049f380ac7f5a4b9a80bdfff4f4)

Author	SHA1	Message	Date
Karsten Jeschkies	c9f3868287	Update dskit to v0.0.0-20220331160727-49faf69f72ca. (#5881)	3 years ago
Bryan Boreham	e2d7ee2cff	GCS: refactor creation of http client (#5838) ... Separate the function into one part which creates the RoundTripper and one which wraps it in instrumentation.	3 years ago
Cyril Tovena	8c1095a554	Moves TSDB to the stores package. (#5852) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833) ... * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Bryan Boreham	c0cc004333	gcs client: use fake credentials in unit test (#5834) ... * gcs client: use fake credentials in unit test 'insecure' mode is only used in unit tests, so I piggy-backed on that to set a fake API key. This avoids the GCS library looking up credentials from the user's environment and possibly contacting Google servers, as described at https://cloud.google.com/docs/authentication/production * Comment Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Sandeep Sukhani	27a027824f	incorporate feedback on merged pr #5830 (#5831)	3 years ago
Sandeep Sukhani	67ba342684	improve mutex locking when writing index in boltdb-shipper (#5830)	3 years ago
李国忠	1f8fc65826	[Optimization] boltdb: reduce the code block affected by the lock tablesMtx in the TableManager.query() function. (#5829)	3 years ago
Owen Diehl	0c826b834a	Tsdb/compaction (#5783) ... * ChunkMetas.finalize uses pool * slight refactoring. Uses pools during chunkmetas.finalize and adds an InclusiveBounds helper * multi idx explicitly wraps tsdb files * TSDB Checksum helper + requires at least 2 indices for MultiIndex constructor * tsdb compaction impl * skips compacting a single file * compaction + testing * tsdb compaction tests * passes reference to &index.ChunkMetasPool for avoiding nocopy * linting * gitignore .vscode * s/Combine/FilePath/g	3 years ago
Tat Chiu Leung	e65f26d30f	storage: make Azure blobID chunk delimiter configurable (#5777) ... * Make Azure chunk delimiter configurable * Changelog: #5777 * doc update	3 years ago
Michel Hollands	b865b81ba0	Add a DeletionMode config variable (#5481) ... * Add deletionEnabled setting and remove delete request manager Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rebase Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove deletion handling from delete requests manager Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * change store so it stores a logql statement Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add validation code for logql statement Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Run deleteRequestsManager when deletion is enabled Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove unused variables Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert "Remove deletion handling from delete requests manager" This reverts commit ce4f774497aa590caff86b0745ec81588592a9e1. * Re-add IsDeleted method Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Re-add tests for IsDeleted Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix delete request store test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix linting issue Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert compactor changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add deletion mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add v1 mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename LogQLRequest to Query Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix linting issues Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use DeleteMode in compactor module Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename logql to query Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Put cancel under delete verb Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert only the API surface area while keeping everything else * Use moved code in syntax package Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove duplicte import Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use renamed field in tests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove duplicates and empty lines in changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog description Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/storage/stores/shipper/compactor/deletion/delete_request.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_manager_test.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_store.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_store.go Co-authored-by: Christian Simon <simon@swine.de> * Update CHANGELOG.md Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_store.go Co-authored-by: Christian Simon <simon@swine.de> * Make DeletionMode struct member non public Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert change to docs re cancellation Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use same variable names Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add parameter validation to changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename v1 to WholeStreamDeletion Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix default value of deletion mode config setting Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * reimplement new api * Add delete request handler when delete mode is set Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove unused variable Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment to change the code when other deletion modes are available Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * create expirationChecker if deletionMode is set Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Address review comments Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/storage/stores/shipper/compactor/compactor.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Rename AddQuery to SetQuery Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com> Co-authored-by: Christian Simon <simon@swine.de> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Christian Haudum	03036f1335	golangci-lint: Check for pointers to enclosing loop variables (#5775) ... * Checks for pointers to enclosing loop variables Using the pointer to the enclosing loop variable in a range loop likely leads to unwanted behaviour. Introducing the check `exportloopref` helps to prevent the usage of such pointers. > The iteration variables may be declared by the "range" clause using a > form of short variable declaration (:=). In this case their types are > set to the types of the respective iteration values and their scope is > the block of the "for" statement; they are re-used in each iteration. https://go.dev/ref/spec#For_statements Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Checks for pointers to enclosing loop variables Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Ed Welch	c158b2c5f5	Loki: Revert #4845 which changed the format of errors from the API (#5772) ... * This reverts the changes from #4845, unfortunately changing the content type from plain text to JSON is a breaking change of the API and we should wait to do this at a major release. Signed-off-by: Ed Welch <edward.welch@grafana.com> * fix tests which were updated after initial change. Signed-off-by: Ed Welch <edward.welch@grafana.com>	3 years ago
Owen Diehl	86cb8d52d3	ensure tsdb is sorted by hash then labelset (#5746)	3 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724)	3 years ago
Owen Diehl	3f28a33d93	TSDB Index reuses slices, adds pools (#5630) ... * adds a pool for ChunkMetas * index takes slice pointers for allocation reasons and adds pools for all index types * reuse slices in tsdb benchmarks * adds count to bench script * properly returns series to pool * more ergonomic index signatures while still supporting slice reuse * tsdb index documentation * aligns tsdb-map tooling with new index signatures	3 years ago
Owen Diehl	d084dc7f4b	Tsdb/hash order (#5631) ... * encode fingerprint into series in tsdb * uses series fingerprint in index reads * order tsdb series by fprint * tsdb writes fingerprint offsets table * shardannotation is part of tsdb/index, fixes testware * loads fingerprint offsets from tsdb * removes unnecessary sortedpostings * adds prefix sharding logic * sharded tsdb postings * updates benchmark to pass nil hash * runs tsdb benches with variable count	3 years ago
Christian Simon	ea62af58e9	Introduce a ChunkRef type as part of logproto (#5643) ... * Change protobuf definitions * Adapt to an embedded ChunkRef struct * Move the Checksum to the ChunkRef	3 years ago
Sandeep Sukhani	6a533e5a56	improve boltdb-shipper logging to help debug index query latency issues (#5651)	3 years ago
Cyril Tovena	59bd44a5df	Remove more chunkstore and schema version below v9 (#5650) ... * Remove more chunkstore and schema version below v9 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes remaining tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes remaining tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Sandeep Sukhani	8c06c546ab	debug log number of index entries cached and retrieved from the cache per index query (#5642)	3 years ago
Owen Diehl	ec809e665f	Tsdb/sharding-benchmarks (#5628) ... * adds shard to tsdb/index pkg * add sharded tsdb benchmark	3 years ago
Travis Patterson	88372c93cc	Query Filtering -- Querier (#5482) ... * Query Filtering -- Querier - Get deletes from the index gateway and attach them to store and ingester requests * lint * Fix test -- sometimes svc is nil * don't test for delete overlap on each selector * lint	3 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539) ... * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
Cyril Tovena	8f693e77b0	Update Prometheus to latest (#5545) ... * Update Prometheus to latest Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update promql.QueryTracker Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes querier test to use new grafana/regexp Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test with new textparse signature Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Owen Diehl	d320873e82	Tsdb/sharding (#5600) ... * sharded support for GetChunkRefs and Series * sharded support for multi-index * linting	3 years ago
Owen Diehl	2eb1046818	Tsdb/multi idx (#5598) ... * begins working on a multi file index for querying across TSDBs * abandons heapsort for multi-index * series struct includes fingerprint and implements rest of multi-idx methods * goimports	3 years ago
Owen Diehl	f6d15fba48	bounds utility for tsdb lib (#5593)	3 years ago
Sandeep Sukhani	d2ce08280f	Compactor improvements (#5587) ... * upload and cleanup index sets concurrently * reuse batch slice used for building a batch of index during compaction	3 years ago
Sandeep Sukhani	7f498e9617	cleanup common index source files at the end while compacting index with boltdb-shipper (#5585)	3 years ago
Owen Diehl	824c65598f	store tsdb bounds (#5582)	3 years ago
Sandeep Sukhani	c76544baf9	build a new batch while starting to process a bucket (#5580)	3 years ago
Owen Diehl	dee6a56df1	Tsdb/index interface (#5577) ... * begins speccing out index ifc, impl * block level labelnames/values * single file index test + bugfix * goimports * LabelNames and LabelValues support matchers * s/Interface/Index/	3 years ago
Sandeep Sukhani	593b510f0e	fix broken index gateway benchmark due to missing limits (#5520)	3 years ago
Travis Patterson	b485b81ff9	Fix potential deadlock in the table manager (#5472) ... * Fix potential deadlock in the table manager * Ensure second lock is released on return * use t.TempDir instead of os.TempDir * fix broken test Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
Sandeep Sukhani	1e56dca741	per user index query readiness with limits overrides (#5484) ... * add support for configuring query readiness for user index with limits overrides config * improve tracking of duration for query time downloading of index * update changelog * use the existing context.Context we have for background operations while ensuring query readiness	3 years ago
Owen Diehl	0a5e01ba01	Rounds chunk bytes to kb in tsdb and includes benchmarking script (#5479) ... * rounds chunk bytes to kb in tsdb and includes benchmarking script * shellcheck * fix querier test	3 years ago
Sandeep Sukhani	87b7dfae87	Stream query responses from boltdb index client (#5468) ... * stream query response from boltdb index client instead of sending one row per batch * add benchmark for testing index gateway query performance * lint and fix broken test * wrap error message while failing to send back response * sort imports	3 years ago
Sandeep Sukhani	55acdbd08f	boltdb shipper index gateway client improvements (#5456) ... Index Gateway query performance improvements by bounding the number of concurrent grpc calls we make from the client.	3 years ago
Sandeep Sukhani	9f301cf87d	use the defined querypages callback in all the types implementing QueryPages function from IndexClient interface (#5463)	3 years ago
Owen Diehl	c36272e31e	Tsdb/idempotent builder (#5424) ... * builder chunks uses heap * idempotent index builder * remove early return * pr suggestions * more thorough equality check on ChunkMetas.finalize	3 years ago
Sandeep Sukhani	333d5da0d0	add a metric for tracking the duration it took to sync the tables in queriers/index-gateways (#5446) ... * add a metric for tracking the duration it took to sync the tables in queriers/index-gateways * changes suggested from PR review	3 years ago
Owen Diehl	a8397161dd	Adds a tool for converting boltdb->tsdb and benchmarking queries (#5430) ... * exports NewChunkIndexIterator * starts building tsdb-bench * adds query benchmark for tsdb prototyping * lint	3 years ago
Sandeep Sukhani	00ca1aa99f	do not send grpc responses concurrently from index-gateway (#5445) ... See https://github.com/grpc/grpc-go/blob/master/stream.go#L120-L123	3 years ago
Owen Diehl	137a28d82b	Adds bytes and entries to chunk metadata in tsdb (#5414) ... * adds bytes and entries to chunk metadata in tsdb * removes comment	3 years ago
Travis Patterson	d8780da454	Revert to working version of Azure client (#5413) ... * Revert "storage: update azure blobstore client to use new sdk (#5302)" This reverts commit 89b04a97b3. * Fix Azure SDK Issue - Revert SDK to working version - Introduce patches to prevent deadlocks * make metrics global	3 years ago
Owen Diehl	1837c9e0b2	Experimental TSDB index (#5376) ... * moves encoding to own pkg * decwrap impl * forks tsdb index * renames tsdb_enc * wraps all tsdb decbufs * use loki encbuf * custom chunk meta replacing ref-id for uint32 checksum * handle overlapping chunks in tsdb encoding * tsdb builder impl * tsdb querier beginnings to use modified tsdb * fixes nonexistent reference * lint/remove unused * adds tsdb testware * lint	3 years ago
Travis Patterson	7378b697f8	Fix Azure issue where 404 not recognized (#5399)	3 years ago
Sandeep Sukhani	e65d1b8c69	add more context to logs, improve comments for clarity in boltdb-shipper code (#5341)	3 years ago
Cyril Tovena	4fafc9c234	Fixes a flaky hedging test. (#5389) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago