400 Commits (47b17b590c354049f380ac7f5a4b9a80bdfff4f4)

Author	SHA1	Message	Date
Owen Diehl	47b17b590c	dont skip writing to TSDB index when chunk exists in cache (#6664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Travis Patterson	68141444f8	Introduce Groupcache (#6644) ... * implement groupcache * use mailgun groupcache * use memberlist * implement peer discovery with the ring * implement retrieval and storage * use a ticker * Correctly configuring chunk cache Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Enabling results cache Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing groupcache ring page Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * groupcache-poc: Add support for index cache. (#6535) * Hacked in prometheus metrics (#6544) Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * add http to addresses * always make a group for each cache * fix peer discovery * Fix groupcache config - Allow it to be enabled/disabled - Move all config to 'common' config - Always write to hot cache * Add http/2 support for groupcache (#6596) * Add groupcache latency metrics (#6619) * Add groupcache latency metrics * lint * lint * lint * Add groupcache test (#6620) * Add groupcache test * lint * Improve 'Fetch' allocs 11 -> 8 (#6623) * Move constants to the file where they're used * remove irrelevant comments Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Marco Pracucci	e7fb10f456	Loki: Update `grafana/dskit` and `weaveworks/common` (#6643) ... * Upgrade dskit and weaveworks/common Signed-off-by: Marco Pracucci <marco@pracucci.com> * 'go mod vendor' again Signed-off-by: Marco Pracucci <marco@pracucci.com>	3 years ago
Owen Diehl	094f8f3b20	[TSDB] fingerprint offsets is now correctly 16 byte aligned (#6630) ... * simpler shard factor calculations, disallows identity factor of 1 Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * log interval span as soon as it finishes Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * another shard bounds test Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * indexclient span logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * more index span logging Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * correctly match single shard Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * shardedPostings test Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * shardedPostings maxOffset is properly exclusive Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * minor posting idiomatic-ness Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * store 16 byte aligned offsets for fingerprint table Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Sandeep Sukhani	f18edcf9d8	use the configured index prefixes while building table names for tsdb index (#6436) ... * use the configured index prefixes while building table names for tsdb index * do not fail compaction when schema for a table can't be found	3 years ago
Sandeep Sukhani	619c2212f5	build and hook tsdb compaction code to the generic index compactor (#6627)	3 years ago
Sandeep Sukhani	9590e5ca4e	Tsdb tenant index querying (#6598) ... * update per tenant index identifier and relevant changes * add a flag to ForEach index callback to tell whether it is multitenant index or not	3 years ago
Travis Patterson	7f24b0f6ab	Remove ourdated comment (#6597)	3 years ago
Travis Patterson	a92e048ec3	avoid making delete requests if deletes are disabled for a user (#6583)	3 years ago
Sandeep Sukhani	25912ea02d	generic index compactor (#6537) ... * make compactor generic to support compacting various index types * implement and register compaction for boltdb-shipper index type * lint and fix broken test * add more comments * fix some code and tests * remove duplicate user-id kv in logger * some improvements in code and test cases * lint	3 years ago
Dylan Guedes	389eee4cc7	Stop compaction interval timer to avoid leak. (#6570)	3 years ago
Michel Hollands	695fa9d7a8	Add client type as label (#6469) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Dylan Guedes	bcb57f100c	During query readiness, log duration of all object-store operations (#6489) ... * During query readiness, log duration of all object-store operations. * Fix duration assignment.	3 years ago
Dylan Guedes	c69605f2af	Loki: Fix flaky IndexGatewayClient test (#6481) ... * Stop gRPC clients in tests.	3 years ago
Michel Hollands	61e3e52d4f	Handle duplicate registration of deletion metrics (#6446) ... * Handle duplicate registration of deletion metrics Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Create metrics struct at Loki level Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix test Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Michel Hollands	f80e487dcf	Remove whole stream deletion mode (#6435) ... * Remove whole-stream-deletion mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove whole-stream-deletion from docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Sort changelog entries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove link to wrong configuration Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix integration test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Set default deletion mode to disabled Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove extra white line in documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix default value in docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add DeletionEnabled method on mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename test Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Michel Hollands	b4e6c59936	Compactor: add per tenant compaction delete enabled flag (#6410) ... * Add per tenant compaction delete enabled flag Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove changes in wrong place Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add compactor deletion enabled field Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use limit in compactor Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use http middleware and add test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix lint issue Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert to default setting if no override Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add default value command line option Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the docs Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Enable access to deletion API for integration test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename flag to allow_deletes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update per review comments Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Owen Diehl	a1c72ad562	Tsdb/wal rotate fix (#6423) ... * rotate tsdb wal under mutex Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * ensure we rotate and doublecheck conditions under lock in tsdb headmanager Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	d6f50ca5e1	[TSDB] Use index sampling to determine shard factor (#6396) ... * use index sampling to determine shard factor * logging/tracing + better parallelism planning * queryrange downstreamer now checks max_query_parallelism * lint * handles interval, offset in tsdb planning and adds logging * fixes ns->ms confusion in index stats proto * handle zero shard value without panics * shardmapper will downstream a single unsharded query * uses concat expr with no shards to avoid query parsing errors * better logging * fixes wrong Size() method call and rounds to nearest KB when calculating chunk size * humanize bytes in log line * only adds defaultLookback to index sampling when interval is zero * removes comment * more logging * adjust through correctly * adds query length for index queries	3 years ago
Jesus Vazquez	db9dbe8e1d	Bump prometheus dependency (#6403) ... * Bump prometheus dependency We had to add a replace of prometheus in go mod because thanos is importing an older version and then replacing it. * Undo textparser.New variable mismatch * modtimevfs path update * Fix labels withlabels and withoutlabels usages WithoutLabels and WithLabels methods have dissappeared. Now labels.NewBuilder has to be used. * Fix RuleManager interface for Update() Update now accepts a new param called ruleGroupPostProcessFunc which is an optional function to be executed after rule group processing. In this case I've set it to nil in the usages i've found. * NewQueryRange now accepts QueryOpts The NewQueryRange function now accepts QueryOptions so I had to update all the calls. I've set the opts to nil since we're not using them. * gofmt	3 years ago
Michel Hollands	20db4df542	Add more logging for delete requests (#6394) ... * Add more logging for delete requests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix typo Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rephrase logging message and add chunk ID Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename userID field to user Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * rename more userID Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix existing log messages Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Michel Hollands	671afe62ba	Use correct counter when deleting lines (#6376) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Travis Patterson	efda5b2af7	Don't interupt the query path if deletes aren't available (#6368)	3 years ago
Wu Zhiyuan	f61cbe4288	GCS: Add service_account option (#6366) ... * gcs client: add service_account support * Update docs for GCS's service_account * Fixed lint check failed	3 years ago
Owen Diehl	9218e4654b	TSDB/query-planning-groundwork (#6367) ... * begins building queryrange compatible types for index sampling * adds MatcherGroups() method to SampleExpr * MatcherGroups test + ShardResolver interface * index stats codec wiring * millisecond precision on index stats codec * lint * cleanMatchers in tsdb.IndexClient for consistency + adds catchall matcher if nil	3 years ago
Dylan Guedes	9686e4ba30	Add total number of unique users involved in query readiness. (#6035)	3 years ago
Owen Diehl	3f4a663dbe	Tsdb/index sampling endpoint (#6347) ... * index sampling endpoint * fix bug in withTenantLabelMatcher * show zero values in stats * handles new errcase * fixes comments	3 years ago
Owen Diehl	0011c2d7b7	TSDB/index-sampling-wiring (#6338) ... * wires up index sampling * regens protos * mockstore stats impl * addtl linting * renames to avoid duplicate via linter	3 years ago
Danny Kopping	3a1fdd03f4	Define a reason why the fifocache has evicted an item (#6335) ... * Define a reason why the fifocache has evicted an item Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Making reason more clear Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Do not increment eviction counter on Stop() Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Owen Diehl	5a6157ecf3	Tsdb/index sampling (#6329) ... * groundwork for index sampling * go mod * bloom filter utils * removes unnecessary atomics * Stats() impl for tsdb index * use passed var * lint	3 years ago
Sandeep Sukhani	086dbf7270	always retry syncing index files from indexshipper when the index list cache is stale (#6327)	3 years ago
Danny Kopping	36e0979cf5	Add cache usage statistics (#6317) ... * Adding cache statistics Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metrics to metrics.go Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Creating new stats context for use in metric queries middleware Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Clean up unnecessary log fields Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding stats tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * CHANGELOG entry Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Documenting function Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving CHANGELOG entry to appropriate section Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Implementing a stats collector cache wrapper to simplify stats collection If we keep the stats collection in pkg/storage/chunk/cache/instrumented.go, then any implementation that wraps it will cause the stats collected to be incomplete. For example: NewBackground(cacheName, cfg.Background, Instrument(cacheName, cache, reg), reg)) - the background cache requests are not collected Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Michel Hollands	65e3148bc9	Update delete integration test (#6192) ... * Add wait until processed Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add query lines Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove Cortex reference Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add deleted lines metric and check in test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Set negative cancel period so default 1 minute delay is compensated for Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Reduce timeout Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add helper function Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Comment out assertions that only work with flush Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove unused metric type param Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Check counter in unit test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename ClientOption to Option as per the linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove redundant check Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * fill in MsgAndArgs field of test assertion Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Evangelos Foutras	a877febf75	Use fake credentials in storage/chunk/storage tests (#6187) ... Enable insecure mode for the GCS client so fake credentials are used when running the storage tests (see commit c0cc004333 ("gcs client: use fake credentials in unit test"). Fixes: https://github.com/grafana/loki/issues/6185	3 years ago
Sandeep Sukhani	82da01ad43	make index shipper read path handle range of tables by type of index (#6304)	3 years ago
Sandeep Sukhani	9a2df5a06e	copy boltdb-shipper cache changes from PR #6054 to generic index shipper (#6316)	3 years ago
Travis Patterson	69eabfcefd	Fix delete cancellation timeout (#6285)	3 years ago
Travis Patterson	ea0a5246cc	Only consider an interval to have possible expired chunks if it overlaps a delete. (#6297)	3 years ago
Sandeep Sukhani	1ad75d0b63	do not initialize tsdb index store when using index gateway client for queries (#6305) ... * do not initialize tsdb index store when using index gateway client for queries * fix test * lint	3 years ago
Sandeep Sukhani	fefc33debc	make tsdb store instance a singleton (#6273) ... * make tsdb store instance a singleton * make it mandatory to have 24h index period config for tsdb * avoid initializing head and tsdb manager when running tsdb store in read only mode	3 years ago
Travis Patterson	1c5e094e16	Only delete data when WholeStreamDeletion or FilterAndDelete (#6286) ... * Only delete data when WholeStreamDeletion or FilterAndDelete * remove broken snyn workflow	3 years ago
Ed Welch	f52fc9d907	Loki: add a run-once flag to the compactor (#6281) ... * add a run once flag to the compactor to allow it to cleanup index files with a single manual invocation. Useful when using migrate tool. Signed-off-by: Edward Welch <edward.welch@grafana.com> * fix lint Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Sandeep Sukhani	2d4ff73cc2	add metrics for tracking index shipper operations (#6278)	3 years ago
Sandeep Sukhani	7369b3785c	use tsdb config for initializing index gateway client (#6267) ... * use tsdb config for initializing index gateway client * review feedback Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Travis Patterson	9b2786bde3	Fix delete updates (#6194) ... * Fix delete updates * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_client.go Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> * Add compactor address to the storage config * review comments * Review feedback Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Steve Hipwell	7cc6aa4021	feat: Add support for Azure user assigned identity (#5891) ... Signed-off-by: Steve Hipwell <steve.hipwell@gmail.com>	3 years ago
Kaviraj Kanagaraj	a795b93897	chore(cache): Make exprimental messages more clear for cache. (#6219) ... Currently we see following message when any of the experimental feature is used for cache (say fifocache or redis). But the problem it doesn't say __what_kind__ of cache is it exactly. So this leads to confusion to track what kind of cache is the source of the experiment feature. This PR fixes it by adding more clear message. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Sandeep Sukhani	6ac955cc7d	make boltdb-shipper use the generic indexshipper for managing index in object storage (#6226)	3 years ago
Callum Styan	3d2282745b	Runtime reloadable config; ring migration setup (#6214) ... * Clean up setting of memberlist and multikv config for Loki services that use the ring. Also sets the multi client runtime config function for all services that use the ring. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add a test for the multi kv setup Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix lint issues. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Travis Patterson	e313d8a873	Remove __name__ from label calls (#6229)	3 years ago