7 Commits (02d88462f39654b85b02d31c1db05e0e2d9cec46)

Author	SHA1	Message	Date
Sandeep Sukhani	02d88462f3	allow applying retention at different interval than compaction with a config (#4736) ... * allow applying retention at different interval than compaction with a config * add changelog entry	4 years ago
Ed Welch	0c9beadaaa	Loki: Add a ring to the compactor used to control concurrency when not running standalone (#4574) ... * add a ring to the compactor * wiring up the start/stop functionality * add some monitoring for multiple compactors running at the same time. * Add compactor to `read` target * fix tests/validation requirements * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * used a shared ring_config between scheduler and compactor to avoid some duplication * fix lint * better handling of the tokens file with a way to easily tell all the rings to persist tokens * fix query scheduler replication factor * removing "scheduler" from log line which may be called by components other than a scheduler in the future. * update changelog Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	43184298b2	Trigger compaction prior retention. (#3665) ... * Trigger compaction prior retention. This should allow to process table where the amount of index files is below 4. Currently they are not processed and we should process them since they may never be compacted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook retention into compaction. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove unused flags. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better cancellation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing ctx in test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flags that are not used. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add compactor retention tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	806d6a503e	Custom Retention (#3642) ... * Playing around with a POC based on the design doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Getting series ID for label matchers is now working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip/ Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * We're parsing the index label now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting to extract interfaces to make the code testable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress will try to add labels to chunk ref iterator Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Iterator for chunks ref with all labels !!!. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Optimize code away. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * with delete into the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleaner but not yet working for v10 and v11. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes series cleaner. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hooking into the compactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking limit retention config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off the marker processor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on sweeper and fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding more tests and founding more bugs along the way. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Bug with path once boltdb is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more bug and more robust test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More test and cleanup getting close. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar bugs with regards to period schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky tests because of boltdb still open. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Save files without using table key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve logging and ability to use more goroutines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes duplicate metrics since histogram contains total too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock bug when too many workers are trying to update the mark file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock when reading and updating db at the same time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes default config test of boltdb. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * PR Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the user of assert to not fail a test if it's incorrect on the first shot. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add experimental notice to the flag documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes empty index detection and table deletion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not delete folder it's not necessary with object store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better working path cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	7b7b1844ca	add a metric in compactor to record timestamp of last successful run (#2788)	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	1bba2a89f8	add some metrics for monitoring compactor (#2548)	5 years ago