apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Owen Diehl	56256bfd5b	Unordered chunks (#4001 ) * merge feature/unordered-replay * interoperable head chunks * memchunk block interop * retain ordered memchunk optimizations when possible * tests+bench for unordered chunk reads * reorder on chunk close * [wip] ingester stream unorderd * unordered writes default in testware config, fixes OOO bug & removes unused lastChunkTimestamp var * validity window is 1/2 of max age & fixes old transfer test * more consistent headblock checking/creation * more cohesive encoding tests * unordered stream test with validity bounds * compat - unordered * reinstates memchunk defaults when rebounding & updates storage test compatibility * lint * reorder across blocks doesnt overflow * respect chunk configs during rebounding when possible * only sync checks on ordered writes	4 years ago
Sandeep Sukhani	8a9abdfe16	Compactor improvements (#4018 ) * use previously compacted file as seed file and copy index into it * allow configuring compactor to compact multiple tables at a time	4 years ago
Sandeep Sukhani	1ed19f7380	add a metric for counting number of failures in opening existing files while loading tables from active index in uploads package (#3977 )	4 years ago
Sandeep Sukhani	1cca922e6d	fix flaky retention tests (#3976 )	4 years ago
Sandeep Sukhani	0cd8f31831	log name of the file failed to open during startup by ingester (#3970 )	4 years ago
Sandeep Sukhani	6a36cf6745	add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests (#3969 ) * add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests * check only for start time	4 years ago
Sandeep Sukhani	39ac2e58ad	fix a panic in compactor when retention is not enabled (#3967 )	4 years ago
Sandeep Sukhani	ae3e49973e	fix panic in compactor when retention is not enabled (#3966 )	4 years ago
Cyril Tovena	5be93e0480	Improve marker file current time metrics. (#3946 ) Currently we'll keep the last recorded time, but if the marker file doesn't exist anymore we should reset it. This would allow us to build consistent alert based on marker files that are too far behind. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	7a7eeb0d63	skip applying retention when data in table is still within retention period and there are no overlapping delete requests (#3934 )	4 years ago
Sandeep Sukhani	43c16ed8a1	remove boltdb files from ingesters on startup which do not have a index bucket (#3929 )	4 years ago
Cyril Tovena	22dcfcde33	Add retention label to loki_distributor_bytes_received_total metrics (#3840 ) * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/operations/observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Move ParseRequest to an apache2 package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Sandeep Sukhani	2c5ce2ce63	feat: index-gateway for boltdb-shipper index store (#3865 ) * feat: index-gateway for boltdb-shipper index store * jsonnet for deploying index-gateway * do not start a goroutine when query count is less than maxQueriesPerGoroutine * changes suggested from PR review	4 years ago
Sandeep Sukhani	d362df7a98	ignore download of missing boltdb files possibly removed during compaction (#3563 )	4 years ago
Cyril Tovena	7e658e8766	Fixes a flaky retention test. (#3833 ) This was due because the condition was using a string array that may be not in correct order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	cd38e987d8	Logs deletion fixes (#3827 ) * fix temp file suffix in delete requests table * log error on failure in removal of temp file only when the error is not os.IsNotExist * fix the cli flag for delete requests cancellation period	4 years ago
Sandeep Sukhani	c90366dbe7	just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled (#3785 ) * just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled * gofmt * Update msg Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Ed Welch <ed@oqqer.com>	4 years ago
Sandeep Sukhani	61b60092af	initialize retention and deletion components only when they are enabled (#3772 )	4 years ago
Sandeep Sukhani	966bed72d3	reduce allocs in delete requests manager by reusing slice for tracing non-deleted intervals for chunks (#3769 )	4 years ago
Cyril Tovena	89b32835c2	Improve mark file processing. (#3762 ) This makes boltdb batch for deleting entries smaller and so goroutines for deletion of chunks are not stuck. Also improve metrics to refresh more often. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	3b26c63c82	Boltdb shipper deletion fixes (#3746 ) * use return instead of break in select case * skip compaction on delete requests table	4 years ago
Cyril Tovena	e1a3ab841c	Improve retention mark files. (#3706 ) * Improve retention mark files. This PR is two fold: 1) Rotate mark files when they reached 100k marks, this is to ensure we max out marks file and don't create file that may be too big. 2) Instead of inserting marks using the chunk id as the key, we insert using a natural sequence. This has two benefits: - Keys are ordered, and so insertion are faster. - Allows us to use boltdb fill percent to 100%, this means boltdb won't over allocate for inserting key in between unordered data. Why ? I realize that when inserting huge amount of marks, this operation can take up to hours since boltdb has to re-allocate pages over and over. This is mainly because chunkid arrive without specific order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve key buf sizing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/storage/stores/shipper/compactor/retention/marker.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Copy the value so that it still available for the whole boltdb tx. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Sandeep Sukhani	2b7a6f201c	Logs deletion support for boltdb-shipper store (#3688 ) * add support for slicing the chunks for partial chunk deletion * add delete requests store and handler for requests * add delete requests manager and a way to check for chunk expiry based on delete requests * add chunk rewriter for partially deleted chunks * hook delete requests manager into compactor and update expiration checker * use chunk client in sweeper for deleting chunks * upload delete requests table during shutdown and some code refactoring * add tests and metrics * fixes post rebase * lint * changes suggested from PR review * avoid allocation when getting userid from chunkid * use bytes.IndexByte instead	4 years ago
Cyril Tovena	14fa7d8f3a	Removes file sync syscall for compaction. (#3693 ) Because we mostly download, read and upload but also because all those operations are idempotent we don't need to try to outsmart the OS by forcefully syncing the file to disk. The OS will use cache on files and we'll most likely not suffer from high disk io. This was a conclusion I made after some digging of why the operation of compaction+retention was a bit slow sometimes. (+5s) I've tried the PR in dev and the whole process execute now below 40s whereas because it was 4min. strace before: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 75.75 12.333637 2122 5812 561 futex 10.81 1.760698 242 7267 nanosleep 7.69 1.252697 88 14287 epoll_pwait 2.86 0.464888 22 21608 write 1.08 0.175313 19479 9 fsync <---------- 1.06 0.172173 15 11249 4334 read 0.40 0.064422 2147 30 15 unlinkat 0.15 0.023945 19 1246 madvise 0.09 0.015288 1529 10 munmap 0.04 0.006098 23 263 tgkill 0.03 0.004134 16 263 getpid ``` strace after: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 77.01 58.085485 4599 12631 840 futex 9.81 7.395901 230 32145 nanosleep 8.34 6.287953 175 35882 epoll_pwait 3.33 2.512512 18 141141 write 0.79 0.596351 16 37911 13536 read 0.51 0.382579 3985 96 48 unlinkat 0.08 0.058465 1720 34 munmap 0.04 0.031904 26 1219 tgkill 0.03 0.023962 22 1082 madvise 0.02 0.013831 11 1219 getpid 0.01 0.011042 9 1219 12 rt_sigreturn 0.01 0.005348 111 48 fdatasync 0.01 0.003785 36 105 openat 0.00 0.003603 8 435 pwrite64 0.00 0.003005 13 229 200 epoll_ctl 0.00 0.001828 19 96 34 newfstatat ``` As you can see the fsync and write are less intensive after. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	5c4ba10fb9	Use model.Duration for easy yaml/json marshalling. (#3687 ) * Use model.Duration for easy yaml/json marshalling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * test fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	1995eaa426	Pass in the now value to the retention. (#3673 ) This allows to call model.Time only once per table which is reasonable for accuracy and this will save some cpu cycle. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	aa544a9dff	cleanup boltdb files failing to open during loading tables which are possibly corrupt (#3605 )	4 years ago
Cyril Tovena	a454f7edb7	Use pgzip in the compactor. (#3672 ) Since index file are expected to be large, this will help to compress/decompress in parallel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	43184298b2	Trigger compaction prior retention. (#3665 ) * Trigger compaction prior retention. This should allow to process table where the amount of index files is below 4. Currently they are not processed and we should process them since they may never be compacted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook retention into compaction. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove unused flags. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better cancellation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing ctx in test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flags that are not used. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add compactor retention tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	806d6a503e	Custom Retention (#3642 ) * Playing around with a POC based on the design doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Getting series ID for label matchers is now working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip/ Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * We're parsing the index label now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting to extract interfaces to make the code testable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress will try to add labels to chunk ref iterator Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Iterator for chunks ref with all labels !!!. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Optimize code away. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * with delete into the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleaner but not yet working for v10 and v11. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes series cleaner. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hooking into the compactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking limit retention config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off the marker processor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on sweeper and fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding more tests and founding more bugs along the way. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Bug with path once boltdb is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more bug and more robust test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More test and cleanup getting close. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar bugs with regards to period schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky tests because of boltdb still open. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Save files without using table key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve logging and ability to use more goroutines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes duplicate metrics since histogram contains total too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock bug when too many workers are trying to update the mark file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock when reading and updating db at the same time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes default config test of boltdb. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * PR Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the user of assert to not fail a test if it's incorrect on the first shot. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add experimental notice to the flag documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes empty index detection and table deletion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not delete folder it's not necessary with object store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better working path cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	835f710bc9	make prefix for keys of objects created by boltdb-shipper configurable (#3491 ) * make prefix for keys of objects created by boltdb-shipper configurable * shared store key prefix should have a path separator	4 years ago
Ed Welch	877f524c36	Loki: Revert reduce list operations and crash after timeout (#3396 ) * revert: boltdbshipper: fix crash of "timeout" panic when boltdbshipper do init (#3369) This appears to have introduced a different problem, still investigating. * Revert "reduce number of list calls on shared object store when using boltdb-shipper (#3283)" This reverts commit `76e713f5`	4 years ago
Ed Welch	cf8be4a55e	remove directories from object store list operation (#3394 )	4 years ago
zhanghjster	a3a2c1a079	boltdbshipper: fix crash of "timeout" panic when boltdbshipper do init (#3369 ) * Fix timeout panic during boltdbshipper init. When boltdbshipper failed to load an invalid local index file, it try to reopen with a newly downloaded index, but it will panic with "timeout" error because the process still hold the flock created by the previous db open. This patch do the flock cleanup. * add a few more comments and change function name. Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Sandeep Sukhani	ba25b0fc80	increase db retain period in ingesters to cover index cache validity period as well (#3300 ) refactor ingester store query config to simplify the code	4 years ago
Sandeep Sukhani	3ff088173f	dedupe index on all the queries for a table instead of query batches (#3338 ) * dedupe index on all the queries for a table instead query batches * convert sync.Mutex to sync.RWMutex in deduper to allow concurrent checks for isSeen	4 years ago
Sandeep Sukhani	76e713f57d	reduce number of list calls on shared object store when using boltdb-shipper (#3283 )	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295 ) * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256 ) * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Sandeep Sukhani	ae9c4b82ec	fix table deletion in table client for boltdb-shipper (#2960 )	5 years ago
Ed Welch	9ba5753e25	Loki: handle faults when opening boltdb files (#2988 ) * we also need to catch faults not just panics when opening boltdb files * fix lint * i'll get it one of these times * fix typo	5 years ago
Ed Welch	6d442673b7	adding a function to safely open boltdb files and recover from any panics thrown by boltdb when opening a file. (#2983 )	5 years ago
Sandeep Sukhani	cd723c0479	Boltdb shipper query readiness (#2911 ) * add support for downloading index on startup in boltdb-shipper for query readiness * tests * pass context to load tables function to be able to cancel the operation * add missing call to tablemanager.stop when loading of tables fail	5 years ago
Sandeep Sukhani	fd451d97f4	fix flaky load tables test for boltdb-shipper uploads table-manager (#2891 )	5 years ago
Owen Diehl	9249ebc022	Fix/validate compactor config (#2824 ) * default compactor config throws error when compactor is a target or subtarget * linting * moves compactor config validation to NewCompactor fn	5 years ago
Cyril Tovena	e6778e9827	Update to go 1.15 (#2877 ) * Upgrade to 1.15. Also upgrade linter and fixes some lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar lint 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moar lint and tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	feb66d62cf	fix flaky tests in boltdb-shipper (#2831 )	5 years ago
Sandeep Sukhani	2793499309	Boltdb shipper metrics changes (#2790 ) * rename boltdb shipper metrics to have table in their name instead of files * Increment tables_sync_operation_total metric during periodic syncs as well * tweak error handling Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Sandeep Sukhani	7b7b1844ca	add a metric in compactor to record timestamp of last successful run (#2788 )	5 years ago
Sandeep Sukhani	f60554ce91	Boltdb shipper query performance improvements (#2770 ) * boltdb-shipper query with cursor to do multiple queries in a single transaction * index deduper * use fifo cache in ingesters * snapshot dbs in ingester for reads * retain fewer files in ingesters for queries * set timeout in cache config instead of index read cache config which saves us from unmarshalling the object for checking the expiry * add comment about fifo cache invalidation * tweaking a few comments Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago

1 2

83 Commits (14f8faf544a25681a9e7f22cdc9d950cde4241d8)