152 Commits (4c2feb32b55dc0431aff659607b67e4ac078325e)

Author	SHA1	Message	Date
Cyril Tovena	4c2feb32b5	Fixes test helper `newChunk` in the storage pkg. (#4056) ... I wanted to test that overlapping chunks are correctly re-ordered and bump into a rounding issue. Keeping both for future will avoid the same mistake. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	8a9abdfe16	Compactor improvements (#4018) ... * use previously compacted file as seed file and copy index into it * allow configuring compactor to compact multiple tables at a time	4 years ago
Sandeep Sukhani	1ed19f7380	add a metric for counting number of failures in opening existing files while loading tables from active index in uploads package (#3977)	4 years ago
Sandeep Sukhani	1cca922e6d	fix flaky retention tests (#3976)	4 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973) ... This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Sandeep Sukhani	0cd8f31831	log name of the file failed to open during startup by ingester (#3970)	4 years ago
Sandeep Sukhani	6a36cf6745	add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests (#3969) ... * add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests * check only for start time	4 years ago
Sandeep Sukhani	39ac2e58ad	fix a panic in compactor when retention is not enabled (#3967)	4 years ago
Sandeep Sukhani	ae3e49973e	fix panic in compactor when retention is not enabled (#3966)	4 years ago
Cyril Tovena	bee456737e	Shards Series API. (#3856) ... * Shards Series API. This PR introduces Series Queries Sharding. It does not check the boundaries of ingesters data since I'm assuming https://github.com/grafana/loki/pull/3852 will be merge first. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	5be93e0480	Improve marker file current time metrics. (#3946) ... Currently we'll keep the last recorded time, but if the marker file doesn't exist anymore we should reset it. This would allow us to build consistent alert based on marker files that are too far behind. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	7a7eeb0d63	skip applying retention when data in table is still within retention period and there are no overlapping delete requests (#3934)	4 years ago
Sandeep Sukhani	43c16ed8a1	remove boltdb files from ingesters on startup which do not have a index bucket (#3929)	4 years ago
Cyril Tovena	22dcfcde33	Add retention label to loki_distributor_bytes_received_total metrics (#3840) ... * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/operations/observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Move ParseRequest to an apache2 package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Sandeep Sukhani	2c5ce2ce63	feat: index-gateway for boltdb-shipper index store (#3865) ... * feat: index-gateway for boltdb-shipper index store * jsonnet for deploying index-gateway * do not start a goroutine when query count is less than maxQueriesPerGoroutine * changes suggested from PR review	4 years ago
Sandeep Sukhani	d362df7a98	ignore download of missing boltdb files possibly removed during compaction (#3563)	4 years ago
Cyril Tovena	7e658e8766	Fixes a flaky retention test. (#3833) ... This was due because the condition was using a string array that may be not in correct order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	cd38e987d8	Logs deletion fixes (#3827) ... * fix temp file suffix in delete requests table * log error on failure in removal of temp file only when the error is not os.IsNotExist * fix the cli flag for delete requests cancellation period	4 years ago
Sandeep Sukhani	c90366dbe7	just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled (#3785) ... * just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled * gofmt * Update msg Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Ed Welch <ed@oqqer.com>	4 years ago
Sandeep Sukhani	61b60092af	initialize retention and deletion components only when they are enabled (#3772)	4 years ago
Sandeep Sukhani	966bed72d3	reduce allocs in delete requests manager by reusing slice for tracing non-deleted intervals for chunks (#3769)	4 years ago
Cyril Tovena	89b32835c2	Improve mark file processing. (#3762) ... This makes boltdb batch for deleting entries smaller and so goroutines for deletion of chunks are not stuck. Also improve metrics to refresh more often. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	3b26c63c82	Boltdb shipper deletion fixes (#3746) ... * use return instead of break in select case * skip compaction on delete requests table	4 years ago
Cyril Tovena	e1a3ab841c	Improve retention mark files. (#3706) ... * Improve retention mark files. This PR is two fold: 1) Rotate mark files when they reached 100k marks, this is to ensure we max out marks file and don't create file that may be too big. 2) Instead of inserting marks using the chunk id as the key, we insert using a natural sequence. This has two benefits: - Keys are ordered, and so insertion are faster. - Allows us to use boltdb fill percent to 100%, this means boltdb won't over allocate for inserting key in between unordered data. Why ? I realize that when inserting huge amount of marks, this operation can take up to hours since boltdb has to re-allocate pages over and over. This is mainly because chunkid arrive without specific order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve key buf sizing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/storage/stores/shipper/compactor/retention/marker.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Copy the value so that it still available for the whole boltdb tx. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Sandeep Sukhani	2b7a6f201c	Logs deletion support for boltdb-shipper store (#3688) ... * add support for slicing the chunks for partial chunk deletion * add delete requests store and handler for requests * add delete requests manager and a way to check for chunk expiry based on delete requests * add chunk rewriter for partially deleted chunks * hook delete requests manager into compactor and update expiration checker * use chunk client in sweeper for deleting chunks * upload delete requests table during shutdown and some code refactoring * add tests and metrics * fixes post rebase * lint * changes suggested from PR review * avoid allocation when getting userid from chunkid * use bytes.IndexByte instead	4 years ago
Cyril Tovena	210dc355cb	Fixes a goroutine leak in the store when doing cancellation. (#3733) ... Found out that we might have no listener and trying to send the information of cancellation. I've moved the error information down the respective log and sample iterators so that we don't need to send it via the channel. This seems to have cause some pretty big memory leak on our side. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	14fa7d8f3a	Removes file sync syscall for compaction. (#3693) ... Because we mostly download, read and upload but also because all those operations are idempotent we don't need to try to outsmart the OS by forcefully syncing the file to disk. The OS will use cache on files and we'll most likely not suffer from high disk io. This was a conclusion I made after some digging of why the operation of compaction+retention was a bit slow sometimes. (+5s) I've tried the PR in dev and the whole process execute now below 40s whereas because it was 4min. strace before: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 75.75 12.333637 2122 5812 561 futex 10.81 1.760698 242 7267 nanosleep 7.69 1.252697 88 14287 epoll_pwait 2.86 0.464888 22 21608 write 1.08 0.175313 19479 9 fsync <---------- 1.06 0.172173 15 11249 4334 read 0.40 0.064422 2147 30 15 unlinkat 0.15 0.023945 19 1246 madvise 0.09 0.015288 1529 10 munmap 0.04 0.006098 23 263 tgkill 0.03 0.004134 16 263 getpid ``` strace after: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 77.01 58.085485 4599 12631 840 futex 9.81 7.395901 230 32145 nanosleep 8.34 6.287953 175 35882 epoll_pwait 3.33 2.512512 18 141141 write 0.79 0.596351 16 37911 13536 read 0.51 0.382579 3985 96 48 unlinkat 0.08 0.058465 1720 34 munmap 0.04 0.031904 26 1219 tgkill 0.03 0.023962 22 1082 madvise 0.02 0.013831 11 1219 getpid 0.01 0.011042 9 1219 12 rt_sigreturn 0.01 0.005348 111 48 fdatasync 0.01 0.003785 36 105 openat 0.00 0.003603 8 435 pwrite64 0.00 0.003005 13 229 200 epoll_ctl 0.00 0.001828 19 96 34 newfstatat ``` As you can see the fsync and write are less intensive after. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	5c4ba10fb9	Use model.Duration for easy yaml/json marshalling. (#3687) ... * Use model.Duration for easy yaml/json marshalling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * test fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	1995eaa426	Pass in the now value to the retention. (#3673) ... This allows to call model.Time only once per table which is reasonable for accuracy and this will save some cpu cycle. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	aa544a9dff	cleanup boltdb files failing to open during loading tables which are possibly corrupt (#3605)	4 years ago
Cyril Tovena	490c19e38c	Deprecate max_look_back_period in the chunk storage. (#3677) ... * Deprecate max_look_back_period in the chunk storage. From now on it will be applied at the frontend, ruler and querier level. The querier validation was missing so I added it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a454f7edb7	Use pgzip in the compactor. (#3672) ... Since index file are expected to be large, this will help to compress/decompress in parallel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	43184298b2	Trigger compaction prior retention. (#3665) ... * Trigger compaction prior retention. This should allow to process table where the amount of index files is below 4. Currently they are not processed and we should process them since they may never be compacted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook retention into compaction. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove unused flags. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better cancellation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing ctx in test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flags that are not used. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add compactor retention tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	806d6a503e	Custom Retention (#3642) ... * Playing around with a POC based on the design doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Getting series ID for label matchers is now working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip/ Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * We're parsing the index label now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting to extract interfaces to make the code testable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress will try to add labels to chunk ref iterator Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Iterator for chunks ref with all labels !!!. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Optimize code away. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * with delete into the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleaner but not yet working for v10 and v11. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes series cleaner. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hooking into the compactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking limit retention config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off the marker processor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on sweeper and fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding more tests and founding more bugs along the way. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Bug with path once boltdb is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more bug and more robust test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More test and cleanup getting close. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar bugs with regards to period schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky tests because of boltdb still open. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Save files without using table key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve logging and ability to use more goroutines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes duplicate metrics since histogram contains total too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock bug when too many workers are trying to update the mark file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock when reading and updating db at the same time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes default config test of boltdb. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * PR Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the user of assert to not fail a test if it's incorrect on the first shot. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add experimental notice to the flag documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes empty index detection and table deletion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not delete folder it's not necessary with object store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better working path cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	6bcd9807d6	replace `time.Duration` -> `model.Duration` for `Limits`. (#3632) ... Rationale 1. support user-friendly duration format (e.g: "1h30m45s") in JSON value. 2. In consistent with Cortex and tempo.	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623) ... * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Cyril Tovena	a22bce7286	Add a chunk filter hook in the store. (#3569) ... * Add a chunk filter hook in the store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	835f710bc9	make prefix for keys of objects created by boltdb-shipper configurable (#3491) ... * make prefix for keys of objects created by boltdb-shipper configurable * shared store key prefix should have a path separator	4 years ago
Ed Welch	877f524c36	Loki: Revert reduce list operations and crash after timeout (#3396) ... * revert: boltdbshipper: fix crash of "timeout" panic when boltdbshipper do init (#3369) This appears to have introduced a different problem, still investigating. * Revert "reduce number of list calls on shared object store when using boltdb-shipper (#3283)" This reverts commit 76e713f5	4 years ago
Ed Welch	cf8be4a55e	remove directories from object store list operation (#3394)	4 years ago
zhanghjster	a3a2c1a079	boltdbshipper: fix crash of "timeout" panic when boltdbshipper do init (#3369) ... * Fix timeout panic during boltdbshipper init. When boltdbshipper failed to load an invalid local index file, it try to reopen with a newly downloaded index, but it will panic with "timeout" error because the process still hold the flock created by the previous db open. This patch do the flock cleanup. * add a few more comments and change function name. Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Sandeep Sukhani	ba25b0fc80	increase db retain period in ingesters to cover index cache validity period as well (#3300) ... refactor ingester store query config to simplify the code	4 years ago
Sandeep Sukhani	3ff088173f	dedupe index on all the queries for a table instead of query batches (#3338) ... * dedupe index on all the queries for a table instead query batches * convert sync.Mutex to sync.RWMutex in deduper to allow concurrent checks for isSeen	4 years ago
Sandeep Sukhani	76e713f57d	reduce number of list calls on shared object store when using boltdb-shipper (#3283)	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295) ... * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256) ... * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Sandeep Sukhani	db14ff193c	limit query range in async store for ingesters when query-ingesters-within flag is set (#3201) ... * limit query range in async store for ingesters when query-ingesters-within flag is set * do not reduce ingester query start time	4 years ago
Cyril Tovena	c98dfa0da1	Fixes a swallowed context err in the batch storage. (#3025) ... Fixes #2999 There's still a potential cases where, the context is done (and so the Err is not nil) and we're finished with iterating over the list of batches. I didn't introduce the error here because I think I would prefer a success in this case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	ae9c4b82ec	fix table deletion in table client for boltdb-shipper (#2960)	5 years ago
Ed Welch	9ba5753e25	Loki: handle faults when opening boltdb files (#2988) ... * we also need to catch faults not just panics when opening boltdb files * fix lint * i'll get it one of these times * fix typo	5 years ago