apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071 )	9 months ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Sandeep Sukhani	c716e498ad	compactor: do not block compation when retention is taking too long (#9884 ) What this PR does / why we need it: Currently, we perform compaction and apply retention in the same loop. Although we have a flag for not applying retention every time we perform compaction, we still see compaction getting blocked when processing some intensive delete requests(processed while applying retention). This PR separates out the compaction and retention to run in a separate loop. I have added a table-locking feature to avoid compaction and retention from processing the same tables at a time. However, compaction and retention would treat locked tables differently, as explained below: * When compaction sees a table is locked: It would skip the table and move on to the following table. However, before skipping, it would check if the table has any uncompacted files and increment the newly added counter called `loki_compactor_skipped_compacting_locked_tables_total` to track how often we are skipping compacting tables which have uncompacted files. * When retention sees a table is locked: It would wait for the lock to be released since we can't skip any tables while processing delete requests. Special notes for your reviewer: * The check for tables with uncompacted files looks for count > 1 because initially, we did not support per tenant index in `boltdb-shipper`, so a table can have a single compacted multi-tenant index file. In a rare case where we have a single file which was supposed to be compacted away, it is okay to have a single uncompacted file for a while. The aim here is to not block compaction for too long in a large cell with too many uncompacted files. * Retention only works on the compacted index, so we first compact down any uncompacted files while applying retention. Checklist - [x] Tests updated --------- Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Ashwanth	91cae03fe1	chore: restructure compactor package (#10775 ) What this PR does / why we need it: This pr moves the compactor pkg from `pkg/storage/stores/shipper/indexshipper/compactor` to `pkg/compactor` - same level as other high level components (querier, distributor...) table compaction code however resides in the respective store pkg closer to other store specific code. `pkg/storage/stores/shipper/indexshipper/boltdb/compactor/table_compactor.go` `pkg/storage/stores/shipper/indexshipper/tsdb/compactor.go` Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Christian Haudum	2e775f3f44	chore(storage): Restructure shipper packages (#10729 ) What this PR does / why we need it: This PR restructures the code for the shipper component of the Loki storage layer. New package layout: ```console $ tree -d pkg/storage/stores/shipper pkg/storage/stores/shipper └── indexshipper ├── boltdb │ └── compactor ├── compactor │ ├── client │ │ └── grpc │ ├── deletion │ ├── deletionmode │ ├── generationnumber │ └── retention ├── downloads ├── gatewayclient ├── index ├── indexgateway ├── storage ├── testutil ├── tsdb │ ├── index │ ├── testdata │ └── testutil ├── uploads └── util 23 directories ``` * TSDB and BoltDB specific code is under `./pkg/storage/stores/shipper/indexshipper/tsdb` and `./pkg/storage/stores/shipper/indexshipper/boltdb` respectively. * Common code for both TSDB and BoltDB is directly under `./pkg/storage/stores/shipper/indexshipper` and subdirectories, such as `uploads/`, `downloads/`, `compactor/`, ... Special notes for your reviewer: This PR is identical to https://github.com/grafana/loki/pull/10724 except of the package `pkg/storage/stores/indexshipper` the package is `pkg/storage/stores/shipper/indexshipper` (one level deeper). Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Sandeep Sukhani	2ed3c15c22	index: allow refreshing index file names cache for specific tables (#9798 ) What this PR does / why we need it: We have a cache for index files available in the object storage, which helps reduce the number of list calls we make to the object storage. Currently, we only support refreshing the whole index cache, which includes all the files for all tables and users. However, in a large cell with many tables and tenants, it is less likely that all the tables are being referenced. Also, most iterative operations would only need a fresh listing of just a few tables at a time so it is wasteful to always refresh the whole index cache. This PR adds support for refreshing the index cache for only specific tables. I am doing it as part of some compactor improvements to run compaction and retention in separate goroutines to avoid blocking compaction when some expensive delete requests are being processed. To make those changes, I needed a way to cost-effectively refresh the index files in the table being picked up for processing. Checklist - [x] Tests updated	2 years ago
afayngelerindbx	4b5b16569d	compactor: adds tunable to control upload parallelism (#6817 ) What this PR does / why we need it: This can help with compaction latency. I patched the compactor to set this to 100. Which issue(s) this PR fixes: Helps with #6775 Checklist - [x] Documentation added	3 years ago
Sandeep Sukhani	882010a810	Index shipper move code (#6704 ) * move compactor code to generic indexshipper * move table client to generic indexshipper * move storage package and relevant code for managing files on storage to generic indexshipper * add missing files * lint	3 years ago
Sandeep Sukhani	25912ea02d	generic index compactor (#6537 ) * make compactor generic to support compacting various index types * implement and register compaction for boltdb-shipper index type * lint and fix broken test * add more comments * fix some code and tests * remove duplicate user-id kv in logger * some improvements in code and test cases * lint	3 years ago
Sandeep Sukhani	2758dc659a	boltdb shipper index list cache improvements (#6054 ) * bypass index list cache when doing query time downloading of index * detect and refresh stale index list cache during sync	3 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833 ) * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Sandeep Sukhani	d2ce08280f	Compactor improvements (#5587 ) * upload and cleanup index sets concurrently * reuse batch slice used for building a batch of index during compaction	3 years ago
Sandeep Sukhani	7f498e9617	cleanup common index source files at the end while compacting index with boltdb-shipper (#5585 )	3 years ago
Sandeep Sukhani	c76544baf9	build a new batch while starting to process a bucket (#5580 )	3 years ago
Sandeep Sukhani	e65d1b8c69	add more context to logs, improve comments for clarity in boltdb-shipper code (#5341 )	3 years ago
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361 ) * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Sandeep Sukhani	d87f2a1c8a	per user index changes on the write path (#5193 ) * per user index changes on the write path - add a flag to enable building per user index from the ingesters - when enabled, ingesters would build boltdb files with buckets per tenant	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Sandeep Sukhani	88b5a767bc	changes on read path for supporting per user index with boltdb-shipper (#5073 )	3 years ago
Sandeep Sukhani	ff47d7419a	compactor changes for building per user index files in boltdb shipper (#5026 ) * compactor changes for building per user index files in boltdb shipper * update comments * lint and changes suggested from PR review * fix broken code after merging main	3 years ago
Kaviraj	14afb13d36	Chore: Copy `cortex/util/math` into Loki (#5036 ) * Copy `cortex/pkg/util/math` into Loki. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Move `MinUint32` from `util` package into `util/math` package Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove NewEWMARate and related helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove unwanted spaces Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix spaces in import paths Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Sandeep Sukhani	3ac9818f82	recreate compacted boltdb files from compactor when they are more than 12 hours old (#4853 )	4 years ago
Sandeep Sukhani	2564a810c5	Logs deletion fixes (#4625 ) * check all tables when we have delete requests to process since chunks could span multiple tables * upload modified index when the it is modified without dropping the chunk * fix broken test * lint * Fix modified check Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * suggested changes from PR review * add changelog Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484 ) * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Sandeep Sukhani	8ef3d5fee9	add a storage client for boltdb-shipper which would do all the object key management for storage operations (#4128 ) * add a storage client for boltdb-shipper which would do all the object key management for storage operations * changes suggested from PR review	4 years ago
Cyril Tovena	32e55c3043	Fork cortex chunk storage into Loki. (#3842 ) The chunk package has been forked from this commit for future reference and licensing. `09fe17c0e6`	4 years ago
Sandeep Sukhani	8a9abdfe16	Compactor improvements (#4018 ) * use previously compacted file as seed file and copy index into it * allow configuring compactor to compact multiple tables at a time	4 years ago
Sandeep Sukhani	7a7eeb0d63	skip applying retention when data in table is still within retention period and there are no overlapping delete requests (#3934 )	4 years ago
Sandeep Sukhani	43c16ed8a1	remove boltdb files from ingesters on startup which do not have a index bucket (#3929 )	4 years ago
Cyril Tovena	14fa7d8f3a	Removes file sync syscall for compaction. (#3693 ) Because we mostly download, read and upload but also because all those operations are idempotent we don't need to try to outsmart the OS by forcefully syncing the file to disk. The OS will use cache on files and we'll most likely not suffer from high disk io. This was a conclusion I made after some digging of why the operation of compaction+retention was a bit slow sometimes. (+5s) I've tried the PR in dev and the whole process execute now below 40s whereas because it was 4min. strace before: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 75.75 12.333637 2122 5812 561 futex 10.81 1.760698 242 7267 nanosleep 7.69 1.252697 88 14287 epoll_pwait 2.86 0.464888 22 21608 write 1.08 0.175313 19479 9 fsync <---------- 1.06 0.172173 15 11249 4334 read 0.40 0.064422 2147 30 15 unlinkat 0.15 0.023945 19 1246 madvise 0.09 0.015288 1529 10 munmap 0.04 0.006098 23 263 tgkill 0.03 0.004134 16 263 getpid ``` strace after: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 77.01 58.085485 4599 12631 840 futex 9.81 7.395901 230 32145 nanosleep 8.34 6.287953 175 35882 epoll_pwait 3.33 2.512512 18 141141 write 0.79 0.596351 16 37911 13536 read 0.51 0.382579 3985 96 48 unlinkat 0.08 0.058465 1720 34 munmap 0.04 0.031904 26 1219 tgkill 0.03 0.023962 22 1082 madvise 0.02 0.013831 11 1219 getpid 0.01 0.011042 9 1219 12 rt_sigreturn 0.01 0.005348 111 48 fdatasync 0.01 0.003785 36 105 openat 0.00 0.003603 8 435 pwrite64 0.00 0.003005 13 229 200 epoll_ctl 0.00 0.001828 19 96 34 newfstatat ``` As you can see the fsync and write are less intensive after. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	43184298b2	Trigger compaction prior retention. (#3665 ) * Trigger compaction prior retention. This should allow to process table where the amount of index files is below 4. Currently they are not processed and we should process them since they may never be compacted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook retention into compaction. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove unused flags. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better cancellation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing ctx in test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flags that are not used. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add compactor retention tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	806d6a503e	Custom Retention (#3642 ) * Playing around with a POC based on the design doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Getting series ID for label matchers is now working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip/ Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * We're parsing the index label now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting to extract interfaces to make the code testable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress will try to add labels to chunk ref iterator Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Iterator for chunks ref with all labels !!!. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Optimize code away. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * with delete into the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleaner but not yet working for v10 and v11. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes series cleaner. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hooking into the compactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking limit retention config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off the marker processor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on sweeper and fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding more tests and founding more bugs along the way. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Bug with path once boltdb is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more bug and more robust test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More test and cleanup getting close. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar bugs with regards to period schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky tests because of boltdb still open. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Save files without using table key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve logging and ability to use more goroutines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes duplicate metrics since histogram contains total too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock bug when too many workers are trying to update the mark file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock when reading and updating db at the same time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes default config test of boltdb. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * PR Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the user of assert to not fail a test if it's incorrect on the first shot. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add experimental notice to the flag documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes empty index detection and table deletion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not delete folder it's not necessary with object store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better working path cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	cf8be4a55e	remove directories from object store list operation (#3394 )	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295 ) * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256 ) * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Ed Welch	6d442673b7	adding a function to safely open boltdb files and recover from any panics thrown by boltdb when opening a file. (#2983 )	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	54c0c5febb	Revendor Cortex (#2755 ) * replaces prometheus & revendors cortex * compat * compat * sd refactor * updates docker * removes prom replace * adds delimiters back to object addrs in boltdb	5 years ago
Davor Kapsa	fffcfa6e81	fix some misspells (#2725 )	5 years ago
Sandeep Sukhani	2bb2eebace	register boltdb shipper compactor cli flags (#2546 )	5 years ago
Sandeep Sukhani	b489130828	compactor for compacting boltdb files uploaded by shipper (#2526 )	5 years ago

4 Commits (9fb2f385529c2444be98161743aa0cd8ad420bf2)