apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Arve Knudsen	cfb4fc1f55	Use flagext from dskit (#4225 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Nick Pillitteri	79a7bdbabc	Chore: Use services and modules from grafana/dskit (#4196 ) * Update Cortex vendor to bcf15611345da65e03604a08a0213aaaaac49e54 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Use services and modules packages from grafana/dskit Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Update promtail test to set default weaveworks server network type Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Punctuation Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Benoît Knecht	488a21bff7	pkg/storage/chunk/aws: Add s3.http.ca-file option (#4211 ) * pkg/storage/chunk/aws: Fix insecure-skip-verify documentation The documentation claimed that the `s3.http.insecure-skip-verify` option needed to be set to `false` in order to skip verifying certificates, but it needs to be set to `true`. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch> * pkg/storage/chunk/aws: Add s3.http.ca-file option This option lets users set a custom CA file for connections to S3, in case they use a local S3 instance with an internal PKI. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>	4 years ago
Arve Knudsen	10c92a901e	Chore: Upgrade to latest Cortex (#4188 ) * Chore: Upgrade to latest Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix formatting Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Sandeep Sukhani	29b07d190d	Retention speedup (#4172 ) * drop the multi table chunk entries from index when the whole table is out of retention * speed up cleaning of deleted series from index * mark partially deleted chunk for deletion when all its tables are processed	4 years ago
Sandeep Sukhani	8ef3d5fee9	add a storage client for boltdb-shipper which would do all the object key management for storage operations (#4128 ) * add a storage client for boltdb-shipper which would do all the object key management for storage operations * changes suggested from PR review	4 years ago
Sandeep Sukhani	67fe2d68a3	fix finding tables which would have out of retention data (#4107 )	4 years ago
Sandeep Sukhani	138ca71dd5	add a method to check whether error is due to a missing object in storage while reading or deleting the object (#4081 )	4 years ago
Cyril Tovena	ac6ace8dbd	Make `ParseExternalKey` from the chunk package alloc free. (#4075 ) ``` ~/go/src/github.com/grafana/loki main* ⇣ ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_ParseExternalKey-16 569 415 -27.03% benchmark old allocs new allocs delta Benchmark_ParseExternalKey-16 2 0 -100.00% benchmark old bytes new bytes delta Benchmark_ParseExternalKey-16 96 0 -100.00% ``` This is a very hot function in Loki being used for every chunk reference we get from the index. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	19aec6e6a3	fix broken TestCachingStorageClientStoreQueries (#4074 )	5 years ago
Sandeep Sukhani	d31f8676fd	Optimize index queries and caching (#4070 ) * Use RangeValuePrefix for building queries for Series IDs * optimize index store queries and caching by not doing broad queries	5 years ago
Cyril Tovena	32e55c3043	Fork cortex chunk storage into Loki. (#3842 ) The chunk package has been forked from this commit for future reference and licensing. `09fe17c0e6`	5 years ago
Owen Diehl	56256bfd5b	Unordered chunks (#4001 ) * merge feature/unordered-replay * interoperable head chunks * memchunk block interop * retain ordered memchunk optimizations when possible * tests+bench for unordered chunk reads * reorder on chunk close * [wip] ingester stream unorderd * unordered writes default in testware config, fixes OOO bug & removes unused lastChunkTimestamp var * validity window is 1/2 of max age & fixes old transfer test * more consistent headblock checking/creation * more cohesive encoding tests * unordered stream test with validity bounds * compat - unordered * reinstates memchunk defaults when rebounding & updates storage test compatibility * lint * reorder across blocks doesnt overflow * respect chunk configs during rebounding when possible * only sync checks on ordered writes	5 years ago
Cyril Tovena	4c2feb32b5	Fixes test helper `newChunk` in the storage pkg. (#4056 ) I wanted to test that overlapping chunks are correctly re-ordered and bump into a rounding issue. Keeping both for future will avoid the same mistake. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	8a9abdfe16	Compactor improvements (#4018 ) * use previously compacted file as seed file and copy index into it * allow configuring compactor to compact multiple tables at a time	5 years ago
Sandeep Sukhani	1ed19f7380	add a metric for counting number of failures in opening existing files while loading tables from active index in uploads package (#3977 )	5 years ago
Sandeep Sukhani	1cca922e6d	fix flaky retention tests (#3976 )	5 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973 ) This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	5 years ago
Sandeep Sukhani	0cd8f31831	log name of the file failed to open during startup by ingester (#3970 )	5 years ago
Sandeep Sukhani	6a36cf6745	add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests (#3969 ) * add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests * check only for start time	5 years ago
Sandeep Sukhani	39ac2e58ad	fix a panic in compactor when retention is not enabled (#3967 )	5 years ago
Sandeep Sukhani	ae3e49973e	fix panic in compactor when retention is not enabled (#3966 )	5 years ago
Cyril Tovena	bee456737e	Shards Series API. (#3856 ) * Shards Series API. This PR introduces Series Queries Sharding. It does not check the boundaries of ingesters data since I'm assuming https://github.com/grafana/loki/pull/3852 will be merge first. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	5be93e0480	Improve marker file current time metrics. (#3946 ) Currently we'll keep the last recorded time, but if the marker file doesn't exist anymore we should reset it. This would allow us to build consistent alert based on marker files that are too far behind. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	7a7eeb0d63	skip applying retention when data in table is still within retention period and there are no overlapping delete requests (#3934 )	5 years ago
Sandeep Sukhani	43c16ed8a1	remove boltdb files from ingesters on startup which do not have a index bucket (#3929 )	5 years ago
Cyril Tovena	22dcfcde33	Add retention label to loki_distributor_bytes_received_total metrics (#3840 ) * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/operations/observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Move ParseRequest to an apache2 package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	5 years ago
Sandeep Sukhani	2c5ce2ce63	feat: index-gateway for boltdb-shipper index store (#3865 ) * feat: index-gateway for boltdb-shipper index store * jsonnet for deploying index-gateway * do not start a goroutine when query count is less than maxQueriesPerGoroutine * changes suggested from PR review	5 years ago
Sandeep Sukhani	d362df7a98	ignore download of missing boltdb files possibly removed during compaction (#3563 )	5 years ago
Cyril Tovena	7e658e8766	Fixes a flaky retention test. (#3833 ) This was due because the condition was using a string array that may be not in correct order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	cd38e987d8	Logs deletion fixes (#3827 ) * fix temp file suffix in delete requests table * log error on failure in removal of temp file only when the error is not os.IsNotExist * fix the cli flag for delete requests cancellation period	5 years ago
Sandeep Sukhani	c90366dbe7	just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled (#3785 ) * just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled * gofmt * Update msg Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Ed Welch <ed@oqqer.com>	5 years ago
Sandeep Sukhani	61b60092af	initialize retention and deletion components only when they are enabled (#3772 )	5 years ago
Sandeep Sukhani	966bed72d3	reduce allocs in delete requests manager by reusing slice for tracing non-deleted intervals for chunks (#3769 )	5 years ago
Cyril Tovena	89b32835c2	Improve mark file processing. (#3762 ) This makes boltdb batch for deleting entries smaller and so goroutines for deletion of chunks are not stuck. Also improve metrics to refresh more often. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	3b26c63c82	Boltdb shipper deletion fixes (#3746 ) * use return instead of break in select case * skip compaction on delete requests table	5 years ago
Cyril Tovena	e1a3ab841c	Improve retention mark files. (#3706 ) * Improve retention mark files. This PR is two fold: 1) Rotate mark files when they reached 100k marks, this is to ensure we max out marks file and don't create file that may be too big. 2) Instead of inserting marks using the chunk id as the key, we insert using a natural sequence. This has two benefits: - Keys are ordered, and so insertion are faster. - Allows us to use boltdb fill percent to 100%, this means boltdb won't over allocate for inserting key in between unordered data. Why ? I realize that when inserting huge amount of marks, this operation can take up to hours since boltdb has to re-allocate pages over and over. This is mainly because chunkid arrive without specific order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve key buf sizing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/storage/stores/shipper/compactor/retention/marker.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Copy the value so that it still available for the whole boltdb tx. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	5 years ago
Sandeep Sukhani	2b7a6f201c	Logs deletion support for boltdb-shipper store (#3688 ) * add support for slicing the chunks for partial chunk deletion * add delete requests store and handler for requests * add delete requests manager and a way to check for chunk expiry based on delete requests * add chunk rewriter for partially deleted chunks * hook delete requests manager into compactor and update expiration checker * use chunk client in sweeper for deleting chunks * upload delete requests table during shutdown and some code refactoring * add tests and metrics * fixes post rebase * lint * changes suggested from PR review * avoid allocation when getting userid from chunkid * use bytes.IndexByte instead	5 years ago
Cyril Tovena	210dc355cb	Fixes a goroutine leak in the store when doing cancellation. (#3733 ) Found out that we might have no listener and trying to send the information of cancellation. I've moved the error information down the respective log and sample iterators so that we don't need to send it via the channel. This seems to have cause some pretty big memory leak on our side. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	14fa7d8f3a	Removes file sync syscall for compaction. (#3693 ) Because we mostly download, read and upload but also because all those operations are idempotent we don't need to try to outsmart the OS by forcefully syncing the file to disk. The OS will use cache on files and we'll most likely not suffer from high disk io. This was a conclusion I made after some digging of why the operation of compaction+retention was a bit slow sometimes. (+5s) I've tried the PR in dev and the whole process execute now below 40s whereas because it was 4min. strace before: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 75.75 12.333637 2122 5812 561 futex 10.81 1.760698 242 7267 nanosleep 7.69 1.252697 88 14287 epoll_pwait 2.86 0.464888 22 21608 write 1.08 0.175313 19479 9 fsync <---------- 1.06 0.172173 15 11249 4334 read 0.40 0.064422 2147 30 15 unlinkat 0.15 0.023945 19 1246 madvise 0.09 0.015288 1529 10 munmap 0.04 0.006098 23 263 tgkill 0.03 0.004134 16 263 getpid ``` strace after: ``` % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 77.01 58.085485 4599 12631 840 futex 9.81 7.395901 230 32145 nanosleep 8.34 6.287953 175 35882 epoll_pwait 3.33 2.512512 18 141141 write 0.79 0.596351 16 37911 13536 read 0.51 0.382579 3985 96 48 unlinkat 0.08 0.058465 1720 34 munmap 0.04 0.031904 26 1219 tgkill 0.03 0.023962 22 1082 madvise 0.02 0.013831 11 1219 getpid 0.01 0.011042 9 1219 12 rt_sigreturn 0.01 0.005348 111 48 fdatasync 0.01 0.003785 36 105 openat 0.00 0.003603 8 435 pwrite64 0.00 0.003005 13 229 200 epoll_ctl 0.00 0.001828 19 96 34 newfstatat ``` As you can see the fsync and write are less intensive after. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	5c4ba10fb9	Use model.Duration for easy yaml/json marshalling. (#3687 ) * Use model.Duration for easy yaml/json marshalling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * test fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	1995eaa426	Pass in the now value to the retention. (#3673 ) This allows to call model.Time only once per table which is reasonable for accuracy and this will save some cpu cycle. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	aa544a9dff	cleanup boltdb files failing to open during loading tables which are possibly corrupt (#3605 )	5 years ago
Cyril Tovena	490c19e38c	Deprecate max_look_back_period in the chunk storage. (#3677 ) * Deprecate max_look_back_period in the chunk storage. From now on it will be applied at the frontend, ruler and querier level. The querier validation was missing so I added it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	a454f7edb7	Use pgzip in the compactor. (#3672 ) Since index file are expected to be large, this will help to compress/decompress in parallel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	43184298b2	Trigger compaction prior retention. (#3665 ) * Trigger compaction prior retention. This should allow to process table where the amount of index files is below 4. Currently they are not processed and we should process them since they may never be compacted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook retention into compaction. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove unused flags. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better cancellation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing ctx in test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flags that are not used. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add compactor retention tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	806d6a503e	Custom Retention (#3642 ) * Playing around with a POC based on the design doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Getting series ID for label matchers is now working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip/ Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * We're parsing the index label now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting to extract interfaces to make the code testable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress will try to add labels to chunk ref iterator Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Iterator for chunks ref with all labels !!!. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Optimize code away. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * with delete into the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleaner but not yet working for v10 and v11. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes series cleaner. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hooking into the compactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking limit retention config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off the marker processor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on sweeper and fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding more tests and founding more bugs along the way. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Bug with path once boltdb is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more bug and more robust test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More test and cleanup getting close. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar bugs with regards to period schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky tests because of boltdb still open. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Save files without using table key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve logging and ability to use more goroutines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes duplicate metrics since histogram contains total too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock bug when too many workers are trying to update the mark file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock when reading and updating db at the same time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes default config test of boltdb. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * PR Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the user of assert to not fail a test if it's incorrect on the first shot. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add experimental notice to the flag documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes empty index detection and table deletion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not delete folder it's not necessary with object store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better working path cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Kaviraj	6bcd9807d6	replace `time.Duration` -> `model.Duration` for `Limits`. (#3632 ) Rationale 1. support user-friendly duration format (e.g: "1h30m45s") in JSON value. 2. In consistent with Cortex and tempo.	5 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	5 years ago
Cyril Tovena	a22bce7286	Add a chunk filter hook in the store. (#3569 ) * Add a chunk filter hook in the store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago

1 2 3 4

165 Commits (a58aea1bf456280aa06fb2f2d09d2d005eefa6ed)