apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Sandeep Sukhani	e65d1b8c69	add more context to logs, improve comments for clarity in boltdb-shipper code (#5341 )	3 years ago
Cyril Tovena	4fafc9c234	Fixes a flaky hedging test. (#5389 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Sandeep Sukhani	da9b4c9369	cleanup common index set after cleaning up all the user index sets (#5372 )	3 years ago
Cyril Tovena	d44b6892eb	Fixes memberlist usage report (#5369 ) * Fixes memberlist usage report Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the linter and improve comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	21fa187fa8	Improve report usage to never fail (#5364 ) * Improve report usage to never fail Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wait until cancelled for the service to not fail Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361 ) * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
gotjosh	02416736d0	Ruler: Rule group not found API message (#5362 ) * Ruler: Rule group not found API message We copied over the ruler code as part of #5089, and it seems that at the same time we stopped depending on the object storage implementation of Cortex. Turns out, there are (slight) differences between the clients which meant that response returned by the API when we try to get a rule group that is not found had changed. This ensures that is consistent with the error assertion that we have in the code for the ruler. * Use the method from the interfaace to determine object storage * also include DeleteObject * appease the linter	3 years ago
Sandeep Sukhani	7fafe94e1a	add a ms instead of ns to end time of the rebound chunk interval (#5351 ) * add a ms instead of ns to end time of the rebound chunk interval conversion between ms and ns loses ns precision which causes us to not include all the logs * improve error when failing to rebound chunk	3 years ago
Sandeep Sukhani	0afd113c96	fix rounding of time to milliseconds (#5352 ) Current implementation of RoundToMilliseconds is broken due to improper handling of floating point operation in go. Fixing it by using simple division and modulo operators	3 years ago
Karsten Jeschkies	8e9009a7e3	Speed up `EntrySortIterator` by 20%. (#5318 ) What this PR does / why we need it: This patch replace the underlying heap of the `EntrySortIterator`. Instead of using a min or max heap the entry iterators are sorted at the beginning. Once the iteration starts only the first entry iterator is re-positioned depending on its current value. This will yield the same runtime complexity as the current heap based solution. However, since the new algorithm accesses the first element it can be cached. Thus the number of access calls to `Entry().Timestamp` are halved. ``` › benchstat before.txt after.txt name old time/op new time/op delta SortIterator/merge_sort-16 4.40ms ± 4% 4.38ms ± 3% ~ (p=0.400 n=9+10) SortIterator/sort-16 3.65ms ± 2% 2.80ms ± 3% -23.25% (p=0.000 n=9+10) ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Dylan Guedes	a3bacd5739	Test `LegacySample` and `LegacyLabelPair` compatibility. (#5313 ) The expected values were built from Cortex' `cortexpb.Sample` and `cortexpb.LabelPair`. They prove that `logproto.LegacyLabelPair` and `logproto.LegacySample` are not compatible with `logproto.LabelPair` and `logproto.Sample`, requiring us to keep both implementations for now.	3 years ago
Sandeep Sukhani	e4d6767412	Fix apply retention issue (#5342 ) * call mark phase started/finished only when applying retention * correctly pass userID to IntervalMayHaveExpiredChunks call on retention and dletion expiry check	3 years ago
Danny Kopping	20041b74d6	Revert "Simplify code for write path in distributor (#5248 )" (#5337 ) This reverts commit `10b93cebe5`. We discovered that these changes can lead to log deduplication issues. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Kaviraj Kanagaraj	2db309271c	Makes `tailer.droppedStreams` slice bounded. (#5334 ) * Makes `tailer.droppedStreams` slice bounded. For slow receiver, this dropped streams may cause ingester memory to keep increasing. Sometimes OOMing the ingester. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make droppedStreams slice size configurable Update tests Adding logging when slice is dropped Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add test for dropped stream Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Bryan Boreham	a50cac7674	filters: use faster regexp package (#5315 ) * regexp-filter: improve benchmark Add a few more cases based on real-world usage. Also simplify the loop to just run the number of times requested. Go benchmarks run 'N' times to reach a duration, default 1 second. Previously the benchmark was running N1000000 times, which took 7 seconds minimum for some cases. regexp filter: use modified package with optimisations See https://github.com/grafana/regexp/tree/speedup#readme Includes the following changes proposed upstream: * [regexp: allow patterns with no alternates to be one-pass](https://go-review.googlesource.com/c/go/+/353711) * [regexp: speed up onepass prefix check](https://go-review.googlesource.com/c/go/+/354909) * [regexp: handle prefix string with fold-case](https://go-review.googlesource.com/c/go/+/358756) * [regexp: avoid copying each instruction executed](https://go-review.googlesource.com/c/go/+/355789) * [regexp: allow prefix string anchored at beginning](https://go-review.googlesource.com/c/go/+/377294) * Add grafana/regexp to vendor directory	3 years ago
Owen Diehl	f598484a94	Release entryBufferPool once (#5324 ) * release entryBufferPool once * reverseEntryIterator.Next is safe to call past expiry	3 years ago
Christian Haudum	10b93cebe5	Simplify code for write path in distributor (#5248 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Cyril Tovena	e3b74158a5	Fixes log deduplication when mutating Labels using LogQL (#5289 ) * Changes the iterator interface to include labelshash. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve unused args and fixes build Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Implement the new hash in the memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add the hash in logproto Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Insert the hash from ingester-querier Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Rename LabelsHash to StreamHash Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working through fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor HeapIterator into merge and sort Iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Order alphabetically when ordering samples. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not JSON encode the hash for legacy model. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more regression tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update CHANGELOG.md Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback	3 years ago
Owen Diehl	2af3ca077e	Logql/parallel binop (#5317 ) * adds justification for keeping Downstreamer parallelism * loads binop legs in parallel * increases downstreamer default concurrency * astmapper spanlogger * always clone expr during mapping to prevent mutability bugs * Revert "astmapper spanlogger" This reverts commit `23f6b55c14`. * cleanup + use errgroup	3 years ago
Vladyslav Diachenko	ea37f4595d	added upgrade notes (#5311 )	3 years ago
Vladyslav Diachenko	d4a4728933	changed logic to respond with '500 Internal Server Error' if rpc error with code 'Cancelled' happened. it's made to allow loki to retry the request to the querier instead of returning '499 The request was cancelled by the client.' back to the client. (#5297 )	3 years ago
Travis Patterson	89b04a97b3	storage: update azure blobstore client to use new sdk (#5302 ) * Update azure blobstore client to use new sdk * changelog * fix hardcoded maxretries * linter * more linting * Fix flaky test	3 years ago
Owen Diehl	af3457b92e	less opaque chunk keys on fs with v12 (#5291 ) * less opaque chunk keys on fs with v12 * lint	3 years ago
Travis Patterson	1925d77f65	cleanup: consolidate util strings contain functions (#5304 ) * Consolidate util strings contain functions * fix test	3 years ago
Dylan Guedes	cf419d1cb8	Implement `LegacySample`, `LegacyLabelPair` proto messages (#5301 ) * Add LegacySample and modify protos to use it. * [Skip this commit] Regenerate protos. * Modify codebase to use LegacySample instead of Sample. For reference, take a look at PR https://github.com/grafana/loki/pull/5151. * Implement LegacyLabelPair. * Use new Legacy implementations accordingly. * Use LegacySample in compat tests. - Since `compat.go` was forked from Cortex, we can reliably change all mentions from `Sample` to `LegacySample` there.	3 years ago
Owen Diehl	a88ee16534	better logic for when to shard wrt disabled lookback (#5303 )	3 years ago
Sandeep Sukhani	40e05198f9	fix a panic in index-gateway caused by double closing of a channel (#5298 )	3 years ago
Cyril Tovena	3af7b7900e	Refactor HeapIterator into Merge and Sort Iterator. (#5281 ) * Refactor HeapIterator into merge and sort Iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Dylan Guedes	ad3c8d07b7	Cleanup `util/validation` code. (#5285 ) * Cleanup `util/validation` code. * Remove `errors.go` file.	3 years ago
Ed Welch	efffe0ce50	add some timing to the logs for downloading and opening index tables (#5042 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Christian Haudum	57b384df8f	Auto-expire old items from FIFO cache (#5148 ) * Remove validity from FIFO cache We can remove the validity setting of 1h in 2.4 for the chunks cache because it doesn't reduce memory usage and instead leads to valid chunks being ignore when querying the cache. Chunks are immutable so it doesn't make sense to not return them if they are present in the cache. Closes #4922 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add periodic task to prune old FIFO cache elements As described in #4921 this PR adds a periodic task that prunes expired items from the FIFO cache to free up memory. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Christian Haudum	8d2b3f7aa5	Return early if push payload does not contain data (#5229 ) Optimize for: * payload does not contain any streams * a stream does not contain any entries * none of the streams contain any entries Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Dylan Guedes	7856db62bb	Copy `cortex/pkg/storage/bucket` package dependency into Loki (#5258 ) * Fork cortex `storage/bucket` into Loki. * Replace storage/bucket usage from Cortex to Loki one. * Remove redundant error check. * Revert SwiftConfig struct and remove cortex pkgs alias. * Add missing test files. * Replace `cortex/pkg/util` with our own implementation. * Remove unused cortex packages.	3 years ago
Kaviraj Kanagaraj	f083aab09d	Remove `cortexpb` dependency from Loki packages. Use only `logproto` now. (#5264 ) * Getting rid of `cortexpb` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Get rid of `cortex/pkg/tenants` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * go mod vendor Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename some import alias Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257 ) * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Michel Hollands	87edab0867	Ingester: remove the labelFilter (#5262 ) * Remove the labelFilter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Sandeep Sukhani	37fcbf9428	use default retention period to check user index may have expired chunks when user does not have custom retention (#5261 )	3 years ago
Owen Diehl	91d837e79c	Refactor/remove global splitby (#5243 ) * cleans up internal limits struct, removes unnecessary logic * moves querier.split-queries-by-interval to limits, simplifies resulting code. * docs, changelog * flag supports model.duration instead of time.duration * lint and adds new error for deprecated field * reregisters queryrange flags * comment lint * ensures we validate queryrange config and simplifies embedded struct validate() * 0 still disables * simplify cache key generation and protect against divide by zero	3 years ago
Dylan Guedes	3d135e5338	Copy `cortex/pkg/configs` package dependency into Loki (#5139 ) * Fork cortex `configs` into Loki. * Use new forked `configs` package. * Remove duplicated import. * Fix lint issues. * Use new forked util/logger. * Drop support for promql rules v1. - This allow us to remove the `legacy_promql` implementation - If a rulesconfig is received in v1 format, we output a specific error, so we can differentiate between a unknown version and a v1 one. * Modify tests to expect error for legacy rules format. * Add changelog and upgrading guide. - Added them as we dropped support for legacy promql rules configuration which might be a breaking change for some environments. * Update vendor/modules after rebasing.	3 years ago
Dylan Guedes	4a70f8ec84	Copy `cortex/pkg/ingester/client` package dependency into Loki (#5164 ) * Fork ingester client. * Document magic numbers.	3 years ago
Dylan Guedes	ba6a2fd41e	Get rid of Cortex chunk. (#5239 )	3 years ago
3JIou_from_home	863408684c	Push: add deflate compression in post requests (#5249 ) * deflate-compression-5247 - add deflate compression in post requests * deflate-compression-5247 - add deflate tests by example gzip	3 years ago
Eng Zer Jun	392fde1e27	refactor: use `T.TempDir` to create temporary test directory (#5253 ) * refactor: use `T.TempDir` to create temporary test directory The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: update CHANGELOG.md Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Sandeep Sukhani	9fdcacff15	v12 chunk format handling in retention (#5254 ) Also add some tests and safety mechanism to avoid dropping whole table when we cant find any chunks	3 years ago
RangerCD	87cbe50b59	Lock consistencyMtx outside locking chunkMtx (#5210 ) * Lock consistencyMtx outside locking chunkMtx * Remove unnecessary RLock	3 years ago
RangerCD	4ee686c22c	Remove ErrStreamMissing (#5234 )	3 years ago
Callum Styan	bfef7bad44	Start refactor of storage (chunks) clients metrics into structs (#5057 ) * Refactor azure metrics to be a struct (+ add an overall client metrics struct), add an azure client metric for total egress bytes. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Add changelog entry for new azure metric. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fixes based on Kavi's review. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix one more merge conflict that I missed in the merge commit. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix failing tests. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Kaviraj Kanagaraj	7713ff78e2	Fix `cortexpb` -> `logproto` rename in some tests (#5231 ) * Fix `cortexpg` -> `logproto` rename in some tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix TimestampMs rename Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * queryrange test Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests on queryrangebase pacakage\ Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix some typos Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Callum Styan	f24b84c15a	[querier] Abort a query in limitedRoundTripper if parallelism is 0 (#5190 ) * Abort a query in limitedRoundTripper if parallelism is 0. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix failing tests related to fakeLimits change. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Change limitedRoundTripper parallelism error message. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Dylan Guedes	3f496e7336	Copy `cortex/pkg/cortexpb` package dependency into Loki (#5151 ) * Fork cortexpb into Cortex `pkg/logproto`. * Use `pkg/logproto` instead of Cortex `pkg/cortexpb`. * Rename `cortexpb` usage to `logproto`. * Fix encoding/decoding test. * Replace cortexpb usage in the ruler/base package. * Remove exemplars from logproto.	3 years ago

1 2 3 4 5 ...

1318 Commits (e65d1b8c692f11240f46f91105f266257696334a)