apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	21fa187fa8	Improve report usage to never fail (#5364 ) * Improve report usage to never fail Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wait until cancelled for the service to not fail Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361 ) * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
gotjosh	02416736d0	Ruler: Rule group not found API message (#5362 ) * Ruler: Rule group not found API message We copied over the ruler code as part of #5089, and it seems that at the same time we stopped depending on the object storage implementation of Cortex. Turns out, there are (slight) differences between the clients which meant that response returned by the API when we try to get a rule group that is not found had changed. This ensures that is consistent with the error assertion that we have in the code for the ruler. * Use the method from the interfaace to determine object storage * also include DeleteObject * appease the linter	3 years ago
Sandeep Sukhani	7fafe94e1a	add a ms instead of ns to end time of the rebound chunk interval (#5351 ) * add a ms instead of ns to end time of the rebound chunk interval conversion between ms and ns loses ns precision which causes us to not include all the logs * improve error when failing to rebound chunk	3 years ago
Sandeep Sukhani	0afd113c96	fix rounding of time to milliseconds (#5352 ) Current implementation of RoundToMilliseconds is broken due to improper handling of floating point operation in go. Fixing it by using simple division and modulo operators	3 years ago
Karsten Jeschkies	8e9009a7e3	Speed up `EntrySortIterator` by 20%. (#5318 ) What this PR does / why we need it: This patch replace the underlying heap of the `EntrySortIterator`. Instead of using a min or max heap the entry iterators are sorted at the beginning. Once the iteration starts only the first entry iterator is re-positioned depending on its current value. This will yield the same runtime complexity as the current heap based solution. However, since the new algorithm accesses the first element it can be cached. Thus the number of access calls to `Entry().Timestamp` are halved. ``` › benchstat before.txt after.txt name old time/op new time/op delta SortIterator/merge_sort-16 4.40ms ± 4% 4.38ms ± 3% ~ (p=0.400 n=9+10) SortIterator/sort-16 3.65ms ± 2% 2.80ms ± 3% -23.25% (p=0.000 n=9+10) ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Dylan Guedes	a3bacd5739	Test `LegacySample` and `LegacyLabelPair` compatibility. (#5313 ) The expected values were built from Cortex' `cortexpb.Sample` and `cortexpb.LabelPair`. They prove that `logproto.LegacyLabelPair` and `logproto.LegacySample` are not compatible with `logproto.LabelPair` and `logproto.Sample`, requiring us to keep both implementations for now.	3 years ago
Josh Arrington	d934139612	fix changelog issue reference to lambda-promtail (#5347 )	3 years ago
Sandeep Sukhani	e4d6767412	Fix apply retention issue (#5342 ) * call mark phase started/finished only when applying retention * correctly pass userID to IntervalMayHaveExpiredChunks call on retention and dletion expiry check	3 years ago
Nate Walters	ffc87055b8	Delete docs/sources/copyright directory (#5325 ) Removing copyright notice per https://raintank-corp.slack.com/archives/CH34H23HD/p1643999482151029	3 years ago
Danny Kopping	20041b74d6	Revert "Simplify code for write path in distributor (#5248 )" (#5337 ) This reverts commit `10b93cebe5`. We discovered that these changes can lead to log deduplication issues. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Kaviraj Kanagaraj	2db309271c	Makes `tailer.droppedStreams` slice bounded. (#5334 ) * Makes `tailer.droppedStreams` slice bounded. For slow receiver, this dropped streams may cause ingester memory to keep increasing. Sometimes OOMing the ingester. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make droppedStreams slice size configurable Update tests Adding logging when slice is dropped Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add test for dropped stream Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Bryan Boreham	a50cac7674	filters: use faster regexp package (#5315 ) * regexp-filter: improve benchmark Add a few more cases based on real-world usage. Also simplify the loop to just run the number of times requested. Go benchmarks run 'N' times to reach a duration, default 1 second. Previously the benchmark was running N1000000 times, which took 7 seconds minimum for some cases. regexp filter: use modified package with optimisations See https://github.com/grafana/regexp/tree/speedup#readme Includes the following changes proposed upstream: * [regexp: allow patterns with no alternates to be one-pass](https://go-review.googlesource.com/c/go/+/353711) * [regexp: speed up onepass prefix check](https://go-review.googlesource.com/c/go/+/354909) * [regexp: handle prefix string with fold-case](https://go-review.googlesource.com/c/go/+/358756) * [regexp: avoid copying each instruction executed](https://go-review.googlesource.com/c/go/+/355789) * [regexp: allow prefix string anchored at beginning](https://go-review.googlesource.com/c/go/+/377294) * Add grafana/regexp to vendor directory	3 years ago
Owen Diehl	f598484a94	Release entryBufferPool once (#5324 ) * release entryBufferPool once * reverseEntryIterator.Next is safe to call past expiry	3 years ago
Andre Ziviani	699fffe9e5	lambda-promtail: Add ability to ingest logs from S3 (#5065 ) * Add ability to ingest logs from S3 on lambda-promtail * fix ci * fix typo * bump golang and alpine version * update changelog * add s3 permissions on terraform * use for_each instead of count * fix typo * improve function naming * add documentation and an example of a s3 file path * refact logic to identify event type * add missing iam permission to allow lambda to run inside a vpc * fix typo * allow lambda to access only specified s3 buckets * configure a default log retention policy on log group * add missing depends_on to make sure iam role is created before lambda function * update docs * fix label naming convention * fix merge conflict * use new backoff lib and update dependencies * add option to limit batch size * cache s3 client * update docs and terraform * address some feedback on PR * fix typo	3 years ago
Karen Miller	d787a0fe28	Docs: remove "regular" from string definition (#5309 )	3 years ago
Vladyslav Diachenko	f0ef4662fe	Documentation for load testing Loki with xk6-loki (#5182 ) * Add initial page for xk6-loki docs Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * added documentation for xk6-loki query scenario * Add simple installation and usage instructions Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add page about log generation with xk6-loki * Add documentation for k6 write scenario Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup: incorporate feedback from Karen Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup: incorporate feedback from Karen Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup: @ppcano's suggestions * fixup: @cyriltovena's suggestions * fixup: make read scenario more generic * Apply suggestions from code review Co-authored-by: Pepe Cano <pepe@loadimpact.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Use correct URL for git clone command Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Pepe Cano <pepe@loadimpact.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Christian Haudum	10b93cebe5	Simplify code for write path in distributor (#5248 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Cyril Tovena	e3b74158a5	Fixes log deduplication when mutating Labels using LogQL (#5289 ) * Changes the iterator interface to include labelshash. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve unused args and fixes build Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Implement the new hash in the memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add the hash in logproto Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Insert the hash from ingester-querier Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Rename LabelsHash to StreamHash Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working through fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor HeapIterator into merge and sort Iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Order alphabetically when ordering samples. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not JSON encode the hash for legacy model. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more regression tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update CHANGELOG.md Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback	3 years ago
Owen Diehl	2af3ca077e	Logql/parallel binop (#5317 ) * adds justification for keeping Downstreamer parallelism * loads binop legs in parallel * increases downstreamer default concurrency * astmapper spanlogger * always clone expr during mapping to prevent mutability bugs * Revert "astmapper spanlogger" This reverts commit `23f6b55c14`. * cleanup + use errgroup	3 years ago
Trevor Whitney	c477e29c3f	Add tooling for local k3d clusters under tools/dev (#5272 ) * add k3d folder with loki-distributed env * add readme * shellcheck create_cluster.sh * lint jsonnet * Update grafana image to latest	3 years ago
Vladyslav Diachenko	ea37f4595d	added upgrade notes (#5311 )	3 years ago
Susana Ferreira	82fac990a1	Update promtail to support duration string formats (#5290 ) * Update promtail to support duration string formats * Address comments	3 years ago
Vladyslav Diachenko	d4a4728933	changed logic to respond with '500 Internal Server Error' if rpc error with code 'Cancelled' happened. it's made to allow loki to retry the request to the querier instead of returning '499 The request was cancelled by the client.' back to the client. (#5297 )	3 years ago
Travis Patterson	89b04a97b3	storage: update azure blobstore client to use new sdk (#5302 ) * Update azure blobstore client to use new sdk * changelog * fix hardcoded maxretries * linter * more linting * Fix flaky test	3 years ago
Travis Patterson	de1faa0671	Fix transitive dependency error (#5308 ) * Fix transitive dependency error - Don't update the gems that come with the logstash image * ci bump * ci bump	3 years ago
Owen Diehl	af3457b92e	less opaque chunk keys on fs with v12 (#5291 ) * less opaque chunk keys on fs with v12 * lint	3 years ago
Travis Patterson	1925d77f65	cleanup: consolidate util strings contain functions (#5304 ) * Consolidate util strings contain functions * fix test	3 years ago
Dylan Guedes	cf419d1cb8	Implement `LegacySample`, `LegacyLabelPair` proto messages (#5301 ) * Add LegacySample and modify protos to use it. * [Skip this commit] Regenerate protos. * Modify codebase to use LegacySample instead of Sample. For reference, take a look at PR https://github.com/grafana/loki/pull/5151. * Implement LegacyLabelPair. * Use new Legacy implementations accordingly. * Use LegacySample in compat tests. - Since `compat.go` was forked from Cortex, we can reliably change all mentions from `Sample` to `LegacySample` there.	3 years ago
Sashank Agarwal	e19c5863ea	Fix `go.mod` path of operator (#5287 ) * fix go mod path of operator * change import path in all operator files	3 years ago
Owen Diehl	a88ee16534	better logic for when to shard wrt disabled lookback (#5303 )	3 years ago
Sandeep Sukhani	40e05198f9	fix a panic in index-gateway caused by double closing of a channel (#5298 )	3 years ago
Christian Haudum	3fb572abda	Fix Promtail watching deadlock (#5283 ) * Reuse channel from fileEventWatcher map if already exists Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove sync() call in FileTarget constructor The constructor `NewFileTarget` is called from within a locked code block by the FileTargetManager. However, the constructor also calls `sync()` which starts and stops watching new directories. These methods `startWatching` and `stopWatching` both send to the `targetEventHandler` channel, causing the `FileTargetManager` to receive the event in a goroutine (`watch()` function) running in the background. Handling the event calls `sendFileCreateEvent` on the syncers, which subsequently tries to write to the `fileEventWatcher` channel which is never read because of the locked mutex. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Sashank Agarwal	c88734b72c	Link `replace` pipeline stage to doc (#5200 )	3 years ago
Karsten Jeschkies	b92b544ca2	Write Promtail position file atomically. (#5266 ) Fixes #5218	3 years ago
Karsten Jeschkies	54bc2dc40e	Restart Docker target if stopped. (#5280 ) Summary: The Docker target would not reconnect to a container if it restarted and kept the same ID. This should introduces a `startIfNotRunning` method that will be called to restart scraping the container's logs. Fixes #5259	3 years ago
Cyril Tovena	3af7b7900e	Refactor HeapIterator into Merge and Sort Iterator. (#5281 ) * Refactor HeapIterator into merge and sort Iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Ed Welch	44ba39055d	logstash: re-add bundle update and update bundler (#5284 ) * update to latest logstash and bundler Signed-off-by: Edward Welch <edward.welch@grafana.com> * re-add bundle update Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Owen Diehl	96ff7f900b	Revert "Adding a docs page that helps users understand memberlist a bit more (#5186 )" (#5286 ) This reverts commit `7e372cbc56`.	3 years ago
Dylan Guedes	ad3c8d07b7	Cleanup `util/validation` code. (#5285 ) * Cleanup `util/validation` code. * Remove `errors.go` file.	3 years ago
Ed Welch	efffe0ce50	add some timing to the logs for downloading and opening index tables (#5042 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Christian Haudum	57b384df8f	Auto-expire old items from FIFO cache (#5148 ) * Remove validity from FIFO cache We can remove the validity setting of 1h in 2.4 for the chunks cache because it doesn't reduce memory usage and instead leads to valid chunks being ignore when querying the cache. Chunks are immutable so it doesn't make sense to not return them if they are present in the cache. Closes #4922 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add periodic task to prune old FIFO cache elements As described in #4921 this PR adds a periodic task that prunes expired items from the FIFO cache to free up memory. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Jennifer Villa	7e372cbc56	Adding a docs page that helps users understand memberlist a bit more (#5186 ) * Added some new docs on memberlist. * Added a page with more memberlist info.	3 years ago
Trevor Whitney	926a0b2a4e	Update local configs to fix issues with incorrect addresses (#5207 ) * Update local configs to fix issues with incorrect addresses * Add back instance_addr for local config	3 years ago
Dan Poltawski	379ca6b409	Promtail docs: Fix link to `unpack` LogQL documentation (#5274 )	3 years ago
Christian Haudum	8d2b3f7aa5	Return early if push payload does not contain data (#5229 ) Optimize for: * payload does not contain any streams * a stream does not contain any entries * none of the streams contain any entries Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Kaviraj Kanagaraj	531905c066	Remove `cortex` references from `go.mod` and makefile (#5269 ) * Remove cortex references from `go.mod` and makefile Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove from `modules.txt` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	7856db62bb	Copy `cortex/pkg/storage/bucket` package dependency into Loki (#5258 ) * Fork cortex `storage/bucket` into Loki. * Replace storage/bucket usage from Cortex to Loki one. * Remove redundant error check. * Revert SwiftConfig struct and remove cortex pkgs alias. * Add missing test files. * Replace `cortex/pkg/util` with our own implementation. * Remove unused cortex packages.	3 years ago
Kaviraj Kanagaraj	858f8ea54a	Fix transitive dependency failure while building logstash plugin (#5268 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Lydia Lee	a4cfab4b84	feat(promtail): fix TargetManager.run() not exit (#5238 )	3 years ago

... 2 3 4 5 6 ...

3237 Commits (2ebbecfcbd7e484fdcfad4669cf33fd8dda64eb4) All Branches Search

3237 Commits (2ebbecfcbd7e484fdcfad4669cf33fd8dda64eb4)

All Branches