apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	490c19e38c	Deprecate max_look_back_period in the chunk storage. (#3677 ) * Deprecate max_look_back_period in the chunk storage. From now on it will be applied at the frontend, ruler and querier level. The querier validation was missing so I added it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a454f7edb7	Use pgzip in the compactor. (#3672 ) Since index file are expected to be large, this will help to compress/decompress in parallel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	dbeb9840d1	Add a dashboard for retention to the loki-mixin. (#3658 ) * Add a dashboard for retention to the loki-mixin. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update dashboard to reflects last changes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	43184298b2	Trigger compaction prior retention. (#3665 ) * Trigger compaction prior retention. This should allow to process table where the amount of index files is below 4. Currently they are not processed and we should process them since they may never be compacted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook retention into compaction. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove unused flags. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better cancellation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing ctx in test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flags that are not used. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add compactor retention tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	d88f3996ea	revendor compatibility: various prom+k8s+cortex (#3664 ) * revendor compatibility: various prom+k8s+cortex * fix logcli client instantiation * go mod tidy	4 years ago
Cyril Tovena	04f4f82b8e	Adds the ability to provide a tripperware to Promtail client. (#3654 ) Fixes #3608 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	806d6a503e	Custom Retention (#3642 ) * Playing around with a POC based on the design doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Getting series ID for label matchers is now working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip/ Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * We're parsing the index label now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting to extract interfaces to make the code testable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress will try to add labels to chunk ref iterator Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Iterator for chunks ref with all labels !!!. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Optimize code away. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * with delete into the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleaner but not yet working for v10 and v11. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes series cleaner. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hooking into the compactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking limit retention config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off the marker processor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on sweeper and fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding more tests and founding more bugs along the way. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Bug with path once boltdb is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more bug and more robust test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More test and cleanup getting close. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar bugs with regards to period schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky tests because of boltdb still open. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Save files without using table key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve logging and ability to use more goroutines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes duplicate metrics since histogram contains total too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock bug when too many workers are trying to update the mark file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock when reading and updating db at the same time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes default config test of boltdb. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * PR Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the user of assert to not fail a test if it's incorrect on the first shot. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add experimental notice to the flag documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes empty index detection and table deletion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not delete folder it's not necessary with object store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better working path cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ruppsn	771d33b69e	Update labels.md in Loki Docs (#3648 ) Complement to https://github.com/grafana/loki/pull/3641	4 years ago
Trevor Whitney	2f29e8e583	add note about wildcard log patterns and log rotation (#3655 ) Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Periklis Tsirakidis	63e25c9c15	Add Red Hat to adopters (#3644 ) * Add Red Hat to adopters * Update ADOPTERS.md	4 years ago
Cyril Tovena	ae23b7e4ed	Rejects push requests with streams without labels. (#3643 ) * Rejects push requests with streams without labels. Fixes #3638 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing commit Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	e71cb454ca	removes naming collision (#3641 )	4 years ago
Michel Hollands	9137ef921b	Update cortex to 1.8 (#3627 ) * Update Cortex to 1.8.1 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Ursula Kallio	f0715a2c8c	Fix wget link. (#3633 )	4 years ago
Kaviraj	6bcd9807d6	replace `time.Duration` -> `model.Duration` for `Limits`. (#3632 ) Rationale 1. support user-friendly duration format (e.g: "1h30m45s") in JSON value. 2. In consistent with Cortex and tempo.	4 years ago
Ed Welch	a9d85de4aa	Re-license to AGPLv3 (#3630 ) For a rationale as to why this was done, please see * https://grafana.com/blog/2021/04/20/grafana-loki-tempo-relicensing-to-agplv3/ * https://grafana.com/blog/2021/04/20/qa-with-our-ceo-on-relicensing/ Signed-off-by: Richard Hartmann <richih@richih.org> Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Kaviraj	3c78579676	Add json struct tags to limits. (#3628 )	4 years ago
Michel Hollands	515f82ee1e	Add interceptor override and make ingester and cfg public (#3618 ) * Add interceptor override and make ingester and cfg public Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove extraneous comment Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Ed Welch	31bb757c3b	update CI for branch rename to main (#3625 )	4 years ago
Ed Welch	0a02aecdb9	fix drone dependencies for manifest (#3624 )	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Ed Welch	b04af08b1b	Remove codecov (#3615 ) * remove codecov from CI, also remove circleci tests as we run these somewhere else now already. * leaving an empty test step for now	4 years ago
Cyril Tovena	72dcda165a	Adds chunk filter hook for ingesters. (#3603 ) * Adds chunk filter hook for ingesters. Follow up on https://github.com/grafana/loki/pull/3569 this is the missing piece. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * That interface is not required to change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	7bac502dda	Dashboard/fix operational vars (#3601 ) * loki operational dashboard uses more refined variables * excludes loki canary	4 years ago
Owen Diehl	965d48222b	adds distributor replication factor metric (#3594 )	4 years ago
Michel Hollands	56af725a08	Make the store field public (#3602 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
timazet	9ee6ba7aef	Update _index.md (#3596 ) Typo in example is corrected	4 years ago
Owen Diehl	d9637a6654	locks trailers during iteration (#3595 )	4 years ago
Ward Bekker	46eff3c12c	Added Architecture Diagram (#3526 ) * Added Architecture Diagram * Update _index.md * Rename LokiArchitectureDiagram.svg to loki_architecture_components.svg	4 years ago
Sandeep Sukhani	28d88b3fb8	add loki resource usage dashboard for read and write path (#3584 )	4 years ago
Cyril Tovena	0107a11769	Fixes a race when using specific tenant and multi-client. (#3573 ) * Fixes a race when using specific tenant and multi-client. This was because each client would try to mutate the original set of labels. Instead of cloning the map which can be expensive, I created a little helper that generate the string from the labelset without a set of label name. Fixes #3571 I also added a test to see if this was reproducible and it was indeed: ``` ~/go/src/github.com/grafana/loki master* ❯ go test -timeout 30s -tags dev,gofuzz -race -run ^TestMultiClient_Handle_Race$ github.com/grafana/loki/pkg/promtail/client -v -count=1 -timeout=0s === RUN TestMultiClient_Handle_Race ================== WARNING: DATA RACE Read at 0x00c0000b77d0 by goroutine 22: runtime.mapaccess2_faststr() /usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:107 +0x0 github.com/grafana/loki/pkg/promtail/client.(client).getTenantID() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:376 +0xcb github.com/grafana/loki/pkg/promtail/client.(client).processEntry() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:407 +0x9a github.com/grafana/loki/pkg/promtail/client.(client).run() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:235 +0x36a Previous write at 0x00c0000b77d0 by goroutine 21: runtime.mapdelete_faststr() /usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:297 +0x0 github.com/grafana/loki/pkg/promtail/client.(client).processEntry() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:408 +0x144 github.com/grafana/loki/pkg/promtail/client.(client).run() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:235 +0x36a Goroutine 22 (running) created at: github.com/grafana/loki/pkg/promtail/client.New() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:198 +0x7dc github.com/grafana/loki/pkg/promtail/client.TestMultiClient_Handle_Race() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/multi_test.go:126 +0x2c7 testing.tRunner() /usr/local/Cellar/go/1.16.2/libexec/src/testing/testing.go:1194 +0x202 Goroutine 21 (running) created at: github.com/grafana/loki/pkg/promtail/client.New() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:198 +0x7dc github.com/grafana/loki/pkg/promtail/client.TestMultiClient_Handle_Race() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/multi_test.go:124 +0x187 testing.tRunner() /usr/local/Cellar/go/1.16.2/libexec/src/testing/testing.go:1194 +0x202 ================== testing.go:1093: race detected during execution of test --- FAIL: TestMultiClient_Handle_Race (0.00s) === CONT testing.go:1093: race detected during execution of test FAIL FAIL github.com/grafana/loki/pkg/promtail/client 0.022s FAIL ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> don't overload variable name Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Ed Welch	21b901db9b	Loki: Distributor log message bodySize should always reflect the compressed size (#3572 )	4 years ago
Cyril Tovena	a22bce7286	Add a chunk filter hook in the store. (#3569 ) * Add a chunk filter hook in the store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Michel Hollands	94af4c4f84	Some small docs fixes (#3568 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	4d1da2edf3	first_over_time and last_over_time (#3050 ) * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * [POC] last_over_time and first_over_time. This is an attempt to allow users to select the first or last discrete sample values from a range vector. This way we can create gauge, the only drawback being the log need to exists at least in every range. May be we could transfer the last or first value to the next range, but it will still be a problem for small ranges. Still wondering how useful this is compare to min/max_over_time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ecca5d345c	Properly release the ticker in Loki client. (#3566 ) I think this was only impacting the docker driver who would start/stop for each new followed containers. My assumption is that this would slowly build up and eat CPU over time. Fixes #3319 I arrived to this conclusion since strace was showing a crazy amount of futex syscall and nothing else seems to be the cause. 🙏 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	eff4ff1231	Improve matchers validations. (#3564 ) 1- It currently 500. 2- Happens too late, at ingester level and querier. (better at the frontend) 3- Improve API surface level, we don't need to expose this validation for anything else than ParseLogSelector. see #3216 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	d450bbb668	Fixes a test from #3216 . (#3562 ) Minor refactoring missing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	b152bc2cf3	check for stream selectors to have atleast one equality matcher (#3216 ) * check for stream selectors to have atleast one equality matcher * move equality matchers check to logql	4 years ago
Lashan	e0f0ee0585	Update README.md (#3552 ) With Go 1.11 it's not possible to build the Fluent Bit plugin with the following error. ``` build github.com/sercand/kuberesolver: cannot find module for path github.com/sercand/kuberesolver ``` Needed to use Go 1.16.2 to resolve the issue.	4 years ago
Cyril Tovena	a76a17877b	Add a target to reproduce fuzz testcase (#3553 ) * Add a make target to facilitate fuzz testing locally. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * missing target.... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sam Jewell	87ca7a4616	Remove key-value pair missing from logfmt output (#3555 ) Also fix typo	4 years ago
Michel Hollands	ffbabc5715	LBAC changes (#3549 ) * Add querier override and tripperware Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use new queryable in engine Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Removed unused Queryable Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Only use QueryFrontEndTripperware if set Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Tom Hu	6ab0e94572	Add Codecov badge to README (#3484 )	4 years ago
Cyril Tovena	b11d2effe2	Fixes a bug in MatrixStepper when sharding queries. (#3550 ) * Fixes a bug in MatrixStepper when sharding queries. Since we split correctly metric queries, this bug has show itself. Basically we were not correctly stepping through time. We should always start from start, add the step until the start is after the end. For more read: https://www.robustperception.io/step-and-query_range Fixes #3541 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. for real. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Not easy to get those test working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Robby Milo	37a7189d4e	docs: add title to Lambda Promtail (#3525 )	4 years ago
Stavros Aronis	22edfe34bc	Fix markdown for output stage (#3472 )	4 years ago
Cyril Tovena	71aa8a631a	Add sprig text/template functions to template stage. (#3515 ) * Add sprig text/template function to template stage. Fixes #3129 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * nit. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * reset vendor file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	24f8f6a4de	compactor docs (#3543 )	4 years ago
Lars Lehtonen	b5930d6d9d	single import of jsoniter in logql subpackages (#3544 )	4 years ago

1 2 3 4 5 ...

2183 Commits (490c19e38c2e78eee8a73541ab4670ed28fb3945) All Branches Search

2183 Commits (490c19e38c2e78eee8a73541ab4670ed28fb3945)

All Branches