928 Commits (04f4f82b8e10d599bb8d66ffbb884bde903ad803)

Author	SHA1	Message	Date
Cyril Tovena	806d6a503e	Custom Retention (#3642) ... * Playing around with a POC based on the design doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Getting series ID for label matchers is now working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip/ Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * We're parsing the index label now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting to extract interfaces to make the code testable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress will try to add labels to chunk ref iterator Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Iterator for chunks ref with all labels !!!. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Optimize code away. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * with delete into the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleaner but not yet working for v10 and v11. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes series cleaner. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hooking into the compactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking limit retention config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off the marker processor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on sweeper and fixing tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding more tests and founding more bugs along the way. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Bug with path once boltdb is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more bug and more robust test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More test and cleanup getting close. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar bugs with regards to period schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky tests because of boltdb still open. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmark. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint code. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Save files without using table key Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve logging and ability to use more goroutines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes duplicate metrics since histogram contains total too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock bug when too many workers are trying to update the mark file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a deadlock when reading and updating db at the same time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes default config test of boltdb. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * PR Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the user of assert to not fail a test if it's incorrect on the first shot. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add experimental notice to the flag documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes empty index detection and table deletion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Do not delete folder it's not necessary with object store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better working path cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ae23b7e4ed	Rejects push requests with streams without labels. (#3643) ... * Rejects push requests with streams without labels. Fixes #3638 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing commit Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Kaviraj	6bcd9807d6	replace `time.Duration` -> `model.Duration` for `Limits`. (#3632) ... Rationale 1. support user-friendly duration format (e.g: "1h30m45s") in JSON value. 2. In consistent with Cortex and tempo.	4 years ago
Ed Welch	a9d85de4aa	Re-license to AGPLv3 (#3630) ... For a rationale as to why this was done, please see * https://grafana.com/blog/2021/04/20/grafana-loki-tempo-relicensing-to-agplv3/ * https://grafana.com/blog/2021/04/20/qa-with-our-ceo-on-relicensing/ Signed-off-by: Richard Hartmann <richih@richih.org> Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Kaviraj	3c78579676	Add json struct tags to limits. (#3628)	4 years ago
Michel Hollands	515f82ee1e	Add interceptor override and make ingester and cfg public (#3618) ... * Add interceptor override and make ingester and cfg public Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove extraneous comment Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623) ... * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Cyril Tovena	72dcda165a	Adds chunk filter hook for ingesters. (#3603) ... * Adds chunk filter hook for ingesters. Follow up on https://github.com/grafana/loki/pull/3569 this is the missing piece. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * That interface is not required to change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	965d48222b	adds distributor replication factor metric (#3594)	4 years ago
Michel Hollands	56af725a08	Make the store field public (#3602) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Owen Diehl	d9637a6654	locks trailers during iteration (#3595)	4 years ago
Cyril Tovena	0107a11769	Fixes a race when using specific tenant and multi-client. (#3573) ... * Fixes a race when using specific tenant and multi-client. This was because each client would try to mutate the original set of labels. Instead of cloning the map which can be expensive, I created a little helper that generate the string from the labelset without a set of label name. Fixes #3571 I also added a test to see if this was reproducible and it was indeed: ``` ~/go/src/github.com/grafana/loki master* ❯ go test -timeout 30s -tags dev,gofuzz -race -run ^TestMultiClient_Handle_Race$ github.com/grafana/loki/pkg/promtail/client -v -count=1 -timeout=0s === RUN TestMultiClient_Handle_Race ================== WARNING: DATA RACE Read at 0x00c0000b77d0 by goroutine 22: runtime.mapaccess2_faststr() /usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:107 +0x0 github.com/grafana/loki/pkg/promtail/client.(*client).getTenantID() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:376 +0xcb github.com/grafana/loki/pkg/promtail/client.(*client).processEntry() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:407 +0x9a github.com/grafana/loki/pkg/promtail/client.(*client).run() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:235 +0x36a Previous write at 0x00c0000b77d0 by goroutine 21: runtime.mapdelete_faststr() /usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:297 +0x0 github.com/grafana/loki/pkg/promtail/client.(*client).processEntry() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:408 +0x144 github.com/grafana/loki/pkg/promtail/client.(*client).run() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:235 +0x36a Goroutine 22 (running) created at: github.com/grafana/loki/pkg/promtail/client.New() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:198 +0x7dc github.com/grafana/loki/pkg/promtail/client.TestMultiClient_Handle_Race() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/multi_test.go:126 +0x2c7 testing.tRunner() /usr/local/Cellar/go/1.16.2/libexec/src/testing/testing.go:1194 +0x202 Goroutine 21 (running) created at: github.com/grafana/loki/pkg/promtail/client.New() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:198 +0x7dc github.com/grafana/loki/pkg/promtail/client.TestMultiClient_Handle_Race() /Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/multi_test.go:124 +0x187 testing.tRunner() /usr/local/Cellar/go/1.16.2/libexec/src/testing/testing.go:1194 +0x202 ================== testing.go:1093: race detected during execution of test --- FAIL: TestMultiClient_Handle_Race (0.00s) === CONT testing.go:1093: race detected during execution of test FAIL FAIL github.com/grafana/loki/pkg/promtail/client 0.022s FAIL ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * don't overload variable name Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Ed Welch	21b901db9b	Loki: Distributor log message bodySize should always reflect the compressed size (#3572)	4 years ago
Cyril Tovena	a22bce7286	Add a chunk filter hook in the store. (#3569) ... * Add a chunk filter hook in the store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	4d1da2edf3	first_over_time and last_over_time (#3050) ... * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * [POC] last_over_time and first_over_time. This is an attempt to allow users to select the first or last discrete sample values from a range vector. This way we can create gauge, the only drawback being the log need to exists at least in every range. May be we could transfer the last or first value to the next range, but it will still be a problem for small ranges. Still wondering how useful this is compare to min/max_over_time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ecca5d345c	Properly release the ticker in Loki client. (#3566) ... I think this was only impacting the docker driver who would start/stop for each new followed containers. My assumption is that this would slowly build up and eat CPU over time. Fixes #3319 I arrived to this conclusion since strace was showing a crazy amount of futex syscall and nothing else seems to be the cause. 🙏 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	eff4ff1231	Improve matchers validations. (#3564) ... 1- It currently 500. 2- Happens too late, at ingester level and querier. (better at the frontend) 3- Improve API surface level, we don't need to expose this validation for anything else than ParseLogSelector. see #3216 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	d450bbb668	Fixes a test from #3216. (#3562) ... Minor refactoring missing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	b152bc2cf3	check for stream selectors to have atleast one equality matcher (#3216) ... * check for stream selectors to have atleast one equality matcher * move equality matchers check to logql	4 years ago
Cyril Tovena	a76a17877b	Add a target to reproduce fuzz testcase (#3553) ... * Add a make target to facilitate fuzz testing locally. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * missing target.... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Michel Hollands	ffbabc5715	LBAC changes (#3549) ... * Add querier override and tripperware Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use new queryable in engine Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Removed unused Queryable Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Only use QueryFrontEndTripperware if set Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	b11d2effe2	Fixes a bug in MatrixStepper when sharding queries. (#3550) ... * Fixes a bug in MatrixStepper when sharding queries. Since we split correctly metric queries, this bug has show itself. Basically we were not correctly stepping through time. We should always start from start, add the step until the start is after the end. For more read: https://www.robustperception.io/step-and-query_range Fixes #3541 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. for real. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Not easy to get those test working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	71aa8a631a	Add sprig text/template functions to template stage. (#3515) ... * Add sprig text/template function to template stage. Fixes #3129 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * nit. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * reset vendor file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Lars Lehtonen	b5930d6d9d	single import of jsoniter in logql subpackages (#3544)	4 years ago
Cyril Tovena	29a3240e17	Support for single step metric query. (#3540) ... When splitting correctly steps in the frontend, some queries might be of a single step, e.g start == end. This PR add supports for those queries, which are already supported by Cortex and Prometheus, but for some reason not Loki. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Michel Hollands	f49a510f38	Loki: Update cortex version and fix resulting changes (#3532) ... * Update cortex version and fix resulting changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix scrapeconfig test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/promtail/scrapeconfig/scrapeconfig_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Remove unused fmt import Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Sandeep Sukhani	8694e7303a	split series api queries by day in query-frontend (#3530)	4 years ago
Garrett	f70c7ea0bc	Implement offset modifier for range vector aggregation in LogQL (#3455) ... * Implement offset modifier for range vector aggregation in LogQL implementation for https://github.com/grafana/loki/issues/2785, the offset modifier allows changing the time offset for range vectors in a query to support e.g. selective timeshift in Grafana Signed-off-by: garrettlish <garrett.li.sh@gmail.com> * add docs for offset modifier * refine offset expr * add offset unit test in engine.go * remove duplicated offset shift * add offset test in range vector test Co-authored-by: garrettlish <garrettlish@163.com>	4 years ago
Cyril Tovena	a7e0770acb	Fixes a race introduced by #3434. (#3517)	4 years ago
Sandeep Sukhani	b06d9703d5	fix live tailing of logs from Loki (#3509)	4 years ago
Kaviraj	14b2c093c1	Add unique promtail_instance id to labels for gcptarget (#3501) ... * Add unique promtail_instance id to labels for gcptarget Rationale: To make labelset unique per promtail instance to avoid out-of-order errors * Update docs/sources/clients/promtail/scraping.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	4 years ago
Sandeep Sukhani	835f710bc9	make prefix for keys of objects created by boltdb-shipper configurable (#3491) ... * make prefix for keys of objects created by boltdb-shipper configurable * shared store key prefix should have a path separator	4 years ago
Cyril Tovena	6148794b9b	Fixes a bug where unpack would mutate log line. (#3502) ... * Fixes a bug where unpack would mutate log line. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add reset. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Neven Miculinic	b67a07e1fb	Added path information to deleted tailed file (#3457)	4 years ago
Aditya C S	0f5fceef8f	Support labelallow stage in Promtail (#3468) ... * add labelallow stage * added docs * add docs * fix lint * update stage name in _index.md	4 years ago
Aditya C S	b1ea8ac050	support math functions in line_format and label_format (#3434) ... * support math functions in line_format and label_format * fix lint and tests * added docs and more tests * update doc * remove toString and int64 * doc adjustement. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	d6ecfd2431	Set the byte slice cap correctly when unsafely converting string. (#3487) ... Fixes #3485 Should also unblock #3481 When unsafely converting string to bytes the cap of the bytes is lost but not when doing bytes to string. It seems that this is only a problem when using go 1.16 or when using delve. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Aditya C S	bf4fa6613a	support max_message_length configuration for syslog parser (#3400) ... * support max_message_length configuration for syslog parser * address review comments	4 years ago
Marco Pracucci	94041307b1	Upgrade Cortex (#3466) ... * Upgrade Cortex Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated vendor/modules.txt Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Cyril Tovena	64decdbcb6	Set a max size for the logql parser to 5k. (#3471) ... * Set a max size for the logql parser to 5k. The API anyway receives the query via querystring which is normally max out by 2k based on RFC2616. This will improve fuzzing too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better error. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	c49e807a55	Fixes Issue 28593: loki:fuzz_parse_expr: Timeout in fuzz_parse_expr. (#3470) ... This was caused by a lot of allocations when parsing range duration. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	e29d53631e	Fixes out-of-memory fuzzing issue. (#3469) ... * Fixes out-of-memory fuzzing issue. Fixes Issue 28535 in oss-fuzz: loki:fuzz_parse_expr: Out-of-memory in fuzz_parse_expr Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	306cc72438	Revendor/cortex (#3446) ... * Upgrade Cortex in Loki Signed-off-by: Marco Pracucci <marco@pracucci.com> * proto compat * gofmt * changelog for deprecated querier.split-queries-by-day Co-authored-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Ed Welch	8012362674	Loki: Per Tenant Runtime Configs (#3460) ... * Add per tenant configs mechanism * Add per tenant logging of stream creation * fix tests * enable per tenant configs on push request logging. * fixing up the stream log, changing log levels to debug	4 years ago
Cyril Tovena	93f2b2d89b	Fixes split interval for metrics queries. (#3459) ... The step needs to be taken into account: - to be removed on each split end. - to not split queries that have a step lower than the split interval. The code is taken from Cortex but adjusted to keep the nanoseconds precision. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a29901d33a	Add limit and line_returned in the query log. (#3423) ... * Add limit and line_returned in the query log. This can be useful to investigate how many wasted queries have been made by the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	ce4f78515f	Improve the parser hint tests. (#3429) ... I was able to find out where was the issue with that test yesterday. Now I realized that we can mutate (re-slice) the given log line, and so other test would fail. This also make the parallel back. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	7e9ac16dbf	Add fgfprof to Loki and Promtail. (#3424) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	6d48fb6942	change ReadStringAsSlice to ReadString so it doesn't parse quotes inside the packed _entry (#3432)	4 years ago
Cyril Tovena	32d668ceb0	Only unpack entry if the key `_entry` exist. (#3426) ... * Only unpack entry if the key `_entry` exist. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * s/kv/lbs/g Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix the test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago