911 Commits (d450bbb668e9edd3f3ee1a00a27abc176b76d93c)

Author	SHA1	Message	Date
Cyril Tovena	d450bbb668	Fixes a test from #3216. (#3562) ... Minor refactoring missing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	b152bc2cf3	check for stream selectors to have atleast one equality matcher (#3216) ... * check for stream selectors to have atleast one equality matcher * move equality matchers check to logql	5 years ago
Cyril Tovena	a76a17877b	Add a target to reproduce fuzz testcase (#3553) ... * Add a make target to facilitate fuzz testing locally. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * missing target.... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Michel Hollands	ffbabc5715	LBAC changes (#3549) ... * Add querier override and tripperware Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use new queryable in engine Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Removed unused Queryable Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Only use QueryFrontEndTripperware if set Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Cyril Tovena	b11d2effe2	Fixes a bug in MatrixStepper when sharding queries. (#3550) ... * Fixes a bug in MatrixStepper when sharding queries. Since we split correctly metric queries, this bug has show itself. Basically we were not correctly stepping through time. We should always start from start, add the step until the start is after the end. For more read: https://www.robustperception.io/step-and-query_range Fixes #3541 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. for real. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Not easy to get those test working. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	71aa8a631a	Add sprig text/template functions to template stage. (#3515) ... * Add sprig text/template function to template stage. Fixes #3129 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * nit. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * reset vendor file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Lars Lehtonen	b5930d6d9d	single import of jsoniter in logql subpackages (#3544)	5 years ago
Cyril Tovena	29a3240e17	Support for single step metric query. (#3540) ... When splitting correctly steps in the frontend, some queries might be of a single step, e.g start == end. This PR add supports for those queries, which are already supported by Cortex and Prometheus, but for some reason not Loki. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Michel Hollands	f49a510f38	Loki: Update cortex version and fix resulting changes (#3532) ... * Update cortex version and fix resulting changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix scrapeconfig test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/promtail/scrapeconfig/scrapeconfig_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Remove unused fmt import Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Sandeep Sukhani	8694e7303a	split series api queries by day in query-frontend (#3530)	5 years ago
Garrett	f70c7ea0bc	Implement offset modifier for range vector aggregation in LogQL (#3455) ... * Implement offset modifier for range vector aggregation in LogQL implementation for https://github.com/grafana/loki/issues/2785, the offset modifier allows changing the time offset for range vectors in a query to support e.g. selective timeshift in Grafana Signed-off-by: garrettlish <garrett.li.sh@gmail.com> * add docs for offset modifier * refine offset expr * add offset unit test in engine.go * remove duplicated offset shift * add offset test in range vector test Co-authored-by: garrettlish <garrettlish@163.com>	5 years ago
Cyril Tovena	a7e0770acb	Fixes a race introduced by #3434. (#3517)	5 years ago
Sandeep Sukhani	b06d9703d5	fix live tailing of logs from Loki (#3509)	5 years ago
Kaviraj	14b2c093c1	Add unique promtail_instance id to labels for gcptarget (#3501) ... * Add unique promtail_instance id to labels for gcptarget Rationale: To make labelset unique per promtail instance to avoid out-of-order errors * Update docs/sources/clients/promtail/scraping.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	5 years ago
Sandeep Sukhani	835f710bc9	make prefix for keys of objects created by boltdb-shipper configurable (#3491) ... * make prefix for keys of objects created by boltdb-shipper configurable * shared store key prefix should have a path separator	5 years ago
Cyril Tovena	6148794b9b	Fixes a bug where unpack would mutate log line. (#3502) ... * Fixes a bug where unpack would mutate log line. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add reset. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Neven Miculinic	b67a07e1fb	Added path information to deleted tailed file (#3457)	5 years ago
Aditya C S	0f5fceef8f	Support labelallow stage in Promtail (#3468) ... * add labelallow stage * added docs * add docs * fix lint * update stage name in _index.md	5 years ago
Aditya C S	b1ea8ac050	support math functions in line_format and label_format (#3434) ... * support math functions in line_format and label_format * fix lint and tests * added docs and more tests * update doc * remove toString and int64 * doc adjustement. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	d6ecfd2431	Set the byte slice cap correctly when unsafely converting string. (#3487) ... Fixes #3485 Should also unblock #3481 When unsafely converting string to bytes the cap of the bytes is lost but not when doing bytes to string. It seems that this is only a problem when using go 1.16 or when using delve. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	bf4fa6613a	support max_message_length configuration for syslog parser (#3400) ... * support max_message_length configuration for syslog parser * address review comments	5 years ago
Marco Pracucci	94041307b1	Upgrade Cortex (#3466) ... * Upgrade Cortex Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated vendor/modules.txt Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Cyril Tovena	64decdbcb6	Set a max size for the logql parser to 5k. (#3471) ... * Set a max size for the logql parser to 5k. The API anyway receives the query via querystring which is normally max out by 2k based on RFC2616. This will improve fuzzing too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better error. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	c49e807a55	Fixes Issue 28593: loki:fuzz_parse_expr: Timeout in fuzz_parse_expr. (#3470) ... This was caused by a lot of allocations when parsing range duration. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	e29d53631e	Fixes out-of-memory fuzzing issue. (#3469) ... * Fixes out-of-memory fuzzing issue. Fixes Issue 28535 in oss-fuzz: loki:fuzz_parse_expr: Out-of-memory in fuzz_parse_expr Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	306cc72438	Revendor/cortex (#3446) ... * Upgrade Cortex in Loki Signed-off-by: Marco Pracucci <marco@pracucci.com> * proto compat * gofmt * changelog for deprecated querier.split-queries-by-day Co-authored-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Ed Welch	8012362674	Loki: Per Tenant Runtime Configs (#3460) ... * Add per tenant configs mechanism * Add per tenant logging of stream creation * fix tests * enable per tenant configs on push request logging. * fixing up the stream log, changing log levels to debug	5 years ago
Cyril Tovena	93f2b2d89b	Fixes split interval for metrics queries. (#3459) ... The step needs to be taken into account: - to be removed on each split end. - to not split queries that have a step lower than the split interval. The code is taken from Cortex but adjusted to keep the nanoseconds precision. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	a29901d33a	Add limit and line_returned in the query log. (#3423) ... * Add limit and line_returned in the query log. This can be useful to investigate how many wasted queries have been made by the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	ce4f78515f	Improve the parser hint tests. (#3429) ... I was able to find out where was the issue with that test yesterday. Now I realized that we can mutate (re-slice) the given log line, and so other test would fail. This also make the parallel back. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	7e9ac16dbf	Add fgfprof to Loki and Promtail. (#3424) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	6d48fb6942	change ReadStringAsSlice to ReadString so it doesn't parse quotes inside the packed _entry (#3432)	5 years ago
Cyril Tovena	32d668ceb0	Only unpack entry if the key `_entry` exist. (#3426) ... * Only unpack entry if the key `_entry` exist. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * s/kv/lbs/g Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix the test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Dmitry Ukolov	6ea7122777	Distributor: append several tests for http parser. (#3416)	5 years ago
Cyril Tovena	7886f35b8f	Introduce a unpack parser. (#3420) ... * Introduce a unpack parser. To go with the new pack stage in promtail https://github.com/grafana/loki/pull/3401 this PR introduces a unpack parser that will automatically unpack all labels packed into the log line and set back the original log line. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tweak promtail docs a little now that we have unpack in Loki Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	20ef66df65	Fixes a race in the scheduling limits. (#3417) ... * Fixes a race in the scheduling limits. Found out the hard way that sharding might not listen correctly to context and so still run queries after the upstream request ends. This can lead to panic when scheduling work which send on a closed channel. I've added a safe guard to wait for all jobs to end before ending the original request but also attemped to fix the sharding downstreamer to properly stop sending work when context is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	2dbdbee5e6	Loki: fix alignment of atomic 64 bit to work with 32 bit OS (#3411) ... * fix alignment of atomic 64 bit to work with 32 bit OS * add comment	5 years ago
gotjosh	144786e67d	Instrumentation: Add histogram for request duration on gRPC client to Ingesters (#3409) ... * Instrumentation: Add histogram for request duration on gRPC client to Ingesters Signed-off-by: gotjosh <josue@grafana.com> * fix lint Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	e57af51c37	Promtail: Add pack stage (#3401) ... * Creating a "wrap" stage which can take labels and embed them into a log line by wrapping the log line in JSON * improve tests, deterministic ordering of json * add test for quotes * add test for ingest timestamp * update docs * update docs * update docs * lint * test flake * rename wrap to pack * fix docs * rename files * typo	5 years ago
Danny Kopping	faa1eab620	Remove default docker pipeline stage - this is undocumented and unexpected behaviour (#3404) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Dr. Jan-Philip Gehrcke	0f881fbf78	distributor: fix snappy pb POST request handling (#3407) (#3408) ... Signed-off-by: Dr. Jan-Philip Gehrcke <jp@opstrace.com>	5 years ago
Ed Welch	877f524c36	Loki: Revert reduce list operations and crash after timeout (#3396) ... * revert: boltdbshipper: fix crash of "timeout" panic when boltdbshipper do init (#3369) This appears to have introduced a different problem, still investigating. * Revert "reduce number of list calls on shared object store when using boltdb-shipper (#3283)" This reverts commit 76e713f5	5 years ago
Ed Welch	cf8be4a55e	remove directories from object store list operation (#3394)	5 years ago
zhanghjster	a3a2c1a079	boltdbshipper: fix crash of "timeout" panic when boltdbshipper do init (#3369) ... * Fix timeout panic during boltdbshipper init. When boltdbshipper failed to load an invalid local index file, it try to reopen with a newly downloaded index, but it will panic with "timeout" error because the process still hold the flock created by the previous db open. This patch do the flock cleanup. * add a few more comments and change function name. Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Sandeep Sukhani	ba25b0fc80	increase db retain period in ingesters to cover index cache validity period as well (#3300) ... refactor ingester store query config to simplify the code	5 years ago
Cyril Tovena	70e87256d6	Add finalizer to zstd. (#3306) ... Fixes #3305 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	afb539fcf3	Fixes some 500 returned by querier when storage cancellation happens. (#3344) ... For unknown reasons, Cortex is transforming all errors to promql.ErrStorage on the storage layer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	f14ddb8bb3	fix nested captured groups indexing in replace stage (#3381) ... * fixes nested captured groups indexing in replace stage * fix typo in test * use lowercase names * fix lint error * fix lint error	5 years ago
Cyril Tovena	55b91fc793	Fixes head chunk iterator direction. (#3383) ... * Fixes head chunk iterator direction. For backward queries, since LogQL parser we are using a heapIterator in the headchunk to re-order properly entries. But we also reverse all iterators in the memchunk code, even the headchunk which causes reversal of already reversed entries. This PR skips reversal of the headchunk. Fixes #3345 Fixes #3208 This has for side effects: - when using replication you would not dedupe data properly anymore, since the data is not correctly ordered accross batches. - when using limit it would miss entries. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix reversal and time filtering of headchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	7613197593	prevents duplicate log lines from being replayed. closes #3378 (#3388)	5 years ago