457 Commits (414f95fedc1b921dc9891d77a4ee16e9729309fd)

Author	SHA1	Message	Date
Owen Diehl	414f95fedc	Literal Expressions in LogQL (#1677) ... * binops in ast * bin op associativity & precedence * binOpEvaluator work * defers close only if constructed without error * tests binary ops * more binops * updates docs * changelog * number literals in ast * [wip] literalExpr parsing * number parsing * signed literals * propagates evaluator close, handles healthchecks * literal evaluator works on non commutative operations * literalExprs cannot be used as legs of logical/set binops * removes comment * single literalExpr tests * reduces binops with literals in ast construction where possible * doc updates * scalar datatypes in logql * scalar serialization type * increases safety and reduces complexity in ast evaluator * recursive literal binop reduction parse test, improves parse errors on literal construction * vector + literal test	5 years ago
Aditya C S	460bb5b857	Support entire extracted value map in template pipeline stage (#1666) ... Refactor template stage processing. This sets all extracted key available, plus if the source is available adds the extra key `Value` for backward compatibility. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Signed-off-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	d6b85876ca	max line lengths (component + tenant overrides) (#1686) ... * bytesize flag support * max line limits * configurable tenant override options for max line size * bytesize yaml support * validator methods * distributor validator type * human friendly line size output, distributor test * invalid labels distributor test * docs + vendor * removes dead limits code	5 years ago
Cyril Tovena	2f061f3a8a	Ensure status codes are set correctly in the frontend. (#1684) ... * Returns httpgrpc error in roundtriper. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Ensure error codes are set correctly in the frontend. Previously we had cases where 400 would endup as 500. We should use httpgrpc.Errorf when possible. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	4210fcef35	--dry-run Promtail. (#1652) ... * --dry-run Promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove changed on gomod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flaky test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Ensure position file is readonly when doing dry-run Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add information about dry run mode in the troubleshooting section. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more spaces. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journaltarget refactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journaltarget refactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journaltarget refactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	24d9956653	This improves the log ouput for statistics in the logcli. (#1644) ... * This improves the log ouput for statistics in the logcli. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Please the linter Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Edward Welch	0202884d27	Automatically prune metrics from the /metrics output of the promtail metrics pipeline stage after an idle period. ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Edward Welch	4aeeeea1c8	Allow a metric defined as a counter to match all lines, useful for creating line count metrics which include all your labels. ... Found a bug in the the test runner where it didn't fail if the actual error was nil but the test expected an error Added some tests to the counters to test valid configs Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Owen Diehl	6bbb61eb2f	Binary operators in LogQL (#1662) ... * binops in ast * bin op associativity & precedence * binOpEvaluator work * defers close only if constructed without error * tests binary ops * more binops * updates docs * changelog * better logql parsing errors for binops Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * adds ^ operator	5 years ago
Cyril Tovena	814cc87eb2	Pipe data to Promtail (#1649) ... * Adds readerTarget that read from io.Reader Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates comment and function names. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds stdin manager Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds default config with labels and support for static configs labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds a test with pipeline stages. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes race on shutdown and adds documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes deadlock on server.run Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/clients/promtail/troubleshooting.md Co-Authored-By: Owen Diehl <ow.diehl@gmail.com> * gofmt Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * gofmt Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * unlock mutex if Promtail.Run is called when Promtail is stopped Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	5 years ago
Cyril Tovena	2af69cb57c	Frontend & Querier query statistics instrumentation. (#1661) ... * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates cortex and add a record metric function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some tests in the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes unused tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Replace global by an injected interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds a test to make sure the stats middleware update response statistics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	b70f691cd4	Support custom prefix name in metrics stage (#1664)	5 years ago
Robert Fratto	267dbe0085	pkg/promtail/positions: handle empty positions file (#1660) ... If the positions file is empty, unmarshaling it will lead to a nil map being created. This commit ensures that positions is always initialized to be non-nil. Potentially handles #1659.	5 years ago
Owen Diehl	e0a77656fd	removes duplicate logRangeExprExt grammar (#1651) ... * removes duplicate logRangeExprExt grammar * linting errors	5 years ago
Owen Diehl	80e938c412	adds query stats and org id options in logcli (#1638)	5 years ago
Cyril Tovena	375fc86af7	Fixes stats summary computation. (#1636) ... * Fixes stats summary computation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes division by zero Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	6930fb2a5c	adds stringer methods for all ast expr types (#1630) ... * adds stringer methods for all ast expr types * Update pkg/logql/ast.go Co-Authored-By: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	93eb4ad27c	Add statistics to query_range and instant_query API. (#1615) ... * Refactor stats package to use proto. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge statistics in the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve engine logging. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes GRPC by rolling back Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test to verify stats in the LogQL engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set circle ci build image to 0.9.1 which contains protobuf regression Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix drone CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Change timing data from nanoseconds to seconds like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve naming of stats Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move stats property under data property like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adjust queryrange frontend to use the new stats property Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup unused structs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for stats API. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Format proto file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds missing godoc on logql.Result Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	1773900659	compiler guarantees for logql exprs (#1626)	5 years ago
Owen Diehl	a53477f8e4	cache key cant be reused when an interval changes (#1616)	5 years ago
Owen Diehl	3e075497cd	Feature/per tenant splitby (#1565) ... * split by overrides by tenant * tenant aware metric query splitting * consistent split_queries_by_interval naming * updates readme * fixes rebase conflict * refactor to support limit based splitby intervals in all locations * simplifies limits cachesplitter * fixes comment	5 years ago
Owen Diehl	018e6bcdfd	bumps cortex to 0.6.1 master (#1612)	5 years ago
Owen Diehl	b7d23f41bd	Decouple logql engine/AST from execution context (#1605) ... * logql engine is an interface * [wip] begins agnostic logql evaluator work * decouples logql AST from execution context * healthcheck comments	5 years ago
Ed Welch	64798e2a98	tweaking the names a little per some discussions to make things easier to understand (#1582) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	f1b8d4d8ad	Improve logql query statistics collection. (#1573) ... * Improve logql query statistics collection. This also add information about ingester queries. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad copy/past in the result log. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ingester tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve headchunk efficiency. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix bad commit on master. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes new interface of ingester grpc server. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve documentations of fields. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	790ba403b1	Add another sleep to the canary reconnects this time if connection succeeds but there is an error receiving the stream. (#1581) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Sandeep Sukhani	c7a3ec54f1	limit for concurrent tail requests (#1562) ... * limit for concurrent tail requests When a new request comes in, querier would ask one of the active ingesters for count of active tail requests for the user If the count exceeds the configured limits for the user then tail request would be rejected right away. * goimports querier_test.go Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * query all ingesters for their max tail count Signed-off-by: Edward Welch <edward.welch@grafana.com> * changed to check only ACTIVE ingesters and tweaked error handling some Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Ed Welch <ed@oqqer.com>	5 years ago
Peter Štibraný	d4cae667ae	loki: use new runtimeconfig package from Cortex (#1484) ... * loki: use new runtimeconfig package from Cortex runtimeconfig is a generalization of previous "overrides" mechanism, and can be used not for per-tenant limits, but other parts of Loki too (only one additional usage for now is "multi" KV client). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed RuntimeConfig from Ingester dependencies. Ingester already depends on RuntimeConfig via Store and Overrides mechanism. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed extra new line. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Remove unused method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix broken code after rebasing. Removed incorrect comment. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated docs with section about runtime configuration. Mostly copied from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated Cortex to current master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Owen Diehl	841befaf8e	Feature/query ingesters within (#1572) ... * querier.query-ingesters-within * avoids an unnecessary heapIter * updates changelog/config readme * variable reuse	5 years ago
Owen Diehl	1269c92663	refactors splitby to not require buffered channels (#1569)	5 years ago
Owen Diehl	5026dfea57	ingester.max-chunk-age (#1558) ... * ingester.max-chunk-age * bumps helm & changelog * adjusts max-chunk-age test * bumps wait period to let ci catch up * updates changelog	5 years ago
Owen Diehl	bde0389979	adds span metadata for split queries (#1567) ... * adds span metadata for split queries * finishes interval span	5 years ago
Joe Elliott	d3972852ee	Minor typo fixes (#1545) ... Signed-off-by: Joe Elliott <number101010@gmail.com>	5 years ago
Owen Diehl	366e06f587	legacy endpoint 400s metric queries (#1541)	5 years ago
Ganesh Vernekar	1761db6d71	Support <selector> <range> <filters> for aggregation (#1526) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Cyril Tovena	4d853b36e5	Adds support for the old query string regexp in the frontend. (#1522) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Robert Fratto	65013218b7	pkg/chunkenc: ignore duplicate lines pushed to a stream (#1519) ... * pkg/chunkenc: ignore duplicate lines pushed to a stream This commit changes the behavior of appending to a stream to ignore an incoming line if its timestamp and contents match the previous line received. This should reduce the chances of ingesters storing duplicate log lines when clients retry push requests whenever a 500 is returned. Fixes #1517	5 years ago
Edward Welch	b43039ed5a	update the position and size metrics at the same time to try to end the race between these two metrics and the two timers that care about position and size ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Chance Zibolski	3a2e5d4bbc	pkg/promtail: Initialize counters to 0 when creating client (#1453) ... This ensures metrics are not missing before they're incremented the first time, as recommended by the upstream Prometheus instrumentation guidelines: https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics. Currently, in the case of dropped_entries_total and dropped_bytes_total, the metric is simply missing until the first occurrence of an entry being dropped.	5 years ago
Sandeep Sukhani	fbcfad13dc	fix panic in tailer when max tail duration exceeds (#1511)	5 years ago
petruha	4f314f3df4	Fileglob parsing fixes (#1485) ... * Update doublestar module to version 1.2.2. * Add absolute path for each element in list. * Remove doublestar test and update vendore module version.	5 years ago
Robert Fratto	9227eed9f4	pkg/ingester: limit total number of errors a stream can return on push (#1071) ... * pkg/ingester: limit total number of errors a stream can return on push * pkg/ingester: s/max_ignored_stream_errors/max_returned_stream_errors	5 years ago
Beorn Facchini	88681a9974	Read websocket close in tail handler (#1383) ... * Read websocket close in tail handler * Send close message to websocket on quit signals	5 years ago
Cyril Tovena	f3332272ed	Update Loki mixin to include frontend QPS and latency. (#1497) ... Also rework frontend route registration to name metrics and spans correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	35dd07c4e0	fixes a panic where prefetch could try to operate on a nil object (#1499) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Marco Pracucci	ec40515d31	Added max streams per user global limit (#1493) ... * Added max streams per user global limit Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated changelog Signed-off-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Marco Pracucci	ce407d3276	Add ingestion rate global limit support (#1486) ... * Added ingestion rate global limit support Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed code comments Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated config doc Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated changelog Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated doc Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Cyril Tovena	33f70b70ce	Add duplicates info and remove timing informations. (#1496) ... * Add duplicates info and remove timing informations. Timing information in the chunkenc iterator are too eating too much resources. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	7c4906786d	pre-fetch the next chunk in parallel to processing the current chunk such that we aren't waiting for fetches from GCS and spend all our CPU time decompressing and processing (#1495) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Owen Diehl	a1d090e161	Fix/deadlock frontend queue (#1490) ... * buffers splitby interval channels to prevent deadlocks * improves codec merge test requiring merging from multiple intervals * bound concurrency by min(parallelism,jobs) * approximates goroutine test	5 years ago