192 Commits (a99c73dd97bf55d912d391339a7b82acccabf915)

Author	SHA1	Message	Date
Cyril Tovena	fa00b90128	Fix error swallowed in the frontend. (#2807) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	071a20803b	Revendor cortex to latest master (#2794)	5 years ago
Cyril Tovena	fd5cb96693	LogQL: Labels and Metrics Extraction (#2769) ... * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Cyril Tovena	3640429532	Bypass sharding middleware when a query can't be sharded. (#2709) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	337db2fedf	revendor cortex to latest master (#2627)	5 years ago
Owen Diehl	164f5cd0ae	Ruler/loki rule validator (#2589) ... * revendors cortex * loki multitenantmanager * vendoring compat * go mod	5 years ago
Owen Diehl	f8ee408180	series endpoint uses normal splits (#2472)	5 years ago
Sandeep Sukhani	4a358bc9ea	revendor cortex to latest master (#2463)	5 years ago
Cyril Tovena	9ad98dfa7c	Split label names queries in the frontend. (#2441) ... * Split the Series API by 24h. We currently split too much by using the same split duration for query range. Let's start with 24h, just tested in ops-tools and it works better. We could later introduce a config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Split label name queries in the frontend. Build on top of #2436 and adds the ability to split label names queries. Those queries are heavy to run for long period and so splitting by day should improve query performance. The frontend also support the old legacy label names API which has a different response. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	48b8504c30	lock fix for flaky test (#2268)	5 years ago
Owen Diehl	70900be57d	avoids further time splitting in querysharding mware (#2252) ... * avoids further time splitting in querysharding mware * removes duplicate utility fns * linting	5 years ago
Owen Diehl	268b7a3d03	bumps cortex & fixes conflicts (#2204) ... * bumps cortex & fixes conflicts * go mod tidy	5 years ago
Owen Diehl	61fc175361	Fix vector-scalar comparisons (#2189) ... * fmts shards in instance logger * vector-scalar comparisons now filter by default * gofmt	5 years ago
Owen Diehl	1530120c2e	Logql comparison ops (#2182) ... * comparison binops * starts comparison operator docs * implements bool modifier for comparison binops * vector-vector binop merging cases, including bool modifier * logql docs formatting * resets protos * linter fixes * goimports local * gofmt	5 years ago
Aditya C S	5b8d9b4ed6	Support series query in query-frontend (#2123)	5 years ago
Cyril Tovena	18f1b0640d	Bytes aggregations (#2150) ... * Add bytes_rate and bytes_over_time. Still need to works on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More lint fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More comments. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix bad conflic resolution. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Peter Štibraný	e4a5360586	Update vendored Cortex to master (1.1+) (#2149) ... * Updated Cortex to master (post 1.1) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use new modules system from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Treat query errors reported by chunk store as client errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed obsolete tests. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix race for ct variable. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Owen Diehl	89d80a6fc4	Feature/querysharding ii (#1927) ... * [wip] sharding evaluator/ast * [wip] continues experimenting with ast mapping * refactoring in preparation for binops * evaluators can pass state to other evaluators * compiler alignment * Evaluator method renamed to StepEvaluator * chained evaluator impl * tidying up sharding code * handling for ConcatSampleExpr * downstream iterator * structure for downstreaming asts * outlines sharding optimizations * work on sharding mapper * ast sharding optimizations * test for different logrange positions * shard mapper tests * stronger ast sharding & tests * shardmapper tests for string->string * removes sharding evaluator code * removes unused ctx arg * Revert "removes sharding evaluator code" This reverts commit 55d41b9519. * interfaces for downstreaming, type conversions * sharding plumbing on frontend * type alignment in queryrange to downstream sharded queriers * downstreaming support for sharding incl storage code * removes chainedevaluator * comment alignment * storage shard injection * speccing out testware for sharding equivalence * [wip] shared engine refactor * sorting streams, sharding eval fixes * downstream evaluator embeds defaultevaluator * other pkgs adopt logql changes * metrics & logs use same middleware instantiation process * wires up shardingware * middleware per metrics/logfilter * empty step populating StepEvaluator promql.Matrix adapter * sharding metrics * log/span injection into sharded engine * sharding metrics avoids multiple instantiation * downstreamhandler tracing * sharding parameterized libsonnet * removes querier replicas * default 32 concurrency for workers * jsonnet correct level override * unquote true in yaml * lowercase error + downstreamEvaluator defaults to embedded defaultEvaluator * makes shardRecorder private * logs query on failed parse * refactors engine to be multi-use, minimizes logger injection, generalizes Query methods, removes Engine interface * basic tests for querysharding mware * [wip] concurrent evaluator * integrates stat propagation into sharding evaluator * splitby histogram * extends le bounds for bytes processed * byte throughput histogram buckets to 40gb * chunk duration mixin * fixes merge w/ field rename * derives logger in sharded engine via ctx & logs some downstream evaluators * moves sharded engine to top, adds comments * logs failed merge results in stats ctx * snapshotting stats merge logic is done more effectively * per query concurrency controlled via downstreamer * unexports decodereq * queryrange testware * downstreamer tests * pr requests	5 years ago
Aditya C S	199e334126	Update loki to cortex master (#2030)	5 years ago
Cyril Tovena	9988ce4b9f	Improve protobuf serialization (#2031) ... * Generate custom proto type for Streams. This is the first step to improve protobuf serialization. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve protobuf serialization allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Saves allocations by stringifying labels only once. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Arve Knudsen	d975b1fdd8	Fix spelling of per second (#2055) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	5 years ago
Cyril Tovena	4fd670dc7a	Support Post request in the frontend queryrange handler. (#2023) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	577d8eb168	Ensure trace propagation in our logs. (#1977) ... * Ensure trace propagation in our logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing import. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix function usage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	49f04ef4bc	update cortex to latest master (#1956)	5 years ago
Aditya C S	f41273182c	Support configurable maximum of the limits parameter (#1798)	5 years ago
Peter Štibraný	5ae6ea71f8	Update Cortex to master (#1785) ... * Update Cortex to v0.7.0-rc.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. queryrange.NewRetryMiddleware in NewLogFilterTripperware is now created without metrics, as that would collide with metrics in NewMetricTripperware. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make sure to start client.Pool before using it. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use shared singleton memberlist KV. Fix queryrange metrics. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make lint happy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Return err early. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Cyril Tovena	4da4d74311	Query range should not support date where start == end. ... Since we are always checking against start inclusively and not end, this means this type of query will always returns no result. Fixes #1702, the frontend was not handling this type of query correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	9be21a7c6b	This logs queries with latency tag when recording stats. (#1733) ... * This logs queries with latency tag when recording stats. Useful for building a dashboard of slow queries. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * mod-check. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	6363e8da0a	Better logql metric status code. (#1718) ... * Fixes querier logql error to be 400 only for ast parsing error. Improve frontend metrics to use a more granular status_code. Improve prometheus histogram granularity. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add latency type dimensions to bytes throughput metric. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use IsParseError in panic recovering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	f9ff2a4b02	Add goimport local-prefixes configuration to .golangci.yml (#1710)	5 years ago
Cyril Tovena	2f061f3a8a	Ensure status codes are set correctly in the frontend. (#1684) ... * Returns httpgrpc error in roundtriper. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Ensure error codes are set correctly in the frontend. Previously we had cases where 400 would endup as 500. We should use httpgrpc.Errorf when possible. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	2af69cb57c	Frontend & Querier query statistics instrumentation. (#1661) ... * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates cortex and add a record metric function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some tests in the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes unused tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Replace global by an injected interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds a test to make sure the stats middleware update response statistics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	93eb4ad27c	Add statistics to query_range and instant_query API. (#1615) ... * Refactor stats package to use proto. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge statistics in the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve engine logging. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes GRPC by rolling back Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test to verify stats in the LogQL engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set circle ci build image to 0.9.1 which contains protobuf regression Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix drone CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Change timing data from nanoseconds to seconds like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve naming of stats Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move stats property under data property like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adjust queryrange frontend to use the new stats property Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup unused structs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for stats API. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Format proto file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds missing godoc on logql.Result Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	a53477f8e4	cache key cant be reused when an interval changes (#1616)	5 years ago
Owen Diehl	3e075497cd	Feature/per tenant splitby (#1565) ... * split by overrides by tenant * tenant aware metric query splitting * consistent split_queries_by_interval naming * updates readme * fixes rebase conflict * refactor to support limit based splitby intervals in all locations * simplifies limits cachesplitter * fixes comment	5 years ago
Owen Diehl	018e6bcdfd	bumps cortex to 0.6.1 master (#1612)	5 years ago
Owen Diehl	1269c92663	refactors splitby to not require buffered channels (#1569)	5 years ago
Owen Diehl	bde0389979	adds span metadata for split queries (#1567) ... * adds span metadata for split queries * finishes interval span	5 years ago
Cyril Tovena	4d853b36e5	Adds support for the old query string regexp in the frontend. (#1522) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	a1d090e161	Fix/deadlock frontend queue (#1490) ... * buffers splitby interval channels to prevent deadlocks * improves codec merge test requiring merging from multiple intervals * bound concurrency by min(parallelism,jobs) * approximates goroutine test	5 years ago
Cyril Tovena	589e53d844	Fixes response json encoding and add regression test. (#1488) ... Also align stream labels alphabetical order with querier. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	4153740a12	Loki Query Frontend (#1442) ... * Adds frontend to Loki. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sneaky bug in entries sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tweak jsonnet deployments and add a way to lint/fmt. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tests and rollback change in loghttp package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a flaky test that might run one more goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes windows build. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tracing in the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test stream to proto conversion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flappy retry test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove err shadowing in stopQueryFrontend as it was confusing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor grpc message size in the libsonnet config file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Don't check auth header for GRPC TransferChunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Query frontend (#3) * frontend codec merging optimizations * codec benchmarks * removes unused bounds code in queryrange ordering * [wip] splitby uses channels instead of sub batching intervals * splitBy channel limit test * single allocation for merging entries from a single stream * skip merging loki responses when limit is already hit * removes checks for unlimited queries in queryrange * removes splitByInterval{,.interval} spans * removes interval_batch_size from jsonnet lib * moves benchmark utils to own file * renames markers -> entries * priority queue comments * Removes unused logRequest. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Sets the cache interval to the same split interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing import libsonnet for the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Frontend should not be a cluster IP. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago