apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	1efeace644	Fixes instant queries in the frontend. (#4091 ) * Fixes instant queries in the frontend. This is a set of fixes for instant queries: - correctly propagate statistics. - correctly convert sampleStream to vector. - correctly marshal to json. - correctly encode the time for the request. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karsten Jeschkies	17f4a73a24	Filter instant queries and shard them. (#3984 ) * Filter instan queries and shard them. * [WIP] Test instant query sharding * Trace casting error. * WRap LokiInstantRequest for Params interface. * Convert paras to request. * Convert vector. * Use proper query time stamp. * Assert number of calls. * Convert vector to multiple samples. * Format code. * Use type switch. * Format code. * Rename Matrix to proto conversion. * Remvoe comment on caching. * Update pkg/querier/queryrange/roundtrip_test.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove nil middlware. * Handle LokiInstantRequest correctly. * Assert reponse type. * Return LokiPromResponse. * Verify err in tests. * Set shards. * Correct result type. * Use limits middleware. * Use shars parameter in querier. * Return vector. * Unify params. * Change direction to FORWARD. Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	fed9115627	Improve decoding of JSON responses. (#4019 ) * Improves frontend allocations Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves also labels Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests and review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	bee456737e	Shards Series API. (#3856 ) * Shards Series API. This PR introduces Series Queries Sharding. It does not check the boundaries of ingesters data since I'm assuming https://github.com/grafana/loki/pull/3852 will be merge first. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	1f6453fb81	exposes loki codec (#3758 )	5 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	5 years ago
Marco Pracucci	94041307b1	Upgrade Cortex (#3466 ) * Upgrade Cortex Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated vendor/modules.txt Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Sandeep Sukhani	071a20803b	Revendor cortex to latest master (#2794 )	5 years ago
Owen Diehl	164f5cd0ae	Ruler/loki rule validator (#2589 ) * revendors cortex * loki multitenantmanager * vendoring compat * go mod	5 years ago
Cyril Tovena	9ad98dfa7c	Split label names queries in the frontend. (#2441 ) * Split the Series API by 24h. We currently split too much by using the same split duration for query range. Let's start with 24h, just tested in ops-tools and it works better. We could later introduce a config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Split label name queries in the frontend. Build on top of #2436 and adds the ability to split label names queries. Those queries are heavy to run for long period and so splitting by day should improve query performance. The frontend also support the old legacy label names API which has a different response. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Owen Diehl	268b7a3d03	bumps cortex & fixes conflicts (#2204 ) * bumps cortex & fixes conflicts * go mod tidy	6 years ago
Owen Diehl	1530120c2e	Logql comparison ops (#2182 ) * comparison binops * starts comparison operator docs * implements bool modifier for comparison binops * vector-vector binop merging cases, including bool modifier * logql docs formatting * resets protos * linter fixes * goimports local * gofmt	6 years ago
Aditya C S	5b8d9b4ed6	Support series query in query-frontend (#2123 )	6 years ago
Owen Diehl	89d80a6fc4	Feature/querysharding ii (#1927 ) * [wip] sharding evaluator/ast * [wip] continues experimenting with ast mapping * refactoring in preparation for binops * evaluators can pass state to other evaluators * compiler alignment * Evaluator method renamed to StepEvaluator * chained evaluator impl * tidying up sharding code * handling for ConcatSampleExpr * downstream iterator * structure for downstreaming asts * outlines sharding optimizations * work on sharding mapper * ast sharding optimizations * test for different logrange positions * shard mapper tests * stronger ast sharding & tests * shardmapper tests for string->string * removes sharding evaluator code * removes unused ctx arg * Revert "removes sharding evaluator code" This reverts commit `55d41b9519`. * interfaces for downstreaming, type conversions * sharding plumbing on frontend * type alignment in queryrange to downstream sharded queriers * downstreaming support for sharding incl storage code * removes chainedevaluator * comment alignment * storage shard injection * speccing out testware for sharding equivalence * [wip] shared engine refactor * sorting streams, sharding eval fixes * downstream evaluator embeds defaultevaluator * other pkgs adopt logql changes * metrics & logs use same middleware instantiation process * wires up shardingware * middleware per metrics/logfilter * empty step populating StepEvaluator promql.Matrix adapter * sharding metrics * log/span injection into sharded engine * sharding metrics avoids multiple instantiation * downstreamhandler tracing * sharding parameterized libsonnet * removes querier replicas * default 32 concurrency for workers * jsonnet correct level override * unquote true in yaml * lowercase error + downstreamEvaluator defaults to embedded defaultEvaluator * makes shardRecorder private * logs query on failed parse * refactors engine to be multi-use, minimizes logger injection, generalizes Query methods, removes Engine interface * basic tests for querysharding mware * [wip] concurrent evaluator * integrates stat propagation into sharding evaluator * splitby histogram * extends le bounds for bytes processed * byte throughput histogram buckets to 40gb * chunk duration mixin * fixes merge w/ field rename * derives logger in sharded engine via ctx & logs some downstream evaluators * moves sharded engine to top, adds comments * logs failed merge results in stats ctx * snapshotting stats merge logic is done more effectively * per query concurrency controlled via downstreamer * unexports decodereq * queryrange testware * downstreamer tests * pr requests	6 years ago
Cyril Tovena	9988ce4b9f	Improve protobuf serialization (#2031 ) * Generate custom proto type for Streams. This is the first step to improve protobuf serialization. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve protobuf serialization allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Saves allocations by stringifying labels only once. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Peter Štibraný	5ae6ea71f8	Update Cortex to master (#1785 ) * Update Cortex to v0.7.0-rc.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fixed errors after updating to latest master. queryrange.NewRetryMiddleware in NewLogFilterTripperware is now created without metrics, as that would collide with metrics in NewMetricTripperware. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make sure to start client.Pool before using it. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use shared singleton memberlist KV. Fix queryrange metrics. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make lint happy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Return err early. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Cyril Tovena	9be21a7c6b	This logs queries with latency tag when recording stats. (#1733 ) * This logs queries with latency tag when recording stats. Useful for building a dashboard of slow queries. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * mod-check. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Aditya C S	f9ff2a4b02	Add goimport local-prefixes configuration to .golangci.yml (#1710 )	6 years ago
Cyril Tovena	2f061f3a8a	Ensure status codes are set correctly in the frontend. (#1684 ) * Returns httpgrpc error in roundtriper. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Ensure error codes are set correctly in the frontend. Previously we had cases where 400 would endup as 500. We should use httpgrpc.Errorf when possible. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	2af69cb57c	Frontend & Querier query statistics instrumentation. (#1661 ) * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates cortex and add a record metric function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some tests in the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes unused tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Replace global by an injected interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds a test to make sure the stats middleware update response statistics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	93eb4ad27c	Add statistics to query_range and instant_query API. (#1615 ) * Refactor stats package to use proto. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge statistics in the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve engine logging. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes GRPC by rolling back Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test to verify stats in the LogQL engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set circle ci build image to 0.9.1 which contains protobuf regression Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix drone CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Change timing data from nanoseconds to seconds like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve naming of stats Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move stats property under data property like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adjust queryrange frontend to use the new stats property Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup unused structs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for stats API. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Format proto file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds missing godoc on logql.Result Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	589e53d844	Fixes response json encoding and add regression test. (#1488 ) Also align stream labels alphabetical order with querier. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	4153740a12	Loki Query Frontend (#1442 ) * Adds frontend to Loki. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sneaky bug in entries sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tweak jsonnet deployments and add a way to lint/fmt. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tests and rollback change in loghttp package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a flaky test that might run one more goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes windows build. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tracing in the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test stream to proto conversion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flappy retry test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove err shadowing in stopQueryFrontend as it was confusing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor grpc message size in the libsonnet config file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Don't check auth header for GRPC TransferChunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Query frontend (#3) * frontend codec merging optimizations * codec benchmarks * removes unused bounds code in queryrange ordering * [wip] splitby uses channels instead of sub batching intervals * splitBy channel limit test * single allocation for merging entries from a single stream * skip merging loki responses when limit is already hit * removes checks for unlimited queries in queryrange * removes splitByInterval{,.interval} spans * removes interval_batch_size from jsonnet lib * moves benchmark utils to own file * renames markers -> entries * priority queue comments * Removes unused logRequest. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Sets the cache interval to the same split interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing import libsonnet for the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Frontend should not be a cluster IP. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	6 years ago

23 Commits (a58aea1bf456280aa06fb2f2d09d2d005eefa6ed)