28 Commits (e8cce09de29dc58c38b333e72dfe11613d49fbb3)

Author	SHA1	Message	Date
Cyril Tovena	3e46067cbb	Fixes a bug where slice of Entries where not zeroed before decoding json. (#3249) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	070246c0a4	Uses custom json-iter decoder for log entries. (#3163) ... * Uses custom json-iter decoder for log entries. Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage. For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line. On top of that we're going to process that json payload 50% faster. ``` ❯ benchcmp before.txt after.txt2 benchmark old ns/op new ns/op delta Benchmark_DecodePushRequest-16 13509236 6677037 -50.57% benchmark old allocs new allocs delta Benchmark_DecodePushRequest-16 106149 38719 -63.52% benchmark old bytes new bytes delta Benchmark_DecodePushRequest-16 10350362 5222989 -49.54% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes query-tee tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	aee064d0a8	Loki: Series API will return all series with no match or empty matcher (#2254) ... * Allow series queries with no match or an empty {} matcher which will return all series for the provided timeframe. * fix test * fix lint	5 years ago
Cyril Tovena	cb76da6c1d	Fixes path prefix in the querier. (#2178) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	89d80a6fc4	Feature/querysharding ii (#1927) ... * [wip] sharding evaluator/ast * [wip] continues experimenting with ast mapping * refactoring in preparation for binops * evaluators can pass state to other evaluators * compiler alignment * Evaluator method renamed to StepEvaluator * chained evaluator impl * tidying up sharding code * handling for ConcatSampleExpr * downstream iterator * structure for downstreaming asts * outlines sharding optimizations * work on sharding mapper * ast sharding optimizations * test for different logrange positions * shard mapper tests * stronger ast sharding & tests * shardmapper tests for string->string * removes sharding evaluator code * removes unused ctx arg * Revert "removes sharding evaluator code" This reverts commit 55d41b9519. * interfaces for downstreaming, type conversions * sharding plumbing on frontend * type alignment in queryrange to downstream sharded queriers * downstreaming support for sharding incl storage code * removes chainedevaluator * comment alignment * storage shard injection * speccing out testware for sharding equivalence * [wip] shared engine refactor * sorting streams, sharding eval fixes * downstream evaluator embeds defaultevaluator * other pkgs adopt logql changes * metrics & logs use same middleware instantiation process * wires up shardingware * middleware per metrics/logfilter * empty step populating StepEvaluator promql.Matrix adapter * sharding metrics * log/span injection into sharded engine * sharding metrics avoids multiple instantiation * downstreamhandler tracing * sharding parameterized libsonnet * removes querier replicas * default 32 concurrency for workers * jsonnet correct level override * unquote true in yaml * lowercase error + downstreamEvaluator defaults to embedded defaultEvaluator * makes shardRecorder private * logs query on failed parse * refactors engine to be multi-use, minimizes logger injection, generalizes Query methods, removes Engine interface * basic tests for querysharding mware * [wip] concurrent evaluator * integrates stat propagation into sharding evaluator * splitby histogram * extends le bounds for bytes processed * byte throughput histogram buckets to 40gb * chunk duration mixin * fixes merge w/ field rename * derives logger in sharded engine via ctx & logs some downstream evaluators * moves sharded engine to top, adds comments * logs failed merge results in stats ctx * snapshotting stats merge logic is done more effectively * per query concurrency controlled via downstreamer * unexports decodereq * queryrange testware * downstreamer tests * pr requests	5 years ago
Ed Welch	c7a1a1fb2e	Loki: Adds an `interval` paramater to query_range queries allowing a sampling of events to be returned based on the provided interval (#1965) ... * Adds an `interval` paramater to query_range queries allowing a sampling of events to be returned based on the provided interval Signed-off-by: Ed Welch <edward.welch@grafana.com> * lint Signed-off-by: Ed Welch <edward.welch@grafana.com> * extract function for parsing parameter Signed-off-by: Ed Welch <edward.welch@grafana.com> * Clarify the experimental nature of interval in the docs. Signed-off-by: Ed Welch <edward.welch@grafana.com> * make min time a constant Signed-off-by: Ed Welch <edward.welch@grafana.com> * Expand the docs around the experimental nature of interval a little more Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Ben Ye	9a62deb829	Support series API in logcli (#1861) ... * support series API in logcli Signed-off-by: yeya24 <yb532204897@gmail.com> add changelog Signed-off-by: yeya24 <yb532204897@gmail.com> * update logcli docs Signed-off-by: yeya24 <yb532204897@gmail.com>	5 years ago
Owen Diehl	726b8cd6d4	support `match[]` encoding (#1875)	5 years ago
Ed Welch	f5d79aee7a	Revert "loki: Allow setting a 'step' when querying logs. (#1773)" (#1777) ... This reverts commit 934a0b8d Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Ed Welch	934a0b8dc6	loki: Allow setting a 'step' when querying logs. (#1773) ... * Setting a step on log queries will result in similar behavior as metrics, only returning logs at multiples of the step Signed-off-by: Edward Welch <edward.welch@grafana.com> * clear up logic a little to make it easier to follow Signed-off-by: Edward Welch <edward.welch@grafana.com> * try to protect future us from future us Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	4da4d74311	Query range should not support date where start == end. ... Since we are always checking against start inclusively and not end, this means this type of query will always returns no result. Fixes #1702, the frontend was not handling this type of query correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Aditya C S	f9ff2a4b02	Add goimport local-prefixes configuration to .golangci.yml (#1710)	5 years ago
Owen Diehl	db6fd7e779	api support for scalars (#1704) ... * adds conversions to loghttp scalar types * logcli prints scalars, lothttp scalars use prometheus json marshaling	5 years ago
Owen Diehl	414f95fedc	Literal Expressions in LogQL (#1677) ... * binops in ast * bin op associativity & precedence * binOpEvaluator work * defers close only if constructed without error * tests binary ops * more binops * updates docs * changelog * number literals in ast * [wip] literalExpr parsing * number parsing * signed literals * propagates evaluator close, handles healthchecks * literal evaluator works on non commutative operations * literalExprs cannot be used as legs of logical/set binops * removes comment * single literalExpr tests * reduces binops with literals in ast construction where possible * doc updates * scalar datatypes in logql * scalar serialization type * increases safety and reduces complexity in ast evaluator * recursive literal binop reduction parse test, improves parse errors on literal construction * vector + literal test	5 years ago
Cyril Tovena	2af69cb57c	Frontend & Querier query statistics instrumentation. (#1661) ... * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates cortex and add a record metric function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some tests in the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes unused tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Replace global by an injected interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds a test to make sure the stats middleware update response statistics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	93eb4ad27c	Add statistics to query_range and instant_query API. (#1615) ... * Refactor stats package to use proto. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge statistics in the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve engine logging. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes GRPC by rolling back Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test to verify stats in the LogQL engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set circle ci build image to 0.9.1 which contains protobuf regression Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix drone CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Change timing data from nanoseconds to seconds like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve naming of stats Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move stats property under data property like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adjust queryrange frontend to use the new stats property Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup unused structs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for stats API. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Format proto file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds missing godoc on logql.Result Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	4153740a12	Loki Query Frontend (#1442) ... * Adds frontend to Loki. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sneaky bug in entries sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tweak jsonnet deployments and add a way to lint/fmt. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tests and rollback change in loghttp package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a flaky test that might run one more goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes windows build. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tracing in the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test stream to proto conversion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flappy retry test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove err shadowing in stopQueryFrontend as it was confusing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor grpc message size in the libsonnet config file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Don't check auth header for GRPC TransferChunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Query frontend (#3) * frontend codec merging optimizations * codec benchmarks * removes unused bounds code in queryrange ordering * [wip] splitby uses channels instead of sub batching intervals * splitBy channel limit test * single allocation for merging entries from a single stream * skip merging loki responses when limit is already hit * removes checks for unlimited queries in queryrange * removes splitByInterval{,.interval} spans * removes interval_batch_size from jsonnet lib * moves benchmark utils to own file * renames markers -> entries * priority queue comments * Removes unused logRequest. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Sets the cache interval to the same split interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing import libsonnet for the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Frontend should not be a cluster IP. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Cyril Tovena	10b5bcab72	Validates limit parameter. (#1470) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	dafb9d8425	Enables Series API in loki (#1419) ... * helpers for parsing /series requests * logproto.SeriesResponse impl * passes unparsed matchers in protobuf for simplicity * ingester support for series queries * labels map coercion * fixes label match loghttp test * removes labelmatcher protobuf adapters * series api impl * tests matchers for dispatching series queries * series looked up serially for /series api * splits series api async code and remove unnecessary type assertion * adds series handle to /api/prom/series and includes docs * compatibility changes w/ upstream merge	6 years ago
Edward Welch	ea5eec2abf	respond to label queries at `loki/api/v1/labels` instead of `loki/api/v1/label`	6 years ago
Owen Diehl	5227f5bee0	application/x-www-form-urlencoded support (#1381) ... * application/x-www-form-urlencoded support * handler func instead of unnecessary struct	6 years ago
Daniel D'Abate	df89859369	HTTP API: Support duration and float formats for step parameter (#1355) ... * Support duration and float formats for step parameter in http api * Support Prometheus duration format and add missing tests * Fixes linter Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	b34c2b0bd3	Move to jsoniter instead of default json package (#1249) ... * Move to jsoniter instead of default json package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes jsoniter on the logcli output Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	27f17bc1de	Fixes linter warning from the yacc file. (#1328) ... * Fixes linter warning from the yacc file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves sed to work on POSIX Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Robert Fratto	822e94fe6a	Fix v1 label API to be Prometheus-compatible (#1322) ... The label and label values API will now return the expected form of: ```json { "status": "success", "data": [ "label1", "label2", "labeln" ] } ```	6 years ago
Cyril Tovena	1abe8841ce	Moves request parsing into the loghttp package (#1171) ... * refactor query requests parsing Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add tests and documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Joe Elliott	bb2b925255	Update `/loki/api/v1/push` to use the v1 json format (#1145) ... Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
Joe Elliott	9614b021d1	Loki HTTP/JSON Model Layer (#1022) ... * First pass data model Signed-off-by: Joe Elliott <number101010@gmail.com> * Use prom model b/c we're serializing promql objects Signed-off-by: Joe Elliott <number101010@gmail.com> * Added legacy query support and tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added legacy label test Signed-off-by: Joe Elliott <number101010@gmail.com> * Added tail response marshalling and tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed marshallers and test Signed-off-by: Joe Elliott <number101010@gmail.com> * Expanded legacy test cases Signed-off-by: Joe Elliott <number101010@gmail.com> * Dropped streams nano test Signed-off-by: Joe Elliott <number101010@gmail.com> * First pass v1 new objects Signed-off-by: Joe Elliott <number101010@gmail.com> * Added failing tail response tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added failing tailresponse test Signed-off-by: Joe Elliott <number101010@gmail.com> * Partial v1 tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Improved legacy labels test Signed-off-by: Joe Elliott <number101010@gmail.com> * Improved legacy query tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved all legacy tests to new method Signed-off-by: Joe Elliott <number101010@gmail.com> * Added v1 tests and fixed stream marshalling bug Signed-off-by: Joe Elliott <number101010@gmail.com> * First pass new Model Signed-off-by: Joe Elliott <number101010@gmail.com> * Added vector test Signed-off-by: Joe Elliott <number101010@gmail.com> * Added matrix tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added conversions for all things except tailed responses Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed mixed case issues Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed tail marshalling Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed unused testStream Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved TailResponse to loghttp Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed legacy tailresponse objects in favor of actual legacy tailresponse objects Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated v1 methods to take legacy tail objects Signed-off-by: Joe Elliott <number101010@gmail.com> * Cleaned up tests. Added some comments Signed-off-by: Joe Elliott <number101010@gmail.com> * Versioned tail endpoint Signed-off-by: Joe Elliott <number101010@gmail.com> * Improved readability on loghttp packages in http.go Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed new as a var name Signed-off-by: Joe Elliott <number101010@gmail.com> * Started all error messages with lowercase alerts Signed-off-by: Joe Elliott <number101010@gmail.com> * new => ret Signed-off-by: Joe Elliott <number101010@gmail.com> * Added comments on exported methods Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed two personal notes Signed-off-by: Joe Elliott <number101010@gmail.com> * Changed legacy package name to loghttp Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved and renamed loghttp v1 package Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved marshalling code out of model Signed-off-by: Joe Elliott <number101010@gmail.com> * Added package comments Signed-off-by: Joe Elliott <number101010@gmail.com> * Added legacy testing Signed-off-by: Joe Elliott <number101010@gmail.com> * Changed DroppedStream slice to value type for consistency Signed-off-by: Joe Elliott <number101010@gmail.com> * gofmt'ed test files Signed-off-by: Joe Elliott <number101010@gmail.com> * Cleaned up linting issues Signed-off-by: Joe Elliott <number101010@gmail.com> * Minor comment cleanup Signed-off-by: Joe Elliott <number101010@gmail.com> * Adjusted GOGC to make CircleCI happy Signed-off-by: Joe Elliott <number101010@gmail.com> * Changed legacy => loghttp for consistency Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed matrix error message to be correct Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved label query over to loghttp response Signed-off-by: Joe Elliott <number101010@gmail.com> * Added marshal loop tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added response type test Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed tail response marshal/unmarshal Signed-off-by: Joe Elliott <number101010@gmail.com> * Passing unmarshal/marshal queryresponse tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed vector and matrix Signed-off-by: Joe Elliott <number101010@gmail.com> * Added output support for streams minus ordering Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed tailing Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed output tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed query tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Order log output Signed-off-by: Joe Elliott <number101010@gmail.com> * Use labels instead of stream Signed-off-by: Joe Elliott <number101010@gmail.com> * Lowered parallelization for CircleCI Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago