apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Owen Diehl	1eb86d4b5f	Only applies entry limits to non-SampleExprs. (#2850 ) * only applies entry limits to non-SampleExprs. updates docs * fixes some test case formatting	5 years ago
Cyril Tovena	fd5cb96693	LogQL: Labels and Metrics Extraction (#2769 ) * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Peter Štibraný	e4a5360586	Update vendored Cortex to master (1.1+) (#2149 ) * Updated Cortex to master (post 1.1) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use new modules system from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Treat query errors reported by chunk store as client errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed obsolete tests. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix race for ct variable. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Owen Diehl	89d80a6fc4	Feature/querysharding ii (#1927 ) * [wip] sharding evaluator/ast * [wip] continues experimenting with ast mapping * refactoring in preparation for binops * evaluators can pass state to other evaluators * compiler alignment * Evaluator method renamed to StepEvaluator * chained evaluator impl * tidying up sharding code * handling for ConcatSampleExpr * downstream iterator * structure for downstreaming asts * outlines sharding optimizations * work on sharding mapper * ast sharding optimizations * test for different logrange positions * shard mapper tests * stronger ast sharding & tests * shardmapper tests for string->string * removes sharding evaluator code * removes unused ctx arg * Revert "removes sharding evaluator code" This reverts commit `55d41b9519`. * interfaces for downstreaming, type conversions * sharding plumbing on frontend * type alignment in queryrange to downstream sharded queriers * downstreaming support for sharding incl storage code * removes chainedevaluator * comment alignment * storage shard injection * speccing out testware for sharding equivalence * [wip] shared engine refactor * sorting streams, sharding eval fixes * downstream evaluator embeds defaultevaluator * other pkgs adopt logql changes * metrics & logs use same middleware instantiation process * wires up shardingware * middleware per metrics/logfilter * empty step populating StepEvaluator promql.Matrix adapter * sharding metrics * log/span injection into sharded engine * sharding metrics avoids multiple instantiation * downstreamhandler tracing * sharding parameterized libsonnet * removes querier replicas * default 32 concurrency for workers * jsonnet correct level override * unquote true in yaml * lowercase error + downstreamEvaluator defaults to embedded defaultEvaluator * makes shardRecorder private * logs query on failed parse * refactors engine to be multi-use, minimizes logger injection, generalizes Query methods, removes Engine interface * basic tests for querysharding mware * [wip] concurrent evaluator * integrates stat propagation into sharding evaluator * splitby histogram * extends le bounds for bytes processed * byte throughput histogram buckets to 40gb * chunk duration mixin * fixes merge w/ field rename * derives logger in sharded engine via ctx & logs some downstream evaluators * moves sharded engine to top, adds comments * logs failed merge results in stats ctx * snapshotting stats merge logic is done more effectively * per query concurrency controlled via downstreamer * unexports decodereq * queryrange testware * downstreamer tests * pr requests	5 years ago
Cyril Tovena	bce4470a5b	Recover from panic in http and grpc handlers. (#2059 ) * Recover from panic in http and grpc handlers. I don't see any good reason to crash any component during a bad request. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add alerts to the mixin for panics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 😡 gomod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	577d8eb168	Ensure trace propagation in our logs. (#1977 ) * Ensure trace propagation in our logs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing import. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix function usage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	c7a1a1fb2e	Loki: Adds an `interval` paramater to query_range queries allowing a sampling of events to be returned based on the provided interval (#1965 ) * Adds an `interval` paramater to query_range queries allowing a sampling of events to be returned based on the provided interval Signed-off-by: Ed Welch <edward.welch@grafana.com> * lint Signed-off-by: Ed Welch <edward.welch@grafana.com> * extract function for parsing parameter Signed-off-by: Ed Welch <edward.welch@grafana.com> * Clarify the experimental nature of interval in the docs. Signed-off-by: Ed Welch <edward.welch@grafana.com> * make min time a constant Signed-off-by: Ed Welch <edward.welch@grafana.com> * Expand the docs around the experimental nature of interval a little more Signed-off-by: Ed Welch <edward.welch@grafana.com>	5 years ago
Aditya C S	f41273182c	Support configurable maximum of the limits parameter (#1798 )	5 years ago
Cyril Tovena	6328146876	Fixes querier cancellation error to return 499 instead of 500. (#1745 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	6363e8da0a	Better logql metric status code. (#1718 ) * Fixes querier logql error to be 400 only for ast parsing error. Improve frontend metrics to use a more granular status_code. Improve prometheus histogram granularity. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add latency type dimensions to bytes throughput metric. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use IsParseError in panic recovering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	b7d23f41bd	Decouple logql engine/AST from execution context (#1605 ) * logql engine is an interface * [wip] begins agnostic logql evaluator work * decouples logql AST from execution context * healthcheck comments	5 years ago
Owen Diehl	366e06f587	legacy endpoint 400s metric queries (#1541 )	5 years ago
Sandeep Sukhani	fbcfad13dc	fix panic in tailer when max tail duration exceeds (#1511 )	5 years ago
Beorn Facchini	88681a9974	Read websocket close in tail handler (#1383 ) * Read websocket close in tail handler * Send close message to websocket on quit signals	5 years ago
Owen Diehl	dafb9d8425	Enables Series API in loki (#1419 ) * helpers for parsing /series requests * logproto.SeriesResponse impl * passes unparsed matchers in protobuf for simplicity * ingester support for series queries * labels map coercion * fixes label match loghttp test * removes labelmatcher protobuf adapters * series api impl * tests matchers for dispatching series queries * series looked up serially for /series api * splits series api async code and remove unnecessary type assertion * adds series handle to /api/prom/series and includes docs * compatibility changes w/ upstream merge	6 years ago
Owen Diehl	5227f5bee0	application/x-www-form-urlencoded support (#1381 ) * application/x-www-form-urlencoded support * handler func instead of unnecessary struct	6 years ago
Cyril Tovena	1abe8841ce	Moves request parsing into the loghttp package (#1171 ) * refactor query requests parsing Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add tests and documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Sandeep Sukhani	569748ff18	logging: removed some noise in logs from live-tailing (#1100 ) * Removed some noise in logs by detecting cases where connection is erroring due to stopped or closed tail request * fixed log formats	6 years ago
Marco Pracucci	96237b5550	logcli: added --step support to query command (#1103 ) * logcli: added --step support to query command * Changed query range API default step value from 1s to a dynamic value based on start/end input params	6 years ago
Joe Elliott	9614b021d1	Loki HTTP/JSON Model Layer (#1022 ) * First pass data model Signed-off-by: Joe Elliott <number101010@gmail.com> * Use prom model b/c we're serializing promql objects Signed-off-by: Joe Elliott <number101010@gmail.com> * Added legacy query support and tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added legacy label test Signed-off-by: Joe Elliott <number101010@gmail.com> * Added tail response marshalling and tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed marshallers and test Signed-off-by: Joe Elliott <number101010@gmail.com> * Expanded legacy test cases Signed-off-by: Joe Elliott <number101010@gmail.com> * Dropped streams nano test Signed-off-by: Joe Elliott <number101010@gmail.com> * First pass v1 new objects Signed-off-by: Joe Elliott <number101010@gmail.com> * Added failing tail response tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added failing tailresponse test Signed-off-by: Joe Elliott <number101010@gmail.com> * Partial v1 tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Improved legacy labels test Signed-off-by: Joe Elliott <number101010@gmail.com> * Improved legacy query tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved all legacy tests to new method Signed-off-by: Joe Elliott <number101010@gmail.com> * Added v1 tests and fixed stream marshalling bug Signed-off-by: Joe Elliott <number101010@gmail.com> * First pass new Model Signed-off-by: Joe Elliott <number101010@gmail.com> * Added vector test Signed-off-by: Joe Elliott <number101010@gmail.com> * Added matrix tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added conversions for all things except tailed responses Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed mixed case issues Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed tail marshalling Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed unused testStream Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved TailResponse to loghttp Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed legacy tailresponse objects in favor of actual legacy tailresponse objects Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated v1 methods to take legacy tail objects Signed-off-by: Joe Elliott <number101010@gmail.com> * Cleaned up tests. Added some comments Signed-off-by: Joe Elliott <number101010@gmail.com> * Versioned tail endpoint Signed-off-by: Joe Elliott <number101010@gmail.com> * Improved readability on loghttp packages in http.go Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed new as a var name Signed-off-by: Joe Elliott <number101010@gmail.com> * Started all error messages with lowercase alerts Signed-off-by: Joe Elliott <number101010@gmail.com> * new => ret Signed-off-by: Joe Elliott <number101010@gmail.com> * Added comments on exported methods Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed two personal notes Signed-off-by: Joe Elliott <number101010@gmail.com> * Changed legacy package name to loghttp Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved and renamed loghttp v1 package Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved marshalling code out of model Signed-off-by: Joe Elliott <number101010@gmail.com> * Added package comments Signed-off-by: Joe Elliott <number101010@gmail.com> * Added legacy testing Signed-off-by: Joe Elliott <number101010@gmail.com> * Changed DroppedStream slice to value type for consistency Signed-off-by: Joe Elliott <number101010@gmail.com> * gofmt'ed test files Signed-off-by: Joe Elliott <number101010@gmail.com> * Cleaned up linting issues Signed-off-by: Joe Elliott <number101010@gmail.com> * Minor comment cleanup Signed-off-by: Joe Elliott <number101010@gmail.com> * Adjusted GOGC to make CircleCI happy Signed-off-by: Joe Elliott <number101010@gmail.com> * Changed legacy => loghttp for consistency Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed matrix error message to be correct Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved label query over to loghttp response Signed-off-by: Joe Elliott <number101010@gmail.com> * Added marshal loop tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added response type test Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed tail response marshal/unmarshal Signed-off-by: Joe Elliott <number101010@gmail.com> * Passing unmarshal/marshal queryresponse tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed vector and matrix Signed-off-by: Joe Elliott <number101010@gmail.com> * Added output support for streams minus ordering Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed tailing Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed output tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed query tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Order log output Signed-off-by: Joe Elliott <number101010@gmail.com> * Use labels instead of stream Signed-off-by: Joe Elliott <number101010@gmail.com> * Lowered parallelization for CircleCI Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
Cyril Tovena	364b5bc4f6	Make Loki HTTP API more similar to Prometheus	6 years ago
Cyril Tovena	8f9cd97880	Improve API compatibility. - keep supporting regex for old endpoint. - use reserved keyword for removed property in grpc	6 years ago
Cyril Tovena	8bd9f752aa	Adds more tests for range queries	6 years ago
Cyril Tovena	7f3db9d7f0	LogQL: Vector and Range Vector Aggregation. - adds avg,min,max,top,bottomk,stddev,stdvar,count - updates api documentation - adds tests Improve yacc & go lexer to understand duration Remove support for regexp in all queries Clean up querier and logselector	6 years ago
sh0rez	38cb093358	feat(loki): extended tailing (#764 ) * refactor(querier/ingester): TailRequest Lookback window Moves the specifications of the Lookback Window out of the logproto.QueryRequest into it's own type logproto.Lookback. This is required, because the Lookback Window will be used in the TailRequest as well. * feat(querier): parse Lookback from HTTP Request * feat(logcli): send Lookback Window spec with tail request * feat(querier): include historic entries in tail mode Extends tailing by sending a configurable amount of historic entries with before the live entries. This enables a behaviour that is closer to kubectl logs -f and docker logs -f. It is implemented by running a regular Query before subscribing to the ingesters. * fix: adapt tests to Lookback change * feat(querier): check all errors to make the linter happy * fix(logproto): flatten Lookback window spec Flattens the Lookback window spec into the individual queries * fix(ingester): adapt test to Lookback flatten	6 years ago
Cyril Tovena	86bc3c0518	Query label values and names are now fetched from the store. (#521 ) * Query label values and names are now fetched from the store. A time range is now required by the /api/prom/label with a sane default (6 hours from now). * fix http querystring and update doc * update vendor * rebased	6 years ago
Sandeep Sukhani	2a8a03795a	Improvements in live tailing of logs (#541 ) * Improvements in live tailing of logs Instead of polling for new logs, a grpc stream is opened between ingester and querier to get live logs Querier reconnects to disconnected or newly added ingesters * Added more comments to code for live log tailing * Some code refactoring in live log tailing * handling delayfor in logcli, max delayfor to be 5 seconds * some changes in tail response in live tailing * Fixed issue with stopping ingesters gracefully when live tailing is being used * Added tests for tailer in querier * Live tailing made logql compatible, some code refactoring suggested in PR	6 years ago
Cyril Tovena	f939e42afd	add support for RFC3339Nano in query timestamps (#656 )	6 years ago
Sandeep Sukhani	f5c8b22b34	Change tail response for backward compatibilty with future changes (#590 ) Changed it to dict with single key called "streams" and value set to list of logproto.Stream	6 years ago
Sandeep Sukhani	1f1706f467	Move live log tailing behind websocket on server (#439 ) * Move live log tailing behind websocket on server Live log tailing now requires client to just open a websocket and keep listening to it for getting live logs for matching query * Fixed linting errors * Unexported tailQuery querier method * Added a delay in querying of logs to not miss delayed entries * Renamed variable for better readability * Some code refactoring suggested in PR review * Fixed linter errors * Code refactoring requested in PR review * Disabling linting for writeHTTPErrorResponse since same value for an input is being passed from all function calls for now * Removed a function which already existed * Improved printing of log entries using cli and some other code refactoring * Checking no-labels flag before printing log stream while tailing	6 years ago
Tom Wilkie	9f257c591a	[WIP] Persistence (#44 ) Persistence chunks using the Cortex chunk store.	7 years ago
Tom Wilkie	5f73598389	Rename Tempo to Loki. (#36 ) * Rename Tempo to Loki. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use new build image, don't delete generated files on clean. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	1fef92bc17	Lint stuff (#27 ) * Check in generated files, to make getting started easier. Add gometalinter, and a lint check for changes to the generated files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make lint pass? Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	3f62b4ec36	s/logish/tempo/ Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	96223322ba	Make queries take nanoseconds since epoch. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	5ec9dfc5d1	Fix logic. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	b247a5450d	Command to fetch label names. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	411e4a8ce4	Plumb through label value queries. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	544eed1b95	Add some unit tests for the iterators. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	d3579bbeef	Add server side regex filtering. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	00afed1eb3	Add start, end and direction options to querier. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	a2c4df0416	Plumb start & end timestamps through the server. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	50fdd2d444	Fix bugs in ingester iterators. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	2031845e7a	Correctly with up query api. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	e5d0be9d27	Update for dependancies. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	45807ba19b	Add HTTP interface to distributor and querier.	7 years ago

46 Commits (d357ad9e58a035bcdb5e0bf0a4ebebaba4de1797)