Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Cyril Tovena dd5fceb348 Logqv2 optimization (#2778) ... * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes race conditions in the batch iterator. We should never advance an iterator in parallel. Unfortunately before the code was building iterators while advancing previous one, building iterator can advance iterator and thus creates a race condition. This changeset make sure we only fetch chunks in advance and build iterator and iterate over them in sequence. Also add support for labels in the cacheIterator which is required for logqlv2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Handle panic in the store goroutine. This could cause Loki to crash if a panic happens in the store since it was happening in another goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Now that races are gone we can use a global decoder. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve labels handling in all LogQL stages. benchmark before: ``` pkg: github.com/grafana/loki/pkg/logql/log Benchmark_Pipeline Benchmark_Pipeline-16 107580 10271 ns/op 6387 B/op 67 allocs/op PASS ok github.com/grafana/loki/pkg/logql/log 1.228s ``` after ``` pkg: github.com/grafana/loki/pkg/logql/log Benchmark_Pipeline Benchmark_Pipeline-16 199170 5670 ns/op 3307 B/op 54 allocs/op PASS ok github.com/grafana/loki/pkg/logql/log 1.201s ``` Way less allocations ! Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes error label filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Reviews comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>		5 years ago
..
build	Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases. (#1095)	6 years ago
canary	Fixes all lint errors. (#2768)	5 years ago
cfg	Fix/promtail yaml config (#2471)	5 years ago
chunkenc	LogQL: Labels and Metrics Extraction (#2769)	5 years ago
distributor	Fixes all lint errors. (#2768)	5 years ago
helpers	Spelling (#2014)	5 years ago
ingester	LogQL: Labels and Metrics Extraction (#2769)	5 years ago
iter	Fixes race conditions in the batch iterator. (#2773)	5 years ago
logcli	Fixes all lint errors. (#2768)	5 years ago
logentry	LogQL: Labels and Metrics Extraction (#2769)	5 years ago
loghttp	Loki: Series API will return all series with no match or empty matcher (#2254)	5 years ago
logproto	rpc for querying ingesters to get chunk ids from its store (#2601)	5 years ago
logql	Logqv2 optimization (#2778)	5 years ago
loki	Boltdb shipper query performance improvements (#2770)	5 years ago
lokifrontend	Fixes all lint errors. (#2768)	5 years ago
promtail	Fixes all lint errors. (#2768)	5 years ago
querier	LogQL: Labels and Metrics Extraction (#2769)	5 years ago
ruler	Fixes all lint errors. (#2768)	5 years ago
storage	Handle panic in the store goroutine. (#2774)	5 years ago
tracing	Add flag to disable tracing (#2191)	5 years ago
util	LogQL: Labels and Metrics Extraction (#2769)	5 years ago