740 Commits (6e79cee85d8d5b62e524f08ae54a7d0f2f55a10b)

Author	SHA1	Message	Date
Cyril Tovena	6e79cee85d	Fixes race condition in tailer since logql v2. (#2917) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	5738611d64	Fixes logql.QueryType. (#2913) ... The bug was introduced with LogQL extensions. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Robert Fratto	ca311c4d52	expose underlying promtail client (#2910)	5 years ago
Cyril Tovena	dfd2e2090e	Fixes metric query issue with no grouping. (#2902) ... This would remove all labels for metrics queries that does not have grouping, instead of adding all labels. It was introduced by https://github.com/grafana/loki/pull/2875 I added a regression test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	095b4a1cb7	Fixes a panic with the logql.NoopPipeline. (#2901) ... Totally forgot that now we can't share this across multiple queries since they hold a map. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	070a145ea6	Improve the chunksize test. (#2892) ... This gives now the ratio, and the characters inserted in human readable format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	fd451d97f4	fix flaky load tables test for boltdb-shipper uploads table-manager (#2891)	5 years ago
Cyril Tovena	e2aee2a9c3	LogQL: Improve template format (#2822) ... * Add error labels on failure and new functions for templates. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Improve docs based off review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedback. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes panic out of boundary Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/logql/functions.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/functions.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>	5 years ago
Keith Byrne	5793c4964e	Configuration: Support environment expansion in configuration (#2837) ... * Support environment expansion in configuration Inspired by https://github.com/cortexproject/cortex/pull/2147 Fixes #2311 Adds -config.expand-env as flag to loki and promtail. * Disable env expansion on logcli * Small nit updates	5 years ago
RangerCD	b9534c65b3	Fix typos (#2853)	5 years ago
Cyril Tovena	53f4aa4ac3	Labels computation LogQLv2 (#2875) ... * Adding a benchmark to compare before and after. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves labels management in logql v2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cache grouped result when no changes has occured. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes unused methods. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds docs and tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Uncomment tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
zhanghjster	bc202ae7ec	documentation: add tail_proxy_url option to query_frontend_config section (#2879) ... * add tail_proxy_url option to the query_frontend_config section of configuration doc * If queryFrontend run in tail proxy mode, append the Host of the tail_proxy_url to the request header, because in some scenario queryFrontend forward tail to the queriers deployed behind nginx and "Host" is required.	5 years ago
Owen Diehl	9249ebc022	Fix/validate compactor config (#2824) ... * default compactor config throws error when compactor is a target or subtarget * linting * moves compactor config validation to NewCompactor fn	5 years ago
Sandeep Sukhani	cb53a47139	fix query intervals when running boltdb-shipper in single binary (#2855)	5 years ago
Cyril Tovena	e6778e9827	Update to go 1.15 (#2877) ... * Upgrade to 1.15. Also upgrade linter and fixes some lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update generated files. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar lint 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moar lint and tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	dc0bd01e35	WAL/marshalable chunks (#2764) ... * marshalable chunks * wal record types custom serialization * proto types for wal checkpoints * byteswith output unaffected by buffer	5 years ago
Karsten Jeschkies	55c768bf8e	Log throughput and total bytes human readable. (#2751)	5 years ago
jkellerer	c394ce9462	LogQL: Add unwrap bytes() conversion function (#2876) ... * Unwrap bytes: Added bytes() op to parser & lexer * Unwrap bytes: Added bytes() conversion function	5 years ago
Owen Diehl	1eb86d4b5f	Only applies entry limits to non-SampleExprs. (#2850) ... * only applies entry limits to non-SampleExprs. updates docs * fixes some test case formatting	5 years ago
Andreas Rammhold	c79878d2ee	fix quoting issues in test output when building with Go 1.15 (#2836)	5 years ago
Sandeep Sukhani	feb66d62cf	fix flaky tests in boltdb-shipper (#2831)	5 years ago
Sandeep Sukhani	7f4d6d5912	fix panic in ingester when not running with boltdb shipper while queriers does (#2830)	5 years ago
Cyril Tovena	fa00b90128	Fix error swallowed in the frontend. (#2807) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	ecc4022d6c	Improve pipeline stages ast errors. (#2805) ... They would throw 500 instead of 4xx before. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	071a20803b	Revendor cortex to latest master (#2794)	5 years ago
Ed Welch	b80f37b724	log any chunk fetch failure (#2804)	5 years ago
Ed Welch	af6e8cb6c9	Update local and docker default config files to use boltdb-shipper with a few other config changes (#2803) ... * change the default local and docker config files to use boltdb-shipper also change them to use chunk_target_size * create ruler local filesystems directories if they don't exist change the ruler paths to be a little more consistent * don't need to create temp dir, also checking for the storage type * add compactor config to docker config file.	5 years ago
Sandeep Sukhani	2793499309	Boltdb shipper metrics changes (#2790) ... * rename boltdb shipper metrics to have table in their name instead of files * Increment tables_sync_operation_total metric during periodic syncs as well * tweak error handling Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	3f93b5bd28	Fixes a bug that would add __error__ labels incorrectly. (#2796) ... Wrong usage of slices would add error label to the original set of labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	edc6215098	Improve the way we reverse iterator for backward queries. (#2793) ... Instead of reversing all blocks, we reverse one block at the time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	7b7b1844ca	add a metric in compactor to record timestamp of last successful run (#2788)	5 years ago
Cyril Tovena	2e14c45ceb	Logqlv2 pushes groups down to edge (#2786) ... Pushes down grouping into range aggregation to reduce labels at edges. This works only for sums. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	dd5fceb348	Logqv2 optimization (#2778) ... * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes race conditions in the batch iterator. We should never advance an iterator in parallel. Unfortunately before the code was building iterators while advancing previous one, building iterator can advance iterator and thus creates a race condition. This changeset make sure we only fetch chunks in advance and build iterator and iterate over them in sequence. Also add support for labels in the cacheIterator which is required for logqlv2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Handle panic in the store goroutine. This could cause Loki to crash if a panic happens in the store since it was happening in another goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Now that races are gone we can use a global decoder. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve labels handling in all LogQL stages. benchmark before: ``` pkg: github.com/grafana/loki/pkg/logql/log Benchmark_Pipeline Benchmark_Pipeline-16 107580 10271 ns/op 6387 B/op 67 allocs/op PASS ok github.com/grafana/loki/pkg/logql/log 1.228s ``` after ``` pkg: github.com/grafana/loki/pkg/logql/log Benchmark_Pipeline Benchmark_Pipeline-16 199170 5670 ns/op 3307 B/op 54 allocs/op PASS ok github.com/grafana/loki/pkg/logql/log 1.201s ``` Way less allocations ! Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes error label filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Reviews comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Cyril Tovena	940529984b	Handle panic in the store goroutine. (#2774) ... * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes race conditions in the batch iterator. We should never advance an iterator in parallel. Unfortunately before the code was building iterators while advancing previous one, building iterator can advance iterator and thus creates a race condition. This changeset make sure we only fetch chunks in advance and build iterator and iterate over them in sequence. Also add support for labels in the cacheIterator which is required for logqlv2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Handle panic in the store goroutine. This could cause Loki to crash if a panic happens in the store since it was happening in another goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Sandeep Sukhani	f60554ce91	Boltdb shipper query performance improvements (#2770) ... * boltdb-shipper query with cursor to do multiple queries in a single transaction * index deduper * use fifo cache in ingesters * snapshot dbs in ingester for reads * retain fewer files in ingesters for queries * set timeout in cache config instead of index read cache config which saves us from unmarshalling the object for checking the expiry * add comment about fifo cache invalidation * tweaking a few comments Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	fe7aadfb9d	Fixes race conditions in the batch iterator. (#2773) ... * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes race conditions in the batch iterator. We should never advance an iterator in parallel. Unfortunately before the code was building iterators while advancing previous one, building iterator can advance iterator and thus creates a race condition. This changeset make sure we only fetch chunks in advance and build iterator and iterate over them in sequence. Also add support for labels in the cacheIterator which is required for logqlv2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Cyril Tovena	fd5cb96693	LogQL: Labels and Metrics Extraction (#2769) ... * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	607ccc4d77	Service discovery refactor (#2761) ... * promtail sd loading refactor + tests * flippin bools like flipper * adds back base scrapeconfig test * passes all service discovery variants	5 years ago
Owen Diehl	54c0c5febb	Revendor Cortex (#2755) ... * replaces prometheus & revendors cortex * compat * compat * sd refactor * updates docker * removes prom replace * adds delimiters back to object addrs in boltdb	5 years ago
Kaviraj	690a387cce	fix: Remove depricated `entry_parser` from scrapeconfig (#2752)	5 years ago
Owen Diehl	6b17831797	better tenant logging in ruler memstore (#2741)	5 years ago
Owen Diehl	b8787163b6	removes r/w pools from block/chunk types (#2711)	5 years ago
Davor Kapsa	69f2eb2314	hack: clean getStore (#2726)	5 years ago
Cyril Tovena	81a697c541	Fixes the frontend logs to include org_id. ... The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	76e73d7c20	set timestamp in instant query done by canaries which otherwise gets set to now by loki (#2732)	5 years ago
Davor Kapsa	fffcfa6e81	fix some misspells (#2725)	5 years ago
Calle Pettersson	426bac7f5b	Add counter promtail_batch_retries_total (#2723) ... Signed-off-by: Calle Pettersson <calle@cape.nu>	5 years ago
Ed Welch	8ea6c381cb	Promtail: Fix deadlock on tailer shutdown. (#2717) ... * to avoid a deadlock on shutdown, leave the goroutine running which reads the Lines channel from the tailer and it will exit when the tailer closes the channel. * remove the flag I added a few weeks ago and instead when updating the position information of the file gracefully fail if the file no longer exists. * update the vendoring of the tail library, more re-work on how tailing shutdown and errors are handled. * fix mistake * updating tail lib to merged commit hash * go mod tidy * update tail library with additional mutex lock * break up the tailer goroutine into two separate goroutines to make it a little more clear and remove some special case logic. * don't forget to close your channels! * make the stop method in the tailer only execute at most one time. * update hp-tail again with improvements to the mutex locking in closeFile * replace the more complicated channel based single execution code with a sync.Once	5 years ago
Ed Welch	d3bf21e774	Promtail: (and also fluent-bit) change the max batch size to 1MB (#2710) ... * change the max batch size to 1MB for all the defaults including helm and fluent-bit, attempt to centralize this config a little where possible. * fix test	5 years ago