756 Commits (a62f81f1cf15d49e02958ae9a6097e84986ea336)

Author	SHA1	Message	Date
Sandeep Sukhani	f60554ce91	Boltdb shipper query performance improvements (#2770) ... * boltdb-shipper query with cursor to do multiple queries in a single transaction * index deduper * use fifo cache in ingesters * snapshot dbs in ingester for reads * retain fewer files in ingesters for queries * set timeout in cache config instead of index read cache config which saves us from unmarshalling the object for checking the expiry * add comment about fifo cache invalidation * tweaking a few comments Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Cyril Tovena	fe7aadfb9d	Fixes race conditions in the batch iterator. (#2773) ... * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes race conditions in the batch iterator. We should never advance an iterator in parallel. Unfortunately before the code was building iterators while advancing previous one, building iterator can advance iterator and thus creates a race condition. This changeset make sure we only fetch chunks in advance and build iterator and iterate over them in sequence. Also add support for labels in the cacheIterator which is required for logqlv2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Cyril Tovena	fd5cb96693	LogQL: Labels and Metrics Extraction (#2769) ... * Adds logfmt, regexp and json logql parser Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook the ast with parsers. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with memchunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with the storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * hook parser with ingesters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes all tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor to pipeline and implement ast parsing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the lexer for duration and range Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests and add some for label filters Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add label and line format. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tests for fmt label and line with validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Polishing parsers and add some more test cases Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish the unwrap parser, still need to add more tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Indent this hell. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar tests and it works. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests which lead me to find a bug in the lexer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests and fix all engine tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes match stage in promtail pipelines. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hook Pipeline into ingester, tailer and storage. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Correctly setup sharding for logqlv2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes precedences issue with label filters and add moar tests ✌️ * Adds quantile_over_time, grouping for non associate range aggregation parsing and moar tests * Extract with grouping * Adds parsing duration on unwrap * Improve the lexer to support more common identifier as functions. Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the frontend logs to include org_id. The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Support byte sizes in label filters. This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`. * Wip on error handling. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes json parser with prometheus label name rules. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixup! Support byte sizes in label filters. * Wip error handling, commit before big refactoring. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactoring in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got something that builds and throw __error__ labels properly now. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error handling + fixes groupins and post filtering. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * 400 on pipeline errors. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a races in the log pipeline. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Unsure the key is parsable and valid. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup and code documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes frontend handler. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes old test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix go1.15 local failing test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Cyril Tovena	62272cdaa1	Fixes all lint errors. (#2768) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	607ccc4d77	Service discovery refactor (#2761) ... * promtail sd loading refactor + tests * flippin bools like flipper * adds back base scrapeconfig test * passes all service discovery variants	5 years ago
Owen Diehl	54c0c5febb	Revendor Cortex (#2755) ... * replaces prometheus & revendors cortex * compat * compat * sd refactor * updates docker * removes prom replace * adds delimiters back to object addrs in boltdb	5 years ago
Kaviraj	690a387cce	fix: Remove depricated `entry_parser` from scrapeconfig (#2752)	5 years ago
Owen Diehl	6b17831797	better tenant logging in ruler memstore (#2741)	5 years ago
Owen Diehl	b8787163b6	removes r/w pools from block/chunk types (#2711)	5 years ago
Davor Kapsa	69f2eb2314	hack: clean getStore (#2726)	5 years ago
Cyril Tovena	81a697c541	Fixes the frontend logs to include org_id. ... The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	76e73d7c20	set timestamp in instant query done by canaries which otherwise gets set to now by loki (#2732)	5 years ago
Davor Kapsa	fffcfa6e81	fix some misspells (#2725)	5 years ago
Calle Pettersson	426bac7f5b	Add counter promtail_batch_retries_total (#2723) ... Signed-off-by: Calle Pettersson <calle@cape.nu>	5 years ago
Ed Welch	8ea6c381cb	Promtail: Fix deadlock on tailer shutdown. (#2717) ... * to avoid a deadlock on shutdown, leave the goroutine running which reads the Lines channel from the tailer and it will exit when the tailer closes the channel. * remove the flag I added a few weeks ago and instead when updating the position information of the file gracefully fail if the file no longer exists. * update the vendoring of the tail library, more re-work on how tailing shutdown and errors are handled. * fix mistake * updating tail lib to merged commit hash * go mod tidy * update tail library with additional mutex lock * break up the tailer goroutine into two separate goroutines to make it a little more clear and remove some special case logic. * don't forget to close your channels! * make the stop method in the tailer only execute at most one time. * update hp-tail again with improvements to the mutex locking in closeFile * replace the more complicated channel based single execution code with a sync.Once	5 years ago
Ed Welch	d3bf21e774	Promtail: (and also fluent-bit) change the max batch size to 1MB (#2710) ... * change the max batch size to 1MB for all the defaults including helm and fluent-bit, attempt to centralize this config a little where possible. * fix test	5 years ago
Benjamin Krämer	9e6afea5f6	Fix timestamp parser for short year format (#2708)	5 years ago
Cyril Tovena	3640429532	Bypass sharding middleware when a query can't be sharded. (#2709) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	6500f82195	Improve lz4 compression (#2614) ... * Improve lz4 compression. - Move to v4. - Remove not required checksuming. - Default will now write 4M blocks, it's backward compatible for reads. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * vendor update Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	2162a0c1b1	Do not mark the position if the file is removed, doing so only results in an error because you can't get the file size of a removed file. (#2658)	5 years ago
BaiLian	0ac1ce4e5a	Querier APIs respond JSON Content-Type (#2665)	5 years ago
Amin Jamali	9fcb8b8e4a	Add fallback_formats for timestamp stage (#2615) ... Fixes grafana/loki#2590 Co-authored-by: Amin Jamali <ajamali@vmware.com>	5 years ago
Lars Lehtonen	0c9f7a58a4	pkg/querier: fix dropped error (#2671) ... Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>	5 years ago
Sandeep Sukhani	6053cdb9d9	fix store query bug when running loki in single binary mode with boltdb-shipper (#2655)	5 years ago
Sandeep Sukhani	79c027550c	run multiple queries per table at once with boltdb-shipper (#2656)	5 years ago
Owen Diehl	89b8ae4b49	improves numeric literal stringer impl (#2663)	5 years ago
Owen Diehl	5729d88302	exposes rule group validation fn (#2662)	5 years ago
Owen Diehl	21d3e69790	Enable local rules backend & disallow configdb. (#2661) ... * revendors cortex * vendoring compat * disallow configdb ruler backend * go mod tidy	5 years ago
Owen Diehl	6f8bfe0c79	Ruler docs + single binary inclusion (#2637) ... * starts alerting docs * ruler in single binary * make docs interactive * alerting docs * ruler prom alerts endpoint * Apply suggestions from code review Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * doc fixes * capitalize ruler * removes double spaces * Update docs/sources/alerting/_index.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> Co-authored-by: Ed Welch <ed@oqqer.com>	5 years ago
Aurélien Rainone	21addf7935	pkg/distributor: fix test by looking for loopback interface (#2645) ... TestDistributor_PushIngestionRateLimiter failed if the system had no interface name in "eth0", "en0" or "lo0". Fix that by looking at the name of the loopback interface in the list of the system's network interfaces, then using that name in the test. Fixes #2356	5 years ago
Aurélien Rainone	85696d00eb	pkg/chunkenc: fix test using string(int) conversion (#2647) ... Since go1.15, there's a new vet check for code such as string(x) where x has an integer type other than rune. This vet check is enabled by default on go test. TestSerialization failed because of that, this commit replaces `string()` conversions with `strconv.Itoa` calls Fixes #2646	5 years ago
Owen Diehl	9e85757d89	adds prometheus ruler routes (#2650)	5 years ago
Ed Welch	0b1dbe26fe	Promtail: Add a stream lagging metric (#2618) ... * add a metric which displays how far a stream lags the last timestamp in a batch * register your metrics * recorded metric in the wrong spot, moved to the correct spot * cardinality is much too high if we add all the stream labels, instead just use filename. * work on using an interface to set/remove metrics * reworking interface again * for now just register this metric based on filename only, we can improve this in the future if the need arises * change the timeout to 1min for expiring the metric * add host label * include host label when removing metric * change the batch size for promtail from 100kb to 1Mb, this will not affect sending for lower volumes which are still driven by the BatchWait setting. Through the addition of this metric it was found that for higher volume files > 100kb/sec this was much too low causing too many batches to be unnecessarily sent.	5 years ago
Sandeep Sukhani	29790ea19b	fix an issue with building loki (#2642)	5 years ago
Sandeep Sukhani	5b361a0c89	improvements for boltdb-shipper compactor (#2640) ... 1. Run the compaction process on startup instead of waiting for the first tick. 2. Run the compactor service when using loki with boltdb-shipper in single binary mode without clustering.	5 years ago
Sandeep Sukhani	2f7e0e50c2	rpc for querying ingesters to get chunk ids from its store (#2601) ... * rpc for querying ingesters to get chunk ids from its store * fix broken tests * separate out code from querier to query ingester to a new type called IngesterQuerier * add AsyncStore which helps with querying and merging results from both chunk store and ingesters * query ingesters local store for labels when active index type is boltdb-shipper or query store is true * better naming * add some tests * add test case for duplicate chunk ids from ingesters * add some logs and spans * check for nil fetcher * use require.Same to assert that two pointers are pointing to same object * some more improvements 1. query chunk-refs parallely from store and ingesters. 2. limit look back for chunk refs and labels from ingesters to start of schema when using boltdb-shipper. * fixes suggested from PR review * fix mode not being set before cortex_storage.NewStore is being called * add IngesterQuerier as a dependency for store when target is either ingester or querier * do not cache index from ingester * rename tests * fix broken tests * update upgrade guide to rollout ingesters before queriers * upgrade guide changes suggested in PR review * runtime check for whether active index store is boltdb-shipper * consider previous schema as well for boltdb-shipper for calculating max look back	5 years ago
Sandeep Sukhani	337db2fedf	revendor cortex to latest master (#2627)	5 years ago
Lars Lehtonen	334cc1e59a	pkg/storage/stores/shipper/uploads: fix test error (#2620) ... Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>	5 years ago
Roger Steneteg	edb5fc5176	Promtail: adding pipeline stage for dropping labels (#2571) ... * adding pipeline stage for dropping labels, our use case for this is to move unbound labels into the log line and dropping the label, fixes #2421 Signed-off-by: Roger Steneteg <rsteneteg@ea.com> * Update docs/sources/clients/promtail/stages/labeldrop.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/_index.md Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * fixed inconsistent naming, removed unused logger in the labeldrop stage Signed-off-by: Roger Steneteg <rsteneteg@ea.com> Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>	5 years ago
Sandeep Sukhani	e53d2032fe	fix a panic when trying to stop boltdb-shipper multiple times using sync.once (#2613)	5 years ago
Ed Welch	2015763327	Loki: Fix query-frontend ready handler (#2610) ... * query frontend should not have a default route as it usurps the ready handler, the ready handler was also not implemented completely for the query frontend. * enable ready check for query-frontend in ksonnet	5 years ago
Robert Fratto	ee67b9c707	Expose UserAgent and fix User-Agent version source (#2603) ... * Use prometheus/common/version for reporting build version in User Agent header Promtail wasn't using github.com/prometheus/common/version for the User-Agent header when obtaining the build version. Instead, it imported github.com/grafana/loki/pkg/build which modifies the Prometheus version package on init. This causes problems for importers that also import and modify github.com/prometheus/common/version with their own build information (i.e., grafana/agent). This PR changes Promtail to obtain the build info from the common dependency to obtain the correct build information regardless of how Promtail is being run. * expose UserAgent to allow it to be modifiable by importers	5 years ago
Sandeep Sukhani	6dd2e9ef34	Read, Write and operational dashboard improvements (#2584) ... 1. Add bigtable, boltdb-shipper to read/write dashboards. 2. Add boltdb-shipper to operational dashboard. 3. Record latency for write operation in boltdb-shipper.	5 years ago
Calle Pettersson	82a8cf425f	Add a counter for empty syslog messages (#2565)	5 years ago
Ben Ye	900f714e9d	add _total suffix to ruler counter metrics (#2582) ... Signed-off-by: Ben Ye <yb532204897@gmail.com>	5 years ago
Owen Diehl	164f5cd0ae	Ruler/loki rule validator (#2589) ... * revendors cortex * loki multitenantmanager * vendoring compat * go mod	5 years ago
Sandeep Sukhani	7566aa4950	fix closing of compressed file from boltdb-shipper compactor (#2574)	5 years ago
Aditya C S	c1f07f08a7	Validate max_query_length in Label API (#2572)	5 years ago
Ed Welch	b8b24f3129	clone labels when sending them to Handle function for stdin target (#2570)	5 years ago
Owen Diehl	f905e77259	exports grouploader (#2578)	5 years ago