35 Commits (a1745c7b47d07c562a208e1273f629d5cc449ed4)

Author	SHA1	Message	Date
Travis Patterson	eb242f305d	Add tailing metrics (#6059) ... * Add tailing metrics * fix test * fix test	4 years ago
Danny Kopping	3fa6cc9fde	Querier: prevent unnecessary calls to ingesters (#5984) ... * Restrict gRPC calls to ingesters when requesting series if there is no overlap within the query_ingesters_within value * Add tests for isWithinIngesterMaxLookbackPeriod * Adding tests for calculateIngesterMaxLookbackPeriod * Only send query for range overlapping with ingesters * Restrict gRPC calls to ingesters when requesting series if there is no overlap within the query_ingesters_within value * Add tests for isWithinIngesterMaxLookbackPeriod * Adding tests for calculateIngesterMaxLookbackPeriod * Only send query for range overlapping with ingesters * Add test for when to query ingester and storage * Fix deadlock in Series when just queriying the ingesters * Refactored slightly for better readability * Added CHANGELOG entry * Fix typo * Rephrase test cases names * Move block to outer loop Co-authored-by: Salva Corts <salva.corts@grafana.com>	4 years ago
Travis Patterson	7299e3bdcd	Fix query filtering (#5951) ... - milli -> nano conversion in the querier - add deletes to store requests from the ingester	4 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833) ... * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Michel Hollands	b865b81ba0	Add a DeletionMode config variable (#5481) ... * Add deletionEnabled setting and remove delete request manager Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rebase Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove deletion handling from delete requests manager Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * change store so it stores a logql statement Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add validation code for logql statement Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Run deleteRequestsManager when deletion is enabled Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove unused variables Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert "Remove deletion handling from delete requests manager" This reverts commit ce4f774497aa590caff86b0745ec81588592a9e1. * Re-add IsDeleted method Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Re-add tests for IsDeleted Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix delete request store test Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix linting issue Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert compactor changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add deletion mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add v1 mode Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename LogQLRequest to Query Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix linting issues Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use DeleteMode in compactor module Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename logql to query Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Put cancel under delete verb Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update documentation Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert only the API surface area while keeping everything else * Use moved code in syntax package Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove duplicte import Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use renamed field in tests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove duplicates and empty lines in changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog description Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/storage/stores/shipper/compactor/deletion/delete_request.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_manager_test.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_store.go Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_store.go Co-authored-by: Christian Simon <simon@swine.de> * Update CHANGELOG.md Co-authored-by: Christian Simon <simon@swine.de> * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_store.go Co-authored-by: Christian Simon <simon@swine.de> * Make DeletionMode struct member non public Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Revert change to docs re cancellation Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use same variable names Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add parameter validation to changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rename v1 to WholeStreamDeletion Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Fix default value of deletion mode config setting Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * reimplement new api * Add delete request handler when delete mode is set Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove unused variable Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment to change the code when other deletion modes are available Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * create expirationChecker if deletionMode is set Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Address review comments Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update pkg/storage/stores/shipper/compactor/compactor.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Rename AddQuery to SetQuery Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com> Co-authored-by: Christian Simon <simon@swine.de> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	4 years ago
Travis Patterson	88372c93cc	Query Filtering -- Querier (#5482) ... * Query Filtering -- Querier - Get deletes from the index gateway and attach them to store and ingester requests * lint * Fix test -- sometimes svc is nil * don't test for delete overlap on each selector * lint	4 years ago
Karsten Jeschkies	ae39676cce	Stub multi tenant querier. (#5490) ... **What this PR does / why we need it**: - Introduce Queries interface and Querier API. - Configure a multi-tenant querier that just passes requests through. **Which issue(s) this PR fixes**: Closes #231 **Special notes for your reviewer**: **Checklist** - [x] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	4 years ago
Susana Ferreira	2d24e2ea64	Migration to dskit/ring (#4641)	4 years ago
Arve Knudsen	cfb4fc1f55	Use flagext from dskit (#4225) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Cyril Tovena	490c19e38c	Deprecate max_look_back_period in the chunk storage. (#3677) ... * Deprecate max_look_back_period in the chunk storage. From now on it will be applied at the frontend, ruler and querier level. The querier validation was missing so I added it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Kaviraj	6bcd9807d6	replace `time.Duration` -> `model.Duration` for `Limits`. (#3632) ... Rationale 1. support user-friendly duration format (e.g: "1h30m45s") in JSON value. 2. In consistent with Cortex and tempo.	5 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623) ... * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	5 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295) ... * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	5 years ago
Sandeep Sukhani	cb53a47139	fix query intervals when running boltdb-shipper in single binary (#2855)	5 years ago
Sandeep Sukhani	6053cdb9d9	fix store query bug when running loki in single binary mode with boltdb-shipper (#2655)	5 years ago
Sandeep Sukhani	2f7e0e50c2	rpc for querying ingesters to get chunk ids from its store (#2601) ... * rpc for querying ingesters to get chunk ids from its store * fix broken tests * separate out code from querier to query ingester to a new type called IngesterQuerier * add AsyncStore which helps with querying and merging results from both chunk store and ingesters * query ingesters local store for labels when active index type is boltdb-shipper or query store is true * better naming * add some tests * add test case for duplicate chunk ids from ingesters * add some logs and spans * check for nil fetcher * use require.Same to assert that two pointers are pointing to same object * some more improvements 1. query chunk-refs parallely from store and ingesters. 2. limit look back for chunk refs and labels from ingesters to start of schema when using boltdb-shipper. * fixes suggested from PR review * fix mode not being set before cortex_storage.NewStore is being called * add IngesterQuerier as a dependency for store when target is either ingester or querier * do not cache index from ingester * rename tests * fix broken tests * update upgrade guide to rollout ingesters before queriers * upgrade guide changes suggested in PR review * runtime check for whether active index store is boltdb-shipper * consider previous schema as well for boltdb-shipper for calculating max look back	5 years ago
Aditya C S	0885a49370	configuration doc improvements (#2318)	6 years ago
Cyril Tovena	0be64fcb34	Improve metric queries by computing samples at the edges. (#2293) ... * First pass breaking the code appart. Wondering how we're going to achieve fast mutation of labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. I realize I need hash for deduping lines. going to benchmark somes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tested some hash and decided which one to use. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting working on ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Trying to find a better hash function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More hash testing we have a winner. xxhash it is. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Settle on xxhash Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better params interfacing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add interface for queryparams for things that exist in both type of params. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add storage sample iterator implementations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests and verifying we don't get collions for the hashing method. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing ingesters tests and refactoring utility function/tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing and testing that stats are still well computed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More engine tests finished. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sharding evaluator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more engine tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix error tests in the engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish fixing all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug where extractor was not passed in correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notes about upgrade. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Renamed and fix a bug. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add memchunk tests and starting test for sampleIterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test heap sample iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * working on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing testing all new iterators. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Making sure all store functions are tested. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Benchmark and verify everything is working well. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make the linter happy. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * use xxhash v2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky test because of map. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Cyril Tovena	b95db8d1fd	Fix error message for max tail connections. (#2161) ... * Fix error message for max tail connections. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Ed Welch	9803eab947	Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper (#2073) ... * Allow configuring query_store_max_look_back_period when running a filesystem store, which allows for boltdb-shipper to horizontally scale filesystem store. Signed-off-by: Ed Welch <edward.welch@grafana.com> * Update docs/operations/storage/boltdb-shipper.md Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * add some more precision to the timestamp used to make the name unique just to make sure there are not collisions if for some reason the ID is empty (which can happen if the hostname doesn't resolve for some reason) Signed-off-by: Ed Welch <edward.welch@grafana.com> * Allow the ingester to query the store for labels if query store is enabled. Signed-off-by: Ed Welch <edward.welch@grafana.com> * make queriers aware of the fact that ingesters may be querying the store so they don't duplicate query efforts Signed-off-by: Ed Welch <edward.welch@grafana.com> * improve the docs Signed-off-by: Ed Welch <edward.welch@grafana.com> * add tests to the calculateMaxLookBack Signed-off-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	6 years ago
Cyril Tovena	9988ce4b9f	Improve protobuf serialization (#2031) ... * Generate custom proto type for Streams. This is the first step to improve protobuf serialization. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve protobuf serialization allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Saves allocations by stringifying labels only once. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Owen Diehl	3347871154	only fetches one chunk per series in /series (#1914) ... * only fetches one chunk per series in /series * Update pkg/storage/store.go * chunksBySeries naming * sorts series response, adds tests for chunks < batchsize	6 years ago
Owen Diehl	e0a77656fd	removes duplicate logRangeExprExt grammar (#1651) ... * removes duplicate logRangeExprExt grammar * linting errors	6 years ago
Sandeep Sukhani	c7a3ec54f1	limit for concurrent tail requests (#1562) ... * limit for concurrent tail requests When a new request comes in, querier would ask one of the active ingesters for count of active tail requests for the user If the count exceeds the configured limits for the user then tail request would be rejected right away. * goimports querier_test.go Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * query all ingesters for their max tail count Signed-off-by: Edward Welch <edward.welch@grafana.com> * changed to check only ACTIVE ingesters and tweaked error handling some Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Ed Welch <ed@oqqer.com>	6 years ago
Peter Štibraný	d4cae667ae	loki: use new runtimeconfig package from Cortex (#1484) ... * loki: use new runtimeconfig package from Cortex runtimeconfig is a generalization of previous "overrides" mechanism, and can be used not for per-tenant limits, but other parts of Loki too (only one additional usage for now is "multi" KV client). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed RuntimeConfig from Ingester dependencies. Ingester already depends on RuntimeConfig via Store and Overrides mechanism. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed extra new line. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Remove unused method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix broken code after rebasing. Removed incorrect comment. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated docs with section about runtime configuration. Mostly copied from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod tidy, go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated Cortex to current master. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Go mod tidy, vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Owen Diehl	841befaf8e	Feature/query ingesters within (#1572) ... * querier.query-ingesters-within * avoids an unnecessary heapIter * updates changelog/config readme * variable reuse	6 years ago
Owen Diehl	dafb9d8425	Enables Series API in loki (#1419) ... * helpers for parsing /series requests * logproto.SeriesResponse impl * passes unparsed matchers in protobuf for simplicity * ingester support for series queries * labels map coercion * fixes label match loghttp test * removes labelmatcher protobuf adapters * series api impl * tests matchers for dispatching series queries * series looked up serially for /series api * splits series api async code and remove unnecessary type assertion * adds series handle to /api/prom/series and includes docs * compatibility changes w/ upstream merge	6 years ago
Peter Štibraný	3df5561336	Updated cortex to latest master. (#1088) ... * Updated cortex to latest master. * Updated cortex to latest master. * Fix tests after updating cortex. * go mod tidy and vendor	6 years ago
Joe Elliott	7794d15b6a	Extended test query to expose broken query. Parsed logql correctly in validate (#1044) ... Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
Sandeep Sukhani	ec5bc7054d	limits: limits implementation for loki (#948) ... * limits implementation for loki Added following new limits: Length of query Number of active streams Number of streams matcher per query Ingestion rate * periodically reload ingestion rate limits to apply latest rates from overrides	6 years ago
Cyril Tovena	8bd9f752aa	Adds more tests for range queries	6 years ago
Cyril Tovena	7f3db9d7f0	LogQL: Vector and Range Vector Aggregation. ... - adds avg,min,max,top,bottomk,stddev,stdvar,count - updates api documentation - adds tests Improve yacc & go lexer to understand duration Remove support for regexp in all queries Clean up querier and logselector	6 years ago
Marco Pracucci	e5d6f3c317	Fix panic in tailer when an ingester is removed from the ring while tailing (#897)	6 years ago
Marco Pracucci	b9185324f4	Improved Tailer loop (dropped entries, logic refactoring, introduced tests)	6 years ago
Marco Pracucci	476c3961c2	Added -querier.query_timeout support (#788)	6 years ago