19 Commits (e2a63e582c4251f2afa35263f0fe04d522444288)

Author	SHA1	Message	Date
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205)	2 years ago
Owen Diehl	acb40ed40e	Eager stream merge (#8968) ... This PR introduces a specialized heap based datastructure to merge incoming log results in the frontend. Recently we've experienced an increase in OOMs on frontends due to logs queries which match lots of data. Sharded requests in loki split based on the amount of data we expect and some queries see thousands of sub requests. For log queries, we'll fetch up the `limit` from each shard, return them to the frontend, and merge. High shard counts * limit log lines, especially combined with large log lines (in byte terms) are accumulated on the frontend. Once they all are received, the frontend merges them. This creates opportunity for OOMs as it can hold up a lot of memory. This PR addresses one of these problems by eagerly accumulating responses as they're received and only retaining a total `limit` number of entries. There's still OOM potential due to race conditions between sub requests returning to the query-frontend and the query-frontend merging other sub requests, but this definitely improves the situation. I've been able to consistently run large limited queries that touch TBs of data (i.e. `{cluster=~".+"} |= "a"`) that previously OOMed frontends. --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
李国忠	25cc9402b7	fix test increase timeout (#7022) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: increase timeout, fix test fail. ``` --- FAIL: TestInstanceFor (0.01s) downstreamer_test.go:205: Error Trace: downstreamer_test.go:205 downstreamer_test.go:256 Error: lock couldn't be acquired Test: TestInstanceFor FAIL ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Owen Diehl	d6f50ca5e1	[TSDB] Use index sampling to determine shard factor (#6396) ... * use index sampling to determine shard factor * logging/tracing + better parallelism planning * queryrange downstreamer now checks max_query_parallelism * lint * handles interval, offset in tsdb planning and adds logging * fixes ns->ms confusion in index stats proto * handle zero shard value without panics * shardmapper will downstream a single unsharded query * uses concat expr with no shards to avoid query parsing errors * better logging * fixes wrong Size() method call and rounds to nearest KB when calculating chunk size * humanize bytes in log line * only adds defaultLookback to index sampling when interval is zero * removes comment * more logging * adjust through correctly * adds query length for index queries	3 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539) ... * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
Dylan Guedes	cf419d1cb8	Implement `LegacySample`, `LegacyLabelPair` proto messages (#5301) ... * Add LegacySample and modify protos to use it. * [Skip this commit] Regenerate protos. * Modify codebase to use LegacySample instead of Sample. For reference, take a look at PR https://github.com/grafana/loki/pull/5151. * Implement LegacyLabelPair. * Use new Legacy implementations accordingly. * Use LegacySample in compat tests. - Since `compat.go` was forked from Cortex, we can reliably change all mentions from `Sample` to `LegacySample` there.	3 years ago
Dylan Guedes	3f496e7336	Copy `cortex/pkg/cortexpb` package dependency into Loki (#5151) ... * Fork cortexpb into Cortex `pkg/logproto`. * Use `pkg/logproto` instead of Cortex `pkg/cortexpb`. * Rename `cortexpb` usage to `logproto`. * Fix encoding/decoding test. * Replace cortexpb usage in the ruler/base package. * Remove exemplars from logproto.	3 years ago
Dylan Guedes	0e3df1938f	Copy querier subpackages dependency into Loki (#5068) ... * Fork cortex `querier/series`. * Fork cortex `querier/chunkstore`. * Fork cortex `querier/lazyquery`. * Use Loki forked cortex code. * Copy cortex `querier/queryrange` as `querier/queryrange/queryrangebase`. * Fork cortex `pkg/querier/queryrange` to `pkg/querier/queryrange/queryrangebase`. * Tell protoc to use gogo.types instead of pb.Any. * Use `pkg/querier/queryrange/queryrangebase`. * Remove cortex `queryrange` vendoring. * Fix lint. * Use `_ = ...` syntax instead of comment to fix lint.	3 years ago
Susana Ferreira	82f8f3c1e5	Log time in queue per request (#4949) ... * Log time in queue per request * Add querierID to log message * Add unescape request url and tenant_ids to log message * Add query enqueue time to metrics.go * Remove "querier request dequeued" log from scheduler * Address comments * Rename enqueue_time to queue_time * Store queue time in nanoseconds (int64) * Use CanonicalMIMEHeaderKey for setting the httpgrpc header	3 years ago
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830) ... * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Ed Welch	d5ec714ae0	Loki: Bug: frontend waiting on results which would never come (#4411) ... * We weren't also checking if the context was canceled while waiting for query results leading to goroutines waiting forever for results that would never come... * return error from context upstream if the context is canceled	4 years ago
Karsten Jeschkies	17f4a73a24	Filter instant queries and shard them. (#3984) ... * Filter instan queries and shard them. * [WIP] Test instant query sharding * Trace casting error. * WRap LokiInstantRequest for Params interface. * Convert paras to request. * Convert vector. * Use proper query time stamp. * Assert number of calls. * Convert vector to multiple samples. * Format code. * Use type switch. * Format code. * Rename Matrix to proto conversion. * Remvoe comment on caching. * Update pkg/querier/queryrange/roundtrip_test.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove nil middlware. * Handle LokiInstantRequest correctly. * Assert reponse type. * Return LokiPromResponse. * Verify err in tests. * Set shards. * Correct result type. * Use limits middleware. * Use shars parameter in querier. * Return vector. * Unify params. * Change direction to FORWARD. Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623) ... * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Marco Pracucci	94041307b1	Upgrade Cortex (#3466) ... * Upgrade Cortex Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated vendor/modules.txt Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Cyril Tovena	20ef66df65	Fixes a race in the scheduling limits. (#3417) ... * Fixes a race in the scheduling limits. Found out the hard way that sharding might not listen correctly to context and so still run queries after the upstream request ends. This can lead to panic when scheduling work which send on a closed channel. I've added a safe guard to wait for all jobs to end before ending the original request but also attemped to fix the sharding downstreamer to properly stop sending work when context is closed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	1530120c2e	Logql comparison ops (#2182) ... * comparison binops * starts comparison operator docs * implements bool modifier for comparison binops * vector-vector binop merging cases, including bool modifier * logql docs formatting * resets protos * linter fixes * goimports local * gofmt	5 years ago
Cyril Tovena	18f1b0640d	Bytes aggregations (#2150) ... * Add bytes_rate and bytes_over_time. Still need to works on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More lint fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More comments. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Review feedbacks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix bad conflic resolution. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Peter Štibraný	e4a5360586	Update vendored Cortex to master (1.1+) (#2149) ... * Updated Cortex to master (post 1.1) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use new modules system from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Treat query errors reported by chunk store as client errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed obsolete tests. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix race for ct variable. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Owen Diehl	89d80a6fc4	Feature/querysharding ii (#1927) ... * [wip] sharding evaluator/ast * [wip] continues experimenting with ast mapping * refactoring in preparation for binops * evaluators can pass state to other evaluators * compiler alignment * Evaluator method renamed to StepEvaluator * chained evaluator impl * tidying up sharding code * handling for ConcatSampleExpr * downstream iterator * structure for downstreaming asts * outlines sharding optimizations * work on sharding mapper * ast sharding optimizations * test for different logrange positions * shard mapper tests * stronger ast sharding & tests * shardmapper tests for string->string * removes sharding evaluator code * removes unused ctx arg * Revert "removes sharding evaluator code" This reverts commit 55d41b9519. * interfaces for downstreaming, type conversions * sharding plumbing on frontend * type alignment in queryrange to downstream sharded queriers * downstreaming support for sharding incl storage code * removes chainedevaluator * comment alignment * storage shard injection * speccing out testware for sharding equivalence * [wip] shared engine refactor * sorting streams, sharding eval fixes * downstream evaluator embeds defaultevaluator * other pkgs adopt logql changes * metrics & logs use same middleware instantiation process * wires up shardingware * middleware per metrics/logfilter * empty step populating StepEvaluator promql.Matrix adapter * sharding metrics * log/span injection into sharded engine * sharding metrics avoids multiple instantiation * downstreamhandler tracing * sharding parameterized libsonnet * removes querier replicas * default 32 concurrency for workers * jsonnet correct level override * unquote true in yaml * lowercase error + downstreamEvaluator defaults to embedded defaultEvaluator * makes shardRecorder private * logs query on failed parse * refactors engine to be multi-use, minimizes logger injection, generalizes Query methods, removes Engine interface * basic tests for querysharding mware * [wip] concurrent evaluator * integrates stat propagation into sharding evaluator * splitby histogram * extends le bounds for bytes processed * byte throughput histogram buckets to 40gb * chunk duration mixin * fixes merge w/ field rename * derives logger in sharded engine via ctx & logs some downstream evaluators * moves sharded engine to top, adds comments * logs failed merge results in stats ctx * snapshotting stats merge logic is done more effectively * per query concurrency controlled via downstreamer * unexports decodereq * queryrange testware * downstreamer tests * pr requests	5 years ago