1979 Commits (3b57ad2a5432f2d13a97b009762e91e8eadd0d2c)

Author	SHA1	Message	Date
Bryan Boreham	3b57ad2a54	WAL: remove ePool that is unused (#8669) ... **What this PR does / why we need it**: Nothing is ever fetched from the pool - `GetEntries()` is never called. And the loop to call `PutEntries()` follows a call where the slice it ranges over is reset to empty, so that loop never added anything to the pool. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - NA Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Bill Franklin	3ed9f0c9ef	Canary: support filtering / parsing logs with LogQL (#8871) ... **What this PR does / why we need it**: The canary logs can be shipped to Loki in various formats. This allows users of the canary to parse out the canaries logs from any format e.g. nested json. The alternative is to do something different for the canary logs to prevent the logs being wrapped / manipulated (e.g. write to Loki directly, or use a different log pipeline) which limits the benefits of the canary. I've confirmed this works locally. **Which issue(s) this PR fixes**: Fixes #7775 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Dylan Guedes	4f94b89fdb	Loki: Add more spans to write path (#8888) ... **What this PR does / why we need it**: Add new spans to our write path to better determine which operations are holding our write performance.	2 years ago
Salva Corts	d24fe3e68b	Max bytes read limit (#8670) ... **What this PR does / why we need it**: This PR implements two new per-tenant limits that are enforced on log and metric queries (both range and instant) when TSDB is used: - `max_query_bytes_read`: Refuse queries that would read more than the configured bytes here. Overall limit regardless of splitting/sharding. The goal is to refuse queries that would take too long. The default value of 0 disables this limit. - `max_querier_bytes_read`: Refuse queries in which any of their subqueries after splitting and sharding would read more than the configured bytes here. The goal is to avoid a querier from running a query that would load too much data in memory and can potentially get OOMed. The default value of 0 disables this limit. These new limits can be configured per tenant and per query (see https://github.com/grafana/loki/pull/8727). The bytes a query would read are estimated through TSDB's index stats. Even though they are not exact, they are good enough to have a rough estimation of whether a query is too big to run or not. For more details on this refer to this discussion in the PR: https://github.com/grafana/loki/pull/8670#discussion_r1124858508. Both limits are implemented in the frontend. Even though we considered implementing `max_querier_bytes_read` in the querier, this way, the limits for pre and post splitting/sharding queries are enforced close to each other on the same component. Moreover, this way we can reduce the number of index stats requests issued to the index gateways by reusing the stats gathered while sharding the query. With regard to how index stats requests are issued: - We parallelize index stats requests by splitting them into queries that span up to 24h since our indices are sharded by 24h periods. On top of that, this prevents a single index gateway from processing a single huge request like `{app=~".+"} for 30d`. - If sharding is enabled and the query is shardable, for `max_querier_bytes_read`, we re-use the stats requests issued by the sharding ware. Specifically, we look at the [bytesPerShard][1] to enforce this limit. Note that once we merge this PR and enable these limits, the load of index stats requests will increase substantially and we may discover bottlenecks in our index gateways and TSDB. After speaking with @owen-d, we think it should be fine as, if needed, we can scale up our index gateways and support caching index stats requests. Here's a demo of this working: <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918478-d4b6c2fd-de4d-478a-9c8b-e38fe148fa95.png"> <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918798-a71b1db8-ea68-4d00-933b-e5eb1524d240.png"> **Which issue(s) this PR fixes**: This PR addresses https://github.com/grafana/loki-private/issues/674. **Special notes for your reviewer**: - @jeschkies has reviewed the changes related to query-time limits. - I've done some refactoring in this PR: - Extracted logic to get stats for a set of matches into a new function [getStatsForMatchers][2]. - Extracted the _Handler_ interface implementation for [queryrangebase.roundTripper][3] into a new type [queryrangebase.roundTripperHandler][4]. This is used to create the handler that skips the rest of configured middlewares when sending an index stat quests ([example][5]). **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: ff847305af/pkg/querier/queryrange/shard_resolver.go (L179-L186) [2]: ff847305af/pkg/querier/queryrange/shard_resolver.go (L72) [3]: 3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L124) [4]: 3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L163) [5]: f422e0a52b/pkg/querier/queryrange/roundtrip.go (L521)	2 years ago
Karsten Jeschkies	94725e7908	Define `RequiredLabels` query limit. (#8851) ... **What this PR does / why we need it**: Some end-users can impose great workload on a cluster by selecting too many streams in their queries. We should be able to limit them. Therefore we introduce a new limit `RequiredLabelMatchers` which list label names that must be included in the stream selectors. The implementation follows the same approach as for max query limit. **Which issue(s) this PR fixes**: Fixes #8745 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Sandeep Sukhani	4e893a0a88	tsdb: sample chunk info from tsdb index to limit the amount of chunkrefs we read from index (#8742) ... **What this PR does / why we need it**: Previously we used to read the info of all the chunks from the index and then filter it out in a layer above within the tsdb code. This wastes a lot of resources when there are too many chunks in the index, but we just need a few of them based on the query range. Before jumping into how and why I went with chunk sampling, here are some points to consider: * Chunks in the index are sorted by the start time of the chunk. Since this does not tell us much about the end time of the chunks, we can only skip chunks that start after the end time of the query, which still would make us process lots of chunks when the query touches chunks that are near the end of the table boundary. * Data is written to tsdb with variable length encoding. This means we can't skip/jump chunks since each chunk info might vary in the number of bytes we write. Here is how I have implemented the sampling approach: * Chunks are sampled considering their end times from the index and stored in memory. * Here is how `chunkSample` is defined: ``` type chunkSample struct { largestMaxt int64 // holds largest chunk end time we have seen so far. In other words all the earlier chunks have maxt <= largestMaxt idx int // index of the chunk in the list which helps with determining position of sampled chunk offset int // offset is relative to beginning chunk info block i.e after series labels info and chunk count etc prevChunkMaxt int64 // chunk times are stored as deltas. This is used for calculating mint of sampled chunk } ``` * When a query comes in, we will find `chunkSample`, which has the largest "largestMaxt" that is less than the given query start time. In other words, find a chunk sample which skips all/most of the chunks that end before the query start time. * Once we have found a chunk sample which skips all/most of the chunks that end before the query start, we will sequentially go through chunks and consider only the once that overlap with the query range. We will stop processing chunks as soon as we see a chunk that starts after the end time of the query since the chunks are sorted by start time. * Sampling of chunks is done lazily for only the series that are queried, so we do not waste any resources on sampling series that are not queried. * To avoid sampling too many chunks, I am sampling chunks at `1h` steps i.e given a sampled chunk with chunk end time `t`, the next chunk would be sampled with end time >= `t + 1h`. This means typically, we should have ~28 chunks sampled for each series queried from each index file, considering 2h default chunk length and chunks overlapping multiple tables. Here are the benchmark results showing the difference it makes: ``` benchmark old ns/op new ns/op delta BenchmarkTSDBIndex_GetChunkRefs-10 12420741 4764309 -61.64% BenchmarkTSDBIndex_GetChunkRefs-10 12412014 4794156 -61.37% BenchmarkTSDBIndex_GetChunkRefs-10 12382716 4748571 -61.65% BenchmarkTSDBIndex_GetChunkRefs-10 12391397 4691054 -62.14% BenchmarkTSDBIndex_GetChunkRefs-10 12272200 5023567 -59.07% benchmark old allocs new allocs delta BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% BenchmarkTSDBIndex_GetChunkRefs-10 345653 40 -99.99% benchmark old bytes new bytes delta BenchmarkTSDBIndex_GetChunkRefs-10 27286536 6398855 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286571 6399276 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286566 6400699 -76.54% BenchmarkTSDBIndex_GetChunkRefs-10 27286561 6399158 -76.55% BenchmarkTSDBIndex_GetChunkRefs-10 27286580 6399643 -76.55% ``` **Checklist** - [x] Tests updated	2 years ago
Christian Haudum	9844fad8b4	Rename LeafQueue to TreeQueue (#8856) ... **What this PR does / why we need it**: `TreeQueue` is the semantically more correct term for this type of queue. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Travis Patterson	9d20bed26a	Short circuit parsers (#8724) ... In many cases we only need a few labels from a line. Because most of our parsers parse lines incrementally, we can stop parsing a line after we have all the labels we want from it. This pr uses `ParserHints` to keep track of the number of extracted labels. It also provides a way for parsers to know when they should stop parsing. Notes: - parsers had inconsistent ordering between the `ShouldExtract` call and adding the `_extracted` label to duplicates. This PR makes appending `_extracted` always happen before `ShouldExtract` to keep counts of what is extracted compared to expected labels consistent. Next Steps: - When the user specifies a query with a grouping containing the `_extracted` label but there is no duplicate between the passed line and labels, short circuiting will not work. I'll address this in a follow up PR. Benchmarks: To try and show a balanced view of what this buys us, this pr picks a label our of the middle of a line. In a best case this might be much better. In the worst case, we have to parse the whole line. ``` benchstat short_circuit_old.txt short_circuit_new.txt name old time/op new time/op delta KeyExtraction/json-8 456ns ± 3% 256ns ± 2% -43.84% (p=0.000 n=9+10) KeyExtraction/logfmt-8 347ns ± 4% 171ns ± 2% -50.86% (p=0.000 n=10+10) KeyExtraction/logfmt-expression-8 552ns ± 2% 368ns ± 2% -33.23% (p=0.000 n=9+10) name old alloc/op new alloc/op delta KeyExtraction/json-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 16.0B ± 0% 16.0B ± 0% ~ (all equal) name old allocs/op new allocs/op delta KeyExtraction/json-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 2.00 ± 0% 2.00 ± 0% ~ (all equal) ``` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
W.T. Chang	183fe85f1c	Loki: Add route_randomly to Redis options (#8852) ... **What this PR does / why we need it**: Current Redis client only reads from master node, which makes master node under heavy pressure. Usually, one or more read replicas are expected on production environments. This PR helps to increase the utilization of read replicas.	2 years ago
Danny Kopping	528ed22dc2	Ruler: remote rule evaluation hardening (#8785) ... **What this PR does / why we need it**: This PR is part 2 of 2 implementing remote rule evaluation. See [part 1](https://github.com/grafana/loki/pull/8744) for more context.	2 years ago
Danny Kopping	2d93952929	(chore): remove incorrect cache config preample (#8859) ... With the addition of the [config generation tool](https://github.com/grafana/loki/pull/7916), a comment prefix was added for all cache types: `Cache config for index entry writing.` https://grafana.com/docs/loki/next/configuration/#cache_config This appears in the description of each cache config option, and is misleading.	2 years ago
Travis Patterson	5a08a6bcb9	Label selector optimizations (#8763) ... This PR introduces the following label selector optimizations: - any selector that is `=~<literal>` is converted to an `Equals/NotEquals` matcher - Use the greedy->non-greedy regex improvements that exist in line and label filters - Never run `.*` label matchers. Prometheus regex matchers anchor the match to the beginning and end of the line. That means matchers of the form `=~<literal_value>` are just testing equality. Benchmarks -- this is just a quick bench to demonstrate equality matchers are faster than their equivalent regex matchers: ``` BenchmarkMatcherTypes/regex-8 27694834 53.74 ns/op 0 B/op 0 allocs/op BenchmarkMatcherTypes/equals-8 627045850 2.153 ns/op 0 B/op 0 allocs/op ```	2 years ago
Ashwanth	7d2eeddfb8	LogQL: Fix panic when using `unpack` with empty line (#8853) ... **What this PR does / why we need it**: Fixes panic in unpack parser when processing empty lines ``` /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/logql/log/parser.go:583 github.com/grafana/loki/pkg/logql/log.(*UnpackParser).unpack(...) panic: runtime error: index out of range [0] with length 0 ``` Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Danny Kopping	5c3d204ebf	Ruler: rule evaluation jitter (#8848) ... **What this PR does / why we need it**: This PR introduces a configurable random sleep before each rule evaluation to prevent contention on the query engine should multiple runs execute concurrently. It's quite likely that rules will be evaluated ~concurrently if they have the same interval.	2 years ago
Karsten Jeschkies	f5f1753851	Print duration in error messages with more readable. (#8816) ... **What this PR does / why we need it**: The old error messages would print only up to hours. E.g. `169h30s`. This change will print it as `7d1h30s`. See [model.Duration](66b493f42b/model/time.go (L259-L290)) for details. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Christian Haudum	be8b4eece3	Scheduler: Add query fairness control across multiple actors within a tenant (#8752) ... **What this PR does / why we need it**: This PR wires up the scheduler with the hierarchical queues. It is the last PR to implement https://github.com/grafana/loki/pull/8585. When these changes are in place, the client performing query requests can control their QoS (query fairness) using the `X-Actor-Path` HTTP header. This header controls in which sub-queue of the tenant's scheduler queue the query request is enqueued. The place within the hierarchy where it is enqueued defines the probability with which the request gets dequeued. A common use-case for this QoS control is giving each Grafana user within a tenant their fair share of query execution time. Any documentation is still missing and will be provided by follow-up PRs. **Special notes for your reviewer**: ```console $ gotest -count=1 -v ./pkg/scheduler/queue/... -test.run=TestQueryFairness === RUN TestQueryFairness === RUN TestQueryFairness/use_hierarchical_queues_=_false dequeue_qos_test.go:109: duration actor a 2.007765568s dequeue_qos_test.go:109: duration actor b 2.209088331s dequeue_qos_test.go:112: total duration 2.209280772s === RUN TestQueryFairness/use_hierarchical_queues_=_true dequeue_qos_test.go:109: duration actor b 605.283144ms dequeue_qos_test.go:109: duration actor a 2.270931324s dequeue_qos_test.go:112: total duration 2.271108551s --- PASS: TestQueryFairness (4.48s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_false (2.21s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_true (2.27s) PASS ok github.com/grafana/loki/pkg/scheduler/queue 4.491s ``` ```console $ gotest -count=5 -v ./pkg/scheduler/queue/... -bench=Benchmark -test.run=^$ -benchtime=10000x -benchmem goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/scheduler/queue cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz BenchmarkGetNextRequest BenchmarkGetNextRequest/without_sub-queues BenchmarkGetNextRequest/without_sub-queues-8 10000 29337 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21348 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21595 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21189 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21602 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33770 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33596 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 34432 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33760 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33664 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 71405 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 59472 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 117163 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 106505 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 64374 ns/op 3200 B/op 300 allocs/op BenchmarkQueueRequest BenchmarkQueueRequest-8 10000 168391 ns/op 320588 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 166203 ns/op 320587 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 149518 ns/op 320584 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 219776 ns/op 320583 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 185198 ns/op 320597 B/op 1156 allocs/op PASS ok github.com/grafana/loki/pkg/scheduler/queue 64.648s ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	d08b87f799	json parser: fix panic when processing empty line (#8835) ... **What this PR does / why we need it**: Fixes panic when an empty line is processed by `JSONExpressionParser` ``` /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/logql/log/parser.go:537 github.com/grafana/loki/pkg/logql/log.isValidJSONStart(...) ... panic: runtime error: index out of range [0] with length 0 ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
René Scheibe	76c2e910cb	chore: Fix spelling of "syncing" (#8829) ... This makes it consistent in the repository.	2 years ago
Dylan Guedes	53b9a351e5	Loki: Add a `prepare-shutdown` endpoint (#8786) ... **What this PR does / why we need it**: Add a new `/ingester/prepare_shutdown` endpoint. This new endpoint configures the ingester to release resources on the next SIGTERM. --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Travis Patterson	37c0b3dec2	Allow math on errors (#8823) ... Currently metric expressions will fail if they contain the parser error label `__error__`. There are cases where you may want to count errors. For example, `sum(count_over_time({foo="bar"} | json | __error__= "JSONParserErr" [$__interval]))` to see how many of your logs are unparsable JSON. This pr uses `ParseHint` to determine whether the `__error__` label was requested in the query and, if it was, do math on it like any other query. Parse hints are only included in VectorAggregations so the label used to signal whether or not to keep errors is never seen by a user. All other operations remain unaffected	2 years ago
Karsten Jeschkies	ffb605fa85	Fix: Ensure query limits tripperware wraps frontend. (#8789) ... **What this PR does / why we need it**: The query frontend would not enforce the per-request query limits because the injected tripperware was wrapped *after* the frontend has been created. That is a bug. The order must be changed so that the `queryrange.LimitsMiddleware` runs *after* the `querylimits.limitsMiddleware`. **Which issue(s) this PR fixes**: Fixes #8790 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Travis Patterson	83ba93c867	fix to-string for noop label filters (#8799) ... Label Filter optimizations omit `.*` label filters from the execution pipeline but intruduced a bug where those stages were also ommitted from calls to `String` on the expression. This results in unparsable queries and panics when queries are cloned via `query.String()` This PR makes `noopLabelFilter` embed matchers and inherit their `String` method just like the `StringLabelFilters` do	2 years ago
Christian Haudum	22b672986a	Revert "Implement hierarchical queues for query scheduler" (#8796) ... Reverts grafana/loki#8691 These changes caused problems with the scheduler in our dev environments. I will conduct further testing and include the reverted changes in https://github.com/grafana/loki/pull/8752	2 years ago
Trevor Whitney	b46e392adb	gofmt is needed for the build (#8678) ... **What this PR does / why we need it**: `gofmt` is needed for the build, `make format` fails without it.	2 years ago
Danny Kopping	33e44ed39d	Ruler: remote rule evaluation (#8744) ... **What this PR does / why we need it**: Adds the ability to evaluate recording & alerting rules against a given `query-frontend`, allowing these queries to be executed with all the parallelisation & optimisation that regular adhoc queries have. This is important because with `local` evaluation all queries are single-threaded, and rules that evaluate a large range/volume of data may timeout or OOM the `ruler` itself, leading to missed metrics or alerts. When `remote` evaluation mode is enabled, the `ruler` effectively just becomes a gRPC client for the `query-frontend`, which will dramatically improve the reliability of the `ruler` and also drastically reduce its resource requirements. **Which issue(s) this PR fixes**: This PR implements the feature discussed in https://github.com/grafana/loki/pull/8129 (**LID 0002: Remote Rule Evaluation**).	2 years ago
Karsten Jeschkies	28ebf8ec12	Impose per-request limit on label queries. (#8777) ... **What this PR does / why we need it**: #8727 introduced per-request limits. These should be enforce for label queries as well. This change adds the required middleware to all label query endpoints. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Ed Welch	0352b75f0d	Loki: Add new logql template functions `bytes`, `duration`, `unixEpochMillis`, `unixEpochNanos`, `toDateInZone`, `b64Enc`, and `b64Dec` (#8774) ... **What this PR does / why we need it**: This adds several new template functions to the logql to be used in `line_format` and `label_format` - bytes - duration (and duration_seconds) - unixEpochMillis - unixEpochNanos - toDateInZone - b64Enc and b64Dec `bytes` and `duration` adds parity with the `unwrap` pipeline stage in logql, allowing you to convert human readable versions of bytes and times into their respective primitives of bytes and seconds. This makes it possible to do math and comparisons of these extracted values in `line_format` and `label_format` `unixEpochMillis` and `unixEpochNanos` extends the existing `unixEpoch` function which can only generate seconds level precision to allow having millisecond and nanosecond level precision. `toDateInZone` allows for specifying a specific timezone when parsing a date. `b64Enc` and `b64Dec` allow base64 encoding and decoding values. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: @JStickler I made a lot of changes to the template_functions.md doc page to try and create some more consistency as well as remove some notices that really aren't relevant anymore: * Tried to make sure every function had a `Signature` and `Examples` block * Removed a lot of `NOTE` about functions being introduced in a specific version * Removed documentation for some deprecated string functions (they still exist in code but I removed the docs for them, their exist better replacements for all of them) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	55c96bfe90	Loki: add lines processed to metrics.go (#8764) ... **What this PR does / why we need it**: Typically `throughput` and `total_bytes` are the most common fields in the `metrics.go` log line which we look at for query performance. However, there are cases where query performance is better measured by `total_lines` and `lines_per_second` (e.g. when processing a lot of high volume but relatively short log lines) and having this information in metrics.go can be very helpful. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	a4eb536fb2	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761) ... **What this PR does / why we need it**: Currently the `metrics.go` log line emitted after every query includes a metric called "subqueries". This currently tracks the number of queries created by the split_by_time operations done in Loki, but does not include any counts for subqueries created as a result of sharding. It becomes difficult to make a single subqueries counter that gives useful information to determine how much a query is split by time and sharded by a shard factor, especially now that sharding in TSDB indexes is dynamic. This PR removes and deprecates the `subqueries` stat and instead creates a `splits` and `shards` statistic which records how much a query was split_by_time and the total number of shards created as well. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Karsten Jeschkies	8eac851962	Remove print statements introduced by #8727. (#8775)	2 years ago
Ed Welch	eadb9d2c00	Loki: Change the behavior of the 0 value for retention_period to disable retention. (#8753) ... **What this PR does / why we need it**: Issue #8359 exposed a very unfortunate inconsistency in a Loki config where the zero value did not disable a feature. By convention in Loki the zero value for a config typically "disables" it. This PR changes the retention_period to honor the zero value of `0` or `0s` to disable retention. **Which issue(s) this PR fixes**: Fixes #8359 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Danny Kopping	e2ac2d50e2	Changes from linter (#8769) ... **What this PR does / why we need it**: We had a few outstanding linter issues, and running `make lint` fixes them all automatically. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727) ... **What this PR does / why we need it**: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. **Which issue(s) this PR fixes**: Fixes #8762 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Travis Patterson	9ac33e216e	Reimplement Unpack in terms of jsonparser rather than jsoniter (#8739) ... This PR reimplements the functionality of `Unpack` using the `jsonparser` library rather than `jsoniter` Benchmarks: ``` name old time/op new time/op delta _Parser/unpack/no_labels_hints-8 900ns ± 4% 309ns ± 3% -65.62% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 1.06µs ±12% 0.32µs ± 2% -69.72% (p=0.000 n=9+10) _Parser/unpack-not_json_line/no_labels_hints-8 37.5ns ± 8% 10.6ns ± 3% -71.70% (p=0.000 n=9+9) _Parser/unpack-not_json_line/labels_hints-8 39.6ns ±11% 10.3ns ± 2% -74.00% (p=0.000 n=10+9) name old alloc/op new alloc/op delta _Parser/unpack/no_labels_hints-8 536B ± 0% 80B ± 0% -85.07% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 536B ± 0% 80B ± 0% -85.07% (p=0.000 n=10+10) _Parser/unpack-not_json_line/no_labels_hints-8 0.00B 0.00B ~ (all equal) _Parser/unpack-not_json_line/labels_hints-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta _Parser/unpack/no_labels_hints-8 17.0 ± 0% 4.0 ± 0% -76.47% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 17.0 ± 0% 4.0 ± 0% -76.47% (p=0.000 n=10+10) _Parser/unpack-not_json_line/no_labels_hints-8 0.00 0.00 ~ (all equal) _Parser/unpack-not_json_line/labels_hints-8 0.00 0.00 ~ (all equal) ```	2 years ago
Travis Patterson	9c6e509a6a	Reimplement JsonExpressionParser in terms of jsonparser (#8734) ... This PR reimplements the functionality of `JSONExpressionParser` using the `jsonparser` library rather than `jsoniter` Benchmarks: ``` benchstat json_expression_old.txt json_expression_new.txt name old time/op new time/op delta JsonExpressionParser/json-expression-8 4.15µs ± 4% 0.82µs ± 3% -80.17% (p=0.000 n=10+10) name old alloc/op new alloc/op delta JsonExpressionParser/json-expression-8 1.07kB ± 0% 0.02kB ± 0% -97.76% (p=0.000 n=10+10) name old allocs/op new allocs/op delta JsonExpressionParser/json-expression-8 72.0 ± 0% 2.0 ± 0% -97.22% (p=0.000 n=10+10) ```	2 years ago
Christian Haudum	4b74a5b815	Implement hierarchical queues for query scheduler (#8691) ... ### What this PR does / why we need it: This PR is the first step towards hierarchical queues within the query scheduler. Hierarchical queues can be used to ensure query fairness across multiple actors within a single tenant, such as individual (human) users using the same tenant. The change is described in the LID ["Query fairness across users within tenants"](https://grafana.com/docs/loki/next/lids/0003-queryfairnessinscheduler/). * Extract metrics for `RequestQueue` into their own `metrics` field, so that they can extended and passed more easily in the future. * Move mapping for tenant queues into its own data structure (ordered map) so its functionality can be reused. It also simplifies the code inside the `tenantQueues` struct. * Add the `LeafQueue` struct that implements hierarchical queues. Sub-queues are dequeued in a round-robin manner using the same implementation of `Mapping` that is also used by the tenant queues. ### Special notes for your reviewer Part of https://github.com/grafana/loki/pull/8585 Contains the changes from PR https://github.com/grafana/loki/pull/8722 **This PR does not yet make use of the new hierarchical queues, it only adds the relevant data structures. Wiring up will be done in a follow up PR.** While the new `Mapping` data structure adds an unsignificant overhead to the time/op I consider the simplification an overall better tradeoff. ```console $ benchstat scheduler-bench-old.txt scheduler-bench-new.txt name old time/op new time/op delta GetNextRequest-8 18.0µs ± 2% 20.5µs ± 3% +13.89% (p=0.008 n=5+5) QueueRequest-8 56.6µs ± 0% 53.5µs ± 3% -5.48% (p=0.008 n=5+5) name old alloc/op new alloc/op delta GetNextRequest-8 1.60kB ± 0% 1.60kB ± 0% ~ (all equal) QueueRequest-8 35.6kB ± 0% 29.8kB ± 0% -16.37% (p=0.008 n=5+5) name old allocs/op new allocs/op delta GetNextRequest-8 100 ± 0% 100 ± 0% ~ (all equal) QueueRequest-8 819 ± 0% 806 ± 0% -1.59% (p=0.008 n=5+5) ``` #### Complexity of operations | Operation | Function | Complexity | | --------- | -------- | ---------- | | Insert | `Put(key string, value *tenantQueue) bool` | `O(1)`| | Select | `GetNext(idx QueueIndex) *tenantQueue` | `O(1)` (best case) `O(n)` (worst case) | | Remove | `Remove(key string) bool` | `O(1)` | --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Sandeep Sukhani	162a2d0057	deletion: fix issue in processing delete requests with tsdb index (#8665) ... **What this PR does / why we need it**: When a delete request creates a new chunk from an existing chunk, we index it to the index store. In TSDB, we [drop](944b488bd2/pkg/storage/stores/tsdb/head_manager.go (L235-L237)) `__name__` label, which was being added to chunkrefs for historical reasons. However, this was not done while indexing chunks from the deletion code, causing deletion to fail with the error [chunk insertion is only allowed on existing streams](619c2212f5/pkg/storage/stores/tsdb/builder.go (L67)). This PR handles the issue by dropping the `__name__` label and adding relevant tests. **Special notes for your reviewer**: I will add an integration test for deletion with TSDB once we have PR https://github.com/grafana/loki/pull/7754 merged since it does some groundwork to run integration tests with both `boltdb-shipper` and `tsdb`. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated	2 years ago
Karsten Jeschkies	28d845d26c	Make Loki Overrides public. (#8696) ... **What this PR does / why we need it**: Follow up to #8689 that will allow to inject a different overrides implementation. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Travis Patterson	7f42137584	Use Line Filters in StringLabelFilter (#8659) ... We've done a ton of work to optimize regexes in line filters but nothing in label filters. To avoid duplicating the work or reworking all of our filter types, this PR creates a new label filter that has an optimized line filter. It uses that against the label value to determine if they match. Note on Benchmarks: the input string for this is 1114195 bytes long. The `not` test tells us that our approach is quite a lot slower when the literal pattern isn't present in the input. The size of the input really emphasizes it. Benchmarks: ``` name old time/op new time/op delta LineLabelFilters/foo-8 50.6ns ± 9% 14.3ns ± 1% -71.67% (p=0.000 n=10+8) LineLabelFilters/not-8 15.7ns ± 1% 64165.6ns ± 2% +407766.77% (p=0.000 n=10+10) LineLabelFilters/(foo)-8 58.4ns ± 1% 14.8ns ±10% -74.68% (p=0.000 n=9+9) LineLabelFilters/(foo|ba)-8 64.9ns ± 0% 18.2ns ± 2% -71.90% (p=0.000 n=7+9) LineLabelFilters/(foo|ba|ar)-8 74.8ns ± 7% 21.4ns ± 2% -71.42% (p=0.000 n=10+8) LineLabelFilters/(foo|(ba|ar))-8 65.1ns ± 1% 18.1ns ± 2% -72.22% (p=0.000 n=8+8) LineLabelFilters/foo.*-8 882ns ± 4% 15ns ±11% -98.30% (p=0.000 n=9+10) LineLabelFilters/.*foo-8 11.5ns ± 8% 14.9ns ± 7% +30.39% (p=0.000 n=10+10) LineLabelFilters/.*foo.*-8 5.46µs ± 7% 0.01µs ± 3% -99.74% (p=0.000 n=10+9) LineLabelFilters/(.*)(foo).*-8 6.26µs ± 7% 0.01µs ± 2% -99.77% (p=0.000 n=10+9) LineLabelFilters/(foo.*|.*ba)-8 5.58µs ± 3% 0.02µs ± 8% -99.67% (p=0.000 n=9+10) LineLabelFilters/(foo.*|.*bar.*)-8 7.02µs ± 7% 0.02µs ± 8% -99.73% (p=0.000 n=10+10) LineLabelFilters/.*foo.*|bar-8 5.53µs ± 6% 0.02µs ± 2% -99.67% (p=0.000 n=10+8) LineLabelFilters/.*foo|bar-8 3.73µs ± 6% 0.02µs ± 3% -99.52% (p=0.000 n=10+9) LineLabelFilters/(?:.*foo.*|bar)-8 5.43µs ± 8% 0.02µs ± 8% -99.65% (p=0.000 n=10+10) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 5.71µs ± 3% 0.02µs ±10% -99.68% (p=0.000 n=9+9) LineLabelFilters/.*foo.*|bar|buzz-8 5.34µs ± 2% 0.02µs ± 3% -99.65% (p=0.000 n=10+9) LineLabelFilters/.*foo.*|bar|uzz-8 5.36µs ± 1% 0.02µs ± 3% -99.60% (p=0.000 n=8+8) LineLabelFilters/foo|bar|b|buzz|zz-8 68.8ns ± 8% 22.6ns ± 7% -67.19% (p=0.000 n=10+10) LineLabelFilters/f|foo|foobar-8 77.1ns ± 1% 18.6ns ± 9% -75.90% (p=0.000 n=8+10) LineLabelFilters/f.*|foobar.*|.*buzz-8 5.64µs ± 7% 0.02µs ±10% -99.60% (p=0.000 n=10+10) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 6.29µs ± 8% 0.02µs ±10% -99.65% (p=0.000 n=10+10) LineLabelFilters/.*-8 963ns ± 8% 2ns ±14% -99.77% (p=0.000 n=10+10) LineLabelFilters/.*|.*-8 4.46µs ± 8% 0.00µs ± 8% -99.95% (p=0.000 n=10+10) LineLabelFilters/.*||||-8 2.99µs ± 1% 0.00µs ±16% -99.93% (p=0.000 n=8+10) LineLabelFilters/#00-8 43.5ns ± 8% 2.1ns ± 1% -95.25% (p=0.000 n=10+8) LineLabelFilters/(?i)foo-8 71.7ns ± 6% 13.1ns ± 1% -81.73% (p=0.000 n=10+8) LineLabelFilters/(?i)界-8 16.5ns ± 8% 150.2ns ± 8% +809.55% (p=0.000 n=10+10) LineLabelFilters/(?i)ïB-8 23.8ns ± 1% 14748979.8ns ± 7% +62074730.81% (p=0.000 n=8+10) LineLabelFilters/(?:)foo|fatal|exception-8 185ns ± 2% 21ns ± 2% -88.38% (p=0.000 n=9+8) LineLabelFilters/(?i)foo|fatal|exception-8 78.4ns ± 3% 20.2ns ± 2% -74.23% (p=0.000 n=9+9) LineLabelFilters/(?i)f|foo|foobar-8 94.3ns ±16% 15.4ns ± 9% -83.66% (p=0.000 n=10+10) LineLabelFilters/(?i)f|fatal|e.*-8 65.7ns ± 5% 18.2ns ± 3% -72.24% (p=0.000 n=9+8) LineLabelFilters/(?i).*foo.*-8 6.04µs ± 8% 0.01µs ± 2% -99.78% (p=0.000 n=10+10) name old alloc/op new alloc/op delta LineLabelFilters/foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/not-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|ba)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|ba|ar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|(ba|ar))-8 0.00B 0.00B ~ (all equal) LineLabelFilters/foo.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(.*)(foo).*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo.*|.*ba)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo.*|.*bar.*)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo|bar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?:.*foo.*|bar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar|buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar|uzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/foo|bar|b|buzz|zz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/f|foo|foobar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/f.*|foobar.*|.*buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*|.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*||||-8 0.00B 0.00B ~ (all equal) LineLabelFilters/#00-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)界-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)ïB-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?:)foo|fatal|exception-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)foo|fatal|exception-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)f|foo|foobar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)f|fatal|e.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i).*foo.*-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta LineLabelFilters/foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/not-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|ba)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|ba|ar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|(ba|ar))-8 0.00 0.00 ~ (all equal) LineLabelFilters/foo.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(.*)(foo).*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo.*|.*ba)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo.*|.*bar.*)-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo|bar-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?:.*foo.*|bar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar|buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar|uzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/foo|bar|b|buzz|zz-8 0.00 0.00 ~ (all equal) LineLabelFilters/f|foo|foobar-8 0.00 0.00 ~ (all equal) LineLabelFilters/f.*|foobar.*|.*buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*|.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*||||-8 0.00 0.00 ~ (all equal) LineLabelFilters/#00-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)界-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)ïB-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?:)foo|fatal|exception-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)foo|fatal|exception-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)f|foo|foobar-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)f|fatal|e.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i).*foo.*-8 0.00 0.00 ~ (all equal) ```	2 years ago
Dylan Guedes	f9a1f2d94e	Loki: Stop running tickers on deletes (#8697) ... **What this PR does / why we need it**: Calls `ticker.Stop()` to avoid leaking ticker resources for `GenNumberLoader` and `deleteRequestsClient`.	2 years ago
Callum Styan	9a2a038f43	Allow passing of context to query related limits functions (#8689) ... In this PR we're allowing for passing of a `context.Context` via the Limits interfaces (some of which are new, to clean up hardcoding/embedding of `validation.Overrides`) This is based on work/ideas by @jeschkies . Fixes #8694 --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Danny Kopping	ab7a970b94	Querier/Ruler: add histogram to track fetched chunk size distribution (#8682) ... **What this PR does / why we need it**: We are looking at making some changes to the chunk caching strategy, and we need this data to know what size chunks we're typically requesting. For example, if the overwhelming majority of chunks are small we may decide to cache only small chunks to decrease the number of requests to the object store; we could fit more chunks in cache this way, which may have a positive performance impact.	2 years ago
Bryan Boreham	08ac5336d5	WAL: store pointers in record pool (#8667) ... **What this PR does / why we need it**: Storing a slice in `sync.Pool` wastes 24 bytes every time it is returned to the heap; see https://staticcheck.io/docs/checks#SA6002 Storing a pointer to a slice allows the Pool to work without waste. This came up in discussion about the benchmark in #8489 - with this change the memory usage comes down: ``` name old time/op new time/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 7.37ms ±13% 7.45ms ± 9% ~ (p=0.841 n=5+5) name old alloc/op new alloc/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 727kB ± 0% 679kB ± 0% -6.63% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 12.0k ± 0% 10.0k ± 0% -16.68% (p=0.000 n=5+4) ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Travis Patterson	c42a0ba868	Optimize .+ regex line filter (#8646) ... Interpolated variables from Grafana dashboards may result in the regex matcher `.+`. Filters with `+` are already being rewritten to be non greedy but there is a further optimization to make replace the regex altogether with a filter that just matchers `len(line) > 0`	2 years ago
Christian Haudum	a5652c8027	Clean up scheduler code to prepare upcoming changes (#8643) ... In preparation of https://github.com/grafana/loki/pull/8585 this PR contains some cleanups. Changes in this PR do not change the implementation, but only rename functions and data structures to make them more generic and/or suitable for extending. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Jan Jansen	84c5e20af1	S3: add support for s3 session token (#8474) ... We want to automate the access account rotation. This is that easy at moment with service accounts in MinIO. Our currently life time of tokens is 3 months with MinIO STS works this fine. Signed-off-by: Jan Jansen <jan.jansen@gdata.de>	2 years ago
Danny Kopping	6bf782b0a7	Ruler: do not keep appending logger keys (#8650) ... **What this PR does / why we need it**: Corrects a logger which was accidentally appending keys in every loop iteration, causing massive lines to be ingested	2 years ago
dependabot[bot]	31f8bf9a73	Bump golang.org/x/net from 0.4.0 to 0.7.0 in /pkg/push (#8562) ... Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. <details> <summary>Commits</summary> <ul> <li><a href="8e2b117aee"><code>8e2b117</code></a> http2/hpack: avoid quadratic complexity in hpack decoding</li> <li><a href="547e7edf38"><code>547e7ed</code></a> http2: avoid referencing ResponseWrite.Write parameter after returning</li> <li><a href="39940adcaa"><code>39940ad</code></a> html: parse comments per HTML spec</li> <li><a href="87ce33ecb4"><code>87ce33e</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="415cb6d518"><code>415cb6d</code></a> all: fix some comments</li> <li><a href="7e3c19ca52"><code>7e3c19c</code></a> all: correct typos in comments</li> <li><a href="296f09aa38"><code>296f09a</code></a> http2: case insensitive handling for 100-continue</li> <li><a href="f8411da775"><code>f8411da</code></a> nettest: fix tests on dragonfly and js/wasm</li> <li><a href="8e0e7d8d38"><code>8e0e7d8</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="7805fdc37d"><code>7805fdc</code></a> http2: rewrite inbound flow control tracking</li> <li>Additional commits viewable in <a href="https://github.com/golang/net/compare/v0.4.0...v0.7.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/grafana/loki/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2 years ago
Bryan Boreham	6fd4b5e89b	Update prometheus/prometheus from 2.41 to 2.42 (#8571) ... **What this PR does / why we need it**: Brings in the latest updates from upstream. These open up some opportunities for optimisations in TSDB indexing. Dependencies updated: * github.com/Azure/go-autorest/autorest/adal v0.9.21 -> v0.9.22 (comment-only change) * github.com/docker/docker v20.10.21 -> v20.10.23 (fixing filters bug) * golang.org/x/exp fae10dda9338 -> d38c7dcee874 (optimisations in `BinarySearch` function) Indirect dependencies also updated: * github.com/digitalocean/godo v1.91.1 -> v1.95.0 (nothing alarming in [release notes](https://github.com/digitalocean/godo/releases)) * github.com/google/pprof aee1124e3a93 -> 76d1ae5aea2b (no changes pulled into vendor) * golang.org/x/tools v0.4.0 -> v0.5.0 (relating to Go compiler utilities) * google.golang.org/genproto 76db0878b65f -> 31e0e69b6fc2 * k8s.io/api v0.26.0 -> v0.26.1 (comment-only change) * k8s.io/apimachinery v0.26.0 -> v0.26.1 (no changes pulled into vendor) * k8s.io/client-go v0.26.0 -> v0.26.1 (small fixes) **Special notes for your reviewer**: A couple of interfaces changed; these have required matching changes in Loki code. Those changes are split into separate commits. I also note that most calls to `relabel` ignore when the rule says "drop". Maybe this is wrong?	2 years ago
Christian Haudum	d4a6b159fa	chore: Cleanup label extraction expression (#8596) ... The aim of this is to remove redundancy from the parser grammar and make the expression name more descriptive. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago