apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Karsten Jeschkies	f5f1753851	Print duration in error messages with more readable. (#8816 ) What this PR does / why we need it: The old error messages would print only up to hours. E.g. `169h30s`. This change will print it as `7d1h30s`. See [model.Duration](`66b493f42b/model/time.go (L259-L290)`) for details. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Christian Haudum	be8b4eece3	Scheduler: Add query fairness control across multiple actors within a tenant (#8752 ) What this PR does / why we need it: This PR wires up the scheduler with the hierarchical queues. It is the last PR to implement https://github.com/grafana/loki/pull/8585. When these changes are in place, the client performing query requests can control their QoS (query fairness) using the `X-Actor-Path` HTTP header. This header controls in which sub-queue of the tenant's scheduler queue the query request is enqueued. The place within the hierarchy where it is enqueued defines the probability with which the request gets dequeued. A common use-case for this QoS control is giving each Grafana user within a tenant their fair share of query execution time. Any documentation is still missing and will be provided by follow-up PRs. Special notes for your reviewer: ```console $ gotest -count=1 -v ./pkg/scheduler/queue/... -test.run=TestQueryFairness === RUN TestQueryFairness === RUN TestQueryFairness/use_hierarchical_queues_=_false dequeue_qos_test.go:109: duration actor a 2.007765568s dequeue_qos_test.go:109: duration actor b 2.209088331s dequeue_qos_test.go:112: total duration 2.209280772s === RUN TestQueryFairness/use_hierarchical_queues_=_true dequeue_qos_test.go:109: duration actor b 605.283144ms dequeue_qos_test.go:109: duration actor a 2.270931324s dequeue_qos_test.go:112: total duration 2.271108551s --- PASS: TestQueryFairness (4.48s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_false (2.21s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_true (2.27s) PASS ok github.com/grafana/loki/pkg/scheduler/queue 4.491s ``` ```console $ gotest -count=5 -v ./pkg/scheduler/queue/... -bench=Benchmark -test.run=^$ -benchtime=10000x -benchmem goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/scheduler/queue cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz BenchmarkGetNextRequest BenchmarkGetNextRequest/without_sub-queues BenchmarkGetNextRequest/without_sub-queues-8 10000 29337 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21348 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21595 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21189 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21602 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33770 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33596 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 34432 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33760 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33664 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 71405 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 59472 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 117163 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 106505 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 64374 ns/op 3200 B/op 300 allocs/op BenchmarkQueueRequest BenchmarkQueueRequest-8 10000 168391 ns/op 320588 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 166203 ns/op 320587 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 149518 ns/op 320584 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 219776 ns/op 320583 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 185198 ns/op 320597 B/op 1156 allocs/op PASS ok github.com/grafana/loki/pkg/scheduler/queue 64.648s ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
René Scheibe	3bed7eef55	chore: Fix typo in "tcp" (#8842 )	2 years ago
Ashwanth	d08b87f799	json parser: fix panic when processing empty line (#8835 ) What this PR does / why we need it: Fixes panic when an empty line is processed by `JSONExpressionParser` ``` /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/logql/log/parser.go:537 github.com/grafana/loki/pkg/logql/log.isValidJSONStart(...) ... panic: runtime error: index out of range [0] with length 0 ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Danny Kopping	a7ff0fd70e	Removing backend target from SSD docker setup (#8836 ) What this PR does / why we need it: The `backend` target is only present in v2.8.	2 years ago
René Scheibe	76c2e910cb	chore: Fix spelling of "syncing" (#8829 ) This makes it consistent in the repository.	2 years ago
李国忠	0295fd4698	[new feature] promtail: Add a Promtail stage for probabilistic sampling (#7127 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: The sampling stage can be directly sampled. The implementation of sampling is to use the algorithm in jaeger go client ``` pipeline_stages: - sampling: rate: 0.1 ``` or it can be used with match for precise sampling. ``` pipeline_stages: - json: expressions: app: - match: pipeline_name: "app2" selector: "{app=\"poki\"}" stages: - sampling: rate: 0.1 ``` Which issue(s) this PR fixes: Fixes #6654 Special notes for your reviewer: The promtail 'rate' stage is also used with the 'match' stage for log filtering.This design makes the code very clean. Other log agents vector have log filtering built into the sampling operator, which I think is too complicated https://vector.dev/docs/reference/configuration/transforms/sample/ ``` transforms: my_transform_id: type: sample inputs: - my-source-or-transform-id exclude: null rate: 10 ``` 'rate' stage review suggestions . https://github.com/grafana/loki/pull/5051 ![image](https://user-images.githubusercontent.com/9583245/189461481-6ee4d835-2573-4b8e-8dec-2814620d758a.png) <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Dylan Guedes	53b9a351e5	Loki: Add a `prepare-shutdown` endpoint (#8786 ) What this PR does / why we need it: Add a new `/ingester/prepare_shutdown` endpoint. This new endpoint configures the ingester to release resources on the next SIGTERM. --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Mitch Hulscher	257225b486	refactor(lambda-promtail): apply terraform best practices (#8750 ) What this PR does / why we need it: I would like to offer this PR as a suggestion to improve the lambda-promtail terraform-module. I forked it to be able to deploy it more than once in an AWS account. I also applied terraform best-practices. I was hoping that perhaps these changes could be merged into upstream as well. Unlike https://github.com/grafana/loki/pull/8549 , I unfortunately did not end up making a separate commit for each change. If you would like me to create one or more issue(s) to address the points below, I'd be happy to do that as well. List of improvements: 1. Added `var.name` (defaults to lambda-promtail) so that this module can be deployed multiple times in the same AWS account. This allows us to define unique, non-conflicting names for: * the Lambda function * the CloudWatch log-group * the IAM role 2. Split IAM role policies per component; only assign permissions when required 3. Scope down permissions of the IAM role policies 4. During terraform-destroy, ensure CloudWatch log-group is removed after the lambda-function. An accidental invocation of the function could re-create an already destroyed log-group, leaving an orphaned log-group List of style changes: 1. Rename resources to `this` when there is only one instance of this resource-type 2. Add newline after `count\|before_each` and before `depends_on` 3. Group resources together and add a section comment 4. Add missing(?) statement-id to S3 AWS lambda permission Misc. 1. I added a `moves.tf` file to facilitate moving renamed resources in existing terraform statefiles. This prevents some resources from recreated. Can also be removed. These changes are backwards compatible, even though some resources will end up being re-created. A `terraform apply` should succeed (it did for me). Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) Signed-off-by: Mitch Hulscher <mitch.hulscher@lib.io>	2 years ago
Langdi	9b5bbf3fb0	Fix word confusions (#8778 ) Fixes two word confusion in the docs: * in multiple occasions `\` is referred to as slash instead of backslash * "most be convertible [...]" probably should be "must be convertible [...]"	2 years ago
J Stickler	858109046f	TOC Sync: Move grafana file under Visualize. (#8806 ) Which issue(s) this PR fixes: Fixes #8716 Special notes for your reviewer: Part of the Database products table of contents alignment project. Moves topic for Grafana dashboard from Operations to new Visualize heading in TOC. Updates front matter for the topic.	2 years ago
J Stickler	a99bbc9440	TOC Sync: Move alerting and recording rules under Alert. (#8805 ) Which issue(s) this PR fixes: Fixes #8717 Special notes for your reviewer: Part of the Database products table of contents alignment project. Moves alerting and recording rules from Rules to new Alert heading in TOC. Updates front matter for the topic. Fixes links broken by the move.	2 years ago
Periklis Tsirakidis	55045add93	operator: Fix publishing operator bundle on quay.io (#8815 )	2 years ago
Periklis Tsirakidis	9b1a427c5b	operator: Add support for memberlist bind network configuration (#8672 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de>	2 years ago
Trevor Whitney	a4d06acca7	Add yaml linting step to helm ci (#8822 ) What this PR does / why we need it: Adds a yaml linting step for the included rules in the loki helm chart. Only way I could think to enforce the comma line ending was to require qutoes around string, otherwise the linter cannot tell if the comma is just part of the string.	2 years ago
Hervé Nicol	9688f83954	helm chart: fix role/PSP mapping (#8656 ) What this PR does / why we need it: Fixes role/PSP and role/SCC mapping in Helm chart. Which issue(s) this PR fixes: Fixes #8652 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Herve Nicol <12008875+hervenicol@users.noreply.github.com>	2 years ago
Travis Patterson	37c0b3dec2	Allow math on errors (#8823 ) Currently metric expressions will fail if they contain the parser error label `__error__`. There are cases where you may want to count errors. For example, `sum(count_over_time({foo="bar"} \| json \| __error__= "JSONParserErr" [$__interval]))` to see how many of your logs are unparsable JSON. This pr uses `ParseHint` to determine whether the `__error__` label was requested in the query and, if it was, do math on it like any other query. Parse hints are only included in VectorAggregations so the label used to signal whether or not to keep errors is never seen by a user. All other operations remain unaffected	2 years ago
Trevor Whitney	8b8d2ca382	prepare changelog for 2.8 release (#8798 ) What this PR does / why we need it: Prepares the changelog for the 2.8 release	2 years ago
Karsten Jeschkies	ffb605fa85	Fix: Ensure query limits tripperware wraps frontend. (#8789 ) What this PR does / why we need it: The query frontend would not enforce the per-request query limits because the injected tripperware was wrapped after the frontend has been created. That is a bug. The order must be changed so that the `queryrange.LimitsMiddleware` runs after the `querylimits.limitsMiddleware`. Which issue(s) this PR fixes: Fixes #8790 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Periklis Tsirakidis	52f6b8253c	operator: Add alertingrule tenant id label for all rules (#8748 )	2 years ago
Periklis Tsirakidis	cb6f7a1a77	operator: Add alerting style guide validation (#8743 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de>	2 years ago
Danny Kopping	c15048b50b	Retention: add selector detail to docs (#8808 ) What this PR does / why we need it: This adds more detail around what can and cannot be used in a `selector` for retention. It was unclear whether arbitrary LogQL could be used or not; this makes it explicit. Also added some additional matchers to the examples to show that it can be combinatorial. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Joel Takvorian	688a392db2	operator: Allow multiple matchers for multi-tenancy with Network tenant (#8192 )	2 years ago
Mohamed-Amine Bouqsimi	f125682802	operator: Promote AlertingRules, RecordingRules and RulerConfig from v1beta1 to v1 (#8800 )	2 years ago
Travis Patterson	83ba93c867	fix to-string for noop label filters (#8799 ) Label Filter optimizations omit `.*` label filters from the execution pipeline but intruduced a bug where those stages were also ommitted from calls to `String` on the expression. This results in unparsable queries and panics when queries are cloned via `query.String()` This PR makes `noopLabelFilter` embed matchers and inherit their `String` method just like the `StringLabelFilters` do	2 years ago
Christian Haudum	22b672986a	Revert "Implement hierarchical queues for query scheduler" (#8796 ) Reverts grafana/loki#8691 These changes caused problems with the scheduler in our dev environments. I will conduct further testing and include the reverted changes in https://github.com/grafana/loki/pull/8752	2 years ago
Danny Kopping	61311957a3	Integration Tests: allow loki logs to be optionally output in integration tests (#8793 ) What this PR does / why we need it: Debugging integration tests can be difficult without Loki's logs being output. Changing the call from `cluster.New()` to `cluster.New(true, level.WarnValue())` now allows the author the option to temporarily or permanently enable logging for their test, as well as configure a level filter to control the noise. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Oren Cohen	2a1bd3da00	operator: Improve documentation for LokiStack installation with ODF Object Storage (#8792 ) Signed-off-by: Oren Cohen <ocohen@redhat.com> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	2 years ago
Periklis Tsirakidis	254fbf72c4	operator: Expand OLM skip range for OpenShift Logging 5.7 release (#8791 )	2 years ago
Trevor Whitney	b46e392adb	gofmt is needed for the build (#8678 ) What this PR does / why we need it: `gofmt` is needed for the build, `make format` fails without it.	2 years ago
Periklis Tsirakidis	4854e4af89	operator: Update Loki operand to v2.7.4 (#8771 )	2 years ago
Gerard Vanloo	a926ae3249	operator: Update golang to 1.20; Update dependencies to latest (#8776 )	2 years ago
Danny Kopping	33e44ed39d	Ruler: remote rule evaluation (#8744 ) What this PR does / why we need it: Adds the ability to evaluate recording & alerting rules against a given `query-frontend`, allowing these queries to be executed with all the parallelisation & optimisation that regular adhoc queries have. This is important because with `local` evaluation all queries are single-threaded, and rules that evaluate a large range/volume of data may timeout or OOM the `ruler` itself, leading to missed metrics or alerts. When `remote` evaluation mode is enabled, the `ruler` effectively just becomes a gRPC client for the `query-frontend`, which will dramatically improve the reliability of the `ruler` and also drastically reduce its resource requirements. Which issue(s) this PR fixes: This PR implements the feature discussed in https://github.com/grafana/loki/pull/8129 (LID 0002: Remote Rule Evaluation).	2 years ago
Paul Tirk	540380fa94	Promtail: fix GeoIP typo (#8783 ) What this PR does / why we need it: Fixes a typo for GeoIP stage. From `geoip_continet_name` to `geoip_continent_name`.	2 years ago
Karsten Jeschkies	28ebf8ec12	Impose per-request limit on label queries. (#8777 ) What this PR does / why we need it: #8727 introduced per-request limits. These should be enforce for label queries as well. This change adds the required middleware to all label query endpoints. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Ed Welch	0352b75f0d	Loki: Add new logql template functions `bytes`, `duration`, `unixEpochMillis`, `unixEpochNanos`, `toDateInZone`, `b64Enc`, and `b64Dec` (#8774 ) What this PR does / why we need it: This adds several new template functions to the logql to be used in `line_format` and `label_format` - bytes - duration (and duration_seconds) - unixEpochMillis - unixEpochNanos - toDateInZone - b64Enc and b64Dec `bytes` and `duration` adds parity with the `unwrap` pipeline stage in logql, allowing you to convert human readable versions of bytes and times into their respective primitives of bytes and seconds. This makes it possible to do math and comparisons of these extracted values in `line_format` and `label_format` `unixEpochMillis` and `unixEpochNanos` extends the existing `unixEpoch` function which can only generate seconds level precision to allow having millisecond and nanosecond level precision. `toDateInZone` allows for specifying a specific timezone when parsing a date. `b64Enc` and `b64Dec` allow base64 encoding and decoding values. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: @JStickler I made a lot of changes to the template_functions.md doc page to try and create some more consistency as well as remove some notices that really aren't relevant anymore: * Tried to make sure every function had a `Signature` and `Examples` block * Removed a lot of `NOTE` about functions being introduced in a specific version * Removed documentation for some deprecated string functions (they still exist in code but I removed the docs for them, their exist better replacements for all of them) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	55c96bfe90	Loki: add lines processed to metrics.go (#8764 ) What this PR does / why we need it: Typically `throughput` and `total_bytes` are the most common fields in the `metrics.go` log line which we look at for query performance. However, there are cases where query performance is better measured by `total_lines` and `lines_per_second` (e.g. when processing a lot of high volume but relatively short log lines) and having this information in metrics.go can be very helpful. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	a4eb536fb2	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761 ) What this PR does / why we need it: Currently the `metrics.go` log line emitted after every query includes a metric called "subqueries". This currently tracks the number of queries created by the split_by_time operations done in Loki, but does not include any counts for subqueries created as a result of sharding. It becomes difficult to make a single subqueries counter that gives useful information to determine how much a query is split by time and sharded by a shard factor, especially now that sharding in TSDB indexes is dynamic. This PR removes and deprecates the `subqueries` stat and instead creates a `splits` and `shards` statistic which records how much a query was split_by_time and the total number of shards created as well. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Karsten Jeschkies	8eac851962	Remove print statements introduced by #8727 . (#8775 )	2 years ago
Ed Welch	eadb9d2c00	Loki: Change the behavior of the 0 value for retention_period to disable retention. (#8753 ) What this PR does / why we need it: Issue #8359 exposed a very unfortunate inconsistency in a Loki config where the zero value did not disable a feature. By convention in Loki the zero value for a config typically "disables" it. This PR changes the retention_period to honor the zero value of `0` or `0s` to disable retention. Which issue(s) this PR fixes: Fixes #8359 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Danny Kopping	e2ac2d50e2	Changes from linter (#8769 ) What this PR does / why we need it: We had a few outstanding linter issues, and running `make lint` fixes them all automatically. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Danny Kopping	56fbb3b771	Expand `which-release.sh` to cover weekly builds as well (#8768 ) What this PR does / why we need it: Expanding tool introduced in https://github.com/grafana/loki/pull/8706 Special notes for your reviewer: Knowing which, if any, weekly builds the commit is in can also be useful. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727 ) What this PR does / why we need it: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. Which issue(s) this PR fixes: Fixes #8762 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Michel Hollands	10e3d80731	Fix distributor latency panel (#8760 ) What this PR does / why we need it: https://github.com/grafana/loki/pull/8247 changes a metric generated by a recording rule. This fixes the "Distributor Panel" to use this different metric. Tested by changing the "Distributor Panel" manually to see it showed results. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Danny Kopping	0bf0b6b718	Modify debug loki build to work on any architecture, including ARM (#8758 ) What this PR does / why we need it: On my M1 Pro, `make loki-debug-image` produces the following output: ```bash qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory ``` This is because we run `delve` as the entrypoint for the debug image, and `delve` is being copied from `loki-build-image` which is exclusively built as `amd64` (we should make this a multi-arch image... created https://github.com/grafana/loki/issues/8759). This change will work with whichever host OS is building the image and download `delve` using that architecture instead of relying on the `loki-build-image` version.	2 years ago
Michel Hollands	15fda6bf39	Change severity of PromtailFileMissing alert to warning (#8757 ) The PromtailFileMissing alert is not critical. This changes it to a warning. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Michel Hollands	1c4dd6ad40	Use changed metric in operational dashboard (#8749 ) What this PR does / why we need it: The metric `job_route` was changed to `cluster_job_route` in https://github.com/grafana/loki/pull/8247. This PR updates this metric in the operational dashboard. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Travis Patterson	9ac33e216e	Reimplement Unpack in terms of jsonparser rather than jsoniter (#8739 ) This PR reimplements the functionality of `Unpack` using the `jsonparser` library rather than `jsoniter` Benchmarks: ``` name old time/op new time/op delta _Parser/unpack/no_labels_hints-8 900ns ± 4% 309ns ± 3% -65.62% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 1.06µs ±12% 0.32µs ± 2% -69.72% (p=0.000 n=9+10) _Parser/unpack-not_json_line/no_labels_hints-8 37.5ns ± 8% 10.6ns ± 3% -71.70% (p=0.000 n=9+9) _Parser/unpack-not_json_line/labels_hints-8 39.6ns ±11% 10.3ns ± 2% -74.00% (p=0.000 n=10+9) name old alloc/op new alloc/op delta _Parser/unpack/no_labels_hints-8 536B ± 0% 80B ± 0% -85.07% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 536B ± 0% 80B ± 0% -85.07% (p=0.000 n=10+10) _Parser/unpack-not_json_line/no_labels_hints-8 0.00B 0.00B ~ (all equal) _Parser/unpack-not_json_line/labels_hints-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta _Parser/unpack/no_labels_hints-8 17.0 ± 0% 4.0 ± 0% -76.47% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 17.0 ± 0% 4.0 ± 0% -76.47% (p=0.000 n=10+10) _Parser/unpack-not_json_line/no_labels_hints-8 0.00 0.00 ~ (all equal) _Parser/unpack-not_json_line/labels_hints-8 0.00 0.00 ~ (all equal) ```	2 years ago
Travis Patterson	9c6e509a6a	Reimplement JsonExpressionParser in terms of jsonparser (#8734 ) This PR reimplements the functionality of `JSONExpressionParser` using the `jsonparser` library rather than `jsoniter` Benchmarks: ``` benchstat json_expression_old.txt json_expression_new.txt name old time/op new time/op delta JsonExpressionParser/json-expression-8 4.15µs ± 4% 0.82µs ± 3% -80.17% (p=0.000 n=10+10) name old alloc/op new alloc/op delta JsonExpressionParser/json-expression-8 1.07kB ± 0% 0.02kB ± 0% -97.76% (p=0.000 n=10+10) name old allocs/op new allocs/op delta JsonExpressionParser/json-expression-8 72.0 ± 0% 2.0 ± 0% -97.22% (p=0.000 n=10+10) ```	2 years ago
Christian Haudum	4b74a5b815	Implement hierarchical queues for query scheduler (#8691 ) ### What this PR does / why we need it: This PR is the first step towards hierarchical queues within the query scheduler. Hierarchical queues can be used to ensure query fairness across multiple actors within a single tenant, such as individual (human) users using the same tenant. The change is described in the LID ["Query fairness across users within tenants"](https://grafana.com/docs/loki/next/lids/0003-queryfairnessinscheduler/). * Extract metrics for `RequestQueue` into their own `metrics` field, so that they can extended and passed more easily in the future. * Move mapping for tenant queues into its own data structure (ordered map) so its functionality can be reused. It also simplifies the code inside the `tenantQueues` struct. * Add the `LeafQueue` struct that implements hierarchical queues. Sub-queues are dequeued in a round-robin manner using the same implementation of `Mapping` that is also used by the tenant queues. ### Special notes for your reviewer Part of https://github.com/grafana/loki/pull/8585 Contains the changes from PR https://github.com/grafana/loki/pull/8722 This PR does not yet make use of the new hierarchical queues, it only adds the relevant data structures. Wiring up will be done in a follow up PR. While the new `Mapping` data structure adds an unsignificant overhead to the time/op I consider the simplification an overall better tradeoff. ```console $ benchstat scheduler-bench-old.txt scheduler-bench-new.txt name old time/op new time/op delta GetNextRequest-8 18.0µs ± 2% 20.5µs ± 3% +13.89% (p=0.008 n=5+5) QueueRequest-8 56.6µs ± 0% 53.5µs ± 3% -5.48% (p=0.008 n=5+5) name old alloc/op new alloc/op delta GetNextRequest-8 1.60kB ± 0% 1.60kB ± 0% ~ (all equal) QueueRequest-8 35.6kB ± 0% 29.8kB ± 0% -16.37% (p=0.008 n=5+5) name old allocs/op new allocs/op delta GetNextRequest-8 100 ± 0% 100 ± 0% ~ (all equal) QueueRequest-8 819 ± 0% 806 ± 0% -1.59% (p=0.008 n=5+5) ``` #### Complexity of operations \| Operation \| Function \| Complexity \| \| --------- \| -------- \| ---------- \| \| Insert \| `Put(key string, value tenantQueue) bool` \| `O(1)`\| \| Select \| `GetNext(idx QueueIndex) tenantQueue` \| `O(1)` (best case) `O(n)` (worst case) \| \| Remove \| `Remove(key string) bool` \| `O(1)` \| --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

1 2 3 4 5 ...

4766 Commits (f5f17538517ce1b8e93eab6cc06cd6d8171857a3) All Branches Search

4766 Commits (f5f17538517ce1b8e93eab6cc06cd6d8171857a3)

All Branches