411 Commits (1b410dbc01fddf2879ee7242de28844184793b0c)

Author	SHA1	Message	Date
Gregor Zeitlinger	b1917a6178	add "alignLeft" and "alignRight" functions (#9672) ... Fixes https://github.com/grafana/loki/issues/9667	2 years ago
Ashwanth	98d1307c7d	config: ensure storage config defaults apply to named stores (#9650) ... **What this PR does / why we need it**: Since named store config does not register any flags, storage configs defined under it do not get the defaults. For example [aws_storage_config](https://grafana.com/docs/loki/latest/configuration/#aws_storage_config) sets the default `storage_class` to `STANDARD`, but the same doesn't get applied by default when using named stores. This PR ensures that named storage configs are always assigned default values when they are unmarshalled by implementing `yaml.Unmarshaler` interface	2 years ago
Salva Corts	1694ad0f9b	Stats cache can be configured independently (#9535) ... **What this PR does / why we need it**: Before this PR, the index stats cache would use the same config as the query results cache. This was a limitation since: 1. We would not be able to point to a different cache for storing the index stats if needed. 2. We would not be able to add specific settings for this cache, without adding it to the results cache. In this PR, we refactor the index stats cache config to be independently configurable. Note that if it's not configured, it will try to use the results cache settings. **Which issue(s) this PR fixes**: This is needed for: - https://github.com/grafana/loki/pull/9537 - https://github.com/grafana/loki/pull/9536 **Special notes for your reviewer**: - This PR also refactors all the tripperwares in rountrip.go to reuse the same stats tripperware instead of each one creating their own. - Configuring a new cache in rountrip.go is a requirement for https://github.com/grafana/loki/pull/9536 so the stats summary can distinguish before the stats cache and the results cache. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Vladyslav Diachenko	3c52034d63	[logcli] fixed query command that is run against user's bucket with TSDB index enabled (#9597) ... **What this PR does / why we need it**: set TSDB shipper mode to ReadOnly and disabled indexGatewayClient during local query run. Also, I increase index downloading timeout to 1m because the users run the query from local machines that not always have more than 100Mbps **Which issue(s) this PR fixes**: Fixes #9555 **Special notes for your reviewer**: Unfortunately, we do not have similar tests and test architecture to cover it with unit tests properly. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Sandeep Sukhani	0a5e149ea5	query-scheduler: fix query distribution in SSD mode (#9471) ... **What this PR does / why we need it**: When we run the `query-scheduler` in `ring` mode, `queriers` and `query-frontend` discover the available `query-scheduler` instances using the ring. However, we have a problem when `query-schedulers` are not running in the same process as queriers and query-frontend since [we try to get the ring client interface from the scheduler instance](abd6131bba/pkg/loki/modules.go (L358)). This causes queries not to be spread across all the available queriers when running in SSD mode because [we point querier workers to query frontend when there is no ring client and scheduler address configured](b05f4fced3/pkg/querier/worker_service.go (L115)). I have fixed this issue by adding a new hidden target to initialize the ring client in `reader`/`member` mode based on which service is initializing it. `reader` mode will be used by `queriers` and `query-frontend` for discovering `query-scheduler` instances from the ring. `member` mode will be used by `query-schedulers` for registering themselves in the ring. I have also made a couple of changes not directly related to the issue but it fixes some problems: * [reset metric registry for each integration test](18c4fe5907) - Previously we were reusing the same registry for all the tests and just [ignored the attempts to register same metrics](01f0ded7fc/integration/cluster/cluster.go (L113)). This causes the registry to have metrics registered only from the first test so any updates from subsequent tests won't reflect in the metrics. metrics was the only reliable way for me to verify that `query-schedulers` were connected to `queriers` and `query-frontend` when running in ring mode in the integration test that I added to test my changes. This should also help with other tests where earlier it was hard to reliably check the metrics. * [load config from cli as well before applying dynamic config](f9e2448fc7) - Previously we were applying dynamic config considering just the config from config file. This results in unexpected config changes, for example, [this config change](4148dd2c51/integration/loki_micro_services_test.go (L66)) was getting ignored and [dynamic config tuning was unexpectedly turning on ring mode](52cd0a39b8/pkg/loki/config_wrapper.go (L94)) in the config. It is better to do any config tuning based on both file and cli args configs. **Which issue(s) this PR fixes**: Fixes #9195	2 years ago
Periklis Tsirakidis	69392de8b6	Fix duplicate label values from ingester streams (#9629)	2 years ago
Danny Kopping	a248e3e970	Querier: configurable writeback queue bytes size (#9604) ... **What this PR does / why we need it**: The background writeback process is used to write chunks that have been fetched from the store to cache. Chunks have a target size, but not all chunks will necessarily be this size, so it is tricky to find the correct value for `writeback_buffer` which specifies how many chunks will be held in the queue at any given time. For example: setting `writeback_buffer=10000` (the default) with all chunks being the default target size (`chunk_target_size=1.5MiB`), in the worst case could result in 15GiB of memory being used for this buffer. Practically speaking, the queue never grows so deep, but if the cache Loki is writing to becomes overwhelmed this could happen. Setting a value too low would result in many chunks not being written back to cache, which defeats the purpose. This PR introduces a configurable _byte size_ limit to the writeback cache, which defaults to 1GB in size. With this setting `writeback_size_limit` in place, we get the best of both worlds: a deep queue which can hold many small chunks or a shallow queue which can hold a few large ones without using too much memory so as to overwhelm the querier. If either the `writeback_buffer` or `writeback_size_limit` reaches capacity, any new chunks which are attempted to be added to the queue will be dropped. Two new metrics are added by this PR: - `loki_cache_background_queue_bytes`: gauge, measures the size of the queue in bytes - `loki_cache_dropped_background_writes_bytes_total`: counter, measures the volume of bytes dropped	2 years ago
Christophe Collot	b709b32d6a	feat(lambda-promtail): add cloudtrail log ingestion support (#9497) ... **What this PR does / why we need it**: ### Add support for AWS CloudTrail audit logs ingestion using lambda-promtail Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for security and debugging purposes. However, I've experienced difficulties with it: + The AWS CloudTrail service is not well integrated with Prometheus (no metrics, no alerts) and I don't want to manage alerts in CloudWatch Alerts + The search experience is painful with CloudTrail via the AWS Console (I will not elaborate 😅). This PR allows ingesting CloudTrail audit logs sent to an S3 bucket using the same approach as VPC flow logs or Load Balancer logs. **Special notes for your reviewer**: + Because the Cloudtrail file format is not text but json, we stream the json CloudTrail records instead of using the already existing scanner. + Because the Cloudtrail filename format is not the same as for the Flow log or the Load balancer log files, we need to split the regexes by service (although many AWS services seem to share the same `defaultFilenameRegex`). + In the `getLabels` function, we expect the `type` parameter to be found in the filename using the Regex. For some log files (ex: Cloudfront log files). The file name has no reference to the service name. This is why, as a default, when no type is found in the name of the file, I set it to use the key of the matching Regex expression. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
oleksii-boiko-ua	f9aba99455	Helm: add hpa for read, write and backend (#8684) ... **What this PR does / why we need it**: It adds missing HPA (HorizontalPodAutoscaler) templates to write, read, and backend components **Which issue(s) this PR fixes**: Fixes #7864 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Oleksii Boiko <oleksii_boiko@external.mckinsey.com>	2 years ago
indX	a55404766c	Logging: Add HTTP API to change the log level at runtime (#9357) ... **What this PR does / why we need it**: To be able to change the log level at runtime, particularly from info to debug to enable debugging information during an incident without restarting Loki. **Which issue(s) this PR fixes**: Fixes #6805 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`: No such changes --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Michel Hollands	478510a1bb	Fix Grouping String function and usage so it prints "without ()" (#9515) ... **What this PR does / why we need it**: The changes rangemapper made to LogQL expressions created invalid LogQL. It added a `without` clause instead of `without ()`. This PR changes the String function of the Grouping struct. This is then used in the rangemapper. This caused the following panic: ``` panic: error cloning query: count without (rate({namespace="apps"}[15s])): parse error at line 1, col 16: syntax error: unexpected RATE, expecting IDENTIFIER or ) ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Piotr	ac4a442291	Promtail: Fix goroutine leak in file tailer (#9495) ... **What this PR does / why we need it**: There's a goroutine leak in promtail file tailer, which has been encountered in https://github.com/grafana/agent/issues/3885. The issue is the following: - There are two goroutines that tailer manages: one for reading lines, the other for updating positions: https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/file/tailer.go#L103 - The tailer can be cleanly stopped & drained via `.Stop()` https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/file/tailer.go#L221 - But if the goroutine reading lines encounters an error from the underlying `tail`, the tailer is stopped without draining using the defer statement here: https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/file/tailer.go#L151 - The above defer statement does not shut down the position updating goroutine and the `.Stop()` won't be called, so we end up with a leaked goroutine. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/agent/issues/3885 As can be seen in the [goroutine dump attached to the issue](https://gist.github.com/amseager/e37f31984facd9cbc51ce0f2153d6864), there were 933 update position goroutines, while we only had 21 files being tailed. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Tests updated - [x] `CHANGELOG.md` updated	2 years ago
Totalus	01f0ded7fc	Update swift_object_client.go : Fetch all objects in List() function (#9463) ... **What this PR does / why we need it**: Allow the Swift Client `List()` function to return all objects that match the query. **Which issue(s) this PR fixes**: Fixes #9360 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Danny Kopping	6141347b79	Adding more buckets to loki_memcache_request_duration_seconds histogram (#9431) ... **What this PR does / why we need it**: Users using extstore (with NVMe drives) may experience latencies higher than the ~250ms max as previously defined. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Jan Jansen	c4048a3adf	Promtail: Break on iterate journal failure (#9155)	2 years ago
Salva Corts	16a0e572de	Update docs 2.8.2 (#9379) ... **What this PR does / why we need it**: Update docs and changelog with Loki 2.8.2 changes. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Danny Kopping	0adf27df3e	Adding CHANGELOG entry missed in #9370 (#9376) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
李国忠	38b298c3fe	LogQL: Introduce `distinct` (#8662) ... **What this PR does / why we need it**: Introduces `distinct` to LogQL. Usage: `{job="varlogs"} | distinct filename` It is similar to `distinct` in SQL and especially useful for sampling. Similar syntax exists in other log query languages. **Example** For the following log lines: ```json {"event": "access", "id": "1", "time": "2023-02-28 15:12:11"} {"event": "access", "id": "1", "time": "2023-02-28 15:13:11"} {"event": "access", "id": "2", "time": "2023-02-28 15:14:11"} {"event": "access", "id": "2", "time": "2023-02-28 15:15:11"} ``` The query below: ```logql {app="order"} | json | distinct id ``` Will return: ```json {"event": "access", `"id": "1",` "time": "2023-02-28 15:13:11"} {"event": "access", `"id": "2", `"time": "2023-02-28 15:15:11"} ``` **Example with multiple labels** ```logql {app="order"} | json | distinct id,time ``` **Which issue(s) this PR fixes**: Fixes #8649 --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Trevor Whitney	50bc1cee71	Rename usagestats to analytics (#9212) ... **What this PR does / why we need it**: Rename usagestats to analytics. The config section in the yaml is already called analytics, this makes things more consistent. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Jeff Cantrill	3d7060bf73	fluent-plugin-grapha-loki: Add config to support tls: ciphers, min_version (#9289)	2 years ago
Trevor Whitney	cfd17090fc	Finalize 2.8.1 changelog and release notes (#9272) ... **What this PR does / why we need it**: Brings over the correct 2.8.1 changelog and release notes from `release-2.8.x`. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	a11921bbb8	Upgrade alpine 3.16.4 -> 3.16.5 (#9264) ... **What this PR does / why we need it**: Upgrade alpine version to address [openssl CVEs](https://www.alpinelinux.org/posts/Alpine-3.17.3-released.html) **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Karsten Jeschkies	7109c56bdc	Use un-escaped regexp literal for string matching. (#9252) ... **What this PR does / why we need it**: The Regexp simplification would use the original string value instead of the escaped literal. E.g. ``` | logfmt | level =~ `foo\.bar` ``` would match the label `level` against `foo\.bar` and not `foo.bar`. That is why the un-escaped literalt should be used in the equal and not-equal matchers. **Which issue(s) this PR fixes**: Fixes #9111 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Dylan Guedes	24d426ce44	Distributor: Add AutoForget support (#8067) ... **What this PR does / why we need it**: - Add auto-forget capabilities to distributors - Change the distributor's ring implementation from Lifecycler to BasicLifecycler. That AutoForget delegate requires it.	2 years ago
Periklis Tsirakidis	3abd260975	Bump dskit from 528d8a7 to 1725bcc (#9184)	2 years ago
Danny Kopping	af896eb5f6	Ruler: log rule evaluation detail (#9206) ... **What this PR does / why we need it**: This logs some extra detail around rule evaluation to enable more effective troubleshooting when a rule is not behaving as expected.	2 years ago
irizzant	7f98b09148	Loki microservices tanka: add some ingester replicas (#9160) ... **What this PR does / why we need it**: This PR makes Ingester pods available in multi zone mode which, based on the previous defaults, were not created. **Which issue(s) this PR fixes**: Fixes #9139	2 years ago
Michel Hollands	abd6131bba	Update prepare shutdown (#9175) ... **What this PR does / why we need it**: This updates the PrepareShutdown method so it supports GET and DELETE methods as well. This makes it similar to Mimir: https://github.com/grafana/mimir/pull/4718. The status is now stored in a local file. A new config setting had to be added for this file as there is no obvious place to store it. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Danny Kopping	52655706b8	Prevent redis client from incorrectly choosing cluster mode with local address (#9185)	2 years ago
Salva Corts	422560b6b1	Flag to disable index stats cache (#9177) ... **What this PR does / why we need it**: At https://github.com/grafana/loki/pull/8972 we started caching all index stats requests. If the results cache gets overloaded, it can quickly take down the rest of the loki cell due to all the increased work. This PR adds a new flag so we can easily disable caching index stats requests. **Which issue(s) this PR fixes**: This PR is a follow up for https://github.com/grafana/loki/pull/8972 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
ndrpnt	afe8f91d0c	logcli: Allow negative filters when omitting stream selector with --stdin (#8566) ... **Which issue(s) this PR fixes**: Fixes #8565 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] `CHANGELOG.md` updated Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Salva Corts	fd16425062	Cache index stats requests (#8972) ... **What this PR does / why we need it**: As described in https://github.com/grafana/loki/issues/8973, we are substantially increasing the load of index stat requests we sent to our index gateways. Many of these requests should be easily re-used by caching them. This PR adds caching for index stat requests by reusing the results cache. Here's a demo ([source][1]):  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/8973 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: https://ops.grafana-ops.net/d/afcaef21-e5ad-49e7-ab06-42a9d7d915eb/index-stats?orgId=1&var-datasource=dev-cortex&var-cluster=dev-eu-west-2&var-namespace=loki-dev-009&var-loki_datasource=Grafana%20Logging&from=1680259907288&to=1680260431814&var-operation=All --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Ashwanth	4544f7b738	expiration: do not drop index if period is a zero value (#9156) ... **What this PR does / why we need it**: `DropFromIndex` should be return false when `retention_period` is set to a zero value. If we take chunks that overlap period boundary and are indexed in two tables, we drop these from the table on the left side of the period boundary even if they have not expired - https://github.com/grafana/loki/blob/ashwanth/drop-from-index-zero-value/pkg/storage/stores/indexshipper/compactor/retention/retention.go#L218 **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Tests updated - [x] `CHANGELOG.md` updated --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Salva Corts	5697bec427	Move misplaced recent changes in changelog to Loki Enhancements (#9137) ... **What this PR does / why we need it**: Some changelog entries were misplaced into _Promtail Enhancements_ instead of _Loki Enhancements_. This PR fixes this by moving the items to the correct section. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/pull/9042 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Salva Corts	8cf921a145	Pass engine opts down to middlewares (#9130) ... **What this PR does / why we need it**: The following middlewares in the query frontend uses a downstream engine: - `NewQuerySizeLimiterMiddleware` and `NewQuerierSizeLimiterMiddleware` - `NewQueryShardMiddleware` - `NewSplitByRangeMiddleware` These were all creating the downstream engine as follows: ```go logql.NewDownstreamEngine(logql.EngineOpts{LogExecutingQuery: false}, DownstreamHandler{next: next, limits: limits}, limits, logger), ``` As can be seen, the [engine options configured in Loki][1] were not being used at all. In the case of `NewQuerySizeLimiterMiddleware`, `NewQuerierSizeLimiterMiddleware` and `NewQueryShardMiddleware`, the downstream engine was created to get the `MaxLookBackPeriod`. When creating a new Downstream Engine as above, the `MaxLookBackPeriod` [would always be the default][2] (30 seconds). This PR fixes this by passing down the engine config to these middlewares, so this config is used to create the new downstream engines. **Which issue(s) this PR fixes**: Adresses some pending tasks from https://github.com/grafana/loki/pull/8670#issuecomment-1507031976. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: 1bcf683513/pkg/querier/querier.go (L52) [2]: edc6b0bff7/pkg/logql/engine.go (L136-L140)	2 years ago
Ashwanth	5cef03d0ea	index-shipper: add support for multiple stores (#7754) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: Currently loki initializes a single instance of index-shipper to [handle all the table ranges](ff7b462973/pkg/storage/factory.go (L188)) (from across periods) for a given index type `boltdb-shipper, tsdb`. Since index-shipper only has the object client handle to the store defined by `shared_store_type`, it limits the index uploads to a single store. Setting `shared_store_type` to a different store at a later point in time would mean losing access to the indexes stored in the previously configured store. With this PR, we initialize a separate index-shipper & table manager for each period if `shared_store_type` is not explicity configured. This offers the flexibility to store index in multiple stores (across providers). **Note**: - usage of `shared_store_type` in this commit text refers to one of these config options depending on the index in use: `-boltdb.shipper.shared-store`, `-tsdb.shipper.shared-store` - `shared_store_type` used to default to the `object_store` from the latest `period_config` if not explicitly configured. This PR removes these defaults in favor of supporting index uploads to multiple stores. **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: All the instances of downloads table manager operate on the same cacheDir. But it shouldn't be a problem as the tableRanges do not overlap across periods. **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Robert Fratto	7bec727c6d	promtail: support configurable polling interval (#8874) ... Currently, Promtail always polls files every 250ms. This PR allows the polling period to be customized, starting at a minimum polling period and exponentially increasing the polling frequency to a bounded maximum if the file has no changes. When file changes are detected, the polling frequency resets to the minimum. The default behavior is the existing behavior; both the minimum and maximum polling intervals are set to 250ms. I'm opening this in draft to get early feedback. There's a few open questions: * Where can I add validations to ensure that the min/max polling intervals are configured properly? * Is this the right approach for implementation? Do I need to make any changes? * Does this need tests? I'll update the CHANGELOG and Documentation after we're agreed on an approach.	2 years ago
Salva Corts	9783757932	Proportional TSDB index stats for chunks partially fitting in queried time range. (#9096) ... **What this PR does / why we need it**: When computing index stats for a query, we sum up the stats from all the chunks the query would read, even if some chunks will only be read partially. Moreover, if the query spans across two different TSDB tables, and there are chunks duplicated on both tables (at the end of the first one and at the beginning of the next one), the chunks' stats will be counted twice. This PR improves how we compute stats by taking care of partially matching chunks. Assuming entries and bytes are evenly distributed in the chunk, we will take the proportional number of entries and the number of bytes that fit on the span. For example, a chunk spans from t1 to t10 and has 10 entries and 10 KBs. A query from t3 through t7 would hit 50% of the chunk time span. Therefore, we estimate that the query would load 5 KBs and 5 entries from the chunk. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
3deep5me	c5453f156c	Helm: Removed support for PodDisruptionBudget in policy/v1alpha1 and upgraded it to policy/v1 (#8894) ... This PR removes the old policy/v1beta1 from pdb objects and removes with it the not needed helper function. This simplifies the helmchart and makes it more compatible with argocd and newer kubernetes versions. **Which issue(s) this PR fixes**: Fixes #8887 **Special notes for your reviewer**: First PR 😅	2 years ago
Salva Corts	fd5d78784f	Estimate chunk size on compaction looking at data uncompressed size (#9099) ... **What this PR does / why we need it**: At `compactedIndex.ToIndexFile` we were not computing the size of the chunk correctly. We were looking at the size of the chunk ref. Since the size of the chunk ref is always a bunk of bytes, the chunk meta KB field would always be 0. 4e893a0a88/pkg/storage/stores/tsdb/compactor.go (L367) Instead, we should get the approx KBs by looking at the chunks data uncompressed size. As we do at `store.IndexChunk`: d405162516/pkg/storage/stores/tsdb/store.go (L189)	2 years ago
Trevor Whitney	ea14d1348e	Fix CHANGELOG to accurately reflect what's in 2.8 (#9042) ... **What this PR does / why we need it**: A few items were incorrectly listed under 2.8 on `main` that are not in 2.8. This PR brings the `CHANGELOG` on `main` in line with the one already on `release-2.8.x` and therefore doesn't need to be backported.	2 years ago
Kaviraj Kanagaraj	cce3d3bed9	chore(format_query): Change response type to `application/json` (#9016)	2 years ago
Trevor Whitney	3f5c73747b	add 2.8 release date (#9015) ... **What this PR does / why we need it**: Fills out the date for the 2.8 release	2 years ago
Dylan Guedes	6dfcd59c1b	Promtail: Add `decompression` configuration (#8994) ... **What this PR does / why we need it**: Adds a new `compression` configuration to Promtail to customize decompression behavior. As of now, two options are supported: `enabled`: false by default but if truthy, Promtail will infer a compression algorithm to use based on the file extension `initialDelay`: 0 by default but if higher, Decompressor will sleep for the given duration before trying to decompress. This is to avoid scenarios where the Decompressor starts to work on a file that is still being compressed/generated. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/8784	2 years ago
Danny Kopping	6442ec1bdd	Mixins: check that `memcached_index_writes` exists before using it in `boltdb_shipper` (#8995) ... **What this PR does / why we need it**: This key is missing when not using boltdb_shipper, but this lib is included by default	2 years ago
Periklis Tsirakidis	2ae80d7166	Align common instance_addr with memberlist advertise_addr (#8959)	2 years ago
René Scheibe	46af2cdcf1	Promtail: Prevent logging errors on normal shutdown (#8988) ... **What this PR does / why we need it**: Since PR #4066 two messages are logged for the errors `ErrExpired` and `EOF`, whereas nothing has been logged beforehand. As `ErrExpired` is returned by the `JournalReader` after sending a message to the `until` channel to signal a normal stop, this should not be logged as an error.	2 years ago
Sandeep Sukhani	1fcefe8efa	index-gateway: fix failure in initializing index gateway when boltdb-shipper is not being used (#8755) ... **What this PR does / why we need it**: When only `tsdb` index is being used in the schema without configuring `boltdb-shipper` index store, we fail to initialize `index-gateway` since it also tries to build `boltdb-shipper` index client. This reference is used for serving `QueryIndex` calls which are for serving index from boltdb files. This PR fixes the issue by adding a check for the existence of `boltdb-shipper` in the schema config before initializing an index client. On the index gateway side, it adds a `failingIndexClient`, which errors when someone tries to query it. **Checklist** - [ ] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <masslessparticle@gmail.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Ed Welch	b892cade6a	Loki: Fixes incorrect query result when querying with start time == end time (#8979) ... **What this PR does / why we need it**: In several places within Loki we need to determine if a query is a `range query` or `instant query`, this is done by checking to see if the start and end time are equal **and** the `step=0` The downstream handler was not checking for `step=0` and thus it incorrectly mapped a range query to an instant query when a query has a start time equal to and end time. There are a few other things at play here, mainly that we should really error anytime someone tries to run an instant query for logs which would have exposed this error much more easily. But that's something I'd like to handle in a different PR as it will be considered a breaking change depending on how we do it. This PR uses an existing function we have for testing the query type and addresses the issue found in #8885 **Which issue(s) this PR fixes**: Fixes #8885 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Danny Kopping	de84737368	Fix fetched chunk from store size in metric (#8971) ... **What this PR does / why we need it**: The `Size()` function on `chunk.Chunk` calls out to `func (ChunkRef) Size() int` in the proto, which doesn't actually include the size of the chunk data! We need to call `chunk.Data.Size()` for that.	2 years ago