* [3627](https://github.com/grafana/loki/pull/3627) **MichelHollands**: Update vendored Cortex to 2d8477c4a325
* [3532](https://github.com/grafana/loki/pull/3532) **MichelHollands**: Update vendored Cortex to 8a2e2c1eeb65
* [3446](https://github.com/grafana/loki/pull/3446) **pracucci, owen-d**: Remove deprecated config `querier.split-queries-by-day` in favor of `querier.split-queries-by-interval`
* [3586](https://github.com/grafana/loki/pull/3587) **rsteneteg** Remove filemanager targets that no longer has any tails, to allow them to be recreated with the proper path
* [3739](https://github.com/grafana/loki/pull/3739) **rsteneteg** Allow users to run instant queries in the past using LogCLI
* [4020](https://github.com/grafana/loki/pull/4020) **simonswine** Prevent path traversal attack from users able to control the HTTP header `X-Scope-OrgID`. (Users only have control of the HTTP header when Loki is not frontend by an auth proxy validating the tenant IDs) (CVE-2021-36156 CVE-2021-36157)
# 2.3.0 (2021/08/06)
Release notes for 2.3.0 can be found on the [release notes page](https://grafana.com/docs/loki/latest/release-notes/v2-3/)
* [3990](https://github.com/grafana/loki/pull/3990) **cyriltovena**: Intern label keys for LogQL parser.
* [3986](https://github.com/grafana/loki/pull/3986) **kavirajk**: Ip matcher for LogQL
* [3984](https://github.com/grafana/loki/pull/3984) **jeschkies**: Filter instant queries and shard them.
* [3983](https://github.com/grafana/loki/pull/3983) **cyriltovena**: Reject labels with invalid runes when using implicit extraction parser.
* [3981](https://github.com/grafana/loki/pull/3981) **owen-d**: fixes chunk size method in facade
* [3979](https://github.com/grafana/loki/pull/3979) **MichelHollands**: Add a chunk filterer field to the config
* [3977](https://github.com/grafana/loki/pull/3977) **sandeepsukhani**: add a metric for counting number of failures in opening existing active index files
* [3973](https://github.com/grafana/loki/pull/3973) **56quarters**: Use the Cortex wrapper for getting tenant ID from a context
* [3972](https://github.com/grafana/loki/pull/3972) **jeschkies**: Return build info under `/loki/api/v1/status/buildinfo`.
* [3970](https://github.com/grafana/loki/pull/3970) **sandeepsukhani**: log name of the file failed to open during startup by ingester
* [3969](https://github.com/grafana/loki/pull/3969) **sandeepsukhani**: add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests
* [3968](https://github.com/grafana/loki/pull/3968) **cyriltovena**: Improve head chunk allocations when reading samples.
* [3967](https://github.com/grafana/loki/pull/3967) **sandeepsukhani**: fix a panic in compactor when retention is not enabled
* [3966](https://github.com/grafana/loki/pull/3966) **sandeepsukhani**: fix panic in compactor when retention is not enabled
* [3957](https://github.com/grafana/loki/pull/3957) **owen-d**: Unordered head block
* [3949](https://github.com/grafana/loki/pull/3949) **cyriltovena**: Allow no overrides config for tenants.
* [3946](https://github.com/grafana/loki/pull/3946) **cyriltovena**: Improve marker file current time metrics.
* [3807](https://github.com/grafana/loki/pull/3807) **dannykopping**: Loki: allow for multiple targets
* [3797](https://github.com/grafana/loki/pull/3797) **dannykopping**: Exposing remote writer for use in integration tests
* [3792](https://github.com/grafana/loki/pull/3792) **MichelHollands**: Add a QueryFrontendTripperware module
* [3785](https://github.com/grafana/loki/pull/3785) **sandeepsukhani**: just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled
* [3772](https://github.com/grafana/loki/pull/3772) **sandeepsukhani**: initialize retention and deletion components only when they are enabled
* [3771](https://github.com/grafana/loki/pull/3771) **sandeepsukhani**: revendor cortex to latest master
* [3769](https://github.com/grafana/loki/pull/3769) **sandeepsukhani**: reduce allocs in delete requests manager by reusing slice for tracing non-deleted intervals for chunks
* [3618](https://github.com/grafana/loki/pull/3618) **MichelHollands**: Add interceptor override and make ingester and cfg public
* [3605](https://github.com/grafana/loki/pull/3605) **sandeepsukhani**: cleanup boltdb files failing to open during loading tables which are possibly corrupt
* [3603](https://github.com/grafana/loki/pull/3603) **cyriltovena**: Adds chunk filter hook for ingesters.
* [3602](https://github.com/grafana/loki/pull/3602) **MichelHollands**: Loli: Make the store field public
* [3595](https://github.com/grafana/loki/pull/3595) **owen-d**: locks trailers during iteration
* [3720](https://github.com/grafana/loki/pull/3720) **fredrikekre**: [docs/clients] fix header for "Unofficial clients" and add a reference to said section.
docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:2.3.0 -config.file=/mnt/config/promtail-config.yaml
```
When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files.
@ -36,10 +36,10 @@ Copy and paste the commands below into your terminal. Note that you will need to
docker run -v <local-path>:/mnt/config -v /var/log:/var/log grafana/promtail:2.3.0 --config.file=/mnt/config/promtail-config.yaml
```
When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files.
@ -51,6 +51,6 @@ Navigate to http://localhost:3100/metrics to view the output.
Run the following commands in your command line. They work for Windows or Linux systems.
The Loki team is excited to announce the release of Loki 2.3!
It's been nearly 6 months since 2.2 was released and we have made good use of that time to bring forward several significant improvements and requested features.
2.3 is also the first version of Loki released under the AGPLv3 license. You can [read more about our licensing here](https://grafana.com/licensing/).
Some parts of the Loki repo will remain Apache-2.0 licensed (mainly clients and some tooling), for more details please read [LICENSING.md](https://github.com/grafana/loki/blob/main/LICENSING.md).
## Features and enhancements
List of features here.
* Loki now has the ability to apply [custom retention](../../operations/storage/retention/) based on stream selectors! This will allow much finer control over log retention all of which is now handled by Loki, no longer requiring the use of object store configs for retention.
* Coming along hand in hand with storing logs for longer durations is the ability to [delete log streams](../../operations/storage/logs-deletion/). The initial implementation lets you submit delete request jobs which will be processed after 24 hours.
* A very exciting new LogQL parser has been introduced: the [pattern parser](../../logql/#parser-expression). Much simpler and faster than regexp for log lines that have a little bit of structure to them such as the [Common Log Format](https://en.wikipedia.org/wiki/Common_Log_Format). This is now Loki's fastest parser so try it out on any of your log lines!
* Extending on the work of Alerting Rules, Loki now accepts [recording rules](../../rules/#recording-rules). This lets you turn your logs into metrics and push them to Prometheus or any Prometheus compatible remote_write endpoint.
* LogQL can understand [IP addresses](../../logql/ip/)! This enables filtering on IP addresses and subnet ranges.
For those of you running Loki as microservices, the following features will improve performance operations significantly for many operations.
* We created an [index gateway](../../operations/storage/boltdb-shipper/#index-gateway) which takes on the task of downloading the boltdb-shipper index files allowing you to run your queriers without any local disk requirements, this is really helpful in Kubernetes environments where you can return your queriers from Statefulsets back to Deployments and save a lot of PVC costs and operational headaches.
* Ingester queriers [are now shardable](https://github.com/grafana/loki/pull/3852), this is a significant performance boost for high volume log streams when querying recent data.
* Instant queries can now be [split and sharded](https://github.com/grafana/loki/pull/3984) making them just as fast as range queries.
A very common feature requested has also been included in 2.3:
Without revisiting the decisions and discussions around the somewhat controversial behavior of _unhealthy_ ingesters, you can now decided how you would like them to be handled: manually or automatically.
Lastly several useful additions to the LogQL query language have been included:
* More text/template functions are included for `label_format` and `line_format` with PR [3515](https://github.com/grafana/loki/pull/3515), please check out the [documentation for template functions](https://grafana.com/docs/loki/latest/logql/template_functions/).
* Also support for math functions withing `label_format` and `line_format` was included with [3434](https://github.com/grafana/loki/pull/3434).
* Two additional metric functions with some interesting use cases `first_over_time` and `last_over_time` were added in PR [3050](https://github.com/grafana/loki/pull/3050). These can be useful for some down sampling approaches where instead of taking an average, max, or min of samples over a range in a metrics query, you can select the first or last log line to use from that range.
## Upgrade considerations
The path from 2.2.1 to 2.3.0 should be smooth, as always, please check out the [Upgrade Guide](https://github.com/grafana/loki/blob/master/docs/sources/upgrading/_index.md#230) for important upgrade guidance.
One change we consider noteworthy however is:
* [3216](https://github.com/grafana/loki/pull/3216) **sandeepsukhani**: check for stream selectors to have at least one equality matcher.
This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:
`{namespace=~".*"}`
This query will now be rejected, however there are several ways to modify it for it to succeed:
Add at least one equals label matcher:
`{cluster="us-east-1",namespace=~".*"}`
Use `.+` instead of `.*`
`{namespace=~".+"}`
This difference may seem subtle but if we break it down `.` matches any character, `*` matches zero or more of the preceding character and `+` matches one or more of the preceding character. The `.*` case will match empty values where `.+` will not, this is the important difference. `{namespace=""}` is an invalid request (unless you add another equals label matcher like the example above).
The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.
## Security fixes
List of security fixes for 2.3.x.
### 2.3.0 security fixes
2.3.0 contains an important security fix:
* [4020](https://github.com/grafana/loki/pull/4020) **simonswine**: Restrict path segments in TenantIDs (CVE-2021-36156 CVE-2021-36157).
**Note** Exploitation of this vulnerability requires the ability for an attacker to craft and send directly to Loki an `X-Scope-OrgID` header, end users should not have the ability to create and send this header directly to Loki as it controls access to tenants and is important to control setting of this header for proper tenant isolation and security. We always recommend having a proxy or gateway be responsible for setting the `X-Scope-OrgID`.
- Pattern parser. [poor docs link to Parser expression](../../logql/#parser-expression)
- Windows event log scraping. [docs link](../../clients/promtail/scraping/#windows-event-log)
- Recording rules.
- Index Gateway.
- Retention.
- `promtail_instance` label.
## Bug fixes
Lists of bug fixes for v2.3.x.
Lists of bug fixes for 2.3.x.
### 2.3.0 bug fixes
### V2.3.0 bug fixes
* An important fix for leaking resources was patched with [3733](https://github.com/grafana/loki/pull/3733), when queries were canceled a goroutine was left running which would hold memory resources creating a memory leak.
* [3686](https://github.com/grafana/loki/pull/3686) fixes a panic with the frontend when use with downstream URL. **Note** we recommend using the [GRPC Pull Model](https://grafana.com/docs/loki/latest/configuration/query-frontend/#grpc-mode-pull-model), better performance and fair scheduling between tenants can be obtained with the GPRC Pull Model.
@ -19,6 +19,33 @@ If possible try to stay current and do sequential updates. If you want to skip v
-_add changes here which are unreleased_
## 2.3.0
### Loki
#### Query restriction introduced for queries which do not have at least one equality matcher
PR [3216](https://github.com/grafana/loki/pull/3216) **sandeepsukhani**: check for stream selectors to have at least one equality matcher
This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:
`{namespace=~".*"}`
This query will now be rejected, however there are several ways to modify it for it to succeed:
Add at least one equals label matcher:
`{cluster="us-east-1",namespace=~".*"}`
Use `.+` instead of `.*`
`{namespace=~".+"}`
This difference may seem subtle but if we break it down `.` matches any character, `*` matches zero or more of the preceding character and `+` matches one or more of the preceding character. The `.*` case will match empty values where `.+` will not, this is the important difference. `{namespace=""}` is an invalid request (unless you add another equals label matcher like the example above)
The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.
Ed Welch
4 years ago
committed by