From 3fef08ca1aaa71d0387859ca632563f8403099b5 Mon Sep 17 00:00:00 2001 From: Ed Welch Date: Fri, 5 Nov 2021 16:21:29 -0400 Subject: [PATCH] 2.4 release notes, changelog, upgrade guide (#4658) * update changelog, release notes, and upgrade guide for 2.4 * tweak release notes a little * make path relative for kafka config * update upgrade guide * more docs fix the cache size not applied to results cache. * fix prefix name in ring_config --- CHANGELOG.md | 337 +++++++++++++++++-- docs/sources/api/_index.md | 13 - docs/sources/clients/promtail/scraping.md | 2 +- docs/sources/configuration/_index.md | 4 +- docs/sources/logql/ip.md | 2 - docs/sources/logql/log_queries.md | 2 - docs/sources/operations/storage/retention.md | 1 - docs/sources/release-notes/v2-4.md | 60 +++- docs/sources/upgrading/_index.md | 191 ++++++++--- pkg/loki/config_wrapper.go | 4 + 10 files changed, 514 insertions(+), 102 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 65cd3fb36e..852a277c3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,24 +1,301 @@ ## Main -* [4400](https://github.com/grafana/loki/pull/4400) **trevorwhitney**: Config: automatically apply memberlist config too all rings when provided -* [4435](https://github.com/grafana/loki/pull/4435) **trevorwhitney**: Change default values for two GRPC settings so querier can connect to frontend/scheduler -* [4443](https://github.com/grafana/loki/pull/4443) **DylanGuedes**: Loki: Change how push API checks for contentType -* [4415](https://github.com/grafana/loki/pull/4415) **DylanGuedes**: Change default limits to common values -* [4473](https://github.com/grafana/loki/pull/4473) **trevorwhitney**: Config: add object storage configuration to common config -* [4425](https://github.com/grafana/loki/pull/4425) **trevorwhitney** and **slim-bean**: Add a ring for the query scheduler -* [4519](https://github.com/grafana/loki/pull/4519) **DylanGuedes** and **replay**: Loki: Enable FIFO cache by default -* [4520](https://github.com/grafana/loki/pull/4520) **jordanrushing** and **owen-d**: Introduce overrides-exporter module for tenant limits -* [4453](https://github.com/grafana/loki/pull/4453) **liguozhong**: Loki: Implement retry to s3 chunk storage -* [4542](https://github.com/grafana/loki/pull/4542) **owen-d**: Introduce the `loki_overrides_defaults` metric and only export diffs for tenant limits -* [4498](https://github.com/grafana/loki/pull/4498) **trevorwhitney**: Feature: add virtual read and write targets -* [4543](https://github.com/grafana/loki/pull/4543) **trevorwhitney**: Change more default values and improve application of common storage config -* [4570](https://github.com/grafana/loki/pull/4570) **DylanGuedes**: Loki: Append loopback to ingester net interface default list -* [4594](https://github.com/grafana/loki/pull/4594) **owen-d**: Configures unordered_writes=true by default + +# 2.4.0 (2021/11/05) + +Release notes for 2.4.0 can be found on the [release notes page](https://grafana.com/docs/loki/latest/release-notes/v2-4/) + +### All Changes + +Here is a list of all changes included in 2.4.0. + +#### Loki + +* [4649](https://github.com/grafana/loki/pull/4649) **cstyan**: Instrument s3 client DeleteObject requests. +* [4643](https://github.com/grafana/loki/pull/4643) **trevorwhitney**: compactor depends on memberlist for memberlist ring option +* [4642](https://github.com/grafana/loki/pull/4642) **slim-bean**: Loki: fix handling of tail requests when using target `all` or `read` +* [4641](https://github.com/grafana/loki/pull/4641) **ssncferreira**: Migration to dskit/ring +* [4638](https://github.com/grafana/loki/pull/4638) **DylanGuedes**: Loki: Revert distributor defaulting to `inmemory` +* [4635](https://github.com/grafana/loki/pull/4635) **owen-d**: dont try to use the scheduler ring when a downstream url is configured +* [4630](https://github.com/grafana/loki/pull/4630) **chaudum**: Allow HTTP POST requests on ring pages +* [4627](https://github.com/grafana/loki/pull/4627) **slim-bean**: Loki: Explicitly define allowed HTTP methods on HTTP endpoints +* [4625](https://github.com/grafana/loki/pull/4625) **sandeepsukhani**: Logs deletion fixes +* [4617](https://github.com/grafana/loki/pull/4617) **trevorwhitney**: Add common ring configuration +* [4615](https://github.com/grafana/loki/pull/4615) **owen-d**: uses ring.Write instead of ring.WriteNoExtend for compactor ring checks +* [4614](https://github.com/grafana/loki/pull/4614) **slim-bean**: Loki: query scheduler should send shutdown to frontends when ReplicationSet changes +* [4608](https://github.com/grafana/loki/pull/4608) **trevorwhitney**: default ingester final sleep to 0 unless otherwise specified +* [4607](https://github.com/grafana/loki/pull/4607) **owen-d**: improves scheduler & compactor ringwatcher checks +* [4603](https://github.com/grafana/loki/pull/4603) **garrettlish**: add date time sprig template functions in logql label/line formatter +* [4598](https://github.com/grafana/loki/pull/4598) **kavirajk**: Fix `ip` matcher lexer to differentiate filter from identifier +* [4596](https://github.com/grafana/loki/pull/4596) **owen-d**: Ignore validity window during wal replay +* [4595](https://github.com/grafana/loki/pull/4595) **owen-d**: Cleans up redundant setting of stream.unorderedWrites=true during replay +* [4594](https://github.com/grafana/loki/pull/4594) **owen-d**: Enable unordered_writes by default +* [4593](https://github.com/grafana/loki/pull/4593) **taisho6339**: Respect gRPC context error when handling errors +* [4592](https://github.com/grafana/loki/pull/4592) **owen-d**: introduces "entry too far behind" instrumentation for unordered writes +* [4589](https://github.com/grafana/loki/pull/4589) **owen-d**: replaces fallthrough statement in InitFrontend +* [4586](https://github.com/grafana/loki/pull/4586) **dannykopping**: Configuring query-frontend interface names with loopback device +* [4585](https://github.com/grafana/loki/pull/4585) **sandeepsukhani**: set wal dir to /loki/wal in docker config +* [4577](https://github.com/grafana/loki/pull/4577) **taisho6339**: Respect shard number in series api * [4574](https://github.com/grafana/loki/pull/4574) **slim-bean**: Loki: Add a ring to the compactor used to control concurrency when not running standalone -* [4603](https://github.com/grafana/loki/pull/4603) **garrettlish**: Add date time sprig template functions in logql label/line formatter -* [4608](https://github.com/grafana/loki/pull/4608) **trevorwhitney**: Change default value of ingester lifecycler's `final_sleep` from `30s` to `0s` -* [4629](https://github.com/grafana/loki/pull/4629) **owen-d**: Default the WAL to enabled in the Loki jsonnet library -* [4625](https://github.com/grafana/loki/pull/4625) **sandeepsukhani**: Logs Deletion: Fix issues in processing of delete requests -* [4556](https://github.com/grafana/loki/pull/4556) **james-callahan**: Remove `promtail_instance` label that was was added by promtail when scraping `gcplog` target. +* [4573](https://github.com/grafana/loki/pull/4573) **sandeepsukhani**: validate default limits config with other configs at startup +* [4570](https://github.com/grafana/loki/pull/4570) **DylanGuedes**: Loki: Append loopback to ingester net interface default list +* [4569](https://github.com/grafana/loki/pull/4569) **DylanGuedes**: Config: Change default RejectOldSamplesMaxAge from 14d to 7d +* [4563](https://github.com/grafana/loki/pull/4563) **cyriltovena**: Fixes the Series function to handle properly sharding. +* [4554](https://github.com/grafana/loki/pull/4554) **cyriltovena**: Fixes a panic in the labels API when no parameters are supplied. +* [4550](https://github.com/grafana/loki/pull/4550) **cyriltovena**: Fixes an edge case in the batch chunk iterator. +* [4546](https://github.com/grafana/loki/pull/4546) **slim-bean**: Loki: Apply the ingester ring config to all other rings (distributor, ruler, query-scheduler) +* [4545](https://github.com/grafana/loki/pull/4545) **trevorwhitney**: Fix race condition in Query Scheduler ring with frontend/worker +* [4543](https://github.com/grafana/loki/pull/4543) **trevorwhitney**: Change a few default config values and improve application of common storage config +* [4542](https://github.com/grafana/loki/pull/4542) **owen-d**: only exports tenant limits which differ from defaults and export defa… +* [4531](https://github.com/grafana/loki/pull/4531) **JordanRushing**: Add quick nil check in TenantLimits for runtime_config +* [4529](https://github.com/grafana/loki/pull/4529) **owen-d**: correctly sets subservicesWatcher on scheduler +* [4525](https://github.com/grafana/loki/pull/4525) **owen-d**: Safely checks read ring for potentially nil scheduler +* [4524](https://github.com/grafana/loki/pull/4524) **dannykopping**: Clarify error message when no valid target scrape config is defined for `promtail` job +* [4520](https://github.com/grafana/loki/pull/4520) **JordanRushing**: Introduce `overrides-exporter` module to Loki +* [4519](https://github.com/grafana/loki/pull/4519) **DylanGuedes**: Loki: Enable FIFO cache by default +* [4518](https://github.com/grafana/loki/pull/4518) **slim-bean**: Loki: Fix bug where items are returned to a sync.Pool incorrectly +* [4510](https://github.com/grafana/loki/pull/4510) **lingpeng0314**: add group_{left,right} to LogQL +* [4508](https://github.com/grafana/loki/pull/4508) **trevorwhitney**: Apply better defaults when boltdb shipper is being used +* [4498](https://github.com/grafana/loki/pull/4498) **trevorwhitney**: Feature: add virtual read and write targets +* [4487](https://github.com/grafana/loki/pull/4487) **cstyan**: Update go.mod to go 1.17 +* [4484](https://github.com/grafana/loki/pull/4484) **dannykopping**: Replacing go-kit/kit/log with go-kit/log +* [4482](https://github.com/grafana/loki/pull/4482) **owen-d**: always expose loki_build_info +* [4479](https://github.com/grafana/loki/pull/4479) **owen-d**: restores for state at seconds(now-forDuration) +* [4478](https://github.com/grafana/loki/pull/4478) **replay**: Update cortex to newer version +* [4473](https://github.com/grafana/loki/pull/4473) **trevorwhitney**: Configuration: add a common config section for object storage +* [4457](https://github.com/grafana/loki/pull/4457) **kavirajk**: Fix return values of Matrix and Vector during query range in QueryShardingMiddleware +* [4453](https://github.com/grafana/loki/pull/4453) **liguozhong**: [querier] s3: add getObject retry +* [4446](https://github.com/grafana/loki/pull/4446) **garrettlish**: make LogQL syntax scope from private to public +* [4443](https://github.com/grafana/loki/pull/4443) **DylanGuedes**: Loki: Change how push API checks for contentType +* [4440](https://github.com/grafana/loki/pull/4440) **DylanGuedes**: Loki: Override distributor's default ring KV store +* [4437](https://github.com/grafana/loki/pull/4437) **dannykopping**: Ruler: Do not clear remote-write HTTP client config +* [4436](https://github.com/grafana/loki/pull/4436) **JordanRushing**: Add metric prefix changes for chunk store and runtime config to upgrading.md +* [4435](https://github.com/grafana/loki/pull/4435) **trevorwhitney**: Change default values for two GRPC setting we have to set so the queriers can connect to a frontend or scheduler +* [4433](https://github.com/grafana/loki/pull/4433) **trevorwhitney**: Add more tests around config parsing changes from common config PR +* [4432](https://github.com/grafana/loki/pull/4432) **owen-d**: tests checkpoints immediately and gives more of a time buffer +* [4431](https://github.com/grafana/loki/pull/4431) **dannykopping**: Ruler: Overwrite instead of merge remote-write headers +* [4429](https://github.com/grafana/loki/pull/4429) **dannykopping**: Ruler: Refactoring remote-write config overrides +* [4424](https://github.com/grafana/loki/pull/4424) **slim-bean**: Loki: Add a ring to the query scheduler to allow discovery via the ring as an alternative to DNS +* [4421](https://github.com/grafana/loki/pull/4421) **owen-d**: Safe per tenant overrides loading +* [4415](https://github.com/grafana/loki/pull/4415) **DylanGuedes**: Loki: Change default limits to common values +* [4413](https://github.com/grafana/loki/pull/4413) **trevorwhitney**: add compactor working dir to auto-configured file paths +* [4411](https://github.com/grafana/loki/pull/4411) **slim-bean**: Loki: Bug: frontend waiting on results which would never come +* [4400](https://github.com/grafana/loki/pull/4400) **trevorwhitney**: auto-apply memberlist ring config when join_members provided +* [4391](https://github.com/grafana/loki/pull/4391) **garrettlish**: add on and ignoring clauses in binOpExpr +* [4388](https://github.com/grafana/loki/pull/4388) **trevorwhitney**: default chunk target size to ~1MB~ 1.5MB +* [4367](https://github.com/grafana/loki/pull/4367) **owen-d**: removes deprecated duplicate per stream rate limit fields +* [4364](https://github.com/grafana/loki/pull/4364) **dannykopping**: Ruler: improve control over marshaling relabel.Config +* [4354](https://github.com/grafana/loki/pull/4354) **dannykopping**: Ruler: adding `pkg/metrics` from agent +* [4349](https://github.com/grafana/loki/pull/4349) **JordanRushing**: Add recovery middleware to Ingester; re-add recovery middleware to Querier when not running in standalone mode +* [4348](https://github.com/grafana/loki/pull/4348) **trevorwhitney**: allow ingester and distributor to run on same instance +* [4347](https://github.com/grafana/loki/pull/4347) **slim-bean**: Loki: Common Config +* [4344](https://github.com/grafana/loki/pull/4344) **dannykopping**: Ruler: per-tenant WAL +* [4327](https://github.com/grafana/loki/pull/4327) **aknuds1**: Chore: Use dskit/limiter +* [4322](https://github.com/grafana/loki/pull/4322) **owen-d**: Hotfix #4308 into k62 +* [4321](https://github.com/grafana/loki/pull/4321) **owen-d**: Hotfix #4308 into k61 +* [4313](https://github.com/grafana/loki/pull/4313) **aknuds1**: Chore: Use middleware package from dskit +* [4312](https://github.com/grafana/loki/pull/4312) **aknuds1**: Chore: Use dskit/grpcclient +* [4308](https://github.com/grafana/loki/pull/4308) **cyriltovena**: Fixes the pattern parser validation. +* [4304](https://github.com/grafana/loki/pull/4304) **aknuds1**: Chore: Reformat Go files +* [4302](https://github.com/grafana/loki/pull/4302) **cyriltovena**: Fixes a bug in the block cache code. +* [4301](https://github.com/grafana/loki/pull/4301) **trevorwhitney**: Feature: allow querier and query frontend targets to run on same process +* [4295](https://github.com/grafana/loki/pull/4295) **aknuds1**: Chore: Upgrade dskit +* [4289](https://github.com/grafana/loki/pull/4289) **kavirajk**: Add custom UnmarshalJSON for bytesize type +* [4282](https://github.com/grafana/loki/pull/4282) **chaudum**: Chore: Update Cortex and use kv package from grafana/dskit +* [4276](https://github.com/grafana/loki/pull/4276) **chaudum**: Export MemberlistKV field on Loki struct +* [4272](https://github.com/grafana/loki/pull/4272) **taisho6339**: Add count to 'loki_ingester_memory_chunks' when recovery from wal +* [4265](https://github.com/grafana/loki/pull/4265) **owen-d**: remove empty streams after wal replay +* [4255](https://github.com/grafana/loki/pull/4255) **owen-d**: replaces old cortex_chunk_store prefix with loki_chunk_store +* [4253](https://github.com/grafana/loki/pull/4253) **JordanRushing**: Change prefix for `runtimeconfig` metrics from `cortex_` to `loki_` +* [4251](https://github.com/grafana/loki/pull/4251) **dannykopping**: Runtime config: do not validate nil limits +* [4246](https://github.com/grafana/loki/pull/4246) **JordanRushing**: Add missing `Inc()` to correctly increment the `dropStage.dropCount` metric on valid dropped log line; update related docs +* [4240](https://github.com/grafana/loki/pull/4240) **bboreham**: Simplify Distributor.push +* [4238](https://github.com/grafana/loki/pull/4238) **liguozhong**: [fix] distributor: fix goroutine leak +* [4236](https://github.com/grafana/loki/pull/4236) **owen-d**: better per stream rate limits configuration options +* [4228](https://github.com/grafana/loki/pull/4228) **owen-d**: bumps per stream default rate limits +* [4227](https://github.com/grafana/loki/pull/4227) **aknuds1**: Chore: Use runtimeconfig from dskit +* [4225](https://github.com/grafana/loki/pull/4225) **aknuds1**: Flagext: Use flagext package from dskit +* [4213](https://github.com/grafana/loki/pull/4213) **owen-d**: Refactor per stream rate limit +* [4212](https://github.com/grafana/loki/pull/4212) **owen-d**: WAL replay discard metrics +* [4211](https://github.com/grafana/loki/pull/4211) **BenoitKnecht**: pkg/storage/chunk/aws: Add s3.http.ca-file option +* [4207](https://github.com/grafana/loki/pull/4207) **cstyan**: Improve error message for stream rate limit. +* [4196](https://github.com/grafana/loki/pull/4196) **56quarters**: Chore: Use services and modules from grafana/dskit +* [4193](https://github.com/grafana/loki/pull/4193) **owen-d**: adds loki_ingester_wal_replay_active metric and records this more acc… +* [4192](https://github.com/grafana/loki/pull/4192) **owen-d**: Cleanup/unordered writes ingester config +* [4191](https://github.com/grafana/loki/pull/4191) **cstyan**: [ingester/stream]: Add a byte stream rate limit. +* [4188](https://github.com/grafana/loki/pull/4188) **aknuds1**: Chore: Upgrade to latest Cortex +* [4185](https://github.com/grafana/loki/pull/4185) **sandeepsukhani**: Canary: allow setting tenant id for querying logs from loki +* [4181](https://github.com/grafana/loki/pull/4181) **owen-d**: initiate grpc health check always +* [4176](https://github.com/grafana/loki/pull/4176) **sokoide**: Authc/z: Enable grpc_client_config to allow mTLS +* [4172](https://github.com/grafana/loki/pull/4172) **sandeepsukhani**: Retention speedup +* [4160](https://github.com/grafana/loki/pull/4160) **owen-d**: safely close nonOverlapping iterators +* [4155](https://github.com/grafana/loki/pull/4155) **owen-d**: Auth followup - Remove unused +* [4153](https://github.com/grafana/loki/pull/4153) **owen-d**: uses more fleshed out cortex auth utility & adds new auth-ignored routes +* [4149](https://github.com/grafana/loki/pull/4149) **owen-d**: add unordered writes to local config +* [4141](https://github.com/grafana/loki/pull/4141) **dannykopping**: Ruler: write meaningful logs when remote-write is disabled or is misconfigured +* [4135](https://github.com/grafana/loki/pull/4135) **slim-bean**: Build: Fix build version info +* [4132](https://github.com/grafana/loki/pull/4132) **owen-d**: Promote/ruler api +* [4130](https://github.com/grafana/loki/pull/4130) **owen-d**: Tenant/unordered +* [4128](https://github.com/grafana/loki/pull/4128) **sandeepsukhani**: add a storage client for boltdb-shipper which would do all the object key management for storage operations +* [4126](https://github.com/grafana/loki/pull/4126) **cstyan**: Allow for loki-canary to generate a percentage of out of order log lines +* [4114](https://github.com/grafana/loki/pull/4114) **owen-d**: Stream iterators account for unordered data +* [4111](https://github.com/grafana/loki/pull/4111) **owen-d**: ingester.index-shards config +* [4107](https://github.com/grafana/loki/pull/4107) **sandeepsukhani**: fix finding tables which would have out of retention data +* [4104](https://github.com/grafana/loki/pull/4104) **owen-d**: Discard/ooo +* [4071](https://github.com/grafana/loki/pull/4071) **jeschkies**: Support frontend V2 with query scheduler. + +#### Promtail +* [4599](https://github.com/grafana/loki/pull/4599) **rsteneteg**: [Promtail] resolve issue with promtail not scraping target if only path changed in a simpler way that dont need mutex to sync threads +* [4588](https://github.com/grafana/loki/pull/4588) **owen-d**: regenerates assets from current vfsgen dependency +* [4568](https://github.com/grafana/loki/pull/4568) **cyriltovena**: Promtail Kafka target +* [4567](https://github.com/grafana/loki/pull/4567) **cyriltovena**: Refactor client configs in Promtail. +* [4556](https://github.com/grafana/loki/pull/4556) **james-callahan**: promtail: no need for GCP promtail_instance label now that loki supports out-of-order writes +* [4516](https://github.com/grafana/loki/pull/4516) **lizzzcai**: promtail: update promtail base image to debian:bullseye-slim +* [4507](https://github.com/grafana/loki/pull/4507) **dannykopping**: Promtail: allow for customisable stream lag labels +* [4495](https://github.com/grafana/loki/pull/4495) **sankalp-r**: Promtail: add static labels in stage +* [4461](https://github.com/grafana/loki/pull/4461) **rsteneteg**: Promtail: fix filetarget to not be stuck if no files was detected on startup +* [4346](https://github.com/grafana/loki/pull/4346) **sandeepsukhani**: add logfmt promtail stage to be able to extract data from logfmt formatted log +* [4336](https://github.com/grafana/loki/pull/4336) **ldb**: clients/promtail: Add ndjson and plaintext formats to loki_push +* [4235](https://github.com/grafana/loki/pull/4235) **kavirajk**: Add metrics for gcplog scrape. +* [3907](https://github.com/grafana/loki/pull/3907) **johanfleury**: promtail: add support for TLS/mTLS in syslog receiver + +#### Logcli +* [4303](https://github.com/grafana/loki/pull/4303) **cyriltovena**: Allow to run local boltdb queries with logcli. +* [4242](https://github.com/grafana/loki/pull/4242) **chaudum**: cli: Register configuration option `store.max-look-back-period` as CLI argument +* [4203](https://github.com/grafana/loki/pull/4203) **invidian**: cmd/logcli: add --follow flag as an alias for --tail + +#### Build +* [4639](https://github.com/grafana/loki/pull/4639) **slim-bean**: Build: simplify how protos are built +* [4609](https://github.com/grafana/loki/pull/4609) **slim-bean**: Build: Update CODEOWNERS to put Karen back in charge of the docs +* [4541](https://github.com/grafana/loki/pull/4541) **cstyan**: Fix drone ECR publish. +* [4481](https://github.com/grafana/loki/pull/4481) **cstyan**: Update golang and loki-build-image image versions. +* [4480](https://github.com/grafana/loki/pull/4480) **cstyan**: Add drone build job for lambda-promtail images. +* [4462](https://github.com/grafana/loki/pull/4462) **cstyan**: Update loki-build-image to drone 1.4.0 +* [4373](https://github.com/grafana/loki/pull/4373) **jeschkies**: Instruct how to sign `drone.yml`. +* [4358](https://github.com/grafana/loki/pull/4358) **JordanRushing**: Add DroneCI pipeline stage to validate loki example configs; create example configuration files +* [4353](https://github.com/grafana/loki/pull/4353) **dannykopping**: CI: Fixing linter deprecations +* [4286](https://github.com/grafana/loki/pull/4286) **slim-bean**: Build: Tweak stalebot message +* [4252](https://github.com/grafana/loki/pull/4252) **slim-bean**: Build: update stalebot message to be more descriptive and friendlier +* [4226](https://github.com/grafana/loki/pull/4226) **aknuds1**: Makefile: Add format target +* [4220](https://github.com/grafana/loki/pull/4220) **slim-bean**: Build: Add github action backport workflow +* [4189](https://github.com/grafana/loki/pull/4189) **mathew-fleisch**: Makefile: Add darwin/arm64 build to release binaries + +#### Project +* [4535](https://github.com/grafana/loki/pull/4535) **carlpett**: Fix branch reference in PR template +* [4604](https://github.com/grafana/loki/pull/4604) **kavirajk**: Update PR template to include `changelog` update in the checklist +* [4494](https://github.com/grafana/loki/pull/4494) **cstyan**: Add a a parameter to keep/drop the stream label from cloudwatch. +* [4315](https://github.com/grafana/loki/pull/4315) **cstyan**: Rewrite lambda-promtail to use subscription filters. + +#### Dashboards +* [4634](https://github.com/grafana/loki/pull/4634) **cyriltovena**: Fixes the operational dashboard using an old metric. +* [4618](https://github.com/grafana/loki/pull/4618) **cstyan**: loki-mixin: fix label selectors + logs dashboard +* [4575](https://github.com/grafana/loki/pull/4575) **dannykopping**: Adding recording rules dashboard +* [4441](https://github.com/grafana/loki/pull/4441) **owen-d**: Revert "loki-mixin: use centralized configuration for dashboard matchers / selectors" +* [4438](https://github.com/grafana/loki/pull/4438) **dannykopping**: Dashboards: adding "logs" into regex +* [4423](https://github.com/grafana/loki/pull/4423) **cstyan**: Add tag/link fix to operational dashboard and promtail mixin dashboard. +* [4401](https://github.com/grafana/loki/pull/4401) **cstyan**: Minor dashboard fixes + + +#### Docker-driver +* [4396](https://github.com/grafana/loki/pull/4396) **owen-d**: Removes docker driver empty log line message +* [4190](https://github.com/grafana/loki/pull/4190) **jeschkies**: Document known Docker driver issues. + +#### FluentD +* [4261](https://github.com/grafana/loki/pull/4261) **MrWong99**: FluentD output plugin: Remove an unused variable when processing chunks + +#### Docs +* [4646](https://github.com/grafana/loki/pull/4646) **KMiller-Grafana**: Docs: revise modes of operation section +* [4631](https://github.com/grafana/loki/pull/4631) **kavirajk**: Add changelog and upgrade guide for #4556 +* [4616](https://github.com/grafana/loki/pull/4616) **owen-d**: index-gw sts doc fix. closes #4583 +* [4612](https://github.com/grafana/loki/pull/4612) **surdaft**: Docs: Fix typo in docs +* [4611](https://github.com/grafana/loki/pull/4611) **KMiller-Grafana**: Docs: revise incendiary language added in PR 4507 +* [4601](https://github.com/grafana/loki/pull/4601) **mustafacansevinc**: docs: fix promtail docs links in loki installation page +* [4597](https://github.com/grafana/loki/pull/4597) **owen-d**: a few doc fixes in preparation for 2.4 +* [4590](https://github.com/grafana/loki/pull/4590) **owen-d**: improves grouping docs examples +* [4579](https://github.com/grafana/loki/pull/4579) **DylanGuedes**: Docs: Modify modes of operation image +* [4576](https://github.com/grafana/loki/pull/4576) **DylanGuedes**: Rename hybrid mode to simple scalable mode +* [4566](https://github.com/grafana/loki/pull/4566) **dannykopping**: Documenting recording rules per-tenant WAL +* [4565](https://github.com/grafana/loki/pull/4565) **DylanGuedes**: Docs: Add virtual targets docs +* [4559](https://github.com/grafana/loki/pull/4559) **chri2547**: docs: Update curl POST example in docs +* [4548](https://github.com/grafana/loki/pull/4548) **cstyan**: Improve lambda-promtail docs based on Owens review. +* [4540](https://github.com/grafana/loki/pull/4540) **JordanRushing**: Update CHANGELOG.md and /docs with info on new `overrides-exporter` module for Loki +* [4539](https://github.com/grafana/loki/pull/4539) **cstyan**: Modify lambda-promtail docs based on rewrite. +* [4527](https://github.com/grafana/loki/pull/4527) **yangkb09**: Docs: add missing quote to log_queries.md +* [4521](https://github.com/grafana/loki/pull/4521) **owen-d**: brings storage architecture up to date +* [4499](https://github.com/grafana/loki/pull/4499) **vdm**: Docs: Remove ListObjects S3 permission +* [4493](https://github.com/grafana/loki/pull/4493) **DylanGuedes**: Docs: Move rule storages configs to their own sections +* [4486](https://github.com/grafana/loki/pull/4486) **KMiller-Grafana**: Docs: correct the page parameter in the Grafana Cloud advertisement +* [4485](https://github.com/grafana/loki/pull/4485) **DylanGuedes**: Document the common config section +* [4422](https://github.com/grafana/loki/pull/4422) **KMiller-Grafana**: Docs: revise wording of Grafana Cloud advertisement +* [4417](https://github.com/grafana/loki/pull/4417) **KMiller-Grafana**: Docs: remove empty section "Generic placeholders" +* [4416](https://github.com/grafana/loki/pull/4416) **KMiller-Grafana**: Docs: correctly represent product name +* [4403](https://github.com/grafana/loki/pull/4403) **KMiller-Grafana**: Docs: introduce a fundamentals section +* [4399](https://github.com/grafana/loki/pull/4399) **KMiller-Grafana**: Docs: prominently advertise free Grafana Cloud availability +* [4374](https://github.com/grafana/loki/pull/4374) **KMiller-Grafana**: Docs: clarify distinction between single binary and microservices. +* [4363](https://github.com/grafana/loki/pull/4363) **KMiller-Grafana**: Docs: Remove wording like "As of version 1.6, you can..." +* [4361](https://github.com/grafana/loki/pull/4361) **JasonGiedymin**: fix(docs): spelling mistake +* [4357](https://github.com/grafana/loki/pull/4357) **carehart**: Correct typo +* [4345](https://github.com/grafana/loki/pull/4345) **pr0PM**: Deduplicating the compactor docs +* [4342](https://github.com/grafana/loki/pull/4342) **KMiller-Grafana**: Docs: Organize and edit the LogQL section +* [4324](https://github.com/grafana/loki/pull/4324) **lingenavd**: Docs: Update _index.md to add value boltdb-shipper for the key store +* [4320](https://github.com/grafana/loki/pull/4320) **KMiller-Grafana**: Docs: improve spelling, grammar, and formatting. +* [4310](https://github.com/grafana/loki/pull/4310) **dannykopping**: Correcting documentation example for `/api/prom/query` +* [4309](https://github.com/grafana/loki/pull/4309) **GneyHabub**: Docs: Fix a link +* [4294](https://github.com/grafana/loki/pull/4294) **mr-karan**: docs: (logs-deletion.md) URL Encode curl command +* [4293](https://github.com/grafana/loki/pull/4293) **Birdi7**: docs: fix link to Promtail documentation +* [4283](https://github.com/grafana/loki/pull/4283) **SeriousM**: Correct the indention for azure storage configuration +* [4277](https://github.com/grafana/loki/pull/4277) **ivanahuckova**: Update example for /series endpoint in _index.md +* [4247](https://github.com/grafana/loki/pull/4247) **KMiller-Grafana**: Docs: inject newlines for configuration section readability +* [4245](https://github.com/grafana/loki/pull/4245) **KMiller-Grafana**: Docs: revise max_query_lookback knob definition +* [4244](https://github.com/grafana/loki/pull/4244) **JordanRushing**: Update limits_config docs to include querier.max_query_lookback flag +* [4237](https://github.com/grafana/loki/pull/4237) **KMiller-Grafana**: Docs: first draft, Loki accepts out-of-order writes +* [4231](https://github.com/grafana/loki/pull/4231) **Aletor93**: doc: fix typo on loki-external-labels for docker client labels +* [4222](https://github.com/grafana/loki/pull/4222) **KMiller-Grafana**: Docs: minor improvements to Loki Canary docs +* [4208](https://github.com/grafana/loki/pull/4208) **cstyan**: Update tanka installation docs to refer to tanka section about `jb` +* [4206](https://github.com/grafana/loki/pull/4206) **jeschkies**: Link Kubernetes service discovery configuration. +* [4199](https://github.com/grafana/loki/pull/4199) **owen-d**: fixes typo +* [4184](https://github.com/grafana/loki/pull/4184) **mcdeck**: Update docker.md +* [4175](https://github.com/grafana/loki/pull/4175) **KMiller-Grafana**: Docs: correct path to Promtail configuration file +* [4163](https://github.com/grafana/loki/pull/4163) **smuth4**: Docs: Update docker install to work out of the box +* [4152](https://github.com/grafana/loki/pull/4152) **charles-woshicai**: Docs: example about using azure storage account as storage +* [4147](https://github.com/grafana/loki/pull/4147) **KMiller-Grafana**: Docs: fluentd client phrasing and formatting +* [4145](https://github.com/grafana/loki/pull/4145) **KMiller-Grafana**: Docs: improve LogQL section +* [4134](https://github.com/grafana/loki/pull/4134) **KMiller-Grafana**: Docs: revise section header (out of order writes) +* [4131](https://github.com/grafana/loki/pull/4131) **owen-d**: updates unordered writes config docs +* [4125](https://github.com/grafana/loki/pull/4125) **owen-d**: Initial out of order docs +* [4122](https://github.com/grafana/loki/pull/4122) **yasharne**: update boltdb-shipper index period +* [4120](https://github.com/grafana/loki/pull/4120) **vitaliyf**: Docs: Fix broken "Upgrading" link +* [4113](https://github.com/grafana/loki/pull/4113) **KMiller-Grafana**: Docs: Fix typos and grammar. Inject newlines for readability. +* [4112](https://github.com/grafana/loki/pull/4112) **slim-bean**: Docs: updated changelog and references to 2.3 +* [4100](https://github.com/grafana/loki/pull/4100) **jeschkies**: Document operation with the query scheduler. +* [4088](https://github.com/grafana/loki/pull/4088) **KMiller-Grafana**: Update Loki README with better links and descriptions +* [3880](https://github.com/grafana/loki/pull/3880) **timothydlister**: Update fluent-plugin-loki documentation URLs + +#### Jsonnet +* [4629](https://github.com/grafana/loki/pull/4629) **owen-d**: Default wal to enabled in jsonnet lib +* [4624](https://github.com/grafana/loki/pull/4624) **chaudum**: Disable chunk transfers in jsonnet lib +* [4530](https://github.com/grafana/loki/pull/4530) **owen-d**: Jsonnet/overrides exporter +* [4496](https://github.com/grafana/loki/pull/4496) **jeschkies**: Use different metrics for `PromtailFileLagging`. +* [4405](https://github.com/grafana/loki/pull/4405) **jdbaldry**: fix: Correct grafana-token creation command +* [4279](https://github.com/grafana/loki/pull/4279) **kevinschoonover**: loki-mixin: use centralized configuration for dashboard matchers / selectors +* [4259](https://github.com/grafana/loki/pull/4259) **eamonryan**: Jsonnet: Update license path argument name +* [4217](https://github.com/grafana/loki/pull/4217) **Duologic**: fix(rules): upstream recording rule switched to sum_irate +* [4182](https://github.com/grafana/loki/pull/4182) **owen-d**: fine tune grpc configs jsonnet +* [4180](https://github.com/grafana/loki/pull/4180) **owen-d**: corrects query scheduler image +* [4165](https://github.com/grafana/loki/pull/4165) **jdbaldry**: Jsonnet: Add Grafana Enterprise Logs library +* [4154](https://github.com/grafana/loki/pull/4154) **owen-d**: updates scheduler libsonnet +* [4102](https://github.com/grafana/loki/pull/4102) **jeschkies**: Define ksonnet lib for query scheduler. + + +### Notes + +This release was created from a branch starting at commit e95d193acf1633a6ec33a328b8a4a3d844e8e5f9 but it may also contain backported changes from main. + +Check the history of the branch `release-2.4`. + +### Dependencies + +* Go Version: 1.17.2 +* Cortex Version: 3f329a21cad432325268717eecf2b77c8d95150f # 2.3.0 (2021/08/06) @@ -71,7 +348,7 @@ Release notes for 2.3.0 can be found on the [release notes page](https://grafana * [3852](https://github.com/grafana/loki/pull/3852) **cyriltovena**: Shard ingester queries. * [3849](https://github.com/grafana/loki/pull/3849) **cyriltovena**: Logs ingester and store queries boundaries. * [3840](https://github.com/grafana/loki/pull/3840) **cyriltovena**: Add retention label to loki_distributor_bytes_received_total metrics -** [3837](https://github.com/grafana/loki/pull/3837) **cyriltovena**: LogQL: Pattern Parser +* [3837](https://github.com/grafana/loki/pull/3837) **cyriltovena**: LogQL: Pattern Parser * [3835](https://github.com/grafana/loki/pull/3835) **sesky4**: lz4: update lz4 version to v4.1.7 to avoid possibly panic * [3833](https://github.com/grafana/loki/pull/3833) **cyriltovena**: Fixes a flaky retention test. * [3827](https://github.com/grafana/loki/pull/3827) **sandeepsukhani**: Logs deletion fixes @@ -280,18 +557,18 @@ Release notes for 2.3.0 can be found on the [release notes page](https://grafana #### Project * [3963](https://github.com/grafana/loki/pull/3963) **rfratto**: Remove Robert Fratto from list of team members -** [3926](https://github.com/grafana/loki/pull/3926) **cyriltovena**: Add Danny Kopping to the Loki Team. +* [3926](https://github.com/grafana/loki/pull/3926) **cyriltovena**: Add Danny Kopping to the Loki Team. * [3732](https://github.com/grafana/loki/pull/3732) **dannykopping**: Issue Templates: Improve wording and add warnings * [3722](https://github.com/grafana/loki/pull/3722) **oddlittlebird**: Update CODEOWNERS * [3951](https://github.com/grafana/loki/pull/3951) **owen-d**: update sizing calc * [3931](https://github.com/grafana/loki/pull/3931) **owen-d**: Hackathon/cluster * [3920](https://github.com/grafana/loki/pull/3920) **owen-d**: adds replication & deduping into cost -** [3630](https://github.com/grafana/loki/pull/3630) **slim-bean**: Re-license to AGPLv3 +* [3630](https://github.com/grafana/loki/pull/3630) **slim-bean**: Re-license to AGPLv3 #### Docker-driver * [3814](https://github.com/grafana/loki/pull/3814) **kavirajk**: Update the docker-driver doc about default labels * [3727](https://github.com/grafana/loki/pull/3727) **3Xpl0it3r**: docker-driver: remove duplicated code -** [3709](https://github.com/grafana/loki/pull/3709) **cyriltovena**: Fixes docker driver that would panic when closed. +* [3709](https://github.com/grafana/loki/pull/3709) **cyriltovena**: Fixes docker driver that would panic when closed. ### Notes @@ -804,14 +1081,14 @@ Check the [upgrade guide](https://github.com/grafana/loki/blob/master/docs/sourc There are too many PR's to list individually for the major improvements which we thought justified a 2.0 but here is the high level: * Significant enhancements to the [LogQL query language](https://grafana.com/docs/loki/latest/logql/)! -** [Parse](https://grafana.com/docs/loki/latest/logql/#parser-expression) your logs to extract labels at query time. -** [Filter](https://grafana.com/docs/loki/latest/logql/#label-filter-expression) on query time extracted labels. -** [Format](https://grafana.com/docs/loki/latest/logql/#line-format-expression) your log lines any way you please! -** [Graph](https://grafana.com/docs/loki/latest/logql/#unwrapped-range-aggregations) the contents of your log lines as metrics, including support for many more of your favorite PromQL functions. +* [Parse](https://grafana.com/docs/loki/latest/logql/#parser-expression) your logs to extract labels at query time. +* [Filter](https://grafana.com/docs/loki/latest/logql/#label-filter-expression) on query time extracted labels. +* [Format](https://grafana.com/docs/loki/latest/logql/#line-format-expression) your log lines any way you please! +* [Graph](https://grafana.com/docs/loki/latest/logql/#unwrapped-range-aggregations) the contents of your log lines as metrics, including support for many more of your favorite PromQL functions. * Generate prometheus [alerts directly from your logs](https://grafana.com/docs/loki/latest/alerting/)! -** Create alerts using the same prometheus alert rule syntax and let Loki send alerts directly to your Prometheus Alertmanager! +* Create alerts using the same prometheus alert rule syntax and let Loki send alerts directly to your Prometheus Alertmanager! * [boltdb-shipper](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/) is now production ready! -** This is it! Now Loki only needs a single object store (S3,GCS,Filesystem...) to store all the data, no more Cassandra, DynamoDB or Bigtable! +* This is it! Now Loki only needs a single object store (S3,GCS,Filesystem...) to store all the data, no more Cassandra, DynamoDB or Bigtable! We are extremely excited about these new features, expect some talks, webinars, and blogs where we explain all this new functionality in detail. @@ -1382,7 +1659,7 @@ Unfortunately after a few more bug reports it seems this change caused some more The docker logging driver received a couple very nice updates, it's always been challenging to configure pipeline stages for the docker driver, with the first PR there are now a few easier ways to do this! In the second PR we added config options to control keeping any log files on the host when using the docker logging driver, allowing you to run with no disk access if you would like, as well as allowing you to control keeping log files available after container restarts. -** [1864](https://github.com/grafana/loki/pull/1864) **cyriltovena**: Sign helm package with GPG. +* [1864](https://github.com/grafana/loki/pull/1864) **cyriltovena**: Sign helm package with GPG. We now GPG sign helm packages! diff --git a/docs/sources/api/_index.md b/docs/sources/api/_index.md index 7d441949c4..dcac2324bd 100644 --- a/docs/sources/api/_index.md +++ b/docs/sources/api/_index.md @@ -951,7 +951,6 @@ Displays a web page with the ruler hash ring status, including the state, health ### List rule groups -This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option. ``` GET /loki/api/v1/rules @@ -999,8 +998,6 @@ List all rules configured for the authenticated tenant. This endpoint returns a ### Get rule groups by namespace -This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option. - ``` GET /loki/api/v1/rules/{namespace} ``` @@ -1024,8 +1021,6 @@ rules: ### Get rule group -This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option. - ``` GET /loki/api/v1/rules/{namespace}/{groupName} ``` @@ -1034,8 +1029,6 @@ Returns the rule group matching the request namespace and group name. ### Set rule group -This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option. - ``` POST /loki/api/v1/rules/{namespace} ``` @@ -1073,8 +1066,6 @@ Deletes a rule group by namespace and group name. This endpoints returns `202` o ### Delete namespace -This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option. - ``` DELETE /loki/api/v1/rules/{namespace} ``` @@ -1083,8 +1074,6 @@ Deletes all the rule groups in a namespace (including the namespace itself). Thi ### List rules -This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option. - ``` GET /prometheus/api/v1/rules ``` @@ -1095,8 +1084,6 @@ For more information, refer to the [Prometheus rules](https://prometheus.io/docs ### List alerts -This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option. - ``` GET /prometheus/api/v1/alerts ``` diff --git a/docs/sources/clients/promtail/scraping.md b/docs/sources/clients/promtail/scraping.md index bea821489c..3cc4e79cef 100644 --- a/docs/sources/clients/promtail/scraping.md +++ b/docs/sources/clients/promtail/scraping.md @@ -301,7 +301,7 @@ scrape_configs: ``` Only the `brokers` and `topics` is required. -see the [configuration](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#kafka) section for more information. +see the [configuration](../../configuration/#kafka) section for more information. ## Relabeling diff --git a/docs/sources/configuration/_index.md b/docs/sources/configuration/_index.md index 2bd4a53f8c..f856686954 100644 --- a/docs/sources/configuration/_index.md +++ b/docs/sources/configuration/_index.md @@ -1842,8 +1842,6 @@ chunks: The `compactor_config` block configures the compactor component. This component periodically compacts index shards to more performant forms. -Retention through the Compactor is experimental. - ```yaml # Directory where files can be downloaded for compaction. # CLI flag: -boltdb.shipper.compactor.working-directory @@ -2364,7 +2362,7 @@ kvstore: # The prefix for the keys in the store. Should end with a /. # CLI flag: -.prefix - [prefix: | default = "schedulers/"] + [prefix: | default = "collectors/"] # The consul_config configures the consul client. [consul: ] diff --git a/docs/sources/logql/ip.md b/docs/sources/logql/ip.md index 905e7f8d83..c0005950cf 100644 --- a/docs/sources/logql/ip.md +++ b/docs/sources/logql/ip.md @@ -5,8 +5,6 @@ weight: 40 # Matching IP addresses -Matching IP addresses is an experimental feature. - LogQL supports matching IP addresses. With logs such as diff --git a/docs/sources/logql/log_queries.md b/docs/sources/logql/log_queries.md index 95eef2b79c..95668274a4 100644 --- a/docs/sources/logql/log_queries.md +++ b/docs/sources/logql/log_queries.md @@ -389,8 +389,6 @@ will get those labels extracted: #### Pattern -The pattern parser is a beta feature. - The pattern parser allows the explicit extraction of fields from log lines by defining a pattern expression (`| pattern ""`). The expression matches the structure of a log line. Consider this NGINX log line. diff --git a/docs/sources/operations/storage/retention.md b/docs/sources/operations/storage/retention.md index 096839c254..e0f8baafbe 100644 --- a/docs/sources/operations/storage/retention.md +++ b/docs/sources/operations/storage/retention.md @@ -8,7 +8,6 @@ Retention in Grafana Loki is achieved either through the [Table Manager](#table- Retention through the [Table Manager](../table-manager/) is achieved by relying on the object store TTL feature, and will work for both [boltdb-shipper](../boltdb-shipper) store and chunk/index store. However retention through the [Compactor](../boltdb-shipper#compactor) is supported only with the [boltdb-shipper](../boltdb-shipper) store. -Retention through the [Compactor](#Compactor) is experimental. The Compactor retention will become the default and have long term support. It supports more granular retention policies on per tenant and per stream use cases. ## Compactor diff --git a/docs/sources/release-notes/v2-4.md b/docs/sources/release-notes/v2-4.md index 01158024d2..5a7ce0d2ba 100644 --- a/docs/sources/release-notes/v2-4.md +++ b/docs/sources/release-notes/v2-4.md @@ -4,11 +4,46 @@ title: V2.4 # Version 2.4 release notes -The Loki team is excited to announce the release of Loki 2.4! +With Loki 2.4 we’ve focused on 2 main things: + +1. Making it easier than ever to get your logs into Loki. +2. Making it easier to run and operate, anyone not using Kubernetes or allergic to microservices should pay close attention, this release has a lot for you. ## Features and enhancements -* Blurb about allowing out-of-order log entries. Configured an a per-tenant basis. Refer to the configuration section's [Accept out-of-order writes](../../configuration/#accept-of-out-of-order-writes) for details. +* [**Loki no longer requires logs to be sent in perfect chronological order.**](../../configuration/#accept-out-of-order-writes) Support for out of order logs is one of the most highly requested features for Loki and in 2.4 the strict ordering constraint has been removed. +* Scaling Loki is now much easier with a hybrid mode between a single binary and microservices we are calling the [**Simple Scalable Deployment**](../../fundamentals/architecture/#modes-of-operation). It’s now possible to scale Loki with new `read` and `write` targets. Where previously you would have needed Kubernetes and the microservices approach to start tapping into Loki’s potential, it’s now possible to do this in a much simpler way. +* We introduced a [common config section](../../configuration/#common_config) and updated and updated a lot of defaults. This results in a **70% smaller Loki config** and comes out of the box with more appropriate defaults and limits. [Check out the example local config](https://github.com/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml) as the new reference for running Loki. +* **[Recording rules](../../rules/#recording-rules) are no longer marked experimental** and have a much more resilient implementation which leverages the existing write ahead log code in Prometheus. +* Do you use Kafka for your logs? You should check out the new [**Promtail Kafka Consumer**](../../clients/promtail/scraping/#kafka) which can easily get your logs out of Kafka and into Loki. +* **LogQL has received some nice enhancements** thanks to the amazing Loki community in the form of [group_left and group_right](../../logql/#many-to-one-and-one-to-many-vector-matches) as well as the `label_format` and `line_format`functions now support [working with dates and times](../../logql/template_functions/#now) +* Another great community contribution to Promtail allows it to [**accept ndjson and plaintext log files over HTTP**](../../clients/promtail/configuration/#loki_push_api). + +All in all about 260 PR’s went into Loki 2.4 and we thank everyone for helping us make the best Loki yet. + +For a full list of all changes please look at the [CHANGELOG](https://github.com/grafana/loki/blob/main/CHANGELOG.md#240-20211105) + +## Upgrade Considerations + +Please read the [upgrade guide](../../upgrading/#240) before updating Loki. + +We made a lot of changes to Loki’s configuration as part of this release, we have tried our best to make sure changes are compatible with existing configurations however some changes to default limits may impact users who didn't have values explicitly set for these limits in their config files. + +Here is a list of limits changed: + +| config | new default | old default | +| --- | --- | --- | +| ingestion_rate_strategy | "global" | "local" | +| max_global_streams_per_user | 5000 | 0 (no limit) | +| max_query_length | "721h" | "0h" (no limit) | +| max_query_parallelism | 32 | 14 | +| max_streams_per_user | 0 (no limit) | 10000 | +| reject_old_samples | true | false | +| reject_old_samples_max_age | "168h" | "336h" | +| per_stream_rate_limit | 3MB | - | +| per_stream_rate_limit_burst | 15MB | - | + +Also we have enabled the [in-memory FIFO cache by default](https://github.com/grafana/loki/pull/4519) when no existing external cache is configured. This helps improve Loki's performance however it will also allow Loki to consume more memory. ## Bug fixes @@ -16,4 +51,23 @@ Lists of bug fixes for 2.4.x. ### 2.4.0 bug fixes -* Bug fix here. +Here is a list of some of the more important fixes in 2.4.0: + +* [4598](https://github.com/grafana/loki/pull/4598) **kavirajk**: Fix `ip` matcher lexer to differentiate filter from identifier +* [4563](https://github.com/grafana/loki/pull/4563) **cyriltovena**: Fixes the Series function to handle properly sharding. +* [4518](https://github.com/grafana/loki/pull/4518) **slim-bean**: Loki: Fix bug where items are returned to a sync.Pool incorrectly +* [4411](https://github.com/grafana/loki/pull/4411) **slim-bean**: Loki: Bug: frontend waiting on results which would never come +* [4238](https://github.com/grafana/loki/pull/4238) **liguozhong**: [fix] distributor: fix goroutine leak + +## Security fixes + +List of security fixes for 2.4.x + +### 2.4.0 security fixes + +2.4.0 contains one fix which is security related + +[4627](https://github.com/grafana/loki/pull/4627) Loki: Explicitly define allowed HTTP methods on HTTP endpoints + +A community user noticed that all Loki endpoints would respond to HTTP OPTIONS requests, and the proxy they placed in front of Loki responsible for HTTP authentication would pass OPTIONS requests unauthenticated to Loki allowing users to make unauthenticated/unauthorized queries to Loki. This PR restricts the type of HTTP methods allowed on each endpoint and disallows OPTIONS requests. + diff --git a/docs/sources/upgrading/_index.md b/docs/sources/upgrading/_index.md index a681e33d88..2393e50240 100644 --- a/docs/sources/upgrading/_index.md +++ b/docs/sources/upgrading/_index.md @@ -14,18 +14,146 @@ If we have any expectation of difficulty upgrading we will document it here. As more versions are released it becomes more likely unexpected problems arise moving between multiple versions at once. If possible try to stay current and do sequential updates. If you want to skip versions, try it in a development environment before attempting to upgrade production. +# Checking for config changes -## Master / Unreleased +Using docker you can check changes between 2 versions of Loki with a command like this: + +``` +export OLD_LOKI=2.3.0 +export NEW_LOKI=2.4.0 +export CONFIG_FILE=loki-local-config.yaml +diff --color=always --side-by-side <(docker run --rm -t -v "${PWD}":/config grafana/loki:${OLD_LOKI} -config.file=/config/${CONFIG_FILE} -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') <(docker run --rm -t -v "${PWD}":/config grafana/loki:${NEW_LOKI} -config.file=/config/${CONFIG_FILE} -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') | less -R +``` + +the `tr -d '\r'` is likely not necessary for most people, seems like WSL2 was sneaking in some windows newline characters... + +The output is incredibly verbose as it shows the entire internal config struct used to run Loki, you can play around with the diff command if you prefer to only show changes or a different style output. + +## Main / Unreleased + +## 2.4.0 + +The following are important changes which should be reviewed and understood prior to upgrading Loki. ### Loki -#### Promtail no longer insert `promtail_instance` label when scraping `gcplog` target -* [4556](https://github.com/grafana/loki/pull/4556) **james-callahan**: Remove `promtail_instance` label that was being added by promtail when scraping `gcplog` target. +The following changes pertain to upgrading Loki. + +#### The single binary no longer runs a table-manager + +Single binary Loki means running loki with `-target=all` which is the default if no `-target` flag is passed. + +This will impact anyone in the following scenarios: + +1. Running a single binary Loki with any index type other than `boltdb-shipper` or `boltdb` +2. Relying on retention with the configs `retention_deletes_enabled` and `retention_period` + +Anyone in situation #1 who is not using `boltdb-shipper` or `boltdb` (e.g. `cassandra` or `bigtable`) should modify their Loki command to include `-target=all,table-manager` this will instruct Loki to run a table-manager for you. + +Anyone in situation #2, you have two options, the first (and not recommended) is to run Loki with a table-manager by adding `-target=all,table-manager`. + +The second and recommended solution, is to use deletes via the compactor: + +``` +compactor: + retention_enabled: true +limits_config: + retention_period: [30d] +``` + +See the [retention docs](../operations/storage/retention) for more info. + +#### Log messages on startup: proto: duplicate proto type registered: + +PR [#3842](https://github.com/grafana/loki/pull/3842) **cyriltovena**: Fork cortex chunk storage into Loki. + +Since Cortex doesn't plan to use the `chunk` package anymore, we decided to fork it into our storage package to +be able to evolve and modify it easily. However, as a side-effect, we still vendor Cortex which includes this forked +code and protobuf files resulting in log messages like these at startup: + +``` +2021-11-04 15:30:02.437911 I | proto: duplicate proto type registered: purgeplan.DeletePlan +2021-11-04 15:30:02.437936 I | proto: duplicate proto type registered: purgeplan.ChunksGroup +2021-11-04 15:30:02.437939 I | proto: duplicate proto type registered: purgeplan.ChunkDetails +... +``` + +The messages are harmless and we will work to remove them in the future. + +#### Change of some default limits to common values + +PR [4415](https://github.com/grafana/loki/pull/4415) **DylanGuedes**: the default value of some limits were changed to protect users from overwhelming their cluster with ingestion load caused by relying on default configs. + +We suggest you double check if the following parameters are +present in your Loki config: `ingestion_rate_strategy`, `max_global_streams_per_user` +`max_query_length` `max_query_parallelism` `max_streams_per_user` +`reject_old_samples` `reject_old_samples_max_age`. If they are not present, we recommend you double check that the new values will not negatively impact your system. The changes are: + +| config | new default | old default | +| --- | --- | --- | +| ingestion_rate_strategy | "global" | "local" | +| max_global_streams_per_user | 5000 | 0 (no limit) | +| max_query_length | "721h" | "0h" (no limit) | +| max_query_parallelism | 32 | 14 | +| max_streams_per_user | 0 (no limit) | 10000 | +| reject_old_samples | true | false | +| reject_old_samples_max_age | "168h" | "336h" | +| per_stream_rate_limit | 3MB | - | +| per_stream_rate_limit_burst | 15MB | - | + +#### Change of configuration defaults + +| config | new default | old default| +| --- | --- | --- | +| chunk_retain_period | 30s | 0s | +| chunk_idle_period | 1h | 30m | +| chunk_target_size | 1048576 | 1572864 | + +* chunk_retain_period is necessary when using an index queries cache which is not enabled by default. If you have configured an index_queries_cache_config section make sure that you set chunk_retain_period larger than your cache TTL +* chunk_idle_period is how long before a chunk which receives no logs is flushed. +* chunk_target_size was increased to flush slightly larger chunks, if using memcache for a chunks store make sure it will accept files up to 1.5MB in size. + +#### In memory FIFO caches enabled by default + +Loki now enables a results cache and chunks cache in memory to improve performance. This can however increase memory usage as the cache's by default are allowed to consume up to 1GB of memory. + +If you would like to disable these caches or change this memory limit: + +Disable: + +``` +chunk_store_config: + chunk_cache_config: + enable_fifocache: false +query_range: + results_cache: + cache: + enable_fifocache: false +``` + +Resize: + +``` +chunk_store_config: + chunk_cache_config: + enable_fifocache: true + fifocache: + max_size_bytes: 500MB +query_range: + results_cache: + cache: + enable_fifocache: true + fifocache: + max_size_bytes: 500MB +``` #### Ingester Lifecycler `final_sleep` now defaults to `0s` + * [4608](https://github.com/grafana/loki/pull/4608) **trevorwhitney**: Change default value of ingester lifecycler's `final_sleep` from `30s` to `0s` -This changes the default value for the `final_sleep` property of the ingester's lifecycler from `30s` to `0s` +This final sleep exists to keep Loki running for long enough to get one final Prometheus scrape before shutting down, however it also causes Loki to sit idle for 30s on shutdown which is an annoying experience for many people. + +We decided the default would be better to disable this sleep behavior but anyone can set this config variable directly to return to the previous behavior. #### Ingester WAL now defaults to on, and chunk transfers are disabled by default @@ -53,7 +181,7 @@ ingester: ``` #### Memberlist config now automatically applies to all non-configured rings -PR [4400](https://github.com/grafana/loki/pull/4400) **trevorwhitney**: Config: automatically apply memberlist config too all rings when provided +* [4400](https://github.com/grafana/loki/pull/4400) **trevorwhitney**: Config: automatically apply memberlist config too all rings when provided This change affects the behavior of the ingester, distributor, and ruler rings. Previously, if you wanted to use memberlist for all of these rings, you had to provide a `memberlist` configuration as well as specify `store: memberlist` for the `kvstore` of each of the rings you wanted to use memberlist. @@ -80,7 +208,7 @@ ruler: Now, if your provide a `memberlist` configuration with at least one `join_members`, loki will default all rings to use a `kvstore` of type `memberlist`. You can change this behavior by overriding specific configurations. For example, if you wanted to use `consul` for you `ruler` rings, but `memberlist` -for the `ingester` and `distributor`, you could do so with the following config: +for the `ingester` and `distributor`, you could do so with the following config (although we don't know why someone would want to do this): ```yaml memberlist: @@ -113,55 +241,24 @@ server: grpc_server_ping_without_stream_allowed: true ``` -Please manually provide the values of `5m` and `true` (respectively) in your config if you rely on those values. - --_add changes here which are unreleased_ - -### Loki Config - -#### Change of some default limits to common values - -PR [4415](https://github.com/grafana/loki/pull/4415) **DylanGuedes**: the default value of some limits were changed to protect users from overwhelming their cluster with ingestion load caused by relying on default configs. - -We suggest you double check if the following parameters are -present in your Loki config: `ingestion_rate_strategy`, `max_global_streams_per_user` -`max_query_length` `max_query_parallelism` `max_streams_per_user` -`reject_old_samples` `reject_old_samples_max_age`. If they are not present, we recommend you double check that the new values will not negatively impact your system. The changes are: - -| config | new default | old default | -| --- | --- | --- | -| ingestion_rate_strategy | "global" | "local" | -| max_global_streams_per_user | 5000 | 0 (no limit) | -| max_query_length | "721h" | "0h" (no limit) | -| max_query_parallelism | 32 | 14 | -| max_streams_per_user | 0 (no limit) | 10000 | -| reject_old_samples | true | false | -| reject_old_samples_max_age | "168h" | "336h" | +[This issue](https://github.com/grafana/loki/issues/4375) has some more information on the change. #### Some metric prefixes have changed from `cortex_` to `loki_` -PR [#3842](https://github.com/grafana/loki/pull/3842)/[#4253](https://github.com/grafana/loki/pull/4253) **jordanrushing**: Metrics related to chunk storage and runtime config have changed their prefixes from `cortex_` to `loki_`. +* [#3842](https://github.com/grafana/loki/pull/3842)/[#4253](https://github.com/grafana/loki/pull/4253) **jordanrushing**: Metrics related to chunk storage and runtime config have changed their prefixes from `cortex_` to `loki_`. -- `cortex_runtime_config*` -> `loki_runtime_config*` -- `cortex_chunks_store*` -> `loki_chunks_store*` - -#### Fork cortex chunk storage into Loki +``` +cortex_runtime_config* -> loki_runtime_config* +cortex_chunks_store* -> loki_chunks_store* +``` -PR [#3842](https://github.com/grafana/loki/pull/3842) **cyriltovena**: Fork cortex chunk storage into Loki. +### Promtail -Since Cortex doesn't plan to use `chunk` package anymore, we decided to fork it into our storage package to -be able to evolve and modify it easily. However, as a side-effect, while Cortex doesn't remove this package from -its source code, we have duplicated protos being registered, which raises messages similar to the ones below. -For now, you should just ignore them. +The following changes pertain to upgrading Promtail. -``` -2021-11-04 15:30:02.437911 I | proto: duplicate proto type registered: purgeplan.DeletePlan -2021-11-04 15:30:02.437936 I | proto: duplicate proto type registered: purgeplan.ChunksGroup -2021-11-04 15:30:02.437939 I | proto: duplicate proto type registered: purgeplan.ChunkDetails -... -``` +#### Promtail no longer insert `promtail_instance` label when scraping `gcplog` target +* [4556](https://github.com/grafana/loki/pull/4556) **james-callahan**: Remove `promtail_instance` label that was being added by promtail when scraping `gcplog` target. --_add changes here which are unreleased_ ## 2.3.0 diff --git a/pkg/loki/config_wrapper.go b/pkg/loki/config_wrapper.go index 5bcb560a8c..537bfa0a29 100644 --- a/pkg/loki/config_wrapper.go +++ b/pkg/loki/config_wrapper.go @@ -442,6 +442,10 @@ func applyFIFOCacheConfig(r *ConfigWrapper) { resultsCacheConfig := r.QueryRange.ResultsCacheConfig.CacheConfig if !isRedisSet(resultsCacheConfig) && !isMemcacheSet(resultsCacheConfig) { r.QueryRange.ResultsCacheConfig.CacheConfig.EnableFifoCache = true + // The query results fifocache is still in Cortex so we couldn't change the flag defaults + // so instead we will override them here. + r.QueryRange.ResultsCacheConfig.CacheConfig.Fifocache.MaxSizeBytes = "1GB" + r.QueryRange.ResultsCacheConfig.CacheConfig.Fifocache.Validity = 1 * time.Hour } }