1460 Commits (f70f9110aa954abe52eca34acdf1e52a114e2400)

Author	SHA1	Message	Date
J Stickler	f70f9110aa	Update version in docker driver command. (#9870) ... **Which issue(s) this PR fixes**: Fixes #9802 There is an 2.5 year old image in Docker Hub with the tag `latest`, so updating the documentation to not reference this tag, as it pulls the incorrect image.	2 years ago
J Stickler	7797a98074	TOC Sync: Move configuration files under Configure (#9879) ... **Which issue(s) this PR fixes**: Fixes #8713 **Special notes for your reviewer**: Replaces PR [#8817](https://github.com/grafana/loki/pull/8817), which was stale and badly out of date. Part of the Database products table of contents alignment project. Moves configuration files from Configuration to new Configure heading in TOC. Updates front matter for the topic. Fixes links broken by the move. Updates the MakeFile to point to the new Configure DOCS_SOURCES_PATH.	2 years ago
David Peleg	c129cfe93c	Docs: add reference to flog CLI args (#9638) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Sandeep Sukhani	e089b4b60d	address feedback on pr 9798 (#9842) ... **What this PR does / why we need it**: PR #9798 got merged by mistake without me addressing the feedback. There is some critical feedback which needs to be addressed, particularly https://github.com/grafana/loki/pull/9798#discussion_r1243526743.	2 years ago
Jonathan Duncan	1d3f0b1001	Documentation: limits_config.enforce_metric_name removed (#9393) ... **What this PR does / why we need it**: ["The config still exists though, it's just not used anywhere."](https://github.com/grafana/loki/issues/2096) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
aryalireza	dc4277fd86	Update tanka.md (#9820) ... Add delete loki's environment and resources from k8s cluster. **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Travis Patterson <masslessparticle@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Karsten Jeschkies	b35bbd80d6	Support content negotiation between query frontend and querier. (#9813) ... **What this PR does / why we need it**: Currently, the querier sends results to the query frontend in JSON which is then decoded to Protobuf. It is more efficient to send the results as Protobuf. This will also allow to extend the results with custom data structures. The change is backwards compatible through content negotiation. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
ZhenYi Liang	0711580154	helm: support endpoint_suffix for azure storage (#9791) ... **What this PR does / why we need it**: According the [official document](https://grafana.com/docs/loki/latest/storage/#using-account-name-and-key), support storage_config.azure.endpoint_suffix, example blob.core.chinacloudapi.cn **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Tiago Posse	a24ef7b206	Helm: support metricRelabelings for loki serviceMonitors (#9784) ... **What this PR does / why we need it**: Add support for `metricRelabelings` in the `ServiceMonitor` resources when monitoring Loki. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Tiago Posse <tiagoposse@gmail.com> Co-authored-by: Tiago Posse <tiago.posse@hi.health>	3 years ago
manohar-koukuntla	999259a4d3	jsonnet: Add TSDB shipper configuration flag (#9790) ... This commit adds the TSDB equivalent of `use_boltdb_shipper` flag to be able to configure `tsdb_shipper` section. Fixes #9789	3 years ago
Ashwanth	f9fc1df4c0	logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default (#9626) ... **What this PR does / why we need it**: Introduces `--strict` flag for logfmt parser to enable strict parsing. With strict parsing enabled, logfmt parser stops scanning the log line and returns early with an error if it encounters any poorly formatted key=value pair. This pr updates the parser to perform non-strict parsing by default where it skips over the invalid tokens. This offers the flexibility to parse semi-structured log lines. Also adds `--keep-empty` flag to retain standalone keys(with empty value) as labels with value set to empty string **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Owen Diehl	e2a63e582c	adds tsdb-max-bytes-per-shard limit (#9811) ... Adds the per tenant limit `tsdb-max-bytes-per-shard` which is used in configuring the shard size for tsdb subqueries. This effectively gives control over how big subqueries should be (assuming they're shardable). The default is no different (`600MB`) than the previously hardcoded initial value. This should help us iterate to find optimal shard sizes to improve operations in the long term.	3 years ago
Victor Login	30fff3cd65	upgrade version nginxinc/nginx-unprivileged to 1.23 (#8984) ... **What this PR does / why we need it**: This PR upgrade nginxinc/nginx-unprivileged to 1.23 **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Alexandru Gologan	f785faa820	Helm: add additionalRuleLabels to PrometheusRule alerts (#9020) ... **What this PR does / why we need it**: Adds the ability to specify other labels on the Prometheus alerts. Specifying: ```yaml monitoring: rules: additionalRuleLabels: custom_label: custom_value ``` will add it to all alerts: ```yaml - alert: LokiRequestErrors annotations: ... expr: ... for: 15m labels: severity: critical custom_label: custom_value ``` This approach is inspired by: [kube-prometheus-stack/rules](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/general.rules.yaml) Though in their case they run [sync_prometheus_rules.py](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/hack/sync_prometheus_rules.py) to create the `tpl`. I can understand if this is not the approach (manual) you want to take as editing the tpl adding the labels everywhere and escaping go tpl `{{ }}` is quite painful. **Which issue(s) this PR fixes**: N/A **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Nyaundi Brian	5add792487	Update _index.md (#9028) ... **What this PR does / why we need it**: Typo fix **Which issue(s) this PR fixes**: None	3 years ago
Hans Kristian Flaatten	86d943d8f4	helm: add updateStategy for canary pods (#9760) ... **What this PR does / why we need it**: Allows configuring Canary DaemonSet rollout. **Which issue(s) this PR fixes**: N/A **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Steven Miller	e1002624bd	Helm: Add priorityClassName for Grafana Agent and Loki Canary (#8506) ... **What this PR does / why we need it**: - Allow configuring PriorityClassName for the GrafanaAgent daemonset - Allow configuring PriorityClassName for the loki canary daemonset without also configuring for read deployment - Helpful when an administrator wants to prioritize logging on their nodes **Special notes for your reviewer**: - Used _coalese_ style like it was already in the case of loki canary - For loki canary, specifying on the canary takes higher precedence than on the reader, but lower than global - For grafana agent, global takes precedence over grafana agent to match the existing style, but maintainers might consider flipping that around so that most-specific is also most-prioritized **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Travis Patterson	806674fdaa	Add log-volume feature flag (#9762) ... Adds a feature flag for use with the new log-volume endpoints so associated features can be rolled out incrementally.	3 years ago
Christian Haudum	ce81895241	Add shuffle sharding to index gateway (#9710) ... This PR introduces shuffle sharding of the index files across index gateways where the shuffle sharding key is the tenant ID. Instead of having a global replication factor on the index gateways which affects all tenants the same, whether they are small or large tenants, the shuffle sharding allows to specify a shard factor per tenant (on top of the replication factor). This a) reduces the amount of indexes that are downloaded on individual index gateways in case of a high replication factor (which is currently the de-factor standard to deal with high load) and b) allows to scale tenants individually. The shuffle sharding algorithm is part of dskit's ring implementation and is not part of this commit. If you have set a high RF for the index gateways, it is required to reduce the RF and instead set the shard factor for the tenants. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Danny Kopping	ccf44f8840	Adding 3 existing maintainers, and our newest one: Ashwanth! (#9741) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
sarah-spang	6157e28998	Loki ECS Client: Update Name in Options config (#9616) ... Configuration options show "Name": "grafana-loki", however, fluentbit specifies this should just be "loki" A few users have reported receiving an error when using the documented "Name" option, but it works as expected when using the Name specified in the fluentbit docs: Reference: https://docs.fluentbit.io/manual/pipeline/outputs/loki#fluent-bit-+-grafana-cloud	3 years ago
Salva Corts	61d407b4bb	Add keep stage to LogQL (#9693) ... **What this PR does / why we need it**: This PR adds a new `keep` stage to LogQL. It allows filtering specific labels and helps to avoid hitting the _max streams per query_ limit by keeping only specific labels in the label set. It works the opposite of [drop stage](https://grafana.com/docs/loki/latest/logql/log_queries/#drop-labels-expression). **Special notes for your reviewer**: We do not drop [special error labels](edc6b0bff7/pkg/logqlmodel/error.go (L18-L20)) added by Loki at query time. Users will have to explicitly drop them by using the drop stage. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
J Stickler	32320736db	TOC Sync: Move files under Get Started. (#9717) ... **Which issue(s) this PR fixes**: Fixes #8711 **Special notes for your reviewer**: Part of the Database products table of contents alignment project. Moves Getting Started and Fundamentals files under Get Started in TOC. Updates front matter for the topic (headings, page weights, descriptions, aliases to redirect from old URL to new URL) Fixes links broken by the move.	3 years ago
J Stickler	11e2375e6d	Removing 'please' from the docs. (#9658) ... **Which issue(s) this PR fixes**: Fixes #8922 **Special notes for your reviewer**: This is to conform with our Style Guide. https://developers.google.com/style/tone#politeness-and-use-of-please	3 years ago
J Stickler	c2f96221dc	Update that Loki accepts out-of-order rewrites by default. (#9670) ... **Which issue(s) this PR fixes**: Fixes #7477	3 years ago
Jan Jansen	8abe080fd6	Promtail: improve behavior of partial lines (#9508) ... **What this PR does / why we need it**: I started to see OOM after upgrading to a promtail version with #8497. So played around to make partial line behavior more stable. You can now truncate partial lines and configure max partial lines buffer. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Jan Jansen <jan.jansen@gdata.de> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Jan-Otto Kröpke	14f96c0c7b	Add loki.memberlistConfig to fully control the memberlist configuration (#9646) ... **What this PR does / why we need it**: Add loki.memberlistConfig to fully control the memberlist configuration On singleBinary with replica 1, memberlist isn't need and its counter-productive since it delays the start of the loki binary. At the moment, it isn't possible to remove the memberlist defaults. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Jack Baldry	8391e557b1	Update doc-validator to v3.0.0 (#9666) ... ## 3.0.0 ### Added - Structured output for use with [`reviewdog`](https://github.com/reviewdog/reviewdog). You can achieve the original error output by piping the output to the following `jq` expression: `jq -r '"ERROR: \(.location.path):\(.location.range.start.line):\(.location.range.start.column): \(.message)"'`. - Suggestions for simple link fixes. In GitHub, [`reviewdog`](https://github.com/reviewdog/reviewdog) comments these suggestions for convenient replacement. - Support for anchors referring to repeated headings. In the case that a page has multiple headings that share the same text, the renderer appends a zero indexed, numbered suffix to the identifier. For example, when there are two headings that are both "Heading text", the first anchor identifier is `heading-text` and the second is `heading-text-1`. - Error when running `doc-validator` on no files. **Notes for reviewer:** This also updates the `make-docs` procedure. Please ensure you can run `make docs` from the `docs/` directory before approving and merging this PR. --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	3 years ago
Terje Sannum	986d4ca756	Helm: add support for additional labels on loki-canary pods (#9673) ... Add support for setting additional labels on the loki-canary pods --------- Signed-off-by: Terje Sannum <terje.sannum@nav.no>	3 years ago
Salva Corts	73ac208981	Improve docs for empty value in cache compression config (#9649) ... **What this PR does / why we need it**: Follow up PR for https://github.com/grafana/loki/pull/9535#discussion_r1218167670 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Christophe Collot	f2394355d8	feat(lambda-promtail): add cloudfront log file ingestion support (#9573) ... **What this PR does / why we need it**: This PR enables ingesting logs from Cloudfront log files stored in s3 (batch). The current setup only supports streaming Cloudfront logs through AWS Kinesis, whereas this PR implements the same flow as for VPC Flow logs, Load Balancer logs, and Cloudtrail logs (s3 --> SQS (optional) --> Lambda Promtail --> Loki) **Special notes for your reviewer**: + The Cloudfront log file format is different from the already implemented services, meaning we had to build yet another regex. AWS never bothered making all services follow the same log file naming convention but the "good" thing is that it's now very unlikely they will change it in the future. + The Cloudfront file name does not have any mention of the AWS account or the time of log it contains, it means we have to infer the log type from the filename format instead of finding the exact string "cloudfront" in the filename. This is why in `getLabels`, if no `type` parameter is found in the regex, we use the key corresponding to the name of the matching parser. + I introduced a new `parser` struct to group together several parameters specific to a type of log (and avoid relying too much on map key string matching and / or if statements for specific use cases) + I've been successfully running this code in several AWS environments for days. + I corrected a typo from my previous PR #9497 (wrong PR number in Changelog.md) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Salva Corts	c6fbff26e1	Add config to avoid caching stats for recent data (#9537) ... **What this PR does / why we need it**: When we query the stats for recent data, we query both the ingesters and the index gateways for the stats. ebdb2b1800/pkg/storage/async_store.go (L112-L114) ebdb2b1800/pkg/storage/async_store.go (L126-L127) Then we merge all the responses, which means summing up all the stats ebdb2b1800/pkg/storage/async_store.go (L157-L158) ebdb2b1800/pkg/storage/stores/index/stats/stats.go (L23-L26) Because we have a replication factor of 3, this means that we will get the stats from the ingesters repeated up to 3 times, hence inflating the stats. In the stats cache, we store the stats for a given matcher set for the whole day, then we extract the stats from the cache by the factor of time from the request that is stored in the cache: 336283acad/pkg/querier/queryrange/index_stats_cache.go (L33) 336283acad/pkg/querier/queryrange/index_stats_cache.go (L40) Inflated stats for recent data will be cached, so subsequent stats extracted from the cache will be inflated regardless of the time. This PR adds a new per-tenant limit `max_stats_cache_freshness` to not cache requests with an end time that falls within Now minus this duration. Here's a scenario illustrating this. The graphs below show the bytes stats queried in the sharding middleware. We are running a log filter query that won't match any log, every 5 seconds with a length of 3h.  As can be seen, after enabling the stats cache and configuring`do_not_cache_request_within` to not cache stats for requests within 30m, the bytes stats used in the sharding middleware stopped increasing. In both cases the stats cache hit ration was 100%.  **Special notes for your reviewer**: - Blocked by https://github.com/grafana/loki/pull/9535 - Note that this PR doesn't fix the root issue of inflated stats form the ingesters, but rather buys us some time to work on that. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Michel Hollands	22779e13c5	Fix date template function with epoch times (#8886) ... **What this PR does / why we need it**: Adds new toUnixEpoch... functions to convert from a string with a Unix/Epoch time to an integer that can be used in the existing `toDate` function. Note that these are the opposites of some of the functions introduced in https://github.com/grafana/loki/pull/8774. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/8624. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Bruno FERNANDO	1b410dbc01	feat(promtail): add CF ClientRequestSource field (#9669) ... **What this PR does / why we need it**: Hey folks 👋 Little contribution here to add a useful log field for cloudflare users. Indeed I add the [ClientRequestSource field](https://developers.cloudflare.com/logs/reference/clientrequestsource/ ) which is pretty useful when debugging some specific traffic handled by cloudflare Extra: Since I was on the documentation I fixed an indentation issue that I spotted Don't hesitate to reach me if you have any questions Cheers 😉 **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: Loki rocks 🚀 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Gregor Zeitlinger	b1917a6178	add "alignLeft" and "alignRight" functions (#9672) ... Fixes https://github.com/grafana/loki/issues/9667	3 years ago
Pepe Cano	4cebc2d85c	Docs: replace `k6 Cloud` mention (#9599) ... k6 is now available as a managed service on Grafana Cloud. This is a small doc changes to remove the mention of `k6 Cloud`. --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Dylan Guedes	609bc22933	Distributor: Make key configurable when logging failures (#9659) ... **What this PR does / why we need it**: Make appending `insight=true` key-value pair to log failures configurable. **Which issue(s) this PR fixes**: N/A	3 years ago
Nils Griebner	d58125885b	Make table manager retention options configurable in helm chart values (#9647) ... **What this PR does / why we need it**: Configuration options for table manager retention are hard-coded in the helm chart values at the moment so that it's not possible to enable retention deletes. **Which issue(s) this PR fixes**: Fixes #8676 **Special notes for your reviewer**: - **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Salva Corts	1694ad0f9b	Stats cache can be configured independently (#9535) ... **What this PR does / why we need it**: Before this PR, the index stats cache would use the same config as the query results cache. This was a limitation since: 1. We would not be able to point to a different cache for storing the index stats if needed. 2. We would not be able to add specific settings for this cache, without adding it to the results cache. In this PR, we refactor the index stats cache config to be independently configurable. Note that if it's not configured, it will try to use the results cache settings. **Which issue(s) this PR fixes**: This is needed for: - https://github.com/grafana/loki/pull/9537 - https://github.com/grafana/loki/pull/9536 **Special notes for your reviewer**: - This PR also refactors all the tripperwares in rountrip.go to reuse the same stats tripperware instead of each one creating their own. - Configuring a new cache in rountrip.go is a requirement for https://github.com/grafana/loki/pull/9536 so the stats summary can distinguish before the stats cache and the results cache. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Eric	c6f809a738	Fix missing ~ in the regexp match for namespace (#9618) ... **What this PR does / why we need it**: Fix documentation which has a missing `~` for the regexp match. I was following the documentation and found it odd that this query didn't follow [LogQL](https://grafana.com/docs/loki/latest/logql/log_queries/#log-queries), but it actually does and was just a typo in the docs. I've tested this with my own Loki 2.8 release and with the `~` this works great! I also added a similar description block to the above example explaining the pipeline, but if this is not desired, I'm happy to remove it. **Which issue(s) this PR fixes**: NA, trivial issue, just a small fix in the docs. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Juhyeong Ko	a21842d740	Fix typo in log_queries docs (#9506)	3 years ago
Danny Kopping	a248e3e970	Querier: configurable writeback queue bytes size (#9604) ... **What this PR does / why we need it**: The background writeback process is used to write chunks that have been fetched from the store to cache. Chunks have a target size, but not all chunks will necessarily be this size, so it is tricky to find the correct value for `writeback_buffer` which specifies how many chunks will be held in the queue at any given time. For example: setting `writeback_buffer=10000` (the default) with all chunks being the default target size (`chunk_target_size=1.5MiB`), in the worst case could result in 15GiB of memory being used for this buffer. Practically speaking, the queue never grows so deep, but if the cache Loki is writing to becomes overwhelmed this could happen. Setting a value too low would result in many chunks not being written back to cache, which defeats the purpose. This PR introduces a configurable _byte size_ limit to the writeback cache, which defaults to 1GB in size. With this setting `writeback_size_limit` in place, we get the best of both worlds: a deep queue which can hold many small chunks or a shallow queue which can hold a few large ones without using too much memory so as to overwhelm the querier. If either the `writeback_buffer` or `writeback_size_limit` reaches capacity, any new chunks which are attempted to be added to the queue will be dropped. Two new metrics are added by this PR: - `loki_cache_background_queue_bytes`: gauge, measures the size of the queue in bytes - `loki_cache_dropped_background_writes_bytes_total`: counter, measures the volume of bytes dropped	3 years ago
FL42	3fc9328862	Fix: HPA for single binary deployment (#9585) ... **What this PR does / why we need it**: Add HPA for single binary deployment. Off by default. **Which issue(s) this PR fixes**: Fixes #7097 **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [N/A] Documentation added - [N/A] Tests updated - [X] `CHANGELOG.md` updated - [N/A] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [X] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Tomasz Abramczyk	1b3f97fde4	Loki helm: Add Ingress labels for Loki gateway helm chart (#9395) ... Hello! I have small change suggestion. I think its too small to add documentation/changelog note, but if I should then just say and I can add. :) **What this PR does / why we need it**: it adds possibility to set labels for Loki gateway Ingress. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Dylan Guedes	7d67b6306f	Loki: Implement initial phase of `limited_log_push_errors` (#9556) ... **What this PR does / why we need it**: Adds an initial implementation of `limited_log_push_errors`. This initial implementation encompass: - Runtime per-tenant configuration - Simple per-tenant rate-limiting based on error message size Notable features that will be added in future phases: - Instead of a single error with the final string, give to the manager the list of all entries - Once it supports per-entry error, make the rate-limiting separated per-reason - Hash the entry error and avoid repeating errors by caching seen errors in memory	3 years ago
oleksii-boiko-ua	558cebf7a2	helm: fix backend hpa missing parameter (#9548) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #9547 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Christophe Collot	b709b32d6a	feat(lambda-promtail): add cloudtrail log ingestion support (#9497) ... **What this PR does / why we need it**: ### Add support for AWS CloudTrail audit logs ingestion using lambda-promtail Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for security and debugging purposes. However, I've experienced difficulties with it: + The AWS CloudTrail service is not well integrated with Prometheus (no metrics, no alerts) and I don't want to manage alerts in CloudWatch Alerts + The search experience is painful with CloudTrail via the AWS Console (I will not elaborate 😅). This PR allows ingesting CloudTrail audit logs sent to an S3 bucket using the same approach as VPC flow logs or Load Balancer logs. **Special notes for your reviewer**: + Because the Cloudtrail file format is not text but json, we stream the json CloudTrail records instead of using the already existing scanner. + Because the Cloudtrail filename format is not the same as for the Flow log or the Load balancer log files, we need to split the regexes by service (although many AWS services seem to share the same `defaultFilenameRegex`). + In the `getLabels` function, we expect the `type` parameter to be found in the filename using the Regex. For some log files (ex: Cloudfront log files). The file name has no reference to the service name. This is why, as a default, when no type is found in the name of the file, I set it to use the key of the matching Regex expression. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Ryan Garcia	028c12f624	[Helm] configStorageType value support (#8526) ... **What this PR does / why we need it**: Adds `loki.configStorageType` & `loki.externalConfigSecretName` values and associated helpers and templates to support choosing either `ConfigMap` or `Secret` as config medium types. **Which issue(s) this PR fixes**: Fixes #8268 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: garcia.ryan <garcia.ryan@solute.us> Co-authored-by: Stefan Büringer <4662360+sbueringer@users.noreply.github.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	3 years ago
risson	41c19ec747	helm: support of annotations to deployments and statefulsets (#9509) ... **What this PR does / why we need it**: Allows annotating loki components with custom annotations. For instance: ```yaml loki: annotations: secret.reloader.stakater.com/reload: loki-aws-credentials ``` **Which issue(s) this PR fixes**: No issue **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - Helm chart section is empty. - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [x] N/A Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>	3 years ago
BlakeFernandez	905d14bee7	Add Prometheus Link for scrape_configs (#9270) ... **What this PR does / why we need it**: Adds some context to the scrape configs section in Promtail Config Docs to ensure clarity around the connection to Prometheus Here is a link to a slack thread where the customer was confused on this: https://raintank-corp.slack.com/archives/C04KY6NJSCE/p1681761335754199 --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago