809 Commits (7aa596752d33e8a0445aec59c23642b83baf4328)

Author	SHA1	Message	Date
nervo	7aa596752d	Helm - Loki: Introduce runtimeConfig (#8179)	3 years ago
Christian Haudum	d97aaca089	[docs] Clarify for what version the Helm chart docs are applicable (#8172) ... **What this PR does / why we need it**: The docs and reference of the Helm chart in the documentation is only applicable for the Loki Helm chart `>= 3.0` and therefore does not apply to the `grafana/loki-stack` chart, which depends on the Loki Helm chart `^2.15.2` ([requirements.yaml](https://github.com/grafana/helm-charts/blob/main/charts/loki-stack/requirements.yaml#L5)). Also added a headline to the "Install Loki using Helm" index page, because it looks awkward without, see screenshot.  **Which issue(s) this PR fixes**: Fixes: #8152 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Prabuddha Chakraborty	20627ad000	Helm: Add ingester config in helm values (#7733) ... **What this PR does / why we need it**: Allow configuring ingester config from helm values: https://grafana.com/docs/loki/latest/configuration/#ingester We wanted to change the default chunk_encoding, which right now not possible in the current values template. Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
TAY TS	94ee5a2930	Helm chart: Add enableServiceLinks (#7129) ... **What this PR does / why we need it**: This is to allow disable `enableServiceLinks` so that the service environments are not injected into the pods where in my scenario Loki is deployed in the same namespace as the Knative services resulted the Loki pod unable to start up due to too many service environment being injected. Reference issue for pod unable to startup due to too many services: https://github.com/kubernetes/kubernetes/issues/84539#issue-514261281 Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
Owen Diehl	a7ea9c1422	Revert "fix(logql): remove __error_details__ label on drop __error__ or other way around" (#8200) ... Reverts grafana/loki#8150 I was talking with @adityacs about reverting this PR and found the following: Looking at why tests don’t keep `__error_details__` when `__error__` is dropped and I found this line of code: ```go // LabelsResult returns the LabelsResult from the builder. // No grouping is applied and the cache is used when possible. func (b *LabelsBuilder) LabelsResult() LabelsResult { // unchanged path. if len(b.del) == 0 && len(b.add) == 0 && b.err == "" { return b.currentResult } return b.toResult(b.labels()) }``` https://github.com/grafana/loki/blob/main/pkg/logql/log/labels.go#L314 We only add err details when err isn’t empty anyway, so I guess we’ll revert for simplicity but just keep the same functionality	3 years ago
Kaviraj Kanagaraj	5031ae1b4e	doc(promtail): Doc about how log rotate works with promtail. (#8096) ... Changes 1. Why log rotation matters? 2. Different components involving in reading, writing and rotating logs 3. Different types of log rotation. 4. How to configure log rotation effectively to work with promtail Fixes # Should resolve all confusions around log rotation errors with promtail Related: #7967 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
openbob	af93f4a11e	Helm: Add Table Manager support to Loki helm chart (#8098) ... The Table Manager is a Loki component which takes care of creating a periodic table before its time period begins, and deleting it once its data time range exceeds the retention period. This PR is to add Table Manager support to the loki helm chart deployment as the current chart is based on the deprecated loki-simple-scalable chart that did not support Table Manager. Fixes #<6830>	3 years ago
Herman	9a516ce813	[docs] fix dead link in the docker client documentation (#8196) ... **What this PR does / why we need it**: Just fixes a dead link in the docker client docs.	3 years ago
Aditya C S	cf23194d05	fix(logql): remove __error_details__ label on drop __error__ or other way around (#8150)	3 years ago
Karsten Jeschkies	f24da4a194	Display replicas, cores and nodes in sizing tool. (#8088) ... **What this PR does / why we need it**: No every users wants to deploy Loki with Helm. That's why we are going to display the required replicas as well.  This change also fixes two bugs in the algorithm and introduces some tests for its invariant. **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: liam-howe-maersk <118169897+liam-howe-maersk@users.noreply.github.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Trevor Whitney	eb39e2645f	[helm] allow provisioned tokens to be created in different namespaces (#8081) ... **What this PR does / why we need it**: _Note_: This PR bumps the chart to 4.0, so it includes a few other breaking changes we've been waiting to make. The new helm chart has a few different ways to create tokens when running in enterprise mode: * The tokengen job creates the initial admin token * The provisioner job creates the self-monitoring tenant if enabled * The provisioner job can also create additional tenants if configured The problem with these jobs is they put the generated read/write tokens for these tenants in secrets in the same namespace as the loki deploiyment. This is problematic if you need these tokens for a specific application, for example a Grafana datasource, that is deployed in a separate namespace. This PR introduces two breaking changes, one of which is breaking as it significantly changes how both the self monitoring and additional tenants are configured. First it adds the value `enterprise.adminToken.additionalNamespaces` and moves `enterprise.adminTokenSecret` to `enterprise.adminToken.secret`. If `additionalNamespaces` are included, additional secrets containing the GEL admin token are created in those namespaces as well as the release namespace. Second, it changes the structure of defining a tenant from just a string name, to an object containing a `name` and `secretNamespace` field. The `secretNamespace` is the namespace the secret containing that tenants token(s) will be placed in. In the case of the self-monitoring token, a secret will still be created in the default namespace in addition to the `secretNamespace` specified as that token is required by multiple components.	3 years ago
JordanRushing	d8a0c6f87f	Revert "[loki-canary] Support passing loki address as environment variable (#8024)" (#8183) ... **What this PR does / why we need it**: This PR reverts commit f9ed9cb7d9 which introduced breaking config changes for the Loki canary. The relevant change introduced support for configuring a new `LOKI_ADDRESS` canary environment variable by adopting a new library for flags. This is the easiest path forward to avoid forcing breaking config changes on canary operators; such a change is likely better suited to join other configuration changes in a future major Loki release. **Which issue(s) this PR fixes**: N/A **Special notes for your reviewer**: Related PR: - https://github.com/grafana/loki/pull/8024 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Danny Kopping	4e16ea6b4d	Minor LID fixes (#8167) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Kaviraj Kanagaraj	9b5fa5e100	doc(distributor): Clarify `distributor.max-line-size` flag. (#8165) ... **What this PR does / why we need it**: Add a clarrification of two flags `distributor.max-line-size` and `distributor.max-line-size-truncate`. And clarity on [discarding the logs](https://github.com/grafana/loki/blob/main/pkg/distributor/validator.go#L84-L92) if later is disabled while still former is set to non-zero values. Whereas if `distributor.max-line-size-truncate` is set, [log line is only truncated.](https://github.com/grafana/loki/blob/main/pkg/distributor/distributor.go#L517-L534) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> **Which issue(s) this PR fixes**: Fixes #NA **Special notes for your reviewer**: Figured while working on PR #8153 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Travis Patterson	5e53303cb8	run 'make doc' and commit the changes (#8142)	3 years ago
Ludovic Cleroux	6e3b62bf3e	Documentation: Add installation instructions for Istio (#4812) ... **What this PR does / why we need it**: Because there is very little information on how to install loki in the istio service mesh, and I spent a good deal of time figuring it out. Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Nils	e4fbd3c949	Loki: Add support for Azure Workload Identity authentication (#7540)	3 years ago
Marcus Teixeira	78c2063e0b	Docs: fix helm ssd typo (#7959) ... Fix docs about helm ssd installation. Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	3 years ago
Danny Kopping	37c1ec442f	Correct template and LID (#8128) ... The extra colon causes a YAML parsing failure, see https://github.com/grafana/loki/actions/runs/3905966435/jobs/6678935989 Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Aditya C S	8df5803d9a	feat(logql): Support drop labels in logql pipeline (#7975) ... This PR introduces `drop` stage in logql pipeline. Fixes #7870, Fixes #7368	3 years ago
Danny Kopping	2c0cc01fa1	Minor fixups to LIDs (#8124) ... Including spelling my own surname right! 🙈 Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Danny Kopping	9cab63d921	Loki Improvement Documents (LIDs) (#8056) ... **What this PR does / why we need it**: This PR adds a _Loki Improvement Document_ process to engage contributors and the community on major changes to Loki, _before_ large amounts of work is done. The aim here is to have a **lightweight documented history of our decisions** which the community can view in perpetuity. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Karsten Jeschkies	907450c9c3	Enhance sizing tool documentation. (#8079)	3 years ago
Jan-Otto Kröpke	59030db186	Add toggle for persistence disk in single-binary mode (#7778)	3 years ago
HTErik	4108aedeea	Promtail Docs: Clarifiy labels and timestamps set by cri stage (#7521) ... Add description that `cri` stage not only adds labels for timestamp, but that it also sets the timestamp of the log-entry before sending it. Honestly I don't know if this is the intention or not, but this is how things are implemented today and it makes the most sense. Even if not, changing the behavior now would not be possible without breaking old implicit assumptions, so better have more explicit documentation. Also clarify that the cri-`flags` get extracted as a label.	3 years ago
Jennifer Villa	fb1ebcb01b	Fixed a typo and made a small edit to the Helm monolithic setup doc. (#8109) ... Its storage type, not filesystem type. Also "before you begin" seemed a bit vague given the list below, so I added that the list below was software requirements.	3 years ago
Trevor Whitney	712bfdd0e0	[helm] add third scalable target `backend` (#7920) ... **What this PR does / why we need it**: Adds the new 3rd scalable deployment mode target (backend) to the helm chart. This new target was added in #7650	3 years ago
chriskuchin	4d5678aa17	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868) ... This add support to the S3 parsing logic for AWS VPC Flow logs. It also makes a small change to allow not printing log lines during processing. It leaves the default of printing the log lines in place. Signed-off-by: Thomas Belian <thomas.belian@bt909.de> Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>	3 years ago
Chris Hodges	eb6ba615f4	Promtail: Add configuration to drop batches when rate limited by Loki (#7973) ... This change will allow for more deterministic behavior by multitenant Promtail instances where one or more tenants are being heavily throttled due to limits exceeded within Loki. Specifically, it will reduce the performance impact of having exponential backoff cause HOL blocking for rate limit response codes while other tenants are ready to send batches. Co-authored-by: chodges15 <chris.hodges@gmail.org>	3 years ago
Karsten Jeschkies	4464abbb70	Use GiB and bytes for sizing tool. (#8078)	3 years ago
MrWong99	380cabaf8c	Fixed typo in Helm values docs (#8075)	3 years ago
Nils	c474f265a7	Add config for optional pod labels (#7594)	3 years ago
Xinbin Huang	7524c4b3dc	[Helm] Add the ability to add extraObjects (#8004) ... **What this PR does / why we need it**: Add the ability to create extraObjects in helm chart. This'll allow users to inject a configmap of loki rules or secrets for object storages. Also, the `extraObjects` is also a common fields for helm charts (i.e. https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L1148)	3 years ago
J Stickler	ca8fd8dfb7	[Docs] Remove outdated canary logo (#8035) ... Remove the rather outdated canary logo on the page describing the Loki canary app. https://grafana.com/docs/loki/latest/operations/loki-canary/ **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [X] Documentation added - [N/A] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Travis Patterson	0db15efd50	Add stream sharding docs (#7901) ... This PR adds documentation for the new stream sharding feature. Co-authored-by: Steven Dungan <114922977+stevendungan@users.noreply.github.com>	3 years ago
Guillaume Durand	55733561f7	[helm] Add a flag to enable/disable persistentVolumeClaimRetentionPolicy feature (#7640) ... **What this PR does / why we need it**: add a flag to enable/disable persistentVolumeClaimRetentionPolicy feature which is available since kubernetes 1.23 but in alpha (StatefulSetAutoDeletePVC) and disabled by default on most cloud provider I've set the flag to true by default, to not change the actual behavior **Which issue(s) this PR fixes**: Fixes #7249	3 years ago
Ashwanth	b3e038c7a8	config_wrapper: Add support for named stores (#7946) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: This PR adds support for named stores which allows users to define multiple object store configurations for a storage type. Named stores can be referred to from other sections of the config similar to how provider type is being used `i.e aws, gcs...`. Named store reference should be of the format: `<store_type>.<key_name>` Below is an example configuration with two named stores defined for aws `store-1, store-2`. Schema config is referring to one of them as `aws.store-1` ``` storage_config: aws: endpoint: s3://common-bucket region: us-east1 access_key_id: abc123 secret_access_key: def789 named_stores: aws: store-1: endpoint: s3://foo-bucket region: us-west1 access_key_id: 123abc secret_access_key: 789def store-2: endpoint: s3://bar-bucket region: us-west2 access_key_id: 456def secret_access_key: 789abc schema_config: configs: - from: "2020-07-31" index: period: 24h prefix: loki_prod_index_ object_store: aws.store-1 schema: v11 store: boltdb-shipper ``` **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: Configuration defined in common storage config has no effect on the named stores. Common storage config only overrides storageConfig if it's not explicitly set - same behavior as before. This should be a non-intrusive change. Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Kim Nylander	4d97dc0cbb	[DOC] Fix broken links in docs (#8065) ... **What this PR does / why we need it**: This PR fixes several broken links in the Promtail, operations, and install sections of the doc.	3 years ago
Ed Welch	2b66c6b126	Loki: add a 'since' query parameter for easier querying using relative time (#7964) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: logcli has a command line parameter called `since` which makes it easy to query logs relative to the current time, however this is some syntactic sugar used in logcli to calculate `start` and `end` query parameters for the query sent to Loki. This PR adds native support for a `since` query parameter to make it a little easier to consume the Loki query API without having to calculate precise start/end times with every request. This is useful if you are polling the Loki API from a third party app which doesn't let you easily build dynamic URL's and all you want to do is poll for say the last 3 hours of logs. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Jijo Thomas John	f9ed9cb7d9	[loki-canary] Support passing loki address as environment variable (#8024) ... **What this PR does / why we need it**: - Replace flag with kingpin which has more capabilities to override defaults - Override default for `addr`, `user` and `pass` from environment the variables `LOKI_ADDRESS`, `LOKI_USERNAME` and `LOKI_PASSWORD` respectively. **Which issue(s) this PR fixes**: Fixes #8021 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Quentin Plessis	fde44609c0	[Helm] Add the ability to specify container lifecycle (#8010) ... **What this PR does / why we need it**: Add the ability to specify container lifecycle in helm chart. This is useful when pods are served by a load balancer that needs time to deregister terminating pods, as we can setup a preStop hook to wait a bit before terminating the container. Example: ```yaml gateway: lifecycle: preStop: exec: command: ["/bin/sh", "-c", "sleep 30"] ``` **Which issue(s) this PR fixes**: None **Special notes for your reviewer**: None **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Ammar Lakis	2a2d1f9178	Docs: Fix promtail installation doc (#8007) ... **What this PR does / why we need it**: Fix promtail installation command using helm in the docs. **Which issue(s) this PR fixes**: None **Special notes for your reviewer**: None **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Ammar Lakis <ammar.lakis@gmail.com>	3 years ago
Constantin Winkler	bc71d81f66	Docs: Removed unnecessary s3 value from example (#8017) ... **What this PR does / why we need it**: removes a confusing value field that isn't necessary. more clarity closes #8036 **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	b4440911e0	[new feature] logql:support sort and sort_desc (#7989) ... **What this PR does / why we need it**: sort() sort(v instant-vector) returns vector elements sorted by their sample values, in ascending order. sort_desc() Same as sort, but sorts in descending order. ex: `sort(rate(({app=~"foo|bar"} |~".+bar")[1m])) without (app) ` prometheud doc: https://prometheus.io/docs/prometheus/latest/querying/functions/#sort **Which issue(s) this PR fixes**: Fixes #7933 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Christian Haudum	17577b4d47	Shut down query frontend gracefully (#7978) ... **What this PR does / why we need it**: This PR fixes a bug in the shutdown process of the query frontend. In order to shut down the frontend gracefully without cancelling any in-flight operations/sub-requests, the frontend service has to wait until all in-flight operations are finished. Only then its sub-services may be stopped. In order to control the deterministic order of the shutdown of the service and its child services, child services need to be started with their own context (not with their parent context). Otherwise, if the context of the frontend is cancelled, also the contexts of the child services are cancelled, leading to a parallel stopping of the parent and its child services. During the time of the shutdown, the `RoundTripGRCP` function still needs to accept any round-tripping requests. This is, because the frontend round tripper is wrapped into a query range tripper ware, which splits incoming (HTTP) query requests into multiple (gRPC) query requests if needed. **Which issue(s) this PR fixes**: This fixes a race condition where queries fail, because the query frontend shut down before the queriers that processed in-flight request could send the query response to the frontend. This manifested in an increased error rate of queries during restarts (rollouts) of the query frontend. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
irizzant	ff7b462973	updated versions to the latest release v2.7.1 (#7940) ... Signed-off-by: irizzant <i.rizzante@gmail.com> **What this PR does / why we need it**: This PR updates Loki version to v2.7.1 **Which issue(s) this PR fixes**: Fixes #7768 Signed-off-by: irizzant <i.rizzante@gmail.com>	3 years ago
Susana Ferreira	0f139e2cb3	Update log queries regex documentation (#8002) ... **What this PR does / why we need it**: Update log queries regex documentation with a clarification on the difference between the newline character in: * label matching operator: `.*` does not match newlines * line filter expressions: `.*` matches newlines **Checklist** - [x] Documentation updated	3 years ago
Michel Hollands	59eff5ca58	Add count to template functions (#7951) ... **What this PR does / why we need it**: Add a `count` template function to count occurrences of a substring. For example if `label="abc abd abc"` ``` {{ .label | count "abc" }} ``` would return 2. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Susana Ferreira	d086c237df	Add missing period_config root block to doc generator (#7981) ... **What this PR does / why we need it**: Update doc generator tool to validate if slice element is a root block. Add missing `period_config` root block to doc generator tool. **Special notes for your reviewer**: **Checklist** - [ ] Documentation updated	3 years ago
Danny Kopping	a4f306399a	Add store & cache download statistics (#7982)	3 years ago