360 Commits (ea798e0f2a3364b4a76f153faf324b4a9ababc4d)

Author	SHA1	Message	Date
Danny Kopping	f5c0fdf028	Upgrade to latest security release: go1.20.4 (#9370) ... **What this PR does / why we need it**: https://groups.google.com/g/golang-announce/c/MEb0UyuSMsU Also updated the build image. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Paul Rogers	6a4ba250a6	Bump grafana and grafana-enterprise versions used for GEL Helm env (#9368) ... **What this PR does / why we need it**: Bumps the version of Grafana/Grafana-Enterprise being used to what is currently current. **Which issue(s) this PR fixes**: This gets rid of a warning displayed about "The current Grafana version is not supported" in relation to the GEL plugin.	2 years ago
Trevor Whitney	50bc1cee71	Rename usagestats to analytics (#9212) ... **What this PR does / why we need it**: Rename usagestats to analytics. The config section in the yaml is already called analytics, this makes things more consistent. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	594ca50f6f	Fix GF_INSTALL_PLUGINS syntax for k3d env GEL plugin (#9300) ... **What this PR does / why we need it**: Not sure if this is a change to Grafana, but while the old sytnax used to work, it no longer does, and needs this `;grafana-enterprise-logs-app` suffix, which appears to fix things.	2 years ago
Owen Diehl	08af8849ee	tsdb migrate versions (#9275) ... Update of https://github.com/grafana/loki/pull/9198 while @sandeepsukhani is on holiday :) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Trevor Whitney	a11921bbb8	Upgrade alpine 3.16.4 -> 3.16.5 (#9264) ... **What this PR does / why we need it**: Upgrade alpine version to address [openssl CVEs](https://www.alpinelinux.org/posts/Alpine-3.17.3-released.html) **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	0f4a7f4d92	[helm] Bump GEL version, default to 3 scalable targets (#9170) ... **What this PR does / why we need it**: This PR bumps the GEL version to match the Loki version that was added in version 5 of the helm chart. With these new versions we can now default to 3 targets in scalable mode `read`, `write`, and `backend`. This PR also fixes 2 bugs when running in scalable mode: * the gateway should forward requests to the admin api to the `backend` target when running 3 targets * the index gateway should default to using `ring` mode, which will allow the queriers in the new `read` target to find the index gateway address for their index gateway clients via the ring.	2 years ago
Owen Diehl	dc9ca8e19a	Tools/index analyzer (#9074) ... Adds a tool that allows downloading and querying statistics across TSDB indices. Example output: ``` analyzed 1046302 series and 7607495 chunks for an average of 7.270841 chunks per series. max chunks/series was 45695. number of series with over 1k chunks: 15 ``` --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Trevor Whitney	38c9c6c460	Bump go from 1.20.1 -> 1.20.3 (#9106) ... **What this PR does / why we need it**: Bumps our go dependency from 1.20.1 to 1.20.3 to address [security fixes](https://groups.google.com/g/golang-announce/c/Xdv6JL9ENs8) **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	e82885da50	Change loki version in docs for 2.8 release (#8985) ... Changes version references in Loki docs for the upcoming release Also adds a script for diffing metrics when doing a release.	2 years ago
Anton Patsev	a4aeb0e651	Update rbac.authorization.k8s.io/v1beta1 to rbac.authorization.k8s.io/v1 (#8270) ... rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole Signed-off-by: Anton Patsev <patsev.anton@gmail.com> Which of these should I fill out in this PR ? **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Anton Patsev <patsev.anton@gmail.com>	2 years ago
Trevor Whitney	f9c90a0e43	Diff configs script (#8944) ... **What this PR does / why we need it**: I always forget this trick for diffing configs when preparing a new release so I added it to a script.	2 years ago
Athira Varghese	ffb961c439	feat(storage): add support for IBM cloud object storage as storage client (#8826) ... **What this PR does / why we need it**: Add support for IBM cloud object storage as storage client **Which issue(s) this PR fixes**: Fixes NA **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated --------- Signed-off-by: Shahul <shahulsonhal@gmail.com> Co-authored-by: Aman Kumar Singh <amankrsingh2110@gmail.com> Co-authored-by: Suruthi-G-K <shruthi.suruthi@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com> Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Tareq Al-Maamari <tariq.mamari@gmail.com>	2 years ago
Christian Haudum	b502ee23ad	Remove ./tools/fetch-tags invocation from Makefile (#8854) ... **What this PR does / why we need it**: This invocation causes problems when make targets are executed in the build image container (`BUILD_IN_CONTAINER=true`), e.g. the `yacc` target, because git tries to fetch tags from `origin` using ssh, causing error message like these: ``` Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. error: Could not fetch origin ``` This is because local ssh keys are not mounted into the build container. In order not to break CI builds (https://github.com/grafana/loki/pull/8232) we need to manually fetch the tags as the first step in release pipelines. This is added to the Drone configuration. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Mitch Hulscher	257225b486	refactor(lambda-promtail): apply terraform best practices (#8750) ... **What this PR does / why we need it**: I would like to offer this PR as a suggestion to improve the lambda-promtail terraform-module. I forked it to be able to deploy it more than once in an AWS account. I also applied terraform best-practices. I was hoping that perhaps these changes could be merged into upstream as well. Unlike https://github.com/grafana/loki/pull/8549 , I unfortunately did not end up making a separate commit for each change. If you would like me to create one or more issue(s) to address the points below, I'd be happy to do that as well. List of improvements: 1. Added `var.name` (defaults to lambda-promtail) so that this module can be deployed multiple times in the same AWS account. This allows us to define unique, non-conflicting names for: * the Lambda function * the CloudWatch log-group * the IAM role 2. Split IAM role policies per component; only assign permissions when required 3. Scope down permissions of the IAM role policies 4. During terraform-destroy, ensure CloudWatch log-group is removed **after** the lambda-function. An accidental invocation of the function could re-create an already destroyed log-group, leaving an orphaned log-group List of style changes: 1. Rename resources to `this` when there is only one instance of this resource-type 2. Add newline after `count|before_each` and before `depends_on` 3. Group resources together and add a section comment 4. Add missing(?) statement-id to S3 AWS lambda permission Misc. 1. I added a `moves.tf` file to facilitate moving renamed resources in existing terraform statefiles. This prevents some resources from recreated. Can also be removed. These changes are backwards compatible, even though some resources will end up being re-created. A `terraform apply` should succeed (it did for me). **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) Signed-off-by: Mitch Hulscher <mitch.hulscher@lib.io>	2 years ago
Trevor Whitney	b46e392adb	gofmt is needed for the build (#8678) ... **What this PR does / why we need it**: `gofmt` is needed for the build, `make format` fails without it.	2 years ago
Karsten Jeschkies	8eac851962	Remove print statements introduced by #8727. (#8775)	2 years ago
Danny Kopping	56fbb3b771	Expand `which-release.sh` to cover weekly builds as well (#8768) ... **What this PR does / why we need it**: Expanding tool introduced in https://github.com/grafana/loki/pull/8706 **Special notes for your reviewer**: Knowing which, if any, weekly builds the commit is in can also be useful. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727) ... **What this PR does / why we need it**: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. **Which issue(s) this PR fixes**: Fixes #8762 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Danny Kopping	6c0f3442cc	Tools: add tool to determine which release (if any) a commit appears in (#8706) ... I've often found myself needing to determine which release a particular PR appears in. This tool makes that easy, and it accepts the merge commit ref of the PR. Example from https://github.com/grafana/loki/pull/7044, in which [a user is asking which version this feature appears in](https://github.com/grafana/loki/pull/7044#issuecomment-1455694087). ```bash $ ./tools/which-release.sh 93a5a71 Commit was found in the following releases: release-2.7.x ```	2 years ago
Trevor Whitney	dac3b84d08	Loki cloud integration instructions (and necessary mixin changes) (#8492) ... **What this PR does / why we need it**: This PR is a first stab at documentation for setting up the Grafana Cloud Loki Integration to monitor a local Loki cluster, installed using the helm chart, using Grafana Cloud. In addition to instructions on how to collect the necessary kubernetes metrics, there were also some things with the Loki mixin (which provides dashboards, alerts, and rules to the integration) that needed to be fixed. For example, our dashboards rely on there being a cluster label on our recording rules, yet our mixin does not include this? Also, we are still including `cortex-gw` panels even when the mixin is compile with `internal_components: false`. **Special notes for your reviewer**: The changes to the mixin will not be reflected in the current version of the integration. We need to merge this PR first then update the integration. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Susana Ferreira	b8118b1e7b	Helm enterprise-logs: update kubernetes secrets creation (#8467) ... **What this PR does / why we need it**: Fix the Kubernetes secrets creation for the tokengen and provisioner jobs for multiple helm installations/uninstallations. The execution flow for the helm installation is the following: 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage and writes it to the `shared` mount. 2) `tokengen` job `create-secret` container creates a new Kubernetes secret for the generated admin token. 3) `provisioner` job `provisioner` container creates the new admin resources via the Admin API and writes the tokens to the `bootstrap` mount. 4) `provisioner` job `create-secret` container creates new Kubernetes secrets for each token of the additional tenants. ## Previous implementation Uninstall and re-install the enterprise-logs helm: 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage, and writes it to the `shared` mount. This is because this job uses an [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) mount which is deleted once the job completes. When a new job is created, the directory will be again empty, forcing the `tokengen` to generate a new token. 2) `tokengen` job `create-secret` container fails, because an admin token secret already exists: ``` Defaulted container "create-secret" out of: create-secret, loki (init) error: failed to create secret secrets "enterprise-logs-admin-token" already exists ``` 3) The helm installation fails ❌ ## Implementation in this PR 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage, and writes it to the `shared` mount. 2) `tokengen` job `create-secret` container updates the previously created Kubernetes secret for the admin token. 3) `provisioner` job `provisioner` container checks that the admin resources already exist and doesn't write the token files to the `bootstrap` mount. 4) `provisioner` job `create-secret` container checks that the token files are not present on the `bootstrap` mount, and does not create new secrets. 5) The helm installation succeeds ✅ **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583) ... **What this PR does / why we need it**: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	2 years ago
Christophe Collot	a013e9f342	Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231)	2 years ago
James Callahan	6d59b4144e	lambda-promtail: Various Terraform fixes (#8549) ... * lambda-promtail: omit_extra_labels_prefix is a boolean This can be changed without a migration as it hasn't been in a release yet * lambda-promtail: fix typo * lambda-promtail: declare variables as a set * lambda-promtail: add a versions file with provider requirements * lambda-promtail: remove provider block - There's no reason to specify a region here - Empty provider blocks are not needed in terraform for quite some time now (and infact they end up producing a warning about their deprecation) * lambda-promtail: bucket_names can be the empty array Without this, you get an error that `resources` cannot be the empty list * lambda-promtail: convert inline policy to a aws_iam_policy_document	2 years ago
dependabot[bot]	3d1ef31af4	Bump golang.org/x/net from 0.5.0 to 0.7.0 in /tools/lambda-promtail (#8556) ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Christian Haudum	20696522fe	lambda-promtail: Add option to omit extra labels prefix `__extra_` (#8548) ... If you want to add an extra label env, lambda-promtail would add the __extra_ prefix, so that the resulting label is __extra_env. However, if you already have the env variable in other stream and you want to have consistent label naming, you would want to omit the prefix so the resulting label is env. Closes https://github.com/grafana/loki/issues/8471 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	ff39654a99	Fix regression in lambda-promtail tool that yields incorrect label names (#8547) ... Commit 4d5678aa17 (diff-c898e72f3b15b46b32f9405b243b7d5b56b68db7bf03541fefd587dc9e49af8eL70-R90) introduced a regression that caused labels to have duplicate `_lb` parts in the label names, as described in https://github.com/grafana/loki/issues/8470 Fixes #8470 Also updated Loki to the latest commit on `main`. Previously the dependency pointed to a Git commit that was out of the tree of this repository.	2 years ago
Susana Ferreira	00d5142c40	Helm: add sleep argument to helm makefile on local development (#8502)	2 years ago
Susana Ferreira	27d5b83c24	Helm: add debug option to helm command for verbose output on local development (#8479)	2 years ago
李国忠	1fc00fa02d	[new] chunk backend: Integrate Alibaba Cloud oss (#8410)	2 years ago
Trevor Whitney	ef8b382c06	[helm] Single binary object storage (#8284) ... Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Trevor Whitney	9920a27f06	Update GEL plugin in k3d env to latest (#8378)	2 years ago
Trevor Whitney	cc39334391	[helm] Fix helm dashboards (#8261) ... Fix the dashboards shipped with the helm chart. Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Danny Kopping	ebfeba5de3	Ruler: implement rule-based and shuffle sharding (#8092)	2 years ago
Christian Haudum	1728d09cba	Use Loki build image 0.27.1 in CI (#8279) ... Second part of updating the Loki build image Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	58e29de988	Update golang to 1.19.5 (#8278) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Taehyun Hwang	b2ea1214fc	Loki: Fix version info issue that shows wrong version (#7669, #8055) (#8232)	2 years ago
Trevor Whitney	71846b0430	[helm] add local k3d loki cluster (#8218)	2 years ago
Trevor Whitney	40ed217620	Make secret job fail when not logged in (#8215) ... Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	2 years ago
Karsten Jeschkies	44e8422dc4	Enalbe multi tenant queries on Helm test chart. (#8228) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Danny Kopping	d0fd4e8ad1	Do not exit when using `stream_lag_labels` setting (#8243)	2 years ago
Trevor Whitney	b6004050a6	[helm] Fix self-monitoring tenant secret (#8198) ... **What this PR does / why we need it**: https://github.com/grafana/loki/pull/8081 introduced a bug that caused the `provisioner` job to create an incorrect secret for the self-monitoring tenant used by the canary. This fixes that bug, as well as adding a readme on how to use the `helm-cluster` k3d env. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	eb39e2645f	[helm] allow provisioned tokens to be created in different namespaces (#8081) ... **What this PR does / why we need it**: _Note_: This PR bumps the chart to 4.0, so it includes a few other breaking changes we've been waiting to make. The new helm chart has a few different ways to create tokens when running in enterprise mode: * The tokengen job creates the initial admin token * The provisioner job creates the self-monitoring tenant if enabled * The provisioner job can also create additional tenants if configured The problem with these jobs is they put the generated read/write tokens for these tenants in secrets in the same namespace as the loki deploiyment. This is problematic if you need these tokens for a specific application, for example a Grafana datasource, that is deployed in a separate namespace. This PR introduces two breaking changes, one of which is breaking as it significantly changes how both the self monitoring and additional tenants are configured. First it adds the value `enterprise.adminToken.additionalNamespaces` and moves `enterprise.adminTokenSecret` to `enterprise.adminToken.secret`. If `additionalNamespaces` are included, additional secrets containing the GEL admin token are created in those namespaces as well as the release namespace. Second, it changes the structure of defining a tenant from just a string name, to an object containing a `name` and `secretNamespace` field. The `secretNamespace` is the namespace the secret containing that tenants token(s) will be placed in. In the case of the self-monitoring token, a secret will still be created in the default namespace in addition to the `secretNamespace` specified as that token is required by multiple components.	2 years ago
Luis Felipe Vaca	d317e659c7	lambda-promtail-: fix parse kinesis timestamp (#7158)	2 years ago
Christophe Collot	ec33a891df	lambda-promtail: remove duplicate key in terraform config sample (#8137) ... **What this PR does / why we need it**: This PR removes a duplicate key in the terraform configuration sample **Which issue(s) this PR fixes**: None **Special notes for your reviewer**: This is a trivial PR	2 years ago
Trevor Whitney	712bfdd0e0	[helm] add third scalable target `backend` (#7920) ... **What this PR does / why we need it**: Adds the new 3rd scalable deployment mode target (backend) to the helm chart. This new target was added in #7650	2 years ago
chriskuchin	4d5678aa17	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868) ... This add support to the S3 parsing logic for AWS VPC Flow logs. It also makes a small change to allow not printing log lines during processing. It leaves the default of printing the log lines in place. Signed-off-by: Thomas Belian <thomas.belian@bt909.de> Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>	2 years ago
Trevor Whitney	3f9b9cc961	[helm] fix race between minio bucket job and enterprise tokengen job (#8064) ... **What this PR does / why we need it**: This PR replaces https://github.com/grafana/loki/pull/7985. It solves the same problem, which is the race condition between the minio create bucket job and the enterprise tokengen job, but without removing minio.	2 years ago
Thomas Belián	6fcc4a2712	Lambda-promtail: Add skip tls verify option (#8059) ... **What this PR does / why we need it**: This PR sets a possibility to use lambda-promtail with self signed certificates sometimes used in development systems. This setting is configurable via environment variable. **Which issue(s) this PR fixes**: Fixes #8013 Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	2 years ago