apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Karsten Jeschkies	b35bbd80d6	Support content negotiation between query frontend and querier. (#9813 ) What this PR does / why we need it: Currently, the querier sends results to the query frontend in JSON which is then decoded to Protobuf. It is more efficient to send the results as Protobuf. This will also allow to extend the results with custom data structures. The change is backwards compatible through content negotiation. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Trevor Whitney	d74c67a9d0	Add version to TSDB index builder struct (#9566 ) Add version as a property to the TSDB Index Builder, forcing users of the builder to specify a version for each builder instance. This is the first of many PRs that will add the TSDB index version to the schema version config.	2 years ago
Dan Poltawski	093ec20b43	packaging: fix postinst on debian oses (#9706 ) Previously the systemd version extract pulled in debian string breaking the detection on debian variants What this PR does / why we need it: Fixes postinst error on Debian variants Which issue(s) this PR fixes: Fixes #9701 #9302 Special notes for your reviewer: Tests on various OS variants: Debian/12 ``` $ systemctl --version systemd 252 (252.6-1) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified $ systemctl --version \| head -1 \| sed 's/systemd $[0-9]\+$./\1/' 252 ``` CentOS/7 ``` $ systemctl --version systemd 219 +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN $ systemctl --version \| head -1 \| sed 's/systemd $[0-9]\+$./\1/' 219 ``` Debian/11 ``` $ systemctl --version systemd 247 (247.3-7+deb11u2) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified $ systemctl --version \| head -1 \| sed 's/systemd $[0-9]\+$./\1/' 247 ``` Ubuntu/20.04 ``` $ systemctl --version systemd 245 (245.4-4ubuntu3.20) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid $ systemctl --version \| head -1 \| sed 's/systemd $[0-9]\+$./\1/' 245 ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Deepansh Mathur	50126731f8	Update README.md (#9698 ) Backticks were missing for a code block What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	2 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588 ) For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	2 years ago
Christophe Collot	f2394355d8	feat(lambda-promtail): add cloudfront log file ingestion support (#9573 ) What this PR does / why we need it: This PR enables ingesting logs from Cloudfront log files stored in s3 (batch). The current setup only supports streaming Cloudfront logs through AWS Kinesis, whereas this PR implements the same flow as for VPC Flow logs, Load Balancer logs, and Cloudtrail logs (s3 --> SQS (optional) --> Lambda Promtail --> Loki) Special notes for your reviewer: + The Cloudfront log file format is different from the already implemented services, meaning we had to build yet another regex. AWS never bothered making all services follow the same log file naming convention but the "good" thing is that it's now very unlikely they will change it in the future. + The Cloudfront file name does not have any mention of the AWS account or the time of log it contains, it means we have to infer the log type from the filename format instead of finding the exact string "cloudfront" in the filename. This is why in `getLabels`, if no `type` parameter is found in the regex, we use the key corresponding to the name of the matching parser. + I introduced a new `parser` struct to group together several parameters specific to a type of log (and avoid relying too much on map key string matching and / or if statements for specific use cases) + I've been successfully running this code in several AWS environments for days. + I corrected a typo from my previous PR #9497 (wrong PR number in Changelog.md) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Ashwanth	98d1307c7d	config: ensure storage config defaults apply to named stores (#9650 ) What this PR does / why we need it: Since named store config does not register any flags, storage configs defined under it do not get the defaults. For example [aws_storage_config](https://grafana.com/docs/loki/latest/configuration/#aws_storage_config) sets the default `storage_class` to `STANDARD`, but the same doesn't get applied by default when using named stores. This PR ensures that named storage configs are always assigned default values when they are unmarshalled by implementing `yaml.Unmarshaler` interface	2 years ago
Karsten Jeschkies	c9a3ff53e4	Set unique timestamp in lambda-promtail. (#9560 )	2 years ago
Christophe Collot	b709b32d6a	feat(lambda-promtail): add cloudtrail log ingestion support (#9497 ) What this PR does / why we need it: ### Add support for AWS CloudTrail audit logs ingestion using lambda-promtail Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for security and debugging purposes. However, I've experienced difficulties with it: + The AWS CloudTrail service is not well integrated with Prometheus (no metrics, no alerts) and I don't want to manage alerts in CloudWatch Alerts + The search experience is painful with CloudTrail via the AWS Console (I will not elaborate 😅). This PR allows ingesting CloudTrail audit logs sent to an S3 bucket using the same approach as VPC flow logs or Load Balancer logs. Special notes for your reviewer: + Because the Cloudtrail file format is not text but json, we stream the json CloudTrail records instead of using the already existing scanner. + Because the Cloudtrail filename format is not the same as for the Flow log or the Load balancer log files, we need to split the regexes by service (although many AWS services seem to share the same `defaultFilenameRegex`). + In the `getLabels` function, we expect the `type` parameter to be found in the filename using the Regex. For some log files (ex: Cloudfront log files). The file name has no reference to the service name. This is why, as a default, when no type is found in the name of the file, I set it to use the key of the matching Regex expression. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Ashwanth	f08794d39a	docs: add memberlist section to the configuration page (#9452 ) What this PR does / why we need it: `memberlist_config` section is not present in 2.8.x release, must have gotten removed by mistake when we moved to the doc generation tool. This PR adds it back. Which issue(s) this PR fixes: Fixes #9423 Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Christophe Collot	1c7f791a1e	lambda-promtail: add terraform code to process s3 logs through SQS (#8700 ) ## Add Terraform sample code to process AWS S3 log files through an SQS queue Lambda-promtail supports processing AWS s3 logs files through an SQS queue since #8231 As explained in the [documentation section of lambda-promtail](https://github.com/grafana/loki/blob/main/docs/sources/clients/lambda-promtail/_index.md#triggering-lambda-promtail-via-sqs:~:text=Triggering%20Lambda%2DPromtail,DLQ%20redrive%20feature.), this can be leveraged to re-process logs that lambda-promtail failed to process (or send to loki) using a main SQS queue and a secondary SQS dead-letter queue. AWS has a feature called `SQS redrive`, which enables routing messages pending in the DLQ back to the main (source) queue. This PR demonstrates how to spin up this architecture using terraform.	2 years ago
Salva Corts	bf56d9b540	Fix docker image tag in release template (#9425 ) What this PR does / why we need it: Releases in GitHub are published automatically from Drone. Drone calls the `tools/release` script which uses the `tools/release-note.md` template to generate the release body. Before this PR, the template used the `$DRONE_TAG` env variable which had a `v` preffix (e. `v2.8.0`). Since we publish the docker images without the `v` preffix, the docker images referenced by the release did not exist. This affected the `v2.8.0`, `v2.8.1` and `v2.8.2` releases. I manually fixed the `v2.8.0`, `v2.8.1` and `v2.8.2` release pages, but this PR should fix the template so we no longer run into this situation. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/9384 Special notes for your reviewer: Not sure where to find the available env variables in Drone. I wonder if there is an existing env var with the version without the `v` preffix. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Danny Kopping	f5c0fdf028	Upgrade to latest security release: go1.20.4 (#9370 ) What this PR does / why we need it: https://groups.google.com/g/golang-announce/c/MEb0UyuSMsU Also updated the build image. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Paul Rogers	6a4ba250a6	Bump grafana and grafana-enterprise versions used for GEL Helm env (#9368 ) What this PR does / why we need it: Bumps the version of Grafana/Grafana-Enterprise being used to what is currently current. Which issue(s) this PR fixes: This gets rid of a warning displayed about "The current Grafana version is not supported" in relation to the GEL plugin.	2 years ago
Trevor Whitney	50bc1cee71	Rename usagestats to analytics (#9212 ) What this PR does / why we need it: Rename usagestats to analytics. The config section in the yaml is already called analytics, this makes things more consistent. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	594ca50f6f	Fix GF_INSTALL_PLUGINS syntax for k3d env GEL plugin (#9300 ) What this PR does / why we need it: Not sure if this is a change to Grafana, but while the old sytnax used to work, it no longer does, and needs this `;grafana-enterprise-logs-app` suffix, which appears to fix things.	2 years ago
Owen Diehl	08af8849ee	tsdb migrate versions (#9275 ) Update of https://github.com/grafana/loki/pull/9198 while @sandeepsukhani is on holiday :) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Trevor Whitney	a11921bbb8	Upgrade alpine 3.16.4 -> 3.16.5 (#9264 ) What this PR does / why we need it: Upgrade alpine version to address [openssl CVEs](https://www.alpinelinux.org/posts/Alpine-3.17.3-released.html) Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	0f4a7f4d92	[helm] Bump GEL version, default to 3 scalable targets (#9170 ) What this PR does / why we need it: This PR bumps the GEL version to match the Loki version that was added in version 5 of the helm chart. With these new versions we can now default to 3 targets in scalable mode `read`, `write`, and `backend`. This PR also fixes 2 bugs when running in scalable mode: * the gateway should forward requests to the admin api to the `backend` target when running 3 targets * the index gateway should default to using `ring` mode, which will allow the queriers in the new `read` target to find the index gateway address for their index gateway clients via the ring.	2 years ago
Owen Diehl	dc9ca8e19a	Tools/index analyzer (#9074 ) Adds a tool that allows downloading and querying statistics across TSDB indices. Example output: ``` analyzed 1046302 series and 7607495 chunks for an average of 7.270841 chunks per series. max chunks/series was 45695. number of series with over 1k chunks: 15 ``` --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Trevor Whitney	38c9c6c460	Bump go from 1.20.1 -> 1.20.3 (#9106 ) What this PR does / why we need it: Bumps our go dependency from 1.20.1 to 1.20.3 to address [security fixes](https://groups.google.com/g/golang-announce/c/Xdv6JL9ENs8) Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Trevor Whitney	e82885da50	Change loki version in docs for 2.8 release (#8985 ) Changes version references in Loki docs for the upcoming release Also adds a script for diffing metrics when doing a release.	2 years ago
Anton Patsev	a4aeb0e651	Update rbac.authorization.k8s.io/v1beta1 to rbac.authorization.k8s.io/v1 (#8270 ) rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole Signed-off-by: Anton Patsev <patsev.anton@gmail.com> Which of these should I fill out in this PR ? Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Anton Patsev <patsev.anton@gmail.com>	2 years ago
Trevor Whitney	f9c90a0e43	Diff configs script (#8944 ) What this PR does / why we need it: I always forget this trick for diffing configs when preparing a new release so I added it to a script.	2 years ago
Athira Varghese	ffb961c439	feat(storage): add support for IBM cloud object storage as storage client (#8826 ) What this PR does / why we need it: Add support for IBM cloud object storage as storage client Which issue(s) this PR fixes: Fixes NA Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated --------- Signed-off-by: Shahul <shahulsonhal@gmail.com> Co-authored-by: Aman Kumar Singh <amankrsingh2110@gmail.com> Co-authored-by: Suruthi-G-K <shruthi.suruthi@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com> Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Tareq Al-Maamari <tariq.mamari@gmail.com>	2 years ago
Christian Haudum	b502ee23ad	Remove ./tools/fetch-tags invocation from Makefile (#8854 ) What this PR does / why we need it: This invocation causes problems when make targets are executed in the build image container (`BUILD_IN_CONTAINER=true`), e.g. the `yacc` target, because git tries to fetch tags from `origin` using ssh, causing error message like these: ``` Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. error: Could not fetch origin ``` This is because local ssh keys are not mounted into the build container. In order not to break CI builds (https://github.com/grafana/loki/pull/8232) we need to manually fetch the tags as the first step in release pipelines. This is added to the Drone configuration. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Mitch Hulscher	257225b486	refactor(lambda-promtail): apply terraform best practices (#8750 ) What this PR does / why we need it: I would like to offer this PR as a suggestion to improve the lambda-promtail terraform-module. I forked it to be able to deploy it more than once in an AWS account. I also applied terraform best-practices. I was hoping that perhaps these changes could be merged into upstream as well. Unlike https://github.com/grafana/loki/pull/8549 , I unfortunately did not end up making a separate commit for each change. If you would like me to create one or more issue(s) to address the points below, I'd be happy to do that as well. List of improvements: 1. Added `var.name` (defaults to lambda-promtail) so that this module can be deployed multiple times in the same AWS account. This allows us to define unique, non-conflicting names for: * the Lambda function * the CloudWatch log-group * the IAM role 2. Split IAM role policies per component; only assign permissions when required 3. Scope down permissions of the IAM role policies 4. During terraform-destroy, ensure CloudWatch log-group is removed after the lambda-function. An accidental invocation of the function could re-create an already destroyed log-group, leaving an orphaned log-group List of style changes: 1. Rename resources to `this` when there is only one instance of this resource-type 2. Add newline after `count\|before_each` and before `depends_on` 3. Group resources together and add a section comment 4. Add missing(?) statement-id to S3 AWS lambda permission Misc. 1. I added a `moves.tf` file to facilitate moving renamed resources in existing terraform statefiles. This prevents some resources from recreated. Can also be removed. These changes are backwards compatible, even though some resources will end up being re-created. A `terraform apply` should succeed (it did for me). Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) Signed-off-by: Mitch Hulscher <mitch.hulscher@lib.io>	2 years ago
Trevor Whitney	b46e392adb	gofmt is needed for the build (#8678 ) What this PR does / why we need it: `gofmt` is needed for the build, `make format` fails without it.	2 years ago
Karsten Jeschkies	8eac851962	Remove print statements introduced by #8727 . (#8775 )	2 years ago
Danny Kopping	56fbb3b771	Expand `which-release.sh` to cover weekly builds as well (#8768 ) What this PR does / why we need it: Expanding tool introduced in https://github.com/grafana/loki/pull/8706 Special notes for your reviewer: Knowing which, if any, weekly builds the commit is in can also be useful. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727 ) What this PR does / why we need it: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. Which issue(s) this PR fixes: Fixes #8762 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Danny Kopping	6c0f3442cc	Tools: add tool to determine which release (if any) a commit appears in (#8706 ) I've often found myself needing to determine which release a particular PR appears in. This tool makes that easy, and it accepts the merge commit ref of the PR. Example from https://github.com/grafana/loki/pull/7044, in which [a user is asking which version this feature appears in](https://github.com/grafana/loki/pull/7044#issuecomment-1455694087). ```bash $ ./tools/which-release.sh `93a5a71` Commit was found in the following releases: release-2.7.x ```	2 years ago
Trevor Whitney	dac3b84d08	Loki cloud integration instructions (and necessary mixin changes) (#8492 ) What this PR does / why we need it: This PR is a first stab at documentation for setting up the Grafana Cloud Loki Integration to monitor a local Loki cluster, installed using the helm chart, using Grafana Cloud. In addition to instructions on how to collect the necessary kubernetes metrics, there were also some things with the Loki mixin (which provides dashboards, alerts, and rules to the integration) that needed to be fixed. For example, our dashboards rely on there being a cluster label on our recording rules, yet our mixin does not include this? Also, we are still including `cortex-gw` panels even when the mixin is compile with `internal_components: false`. Special notes for your reviewer: The changes to the mixin will not be reflected in the current version of the integration. We need to merge this PR first then update the integration. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Susana Ferreira	b8118b1e7b	Helm enterprise-logs: update kubernetes secrets creation (#8467 ) What this PR does / why we need it: Fix the Kubernetes secrets creation for the tokengen and provisioner jobs for multiple helm installations/uninstallations. The execution flow for the helm installation is the following: 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage and writes it to the `shared` mount. 2) `tokengen` job `create-secret` container creates a new Kubernetes secret for the generated admin token. 3) `provisioner` job `provisioner` container creates the new admin resources via the Admin API and writes the tokens to the `bootstrap` mount. 4) `provisioner` job `create-secret` container creates new Kubernetes secrets for each token of the additional tenants. ## Previous implementation Uninstall and re-install the enterprise-logs helm: 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage, and writes it to the `shared` mount. This is because this job uses an [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) mount which is deleted once the job completes. When a new job is created, the directory will be again empty, forcing the `tokengen` to generate a new token. 2) `tokengen` job `create-secret` container fails, because an admin token secret already exists: ``` Defaulted container "create-secret" out of: create-secret, loki (init) error: failed to create secret secrets "enterprise-logs-admin-token" already exists ``` 3) The helm installation fails ❌ ## Implementation in this PR 1) `tokengen` job `loki` container generates a new admin token, stores it in the object storage, and writes it to the `shared` mount. 2) `tokengen` job `create-secret` container updates the previously created Kubernetes secret for the admin token. 3) `provisioner` job `provisioner` container checks that the admin resources already exist and doesn't write the token files to the `bootstrap` mount. 4) `provisioner` job `create-secret` container checks that the token files are not present on the `bootstrap` mount, and does not create new secrets. 5) The helm installation succeeds ✅ Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583 ) What this PR does / why we need it: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	2 years ago
Christophe Collot	a013e9f342	Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231 )	2 years ago
James Callahan	6d59b4144e	lambda-promtail: Various Terraform fixes (#8549 ) * lambda-promtail: omit_extra_labels_prefix is a boolean This can be changed without a migration as it hasn't been in a release yet * lambda-promtail: fix typo * lambda-promtail: declare variables as a set * lambda-promtail: add a versions file with provider requirements * lambda-promtail: remove provider block - There's no reason to specify a region here - Empty provider blocks are not needed in terraform for quite some time now (and infact they end up producing a warning about their deprecation) * lambda-promtail: bucket_names can be the empty array Without this, you get an error that `resources` cannot be the empty list * lambda-promtail: convert inline policy to a aws_iam_policy_document	2 years ago
dependabot[bot]	3d1ef31af4	Bump golang.org/x/net from 0.5.0 to 0.7.0 in /tools/lambda-promtail (#8556 ) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Christian Haudum	20696522fe	lambda-promtail: Add option to omit extra labels prefix `__extra_` (#8548 ) If you want to add an extra label env, lambda-promtail would add the __extra_ prefix, so that the resulting label is __extra_env. However, if you already have the env variable in other stream and you want to have consistent label naming, you would want to omit the prefix so the resulting label is env. Closes https://github.com/grafana/loki/issues/8471 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	ff39654a99	Fix regression in lambda-promtail tool that yields incorrect label names (#8547 ) Commit `4d5678aa17 (diff-c898e72f3b15b46b32f9405b243b7d5b56b68db7bf03541fefd587dc9e49af8eL70-R90)` introduced a regression that caused labels to have duplicate `_lb` parts in the label names, as described in https://github.com/grafana/loki/issues/8470 Fixes #8470 Also updated Loki to the latest commit on `main`. Previously the dependency pointed to a Git commit that was out of the tree of this repository.	2 years ago
Susana Ferreira	00d5142c40	Helm: add sleep argument to helm makefile on local development (#8502 )	2 years ago
Susana Ferreira	27d5b83c24	Helm: add debug option to helm command for verbose output on local development (#8479 )	2 years ago
李国忠	1fc00fa02d	[new] chunk backend: Integrate Alibaba Cloud oss (#8410 )	2 years ago
Trevor Whitney	ef8b382c06	[helm] Single binary object storage (#8284 ) Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Trevor Whitney	9920a27f06	Update GEL plugin in k3d env to latest (#8378 )	2 years ago
Trevor Whitney	cc39334391	[helm] Fix helm dashboards (#8261 ) Fix the dashboards shipped with the helm chart. Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Danny Kopping	ebfeba5de3	Ruler: implement rule-based and shuffle sharding (#8092 )	2 years ago
Christian Haudum	1728d09cba	Use Loki build image 0.27.1 in CI (#8279 ) Second part of updating the Loki build image Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	58e29de988	Update golang to 1.19.5 (#8278 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

... 3 4 5 6 7 ...

423 Commits (ccf3e4e5ecd5f237d22c82c7e3623aafd216245a)