71 Commits (4cae003ecedd474e4c15feab4ea2ef435afff83f)

Author	SHA1	Message	Date
renovate[bot]	a627fb6aa1	chore(deps): update module golang.org/x/crypto to v0.17.0 [security] (main) (#11522) ... [](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | golang.org/x/crypto | `v0.14.0` -> `v0.17.0` | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | ### GitHub Vulnerability Alerts #### [CVE-2023-48795](https://togithub.com/warp-tech/russh/security/advisories/GHSA-45x7-px36-x8w8) ### Summary Terrapin is a prefix truncation attack targeting the SSH protocol. More precisely, Terrapin breaks the integrity of SSH's secure channel. By carefully adjusting the sequence numbers during the handshake, an attacker can remove an arbitrary amount of messages sent by the client or server at the beginning of the secure channel without the client or server noticing it. ### Mitigations To mitigate this protocol vulnerability, OpenSSH suggested a so-called "strict kex" which alters the SSH handshake to ensure a Man-in-the-Middle attacker cannot introduce unauthenticated messages as well as convey sequence number manipulation across handshakes. **Warning: To take effect, both the client and server must support this countermeasure.** As a stop-gap measure, peers may also (temporarily) disable the affected algorithms and use unaffected alternatives like AES-GCM instead until patches are available. ### Details The SSH specifications of ChaCha20-Poly1305 (chacha20-poly1305@&#8203;openssh.com) and Encrypt-then-MAC (*-etm@openssh.com MACs) are vulnerable against an arbitrary prefix truncation attack (a.k.a. Terrapin attack). This allows for an extension negotiation downgrade by stripping the SSH_MSG_EXT_INFO sent after the first message after SSH_MSG_NEWKEYS, downgrading security, and disabling attack countermeasures in some versions of OpenSSH. When targeting Encrypt-then-MAC, this attack requires the use of a CBC cipher to be practically exploitable due to the internal workings of the cipher mode. Additionally, this novel attack technique can be used to exploit previously unexploitable implementation flaws in a Man-in-the-Middle scenario. The attack works by an attacker injecting an arbitrary number of SSH_MSG_IGNORE messages during the initial key exchange and consequently removing the same number of messages just after the initial key exchange has concluded. This is possible due to missing authentication of the excess SSH_MSG_IGNORE messages and the fact that the implicit sequence numbers used within the SSH protocol are only checked after the initial key exchange. In the case of ChaCha20-Poly1305, the attack is guaranteed to work on every connection as this cipher does not maintain an internal state other than the message's sequence number. In the case of Encrypt-Then-MAC, practical exploitation requires the use of a CBC cipher; while theoretical integrity is broken for all ciphers when using this mode, message processing will fail at the application layer for CTR and stream ciphers. For more details see [https://terrapin-attack.com](https://terrapin-attack.com). ### Impact This attack targets the specification of ChaCha20-Poly1305 (chacha20-poly1305@&#8203;openssh.com) and Encrypt-then-MAC (*-etm@openssh.com), which are widely adopted by well-known SSH implementations and can be considered de-facto standard. These algorithms can be practically exploited; however, in the case of Encrypt-Then-MAC, we additionally require the use of a CBC cipher. As a consequence, this attack works against all well-behaving SSH implementations supporting either of those algorithms and can be used to downgrade (but not fully strip) connection security in case SSH extension negotiation (RFC8308) is supported. The attack may also enable attackers to exploit certain implementation flaws in a man-in-the-middle (MitM) scenario. --- ### Configuration 📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy45My4xIiwidXBkYXRlZEluVmVyIjoiMzcuMTM1LjAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIn0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Trevor Whitney	8b48a18d77	Bump build image (#11608) ... **What this PR does / why we need it**: Bumps alpine in the loki build image and ass the build image to the `scan-vulnerabilities` target to make it easier to scan in the future --------- Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Elliot Dobson	0945b186ff	lambda-promtail: cloudwatch: add '__aws_log_type' label (#11335) ... **What this PR does / why we need it**: Adds the `__aws_log_type` label to AWS CloudWatch logs in `lambda-promtail`. **Which issue(s) this PR fixes**: N/A **Special notes for your reviewer**: AWS S3 & Kinesis log types already have this label. The `lambda-promtail` documentation [here](https://github.com/grafana/loki/blob/main/docs/sources/send-data/lambda-promtail/_index.md?plain=1#L154) suggests that CloudWatch logs has this label added as well, but in practice it does not AFAICT. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
renovate[bot]	10fe48b815	chore(deps): update alpine docker tag to v3.18.4 (main) (#11167) ... [](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | alpine | final | patch | `3.18.3` -> `3.18.4` | | alpine | final | minor | `3.16.7` -> `3.18.4` | --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40Ni4wIiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
keyolk	8328345a8b	lambda-promtail: fix IAM policy for clouddwatch log stream (#10909)	2 years ago
Đỗ Trọng Hải	94169a0e6b	fix(tools/lambda-promtail): Do not evaluate empty string for drop_labels (#11074) ... Fixes #11005 Signed-off-by: hainenber <dotronghai96@gmail.com>	2 years ago
renovate[bot]	ab94c70c4c	chore(deps): update module golang.org/x/net to v0.17.0 [security] (main) (#10831) ... [](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | golang.org/x/net | indirect | minor | `v0.7.0` -> `v0.17.0` | | golang.org/x/net | indirect | minor | `v0.8.0` -> `v0.17.0` | ### GitHub Vulnerability Alerts #### [CVE-2023-3978](https://nvd.nist.gov/vuln/detail/CVE-2023-3978) Text nodes not in the HTML namespace are incorrectly literally rendered, causing text which should be escaped to not be. This could lead to an XSS attack. #### [CVE-2023-39325](https://togithub.com/golang/go/issues/63417) A malicious HTTP/2 client which rapidly creates requests and immediately resets them can cause excessive server resource consumption. While the total number of requests is bounded by the http2.Server.MaxConcurrentStreams setting, resetting an in-progress request allows the attacker to create a new request while the existing one is still executing. With the fix applied, HTTP/2 servers now bound the number of simultaneously executing handler goroutines to the stream concurrency limit (MaxConcurrentStreams). New requests arriving when at the limit (which can only happen after the client has reset an existing, in-flight request) will be queued until a handler exits. If the request queue grows too large, the server will terminate the connection. This issue is also fixed in golang.org/x/net/http2 for users manually configuring HTTP/2. The default stream concurrency limit is 250 streams (requests) per HTTP/2 connection. This value may be adjusted using the golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams setting and the ConfigureServer function. #### [CVE-2023-44487](https://togithub.com/apple/swift-nio-http2/security/advisories/GHSA-qppj-fm5r-hxr3) swift-nio-http2 is vulnerable to a denial-of-service vulnerability in which a malicious client can create and then reset a large number of HTTP/2 streams in a short period of time. This causes swift-nio-http2 to commit to a large amount of expensive work which it then throws away, including creating entirely new `Channel`s to serve the traffic. This can easily overwhelm an `EventLoop` and prevent it from making forward progress. swift-nio-http2 1.28 contains a remediation for this issue that applies reset counter using a sliding window. This constrains the number of stream resets that may occur in a given window of time. Clients violating this limit will have their connections torn down. This allows clients to continue to cancel streams for legitimate reasons, while constraining malicious actors. --- ### Configuration 📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy44LjEiLCJ1cGRhdGVkSW5WZXIiOiIzNy4xOS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Đỗ Trọng Hải	ec54c723eb	feat(lambda/promtail): support dropping labels (#10755) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #10669 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: hainenber <dotronghai96@gmail.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Ashwanth	98551ceb9a	use go 1.21.3 and go-grpc v1.56.3 (#10869) ... **What this PR does / why we need it**: Fixes CVE-2023-39325 / CVE-2023-44487 https://github.com/grpc/grpc-go/releases/tag/v1.56.3 **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Michel Hollands	eb3cff4625	Update Go version to 1.21.2 in Dockerfiles (#10805) ... **What this PR does / why we need it**: Update the go version in the Dockerfiles to 1.21.2 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Pablo	222ffc52d3	Add CloudFormation template for lambda + EventBridge setup (#10540) ... **What this PR does / why we need it**: https://github.com/grafana/loki/pull/10449 added support so that lambda-promtail can handle s3 events through AWS EventBridge. This PR is a follow up adding a CloudFormation template to deploy lambda-promtail, with the corresponding EventBridge rule to deliver s3 events to the lambda. **Which issue(s) this PR fixes**: Rel https://github.com/grafana/loki/issues/10209 --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Dylan Guedes	4681386775	Update alpine versions (#10573) ... **What this PR does / why we need it**: Update alpine versions used by our images to `3.18.3` **Which issue(s) this PR fixes**: N/A **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Pablo	5034b1b337	Add support for EventBridge s3 events in lambda-promtail (#10449) ... **What this PR does / why we need it**: Lambda promtail supports s3 events, which are used for scraping several log sources such as ALB access logs. This works by configuring at the S3 bucket level "s3 event notification", that are configured to target the lambda deployment of lambda-promtail. However, if one is configuring this through CloudFormation, there's a known issue with AWS that doesn't allow to configure both the lambda, the bucket, and the notifications in the same stack. See [this issue](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/79) for details. For that, AWS introduced EventBridge notifications, which can be used to ship s3 events to a lambda deployment as well. This flow looks like: s3 -> eventbridge bus -> eventbridge rule -> lambda EventBridge has it's own message structure for s3 notifications. This PR adds a translation layer, just for `Object created` events (since they are the only ones we should take into account), so that EventBridge events can be received, and trigger the lambda as if they were from s3. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/10209 **Special notes for your reviewer**: - [x] Pending testing this with an actual deployment of the s3 -> event bridge -> lambda flow - [x] ~~Add CF template for the `s3 -> event bridge -> lambda` deployment~~ Follow up PR **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Marcus Morris	e8f4b42eaa	Lambda Promtail: Add logs:PutSubscriptionFilter to Lambda Promtail role (#10426) ... **What this PR does / why we need it**: I ran into the below error during the CloudFormation stack creation when trying to use the [example CloudFormation config](https://github.com/grafana/loki/blob/main/tools/lambda-promtail/template.yaml): `"Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function"` The issue is that the config does not give the `logs:PutSubscriptionFilter` permission to the Lambda Promtail Role. This adds that permission to fix the error **Which issue(s) this PR fixes**: **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
lpugoy	965f19a3b9	lambda-promtail: Add support for WAF logs in S3 (#10416) ... **What this PR does / why we need it**: Adds support for WAF logs in S3 in lambda-promtail **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Elliot Dobson	6a8225d533	lambda-promtail: s3: update log parsing to support NLBs (#7194) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: Fixes the filename and timestamp regex for AWS Network Load Balancer (NLB) logs and adds logic to make NLB log timestamps RFC3339 compatible. Also adds a test to check the parsed timestamp is as expected. **Which issue(s) this PR fixes**: Fixes #6455 **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Shantanu Alsi	b57d260ddc	Upgrade alpine 3.16.5 -> 3.16.7 (#10188) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #10181 Fixes Open SSL vulnerabilities [CVE-2023-1255](https://security.alpinelinux.org/vuln/CVE-2023-1255), [CVE-2023-2650](https://security.alpinelinux.org/vuln/CVE-2023-2650), [CVE-2023-2975](https://security.alpinelinux.org/vuln/CVE-2023-2975), [CVE-2023-3446](https://security.alpinelinux.org/vuln/CVE-2023-3446), [CVE-2023-3817](https://security.alpinelinux.org/vuln/CVE-2023-3817) Alpine Release Notes: [3.16.7](https://www.alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html) [3.16.6](https://www.alpinelinux.org/posts/Alpine-3.15.9-3.16.6-3.17.4-3.18.2-released.html) **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Justin Roberson	8998539981	lambda-promtail: Support AWS Organization CloudTrail (#10147) ... **What this PR does / why we need it**: The S3 object key for Organization CloudTrail events includes the org's ID at the beginning. Currently, trying to ingest Organization CloudTrail events with lambda-promtail fails due to the object key not matching the regex. This PR adds support for both matching and parsing out that org ID. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: - instead of adding a new regex I just amended the existing regex just to avoid too heavy of a refactor - I also threw in error logging on the handler because that was _invaluable_ debugging why lambda-promtail wasn't working in our environment. I figure others might get some benefit from it. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Justin Roberson	dd8226e63b	lambda-promtail: fix/refactor SQS integration in Terraform module (#9909) ... **What this PR does / why we need it**: Currently, the Terraform module for lambda-promtail is not working due to a misnamed reference: ``` ╷ │ Error: Reference to undeclared resource │ │ on .terraform/modules/lambda_promtail/tools/lambda-promtail/sqs.tf line 66, in resource "aws_iam_role_policy_attachment" "lambda_sqs_execution": │ 66: role = aws_iam_role.iam_for_lambda.name │ │ A managed resource "aws_iam_role" "iam_for_lambda" has not been declared in │ module.lambda_promtail. ╵ ``` It looks like this was the result of a conflict with #8231 and #8750. While I was in there I also refactored the SQS work to be more consistent with #8750. I also made the queue name prefix configurable with the `sqs_queue_name_prefix` variable, as it was hardcoded before which could cause issues if multiple instances of this module are initialized in the same AWS account and region. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Andrew Gleeson	de77777a05	Lambda Promtail - Add functions to ungzip incoming data from Cross-Account AWS Kinesis/CloudWatch (#10077) ... **What this PR does / why we need it**: When using Lambda Promtail to receive logs via Kinesis that have come from a CloudWatch Logs Subscription Filter in another account, the data is gzipped. This PR add a check for gzipped content and if detected unzips and sends to Loki. [AWS Docs - Cross Account Subscriptions - Kinesis](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CrossAccountSubscriptions-Kinesis.html) **Which issue(s) this PR fixes**: n/a **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Justin Roberson	dd9f30fcb0	lambda-promtail: Support SNS events (#9964) ... **What this PR does / why we need it**: Some use cases (such as S3 bucket notifications) are better suited for SNS rather than SQS when you need to send events to multiple destinations. Adding a little bit of extra parsing logic for SNS events allows directly connecting an SNS topic to the Lambda Promtail function and bypasses the extra complexity of the SQS queue. This lower complexity is very helpful in cross-AWS-account use cases like CloudTrail event forwarding. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: - I just updated the SQS section of the docs to also reference SNS, since the functionality is basically identical. Happy to take suggestions on improvements there. Not sure if `CHANGELOG.md` needs to be updated as well or not. - This is semi-difficult to unit test since there isn't an existing test harness for event matching. Testing in situ on our own infrastructure seems to work. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Michel Hollands	7be9deacf3	Update go version to 1.20.6 (#9913) ... Update the go version. Another PR will follow that uses the new build image --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Deepansh Mathur	50126731f8	Update README.md (#9698) ... Backticks were missing for a code block **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601) ... **What this PR does / why we need it**: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
Christophe Collot	f2394355d8	feat(lambda-promtail): add cloudfront log file ingestion support (#9573) ... **What this PR does / why we need it**: This PR enables ingesting logs from Cloudfront log files stored in s3 (batch). The current setup only supports streaming Cloudfront logs through AWS Kinesis, whereas this PR implements the same flow as for VPC Flow logs, Load Balancer logs, and Cloudtrail logs (s3 --> SQS (optional) --> Lambda Promtail --> Loki) **Special notes for your reviewer**: + The Cloudfront log file format is different from the already implemented services, meaning we had to build yet another regex. AWS never bothered making all services follow the same log file naming convention but the "good" thing is that it's now very unlikely they will change it in the future. + The Cloudfront file name does not have any mention of the AWS account or the time of log it contains, it means we have to infer the log type from the filename format instead of finding the exact string "cloudfront" in the filename. This is why in `getLabels`, if no `type` parameter is found in the regex, we use the key corresponding to the name of the matching parser. + I introduced a new `parser` struct to group together several parameters specific to a type of log (and avoid relying too much on map key string matching and / or if statements for specific use cases) + I've been successfully running this code in several AWS environments for days. + I corrected a typo from my previous PR #9497 (wrong PR number in Changelog.md) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Karsten Jeschkies	c9a3ff53e4	Set unique timestamp in lambda-promtail. (#9560)	3 years ago
Christophe Collot	b709b32d6a	feat(lambda-promtail): add cloudtrail log ingestion support (#9497) ... **What this PR does / why we need it**: ### Add support for AWS CloudTrail audit logs ingestion using lambda-promtail Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for security and debugging purposes. However, I've experienced difficulties with it: + The AWS CloudTrail service is not well integrated with Prometheus (no metrics, no alerts) and I don't want to manage alerts in CloudWatch Alerts + The search experience is painful with CloudTrail via the AWS Console (I will not elaborate 😅). This PR allows ingesting CloudTrail audit logs sent to an S3 bucket using the same approach as VPC flow logs or Load Balancer logs. **Special notes for your reviewer**: + Because the Cloudtrail file format is not text but json, we stream the json CloudTrail records instead of using the already existing scanner. + Because the Cloudtrail filename format is not the same as for the Flow log or the Load balancer log files, we need to split the regexes by service (although many AWS services seem to share the same `defaultFilenameRegex`). + In the `getLabels` function, we expect the `type` parameter to be found in the filename using the Regex. For some log files (ex: Cloudfront log files). The file name has no reference to the service name. This is why, as a default, when no type is found in the name of the file, I set it to use the key of the matching Regex expression. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Christophe Collot	1c7f791a1e	lambda-promtail: add terraform code to process s3 logs through SQS (#8700) ... ## Add Terraform sample code to process AWS S3 log files through an SQS queue Lambda-promtail supports processing AWS s3 logs files through an SQS queue since #8231 As explained in the [documentation section of lambda-promtail](https://github.com/grafana/loki/blob/main/docs/sources/clients/lambda-promtail/_index.md#triggering-lambda-promtail-via-sqs:~:text=Triggering%20Lambda%2DPromtail,DLQ%20redrive%20feature.), this can be leveraged to re-process logs that lambda-promtail failed to process (or send to loki) using a main SQS queue and a secondary SQS dead-letter queue. AWS has a feature called `SQS redrive`, which enables routing messages pending in the DLQ back to the main (source) queue. This PR demonstrates how to spin up this architecture using terraform.	3 years ago
Danny Kopping	f5c0fdf028	Upgrade to latest security release: go1.20.4 (#9370) ... **What this PR does / why we need it**: https://groups.google.com/g/golang-announce/c/MEb0UyuSMsU Also updated the build image. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Trevor Whitney	a11921bbb8	Upgrade alpine 3.16.4 -> 3.16.5 (#9264) ... **What this PR does / why we need it**: Upgrade alpine version to address [openssl CVEs](https://www.alpinelinux.org/posts/Alpine-3.17.3-released.html) **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Trevor Whitney	38c9c6c460	Bump go from 1.20.1 -> 1.20.3 (#9106) ... **What this PR does / why we need it**: Bumps our go dependency from 1.20.1 to 1.20.3 to address [security fixes](https://groups.google.com/g/golang-announce/c/Xdv6JL9ENs8) **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Mitch Hulscher	257225b486	refactor(lambda-promtail): apply terraform best practices (#8750) ... **What this PR does / why we need it**: I would like to offer this PR as a suggestion to improve the lambda-promtail terraform-module. I forked it to be able to deploy it more than once in an AWS account. I also applied terraform best-practices. I was hoping that perhaps these changes could be merged into upstream as well. Unlike https://github.com/grafana/loki/pull/8549 , I unfortunately did not end up making a separate commit for each change. If you would like me to create one or more issue(s) to address the points below, I'd be happy to do that as well. List of improvements: 1. Added `var.name` (defaults to lambda-promtail) so that this module can be deployed multiple times in the same AWS account. This allows us to define unique, non-conflicting names for: * the Lambda function * the CloudWatch log-group * the IAM role 2. Split IAM role policies per component; only assign permissions when required 3. Scope down permissions of the IAM role policies 4. During terraform-destroy, ensure CloudWatch log-group is removed **after** the lambda-function. An accidental invocation of the function could re-create an already destroyed log-group, leaving an orphaned log-group List of style changes: 1. Rename resources to `this` when there is only one instance of this resource-type 2. Add newline after `count|before_each` and before `depends_on` 3. Group resources together and add a section comment 4. Add missing(?) statement-id to S3 AWS lambda permission Misc. 1. I added a `moves.tf` file to facilitate moving renamed resources in existing terraform statefiles. This prevents some resources from recreated. Can also be removed. These changes are backwards compatible, even though some resources will end up being re-created. A `terraform apply` should succeed (it did for me). **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) Signed-off-by: Mitch Hulscher <mitch.hulscher@lib.io>	3 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583) ... **What this PR does / why we need it**: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	3 years ago
Christophe Collot	a013e9f342	Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231)	3 years ago
James Callahan	6d59b4144e	lambda-promtail: Various Terraform fixes (#8549) ... * lambda-promtail: omit_extra_labels_prefix is a boolean This can be changed without a migration as it hasn't been in a release yet * lambda-promtail: fix typo * lambda-promtail: declare variables as a set * lambda-promtail: add a versions file with provider requirements * lambda-promtail: remove provider block - There's no reason to specify a region here - Empty provider blocks are not needed in terraform for quite some time now (and infact they end up producing a warning about their deprecation) * lambda-promtail: bucket_names can be the empty array Without this, you get an error that `resources` cannot be the empty list * lambda-promtail: convert inline policy to a aws_iam_policy_document	3 years ago
dependabot[bot]	3d1ef31af4	Bump golang.org/x/net from 0.5.0 to 0.7.0 in /tools/lambda-promtail (#8556) ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Christian Haudum	20696522fe	lambda-promtail: Add option to omit extra labels prefix `__extra_` (#8548) ... If you want to add an extra label env, lambda-promtail would add the __extra_ prefix, so that the resulting label is __extra_env. However, if you already have the env variable in other stream and you want to have consistent label naming, you would want to omit the prefix so the resulting label is env. Closes https://github.com/grafana/loki/issues/8471 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	ff39654a99	Fix regression in lambda-promtail tool that yields incorrect label names (#8547) ... Commit 4d5678aa17 (diff-c898e72f3b15b46b32f9405b243b7d5b56b68db7bf03541fefd587dc9e49af8eL70-R90) introduced a regression that caused labels to have duplicate `_lb` parts in the label names, as described in https://github.com/grafana/loki/issues/8470 Fixes #8470 Also updated Loki to the latest commit on `main`. Previously the dependency pointed to a Git commit that was out of the tree of this repository.	3 years ago
Christian Haudum	58e29de988	Update golang to 1.19.5 (#8278) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Luis Felipe Vaca	d317e659c7	lambda-promtail-: fix parse kinesis timestamp (#7158)	3 years ago
Christophe Collot	ec33a891df	lambda-promtail: remove duplicate key in terraform config sample (#8137) ... **What this PR does / why we need it**: This PR removes a duplicate key in the terraform configuration sample **Which issue(s) this PR fixes**: None **Special notes for your reviewer**: This is a trivial PR	3 years ago
chriskuchin	4d5678aa17	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868) ... This add support to the S3 parsing logic for AWS VPC Flow logs. It also makes a small change to allow not printing log lines during processing. It leaves the default of printing the log lines in place. Signed-off-by: Thomas Belian <thomas.belian@bt909.de> Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>	3 years ago
Thomas Belián	6fcc4a2712	Lambda-promtail: Add skip tls verify option (#8059) ... **What this PR does / why we need it**: This PR sets a possibility to use lambda-promtail with self signed certificates sometimes used in development systems. This setting is configurable via environment variable. **Which issue(s) this PR fixes**: Fixes #8013 Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	3 years ago
loanshark	3f59fa96d8	lambda-promtail: Add kinesis data stream to use in terraform (#7632) ... **What this PR does / why we need it**: https://github.com/grafana/loki/pull/5977 With the addition of the kinesis data stream function add kinesis data stream to use in terraform	3 years ago
Danny Kopping	355175ff89	Upgrade to Go 1.19 (#7243) ... Upgrade Loki and all related components to Go 1.19.2.	3 years ago
Thomas Belian	b795672bba	feat(lambda-promtail): add bearer token support (#7333) ... It adds bearer token support for lambda-promtail, as it supports only http basic auth. Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	3 years ago
Sandeep Sukhani	b23687ad96	update regex for matching nlb logs while using lambda promtail (#7305) ... **What this PR does / why we need it**: Regex in lambda promtail looks for specific file patterns for reading and pushing logs from it. It currently does not read logs generated by nlb i.e having `_net` in the name. This PR updates the regex. Cross checked it with was docs here https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212) ... The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Jonathan	6b9bf0b5f4	feat: add kms and ec2 permission (#7058) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: This PR will help to increase a necessary policy (kms and ec2) to resource log group and lambda. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Callum Styan	4a21923913	update to alpine:3.6.2 (#6979) ... Updates the alpine image version we use to 3.16.2 and will also build/publish a new loki build image Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago