apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
renovate[bot]	9d7a6ea680	fix(deps): update github.com/grafana/dskit digest to b69ac1b (#14355 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	1 year ago
renovate[bot]	a4f3edfb52	fix(deps): update github.com/grafana/dskit digest to f52de24 (#14319 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	1 year ago
renovate[bot]	f39cdbd541	fix(deps): update github.com/grafana/dskit digest to 7c41a40 (#14277 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
renovate[bot]	7c18642521	fix(deps): update github.com/grafana/dskit digest to 931a021 (#14032 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	1 year ago
renovate[bot]	6f491233ca	fix(deps): update aws-sdk-go-v2 monorepo (#13986 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	1 year ago
Paul Rogers	07f916936b	chore(deps): lamba-promtail, move back to al2, update krb5-libs (#13939 )	1 year ago
Paul Rogers	1527eadfb5	chore(deps): LambdaPromtail - Bump to AmazonLinux 2023 (#13938 )	1 year ago
Paul Rogers	8319d1cb81	chore(deps): Update dependencies, build version, and go version for lambda-promtail (#13904 )	1 year ago
Max Forasteiro	9bf08f7cc0	feat(lambda-promtail): build lambda with zip file (#13787 ) Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	1 year ago
renovate[bot]	ea6395a2eb	chore(deps): update golang docker tag to v1.22.6 (#13847 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	1 year ago
samuelebstein	2d92fff2aa	feat(lambda-promtail): Adding S3 log parser support for AWS GuardDuty (#13148 )	1 year ago
Alexandre Steppé	dbfb19be49	feat: lambda-promtail; ensure messages to Kinesis are usable by refactoring parsing of KinesisEvent to match parsing of CWEvents + code cleanup (#13098 )	1 year ago
Paul Rogers	ca2f11d93e	chore: Update Docker images to Go 1.22.5 (#13398 )	2 years ago
Tanat Lokejaroenlarb	8892dc8923	feat: parameterise the MaximumEventAgeInSeconds, LogGroupName, and IAMRoleName for lambda-promtail CloudFormation template (#12728 )	2 years ago
dependabot[bot]	0e8a247b58	chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 in /tools/lambda-promtail (#12696 ) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	2 years ago
Callum Styan	71a8f2c2b1	fix: updated all dockerfiles go1.22 (#12708 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Alan Edwardes	8c18463285	fix: Remove Hardcoded Bucket Name from EventBridge Example CloudFormation Template (#12609 ) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Joshua Ford	7a81d264a4	fix: lambda-promtail, update s3 filename regex to allow finding of log files from AWS GovCloud regions (#12482 )	2 years ago
Periklis Tsirakidis	131144bcab	chore(deps): Bump golang to 1.21.9 (#12467 )	2 years ago
Callum Styan	39fc41bda3	chore: lambda-promtail image is failing to build in CI, maybe this will fix it (#12459 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Callum Styan	2951b38e15	chore: update base image used for the lambda-promtail image (#12454 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
renovate[bot]	bdad1d331d	chore(deps): update module google.golang.org/protobuf to v1.33.0 [security] (main) (#12137 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
renovate[bot]	a627fb6aa1	chore(deps): update module golang.org/x/crypto to v0.17.0 [security] (main) (#11522 ) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: \| Package \| Change \| Age \| Adoption \| Passing \| Confidence \| \|---\|---\|---\|---\|---\|---\| \| golang.org/x/crypto \| `v0.14.0` -> `v0.17.0` \| [![age](https://developer.mend.io/api/mc/badges/age/go/golang.org%2fx%2fcrypto/v0.17.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) \| [![adoption](https://developer.mend.io/api/mc/badges/adoption/go/golang.org%2fx%2fcrypto/v0.17.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) \| [![passing](https://developer.mend.io/api/mc/badges/compatibility/go/golang.org%2fx%2fcrypto/v0.14.0/v0.17.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) \| [![confidence](https://developer.mend.io/api/mc/badges/confidence/go/golang.org%2fx%2fcrypto/v0.14.0/v0.17.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) \| ### GitHub Vulnerability Alerts #### [CVE-2023-48795](https://togithub.com/warp-tech/russh/security/advisories/GHSA-45x7-px36-x8w8) ### Summary Terrapin is a prefix truncation attack targeting the SSH protocol. More precisely, Terrapin breaks the integrity of SSH's secure channel. By carefully adjusting the sequence numbers during the handshake, an attacker can remove an arbitrary amount of messages sent by the client or server at the beginning of the secure channel without the client or server noticing it. ### Mitigations To mitigate this protocol vulnerability, OpenSSH suggested a so-called "strict kex" which alters the SSH handshake to ensure a Man-in-the-Middle attacker cannot introduce unauthenticated messages as well as convey sequence number manipulation across handshakes. Warning: To take effect, both the client and server must support this countermeasure. As a stop-gap measure, peers may also (temporarily) disable the affected algorithms and use unaffected alternatives like AES-GCM instead until patches are available. ### Details The SSH specifications of ChaCha20-Poly1305 (chacha20-poly1305@openssh.com) and Encrypt-then-MAC (-etm@openssh.com MACs) are vulnerable against an arbitrary prefix truncation attack (a.k.a. Terrapin attack). This allows for an extension negotiation downgrade by stripping the SSH_MSG_EXT_INFO sent after the first message after SSH_MSG_NEWKEYS, downgrading security, and disabling attack countermeasures in some versions of OpenSSH. When targeting Encrypt-then-MAC, this attack requires the use of a CBC cipher to be practically exploitable due to the internal workings of the cipher mode. Additionally, this novel attack technique can be used to exploit previously unexploitable implementation flaws in a Man-in-the-Middle scenario. The attack works by an attacker injecting an arbitrary number of SSH_MSG_IGNORE messages during the initial key exchange and consequently removing the same number of messages just after the initial key exchange has concluded. This is possible due to missing authentication of the excess SSH_MSG_IGNORE messages and the fact that the implicit sequence numbers used within the SSH protocol are only checked after the initial key exchange. In the case of ChaCha20-Poly1305, the attack is guaranteed to work on every connection as this cipher does not maintain an internal state other than the message's sequence number. In the case of Encrypt-Then-MAC, practical exploitation requires the use of a CBC cipher; while theoretical integrity is broken for all ciphers when using this mode, message processing will fail at the application layer for CTR and stream ciphers. For more details see [https://terrapin-attack.com](https://terrapin-attack.com). ### Impact This attack targets the specification of ChaCha20-Poly1305 (chacha20-poly1305@openssh.com) and Encrypt-then-MAC (-etm@openssh.com), which are widely adopted by well-known SSH implementations and can be considered de-facto standard. These algorithms can be practically exploited; however, in the case of Encrypt-Then-MAC, we additionally require the use of a CBC cipher. As a consequence, this attack works against all well-behaving SSH implementations supporting either of those algorithms and can be used to downgrade (but not fully strip) connection security in case SSH extension negotiation (RFC8308) is supported. The attack may also enable attackers to exploit certain implementation flaws in a man-in-the-middle (MitM) scenario. --- ### Configuration 📅 Schedule: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied. ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 Ignore: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy45My4xIiwidXBkYXRlZEluVmVyIjoiMzcuMTM1LjAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIn0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Trevor Whitney	8b48a18d77	Bump build image (#11608 ) What this PR does / why we need it: Bumps alpine in the loki build image and ass the build image to the `scan-vulnerabilities` target to make it easier to scan in the future --------- Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Elliot Dobson	0945b186ff	lambda-promtail: cloudwatch: add '__aws_log_type' label (#11335 ) What this PR does / why we need it: Adds the `__aws_log_type` label to AWS CloudWatch logs in `lambda-promtail`. Which issue(s) this PR fixes: N/A Special notes for your reviewer: AWS S3 & Kinesis log types already have this label. The `lambda-promtail` documentation [here](https://github.com/grafana/loki/blob/main/docs/sources/send-data/lambda-promtail/_index.md?plain=1#L154) suggests that CloudWatch logs has this label added as well, but in practice it does not AFAICT. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
renovate[bot]	10fe48b815	chore(deps): update alpine docker tag to v3.18.4 (main) (#11167 ) [![Mend Renovate logo banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: \| Package \| Type \| Update \| Change \| \|---\|---\|---\|---\| \| alpine \| final \| patch \| `3.18.3` -> `3.18.4` \| \| alpine \| final \| minor \| `3.16.7` -> `3.18.4` \| --- ### Configuration 📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied. ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 Ignore: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40Ni4wIiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
keyolk	8328345a8b	lambda-promtail: fix IAM policy for clouddwatch log stream (#10909 )	2 years ago
Đỗ Trọng Hải	94169a0e6b	fix(tools/lambda-promtail): Do not evaluate empty string for drop_labels (#11074 ) Fixes #11005 Signed-off-by: hainenber <dotronghai96@gmail.com>	2 years ago
renovate[bot]	ab94c70c4c	chore(deps): update module golang.org/x/net to v0.17.0 [security] (main) (#10831 ) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: \| Package \| Type \| Update \| Change \| \|---\|---\|---\|---\| \| golang.org/x/net \| indirect \| minor \| `v0.7.0` -> `v0.17.0` \| \| golang.org/x/net \| indirect \| minor \| `v0.8.0` -> `v0.17.0` \| ### GitHub Vulnerability Alerts #### [CVE-2023-3978](https://nvd.nist.gov/vuln/detail/CVE-2023-3978) Text nodes not in the HTML namespace are incorrectly literally rendered, causing text which should be escaped to not be. This could lead to an XSS attack. #### [CVE-2023-39325](https://togithub.com/golang/go/issues/63417) A malicious HTTP/2 client which rapidly creates requests and immediately resets them can cause excessive server resource consumption. While the total number of requests is bounded by the http2.Server.MaxConcurrentStreams setting, resetting an in-progress request allows the attacker to create a new request while the existing one is still executing. With the fix applied, HTTP/2 servers now bound the number of simultaneously executing handler goroutines to the stream concurrency limit (MaxConcurrentStreams). New requests arriving when at the limit (which can only happen after the client has reset an existing, in-flight request) will be queued until a handler exits. If the request queue grows too large, the server will terminate the connection. This issue is also fixed in golang.org/x/net/http2 for users manually configuring HTTP/2. The default stream concurrency limit is 250 streams (requests) per HTTP/2 connection. This value may be adjusted using the golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams setting and the ConfigureServer function. #### [CVE-2023-44487](https://togithub.com/apple/swift-nio-http2/security/advisories/GHSA-qppj-fm5r-hxr3) swift-nio-http2 is vulnerable to a denial-of-service vulnerability in which a malicious client can create and then reset a large number of HTTP/2 streams in a short period of time. This causes swift-nio-http2 to commit to a large amount of expensive work which it then throws away, including creating entirely new `Channel`s to serve the traffic. This can easily overwhelm an `EventLoop` and prevent it from making forward progress. swift-nio-http2 1.28 contains a remediation for this issue that applies reset counter using a sliding window. This constrains the number of stream resets that may occur in a given window of time. Clients violating this limit will have their connections torn down. This allows clients to continue to cancel streams for legitimate reasons, while constraining malicious actors. --- ### Configuration 📅 Schedule: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied. ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 Ignore: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy44LjEiLCJ1cGRhdGVkSW5WZXIiOiIzNy4xOS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Đỗ Trọng Hải	ec54c723eb	feat(lambda/promtail): support dropping labels (#10755 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #10669 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: hainenber <dotronghai96@gmail.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Ashwanth	98551ceb9a	use go 1.21.3 and go-grpc v1.56.3 (#10869 ) What this PR does / why we need it: Fixes CVE-2023-39325 / CVE-2023-44487 https://github.com/grpc/grpc-go/releases/tag/v1.56.3 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Michel Hollands	eb3cff4625	Update Go version to 1.21.2 in Dockerfiles (#10805 ) What this PR does / why we need it: Update the go version in the Dockerfiles to 1.21.2 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Pablo	222ffc52d3	Add CloudFormation template for lambda + EventBridge setup (#10540 ) What this PR does / why we need it: https://github.com/grafana/loki/pull/10449 added support so that lambda-promtail can handle s3 events through AWS EventBridge. This PR is a follow up adding a CloudFormation template to deploy lambda-promtail, with the corresponding EventBridge rule to deliver s3 events to the lambda. Which issue(s) this PR fixes: Rel https://github.com/grafana/loki/issues/10209 --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Dylan Guedes	4681386775	Update alpine versions (#10573 ) What this PR does / why we need it: Update alpine versions used by our images to `3.18.3` Which issue(s) this PR fixes: N/A Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Pablo	5034b1b337	Add support for EventBridge s3 events in lambda-promtail (#10449 ) What this PR does / why we need it: Lambda promtail supports s3 events, which are used for scraping several log sources such as ALB access logs. This works by configuring at the S3 bucket level "s3 event notification", that are configured to target the lambda deployment of lambda-promtail. However, if one is configuring this through CloudFormation, there's a known issue with AWS that doesn't allow to configure both the lambda, the bucket, and the notifications in the same stack. See [this issue](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/79) for details. For that, AWS introduced EventBridge notifications, which can be used to ship s3 events to a lambda deployment as well. This flow looks like: s3 -> eventbridge bus -> eventbridge rule -> lambda EventBridge has it's own message structure for s3 notifications. This PR adds a translation layer, just for `Object created` events (since they are the only ones we should take into account), so that EventBridge events can be received, and trigger the lambda as if they were from s3. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/10209 Special notes for your reviewer: - [x] Pending testing this with an actual deployment of the s3 -> event bridge -> lambda flow - [x] ~~Add CF template for the `s3 -> event bridge -> lambda` deployment~~ Follow up PR Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Marcus Morris	e8f4b42eaa	Lambda Promtail: Add logs:PutSubscriptionFilter to Lambda Promtail role (#10426 ) What this PR does / why we need it: I ran into the below error during the CloudFormation stack creation when trying to use the [example CloudFormation config](https://github.com/grafana/loki/blob/main/tools/lambda-promtail/template.yaml): `"Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function"` The issue is that the config does not give the `logs:PutSubscriptionFilter` permission to the Lambda Promtail Role. This adds that permission to fix the error Which issue(s) this PR fixes: Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
lpugoy	965f19a3b9	lambda-promtail: Add support for WAF logs in S3 (#10416 ) What this PR does / why we need it: Adds support for WAF logs in S3 in lambda-promtail Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Elliot Dobson	6a8225d533	lambda-promtail: s3: update log parsing to support NLBs (#7194 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: Fixes the filename and timestamp regex for AWS Network Load Balancer (NLB) logs and adds logic to make NLB log timestamps RFC3339 compatible. Also adds a test to check the parsed timestamp is as expected. Which issue(s) this PR fixes: Fixes #6455 Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Shantanu Alsi	b57d260ddc	Upgrade alpine 3.16.5 -> 3.16.7 (#10188 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #10181 Fixes Open SSL vulnerabilities [CVE-2023-1255](https://security.alpinelinux.org/vuln/CVE-2023-1255), [CVE-2023-2650](https://security.alpinelinux.org/vuln/CVE-2023-2650), [CVE-2023-2975](https://security.alpinelinux.org/vuln/CVE-2023-2975), [CVE-2023-3446](https://security.alpinelinux.org/vuln/CVE-2023-3446), [CVE-2023-3817](https://security.alpinelinux.org/vuln/CVE-2023-3817) Alpine Release Notes: [3.16.7](https://www.alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html) [3.16.6](https://www.alpinelinux.org/posts/Alpine-3.15.9-3.16.6-3.17.4-3.18.2-released.html) Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Justin Roberson	8998539981	lambda-promtail: Support AWS Organization CloudTrail (#10147 ) What this PR does / why we need it: The S3 object key for Organization CloudTrail events includes the org's ID at the beginning. Currently, trying to ingest Organization CloudTrail events with lambda-promtail fails due to the object key not matching the regex. This PR adds support for both matching and parsing out that org ID. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: - instead of adding a new regex I just amended the existing regex just to avoid too heavy of a refactor - I also threw in error logging on the handler because that was _invaluable_ debugging why lambda-promtail wasn't working in our environment. I figure others might get some benefit from it. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Justin Roberson	dd8226e63b	lambda-promtail: fix/refactor SQS integration in Terraform module (#9909 ) What this PR does / why we need it: Currently, the Terraform module for lambda-promtail is not working due to a misnamed reference: ``` ╷ │ Error: Reference to undeclared resource │ │ on .terraform/modules/lambda_promtail/tools/lambda-promtail/sqs.tf line 66, in resource "aws_iam_role_policy_attachment" "lambda_sqs_execution": │ 66: role = aws_iam_role.iam_for_lambda.name │ │ A managed resource "aws_iam_role" "iam_for_lambda" has not been declared in │ module.lambda_promtail. ╵ ``` It looks like this was the result of a conflict with #8231 and #8750. While I was in there I also refactored the SQS work to be more consistent with #8750. I also made the queue name prefix configurable with the `sqs_queue_name_prefix` variable, as it was hardcoded before which could cause issues if multiple instances of this module are initialized in the same AWS account and region. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Andrew Gleeson	de77777a05	Lambda Promtail - Add functions to ungzip incoming data from Cross-Account AWS Kinesis/CloudWatch (#10077 ) What this PR does / why we need it: When using Lambda Promtail to receive logs via Kinesis that have come from a CloudWatch Logs Subscription Filter in another account, the data is gzipped. This PR add a check for gzipped content and if detected unzips and sends to Loki. [AWS Docs - Cross Account Subscriptions - Kinesis](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CrossAccountSubscriptions-Kinesis.html) Which issue(s) this PR fixes: n/a Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Justin Roberson	dd9f30fcb0	lambda-promtail: Support SNS events (#9964 ) What this PR does / why we need it: Some use cases (such as S3 bucket notifications) are better suited for SNS rather than SQS when you need to send events to multiple destinations. Adding a little bit of extra parsing logic for SNS events allows directly connecting an SNS topic to the Lambda Promtail function and bypasses the extra complexity of the SQS queue. This lower complexity is very helpful in cross-AWS-account use cases like CloudTrail event forwarding. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: - I just updated the SQS section of the docs to also reference SNS, since the functionality is basically identical. Happy to take suggestions on improvements there. Not sure if `CHANGELOG.md` needs to be updated as well or not. - This is semi-difficult to unit test since there isn't an existing test harness for event matching. Testing in situ on our own infrastructure seems to work. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Michel Hollands	7be9deacf3	Update go version to 1.20.6 (#9913 ) Update the go version. Another PR will follow that uses the new build image --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Deepansh Mathur	50126731f8	Update README.md (#9698 ) Backticks were missing for a code block What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
Christophe Collot	f2394355d8	feat(lambda-promtail): add cloudfront log file ingestion support (#9573 ) What this PR does / why we need it: This PR enables ingesting logs from Cloudfront log files stored in s3 (batch). The current setup only supports streaming Cloudfront logs through AWS Kinesis, whereas this PR implements the same flow as for VPC Flow logs, Load Balancer logs, and Cloudtrail logs (s3 --> SQS (optional) --> Lambda Promtail --> Loki) Special notes for your reviewer: + The Cloudfront log file format is different from the already implemented services, meaning we had to build yet another regex. AWS never bothered making all services follow the same log file naming convention but the "good" thing is that it's now very unlikely they will change it in the future. + The Cloudfront file name does not have any mention of the AWS account or the time of log it contains, it means we have to infer the log type from the filename format instead of finding the exact string "cloudfront" in the filename. This is why in `getLabels`, if no `type` parameter is found in the regex, we use the key corresponding to the name of the matching parser. + I introduced a new `parser` struct to group together several parameters specific to a type of log (and avoid relying too much on map key string matching and / or if statements for specific use cases) + I've been successfully running this code in several AWS environments for days. + I corrected a typo from my previous PR #9497 (wrong PR number in Changelog.md) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Karsten Jeschkies	c9a3ff53e4	Set unique timestamp in lambda-promtail. (#9560 )	3 years ago
Christophe Collot	b709b32d6a	feat(lambda-promtail): add cloudtrail log ingestion support (#9497 ) What this PR does / why we need it: ### Add support for AWS CloudTrail audit logs ingestion using lambda-promtail Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for security and debugging purposes. However, I've experienced difficulties with it: + The AWS CloudTrail service is not well integrated with Prometheus (no metrics, no alerts) and I don't want to manage alerts in CloudWatch Alerts + The search experience is painful with CloudTrail via the AWS Console (I will not elaborate 😅). This PR allows ingesting CloudTrail audit logs sent to an S3 bucket using the same approach as VPC flow logs or Load Balancer logs. Special notes for your reviewer: + Because the Cloudtrail file format is not text but json, we stream the json CloudTrail records instead of using the already existing scanner. + Because the Cloudtrail filename format is not the same as for the Flow log or the Load balancer log files, we need to split the regexes by service (although many AWS services seem to share the same `defaultFilenameRegex`). + In the `getLabels` function, we expect the `type` parameter to be found in the filename using the Regex. For some log files (ex: Cloudfront log files). The file name has no reference to the service name. This is why, as a default, when no type is found in the name of the file, I set it to use the key of the matching Regex expression. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Christophe Collot	1c7f791a1e	lambda-promtail: add terraform code to process s3 logs through SQS (#8700 ) ## Add Terraform sample code to process AWS S3 log files through an SQS queue Lambda-promtail supports processing AWS s3 logs files through an SQS queue since #8231 As explained in the [documentation section of lambda-promtail](https://github.com/grafana/loki/blob/main/docs/sources/clients/lambda-promtail/_index.md#triggering-lambda-promtail-via-sqs:~:text=Triggering%20Lambda%2DPromtail,DLQ%20redrive%20feature.), this can be leveraged to re-process logs that lambda-promtail failed to process (or send to loki) using a main SQS queue and a secondary SQS dead-letter queue. AWS has a feature called `SQS redrive`, which enables routing messages pending in the DLQ back to the main (source) queue. This PR demonstrates how to spin up this architecture using terraform.	3 years ago

1 2

93 Commits (39b57ec4eac3cbdc718aacae32ab8ff4e989709b)