Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Pablo 5034b1b337 Add support for EventBridge s3 events in lambda-promtail (#10449) ... **What this PR does / why we need it**: Lambda promtail supports s3 events, which are used for scraping several log sources such as ALB access logs. This works by configuring at the S3 bucket level "s3 event notification", that are configured to target the lambda deployment of lambda-promtail. However, if one is configuring this through CloudFormation, there's a known issue with AWS that doesn't allow to configure both the lambda, the bucket, and the notifications in the same stack. See [this issue](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/79) for details. For that, AWS introduced EventBridge notifications, which can be used to ship s3 events to a lambda deployment as well. This flow looks like: s3 -> eventbridge bus -> eventbridge rule -> lambda EventBridge has it's own message structure for s3 notifications. This PR adds a translation layer, just for `Object created` events (since they are the only ones we should take into account), so that EventBridge events can be received, and trigger the lambda as if they were from s3. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/10209 **Special notes for your reviewer**: - [x] Pending testing this with an actual deployment of the s3 -> event bridge -> lambda flow - [x] ~~Add CF template for the `s3 -> event bridge -> lambda` deployment~~ Follow up PR **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)		2 years ago
..
albaccesslog.log.gz	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868)	3 years ago
cloudfront.log.gz	feat(lambda-promtail): add cloudfront log file ingestion support (#9573)	3 years ago
cloudtrail-log-file.json.gz	feat(lambda-promtail): add cloudtrail log ingestion support (#9497)	3 years ago
eventbridge-s3-event.json	Add support for EventBridge s3 events in lambda-promtail (#10449)	2 years ago
kinesis-event.json	…
nlbaccesslog.log.gz	lambda-promtail: s3: update log parsing to support NLBs (#7194)	2 years ago
vpcflowlog.log.gz	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868)	3 years ago
waflog.log.gz	lambda-promtail: Add support for WAF logs in S3 (#10416)	2 years ago