Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Pablo 5034b1b337 Add support for EventBridge s3 events in lambda-promtail (#10449) ... **What this PR does / why we need it**: Lambda promtail supports s3 events, which are used for scraping several log sources such as ALB access logs. This works by configuring at the S3 bucket level "s3 event notification", that are configured to target the lambda deployment of lambda-promtail. However, if one is configuring this through CloudFormation, there's a known issue with AWS that doesn't allow to configure both the lambda, the bucket, and the notifications in the same stack. See [this issue](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/79) for details. For that, AWS introduced EventBridge notifications, which can be used to ship s3 events to a lambda deployment as well. This flow looks like: s3 -> eventbridge bus -> eventbridge rule -> lambda EventBridge has it's own message structure for s3 notifications. This PR adds a translation layer, just for `Object created` events (since they are the only ones we should take into account), so that EventBridge events can be received, and trigger the lambda as if they were from s3. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/10209 **Special notes for your reviewer**: - [x] Pending testing this with an actual deployment of the s3 -> event bridge -> lambda flow - [x] ~~Add CF template for the `s3 -> event bridge -> lambda` deployment~~ Follow up PR **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)		2 years ago
..
cloudtrail.go	feat(lambda-promtail): add cloudtrail log ingestion support (#9497)	3 years ago
cloudtrail_test.go	Set unique timestamp in lambda-promtail. (#9560)	3 years ago
cw.go	Set unique timestamp in lambda-promtail. (#9560)	3 years ago
eventbridge.go	Add support for EventBridge s3 events in lambda-promtail (#10449)	2 years ago
eventbridge_test.go	Add support for EventBridge s3 events in lambda-promtail (#10449)	2 years ago
json_stream.go	Set unique timestamp in lambda-promtail. (#9560)	3 years ago
kinesis.go	Lambda Promtail - Add functions to ungzip incoming data from Cross-Account AWS Kinesis/CloudWatch (#10077)	3 years ago
kinesis_test.go	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
logger.go	Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231)	3 years ago
main.go	Add support for EventBridge s3 events in lambda-promtail (#10449)	2 years ago
main_test.go	Set unique timestamp in lambda-promtail. (#9560)	3 years ago
promtail.go	Set unique timestamp in lambda-promtail. (#9560)	3 years ago
promtail_client.go	Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231)	3 years ago
s3.go	lambda-promtail: Add support for WAF logs in S3 (#10416)	2 years ago
s3_test.go	lambda-promtail: Add support for WAF logs in S3 (#10416)	2 years ago