From 222ffc52d3e9d2830f6b8678df14529f67dde1db Mon Sep 17 00:00:00 2001 From: Pablo <2617411+thepalbi@users.noreply.github.com> Date: Mon, 18 Sep 2023 12:50:36 -0300 Subject: [PATCH] Add CloudFormation template for lambda + EventBridge setup (#10540) **What this PR does / why we need it**: https://github.com/grafana/loki/pull/10449 added support so that lambda-promtail can handle s3 events through AWS EventBridge. This PR is a follow up adding a CloudFormation template to deploy lambda-promtail, with the corresponding EventBridge rule to deliver s3 events to the lambda. **Which issue(s) this PR fixes**: Rel https://github.com/grafana/loki/issues/10209 --------- Co-authored-by: J Stickler Co-authored-by: Christian Haudum --- .../send-data/lambda-promtail/_index.md | 23 +++ tools/lambda-promtail/README.md | 12 ++ .../lambda-promtail/template-eventbridge.yaml | 160 ++++++++++++++++++ 3 files changed, 195 insertions(+) create mode 100644 tools/lambda-promtail/template-eventbridge.yaml diff --git a/docs/sources/send-data/lambda-promtail/_index.md b/docs/sources/send-data/lambda-promtail/_index.md index 0c2ca012c6..170665713a 100644 --- a/docs/sources/send-data/lambda-promtail/_index.md +++ b/docs/sources/send-data/lambda-promtail/_index.md @@ -85,6 +85,8 @@ To add a tenant ID, add `ParameterKey=TenantID,ParameterValue=value`. To modify an existing CloudFormation stack, use [update-stack](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/update-stack.html). +If using CloudFormation to write your infrastructure code, you should consider the [EventBridge based solution](#s3-based-logging-and-cloudformation) for easier deployment. + ## Uses ### Ephemeral Jobs @@ -124,6 +126,27 @@ For AWS services supporting sending messages to SQS (for example, S3 with an S3 ### On-Failure log recovery using SQS Triggering lambda-promtail through SQS allows handling on-failure recovery of the logs using a secondary SQS queue as a dead-letter-queue (DLQ). You can configure lambda so that unsuccessfully processed messages will be sent to the DLQ. After fixing the issue, operators will be able to reprocess the messages by sending back messages from the DLQ to the source queue using the [SQS DLQ redrive](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue-redrive.html) feature. +### S3 based logging and CloudFormation + +Lambda-promtail lets you send logs from different services that use S3 as their logs destination (ALB, VPC Flow, CloudFront access logs, etc.). For this, you need to configure S3 bucket notifications to trigger the lambda-promtail deployment. However, when using CloudFormation to encode infrastructure, there is a [known issue](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/79) when configuring `AWS::S3::BucketNotification` and the resource that will be triggered by the notification in the same stack. + +To manage this issue, AWS introduced [S3 event notifications with Event Bridge](https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/). When an object gets created in a S3 bucket, this sends an event to an EventBridge bus, and you can create a rule to send those events to Lambda-promtail. + +The diagram below shows how notifications logs will be written from the source service into an S3 bucket. From there on, the S3 bucket will send an `Object created` notification into the EventBridge `default` bus, where we can configure a rule to trigger Lambda Promtail. + +![](https://grafana.com/media/docs/loki/lambda-promtail-with-eventbridge.png) + +The [template-eventbridge.yaml](https://github.com/grafana/loki/blob/main/tools/lambda-promtail/template-eventbridge.yaml) CloudFormation template configures Lambda-promtail with EventBridge to address this known issue. To deploy the template, use the snippet below, completing appropriately the `ParameterValue` arguments. + +```bash +aws cloudformation create-stack \ + --stack-name lambda-promtail-stack \ + --template-body file://template-eventbridge.yaml \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ + --region us-east-2 \ + --parameters ParameterKey=WriteAddress,ParameterValue=https://your-loki-url/loki/api/v1/push ParameterKey=Username,ParameterValue= ParameterKey=Password,ParameterValue= ParameterKey=BearerToken,ParameterValue= ParameterKey=LambdaPromtailImage,ParameterValue=: ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2" ParameterKey=TenantID,ParameterValue= ParameterKey=SkipTlsVerify,ParameterValue="false" ParameterKey=EventSourceS3Bucket,ParameterValue= +``` + ## Propagated Labels Incoming logs can have seven special labels assigned to them which can be used in [relabeling]({{< relref "../../send-data/promtail/configuration#relabel_configs" >}}) or later stages in a Promtail [pipeline]({{< relref "../../send-data/promtail/pipelines" >}}): diff --git a/tools/lambda-promtail/README.md b/tools/lambda-promtail/README.md index 39ea06509b..2f4ab27fd1 100644 --- a/tools/lambda-promtail/README.md +++ b/tools/lambda-promtail/README.md @@ -119,6 +119,18 @@ If it's already installed, run the following command to ensure it's the latest v choco upgrade golang ``` +## CloudFormation and S3 events + +Lambda-promtail lets you send logs from different services that use S3 as their logs destination (ALB, VPC Flow, CloudFront access logs, etc.). For this, you need to configure S3 bucket notifications to trigger the lambda-promtail deployment. However, when using CloudFormation to encode infrastructure, there is a [known issue](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/79) when configuring `AWS::S3::BucketNotification` and the resource that will be triggered by the notification in the same stack. + +To manage the issue, AWS introduced [S3 event notifications with Event Bridge](https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/). In that way, when an object gets created in a S3 bucket, this sends an event to an EventBridge bus, and you can create a rule to send those events to Lambda-promtail. + +The [template-eventbridge.yaml](./template-eventbridge.yaml) CloudFormation template configures Lambda-promtail with EventBridge, for the use case mentioned above: + +```bash +aws cloudformation create-stack --stack-name lambda-promtail-stack --template-body file://template-eventbridge.yaml --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region us-east-2 --parameters ParameterKey=WriteAddress,ParameterValue=https://your-loki-url/loki/api/v1/push ParameterKey=Username,ParameterValue= ParameterKey=Password,ParameterValue= ParameterKey=BearerToken,ParameterValue= ParameterKey=LambdaPromtailImage,ParameterValue=: ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2" ParameterKey=TenantID,ParameterValue= ParameterKey=SkipTlsVerify,ParameterValue="false" ParameterKey=EventSourceS3Bucket,ParameterValue="alb-logs-bucket-name" +``` + ## Limitations - Error handling: If promtail is unresponsive, `lambda-promtail` will drop logs after `retry_count`, which defaults to 2. - AWS CloudWatch [quotas](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html) state that the event size is limited to 256kb. `256 KB (maximum). This quota can't be changed.` diff --git a/tools/lambda-promtail/template-eventbridge.yaml b/tools/lambda-promtail/template-eventbridge.yaml new file mode 100644 index 0000000000..a6c2789d2c --- /dev/null +++ b/tools/lambda-promtail/template-eventbridge.yaml @@ -0,0 +1,160 @@ +AWSTemplateFormatVersion: "2010-09-09" +Description: > + lambda-promtail: + + propagate Cloudwatch Logs to Loki/Promtail via Loki Write API. + +Parameters: + WriteAddress: + Description: "Address to write to in the form of: http://<:port>/loki/api/v1/push" + Type: String + Default: "http://localhost:8080/loki/api/v1/push" + ReservedConcurrency: + Description: The maximum of concurrent executions you want to reserve for the function. + Type: Number + Default: 2 + Username: + Description: The basic auth username, necessary if writing directly to Grafana Cloud Loki. + Type: String + Default: "" + Password: + Description: The basic auth password, necessary if writing directly to Grafana Cloud Loki. + Type: String + Default: "" + NoEcho: true + BearerToken: + Description: The bearer token, necessary if target endpoint requires it. + Type: String + Default: "" + NoEcho: true + LambdaPromtailImage: + Description: The ECR image URI to pull and use for lambda-promtail. + Type: String + Default: "" + KeepStream: + Description: Determines whether to keep the CloudWatch Log Stream value as a Loki label when writing logs from lambda-promtail. + Type: String + Default: "false" + ExtraLabels: + Description: Comma separated list of extra labels, in the format 'name1,value1,name2,value2,...,nameN,valueN' to add to entries forwarded by lambda-promtail. + Type: String + Default: "" + OmitExtraLabelsPrefix: + Description: Whether or not to omit the prefix `__extra_` from extra labels defined in `ExtraLabels`. + Type: String + Default: "false" + TenantID: + Description: Tenant ID to be added when writing logs from lambda-promtail. + Type: String + Default: "" + SkipTlsVerify: + Description: Determines whether to verify the TLS certificate + Type: String + Default: "false" + EventSourceS3Bucket: + Description: The S3 bucket to listen event notifications from. + Type: String + Default: "" + +Resources: + LambdaPromtailRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: + Service: + - lambda.amazonaws.com + Action: + - sts:AssumeRole + Description: "Lambda Promtail Role" + Policies: + - PolicyName: logs + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: + - logs:CreateLogGroup + - logs:CreateLogStream + - logs:PutLogEvents + Resource: arn:aws:logs:*:*:* + - PolicyName: s3 + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: + - s3:GetObject + Resource: arn:aws:s3:::thepalbi-lambda-lb-access-logs/* + RoleName: iam_for_lambda + LambdaPromtailFunction: + Type: AWS::Lambda::Function + Properties: + Code: + ImageUri: !Ref LambdaPromtailImage + MemorySize: 128 + PackageType: Image + Timeout: 60 + Role: !GetAtt LambdaPromtailRole.Arn + ReservedConcurrentExecutions: !Ref ReservedConcurrency + # # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-vpcconfig.html + # VpcConfig: + Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object + Variables: + WRITE_ADDRESS: !Ref WriteAddress + USERNAME: !Ref Username + PASSWORD: !Ref Password + BEARER_TOKEN: !Ref BearerToken + KEEP_STREAM: !Ref KeepStream + EXTRA_LABELS: !Ref ExtraLabels + OMIT_EXTRA_LABELS_PREFIX: !Ref OmitExtraLabelsPrefix + TENANT_ID: !Ref TenantID + SKIP_TLS_VERIFY: !Ref SkipTlsVerify + + LambdaPromtailVersion: + Type: AWS::Lambda::Version + Properties: + FunctionName: !Ref LambdaPromtailFunction + + LambdaPromtailEventInvokeConfig: + Type: AWS::Lambda::EventInvokeConfig + Properties: + FunctionName: !Ref LambdaPromtailFunction + MaximumRetryAttempts: 2 + Qualifier: !GetAtt LambdaPromtailVersion.Version + + # EventBridge rule to route s3 object created events to lambda promtail + EventRule: + Type: AWS::Events::Rule + Properties: + Description: EventRule + State: ENABLED + EventPattern: # https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html#eb-filtering-data- + source: + - aws.s3 + detail-type: + - "Object Created" + detail: + bucket: + name: + - !Ref EventSourceS3Bucket + Targets: + - Arn: !GetAtt LambdaPromtailFunction.Arn + Id: LambdaPromtailTarget + + # Permission that allows EventBridge rule to trigger lambda promtail + EventRuleLambdaPermission: + Type: AWS::Lambda::Permission + Properties: + FunctionName: !GetAtt LambdaPromtailFunction.Arn + Action: lambda:InvokeFunction + Principal: events.amazonaws.com + SourceArn: !GetAtt EventRule.Arn + +Outputs: + LambdaPromtailFunction: + Description: "Lambda Promtail Function ARN" + Value: !GetAtt LambdaPromtailFunction.Arn