apitech
/
loki
mirror of https://github.com/grafana/loki
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Add CloudFormation template for lambda + EventBridge setup (#10540)

Browse Source 
**What this PR does / why we need it**:

https://github.com/grafana/loki/pull/10449 added support so that lambda-promtail can handle s3 events through AWS EventBridge. This PR is a follow up adding a CloudFormation template to deploy lambda-promtail, with the corresponding EventBridge rule to deliver s3 events to the lambda.

**Which issue(s) this PR fixes**:

Rel https://github.com/grafana/loki/issues/10209

---------

Co-authored-by: J Stickler <julie.stickler@grafana.com>
Co-authored-by: Christian Haudum <christian.haudum@gmail.com>
pull/10633/head
Pablo 2 years ago committed by GitHub
parent 3a8a5e54dd
commit 222ffc52d3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 195 additions and 0 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 23
      docs/sources/send-data/lambda-promtail/_index.md
  2. 12
      tools/lambda-promtail/README.md
  3. 160
      tools/lambda-promtail/template-eventbridge.yaml

23
docs/sources/send-data/lambda-promtail/_index.md
Unescape View File

@ -85,6 +85,8 @@ To add a tenant ID, add `ParameterKey=TenantID,ParameterValue=value`.
To modify an existing CloudFormation stack, use [update-stack](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/update-stack.html).
If using CloudFormation to write your infrastructure code, you should consider the [EventBridge based solution](#s3-based-logging-and-cloudformation) for easier deployment.
## Uses
### Ephemeral Jobs
@ -124,6 +126,27 @@ For AWS services supporting sending messages to SQS (for example, S3 with an S3
### On-Failure log recovery using SQS
Triggering lambda-promtail through SQS allows handling on-failure recovery of the logs using a secondary SQS queue as a dead-letter-queue (DLQ). You can configure lambda so that unsuccessfully processed messages will be sent to the DLQ. After fixing the issue, operators will be able to reprocess the messages by sending back messages from the DLQ to the source queue using the [SQS DLQ redrive](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue-redrive.html) feature.
### S3 based logging and CloudFormation
Lambda-promtail lets you send logs from different services that use S3 as their logs destination (ALB, VPC Flow, CloudFront access logs, etc.). For this, you need to configure S3 bucket notifications to trigger the lambda-promtail deployment. However, when using CloudFormation to encode infrastructure, there is a [known issue](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/79) when configuring `AWS::S3::BucketNotification` and the resource that will be triggered by the notification in the same stack.
To manage this issue, AWS introduced [S3 event notifications with Event Bridge](https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/). When an object gets created in a S3 bucket, this sends an event to an EventBridge bus, and you can create a rule to send those events to Lambda-promtail.
The diagram below shows how notifications logs will be written from the source service into an S3 bucket. From there on, the S3 bucket will send an `Object created` notification into the EventBridge `default` bus, where we can configure a rule to trigger Lambda Promtail.
![](https://grafana.com/media/docs/loki/lambda-promtail-with-eventbridge.png)
The [template-eventbridge.yaml](https://github.com/grafana/loki/blob/main/tools/lambda-promtail/template-eventbridge.yaml) CloudFormation template configures Lambda-promtail with EventBridge to address this known issue. To deploy the template, use the snippet below, completing appropriately the `ParameterValue` arguments.
```bash
aws cloudformation create-stack \
--stack-name lambda-promtail-stack \
--template-body file://template-eventbridge.yaml \
--capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
--region us-east-2 \
--parameters ParameterKey=WriteAddress,ParameterValue=https://your-loki-url/loki/api/v1/push ParameterKey=Username,ParameterValue=<basic-auth-username> ParameterKey=Password,ParameterValue=<basic-auth-pw> ParameterKey=BearerToken,ParameterValue=<bearer-token> ParameterKey=LambdaPromtailImage,ParameterValue=<ecr-repo>:<tag> ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2" ParameterKey=TenantID,ParameterValue=<value> ParameterKey=SkipTlsVerify,ParameterValue="false" ParameterKey=EventSourceS3Bucket,ParameterValue=<S3 where target logs are stored>
```
## Propagated Labels
Incoming logs can have seven special labels assigned to them which can be used in [relabeling]({{< relref "../../send-data/promtail/configuration#relabel_configs" >}}) or later stages in a Promtail [pipeline]({{< relref "../../send-data/promtail/pipelines" >}}):

12
tools/lambda-promtail/README.md
Unescape View File

@ -119,6 +119,18 @@ If it's already installed, run the following command to ensure it's the latest v
choco upgrade golang
```
## CloudFormation and S3 events
Lambda-promtail lets you send logs from different services that use S3 as their logs destination (ALB, VPC Flow, CloudFront access logs, etc.). For this, you need to configure S3 bucket notifications to trigger the lambda-promtail deployment. However, when using CloudFormation to encode infrastructure, there is a [known issue](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/79) when configuring `AWS::S3::BucketNotification` and the resource that will be triggered by the notification in the same stack.
To manage the issue, AWS introduced [S3 event notifications with Event Bridge](https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/). In that way, when an object gets created in a S3 bucket, this sends an event to an EventBridge bus, and you can create a rule to send those events to Lambda-promtail.
The [template-eventbridge.yaml](./template-eventbridge.yaml) CloudFormation template configures Lambda-promtail with EventBridge, for the use case mentioned above:
```bash
aws cloudformation create-stack --stack-name lambda-promtail-stack --template-body file://template-eventbridge.yaml --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region us-east-2 --parameters ParameterKey=WriteAddress,ParameterValue=https://your-loki-url/loki/api/v1/push ParameterKey=Username,ParameterValue=<basic-auth-username> ParameterKey=Password,ParameterValue=<basic-auth-pw> ParameterKey=BearerToken,ParameterValue=<bearer-token> ParameterKey=LambdaPromtailImage,ParameterValue=<ecr-repo>:<tag> ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2" ParameterKey=TenantID,ParameterValue=<value> ParameterKey=SkipTlsVerify,ParameterValue="false" ParameterKey=EventSourceS3Bucket,ParameterValue="alb-logs-bucket-name"
```
## Limitations
- Error handling: If promtail is unresponsive, `lambda-promtail` will drop logs after `retry_count`, which defaults to 2.
- AWS CloudWatch [quotas](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html) state that the event size is limited to 256kb. `256 KB (maximum). This quota can't be changed.`

160
tools/lambda-promtail/template-eventbridge.yaml
Unescape View File

@ -0,0 +1,160 @@
AWSTemplateFormatVersion: "2010-09-09"
Description: >
lambda-promtail:
propagate Cloudwatch Logs to Loki/Promtail via Loki Write API.
Parameters:
WriteAddress:
Description: "Address to write to in the form of: http<s>://<location><:port>/loki/api/v1/push"
Type: String
Default: "http://localhost:8080/loki/api/v1/push"
ReservedConcurrency:
Description: The maximum of concurrent executions you want to reserve for the function.
Type: Number
Default: 2
Username:
Description: The basic auth username, necessary if writing directly to Grafana Cloud Loki.
Type: String
Default: ""
Password:
Description: The basic auth password, necessary if writing directly to Grafana Cloud Loki.
Type: String
Default: ""
NoEcho: true
BearerToken:
Description: The bearer token, necessary if target endpoint requires it.
Type: String
Default: ""
NoEcho: true
LambdaPromtailImage:
Description: The ECR image URI to pull and use for lambda-promtail.
Type: String
Default: ""
KeepStream:
Description: Determines whether to keep the CloudWatch Log Stream value as a Loki label when writing logs from lambda-promtail.
Type: String
Default: "false"
ExtraLabels:
Description: Comma separated list of extra labels, in the format 'name1,value1,name2,value2,...,nameN,valueN' to add to entries forwarded by lambda-promtail.
Type: String
Default: ""
OmitExtraLabelsPrefix:
Description: Whether or not to omit the prefix `__extra_` from extra labels defined in `ExtraLabels`.
Type: String
Default: "false"
TenantID:
Description: Tenant ID to be added when writing logs from lambda-promtail.
Type: String
Default: ""
SkipTlsVerify:
Description: Determines whether to verify the TLS certificate
Type: String
Default: "false"
EventSourceS3Bucket:
Description: The S3 bucket to listen event notifications from.
Type: String
Default: ""
Resources:
LambdaPromtailRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Description: "Lambda Promtail Role"
Policies:
- PolicyName: logs
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
- PolicyName: s3
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- s3:GetObject
Resource: arn:aws:s3:::thepalbi-lambda-lb-access-logs/*
RoleName: iam_for_lambda
LambdaPromtailFunction:
Type: AWS::Lambda::Function
Properties:
Code:
ImageUri: !Ref LambdaPromtailImage
MemorySize: 128
PackageType: Image
Timeout: 60
Role: !GetAtt LambdaPromtailRole.Arn
ReservedConcurrentExecutions: !Ref ReservedConcurrency
# # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-vpcconfig.html
# VpcConfig:
Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
Variables:
WRITE_ADDRESS: !Ref WriteAddress
USERNAME: !Ref Username
PASSWORD: !Ref Password
BEARER_TOKEN: !Ref BearerToken
KEEP_STREAM: !Ref KeepStream
EXTRA_LABELS: !Ref ExtraLabels
OMIT_EXTRA_LABELS_PREFIX: !Ref OmitExtraLabelsPrefix
TENANT_ID: !Ref TenantID
SKIP_TLS_VERIFY: !Ref SkipTlsVerify
LambdaPromtailVersion:
Type: AWS::Lambda::Version
Properties:
FunctionName: !Ref LambdaPromtailFunction
LambdaPromtailEventInvokeConfig:
Type: AWS::Lambda::EventInvokeConfig
Properties:
FunctionName: !Ref LambdaPromtailFunction
MaximumRetryAttempts: 2
Qualifier: !GetAtt LambdaPromtailVersion.Version
# EventBridge rule to route s3 object created events to lambda promtail
EventRule:
Type: AWS::Events::Rule
Properties:
Description: EventRule
State: ENABLED
EventPattern: # https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html#eb-filtering-data-
source:
- aws.s3
detail-type:
- "Object Created"
detail:
bucket:
name:
- !Ref EventSourceS3Bucket
Targets:
- Arn: !GetAtt LambdaPromtailFunction.Arn
Id: LambdaPromtailTarget
# Permission that allows EventBridge rule to trigger lambda promtail
EventRuleLambdaPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt LambdaPromtailFunction.Arn
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt EventRule.Arn
Outputs:
LambdaPromtailFunction:
Description: "Lambda Promtail Function ARN"
Value: !GetAtt LambdaPromtailFunction.Arn
Write Preview
Loading…
Cancel
Save