Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
loki/tools/lambda-promtail
Michel Hollands eb3cff4625
Update Go version to 1.21.2 in Dockerfiles (#10805)
2 years ago
..
lambda-promtail Add support for EventBridge s3 events in lambda-promtail (#10449) 2 years ago
testdata Add support for EventBridge s3 events in lambda-promtail (#10449) 2 years ago
Dockerfile Update Go version to 1.21.2 in Dockerfiles (#10805) 2 years ago
Makefile Fix regression in lambda-promtail tool that yields incorrect label names (#8547) 2 years ago
README.md Add CloudFormation template for lambda + EventBridge setup (#10540) 2 years ago
go.mod Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231) 2 years ago
go.sum Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231) 2 years ago
main.tf lambda-promtail: add terraform code to process s3 logs through SQS (#8700) 2 years ago
moves.tf lambda-promtail: fix/refactor SQS integration in Terraform module (#9909) 2 years ago
sqs.tf lambda-promtail: fix/refactor SQS integration in Terraform module (#9909) 2 years ago
template-eventbridge.yaml Add CloudFormation template for lambda + EventBridge setup (#10540) 2 years ago
template.yaml Lambda Promtail: Add logs:PutSubscriptionFilter to Lambda Promtail role (#10426) 2 years ago
variables.tf lambda-promtail: fix/refactor SQS integration in Terraform module (#9909) 2 years ago
versions.tf lambda-promtail: Various Terraform fixes (#8549) 2 years ago

README.md

lambda-promtail

This is a sample deployment for lambda-promtail - Below is a brief explanation of what we have generated for you:

.
├── Makefile                    <-- Make to automate build
├── Dockerfile                  <-- Uses the AWS Lambda Go base image
├── README.md                   <-- This instructions file
├── lambda-promtail             <-- Source code for a lambda function
│   └── main.go                 <-- Lambda function code

Requirements

  • AWS CLI already configured with Administrator permission
  • Terraform

If you want to modify the lambda-promtail code you will also need:

Setup process

Building and Packaging

The provided Makefile has targets build, and clean.

build builds the lambda-promtail as a Go static binary. To build the container image properly you should run docker build . -f tools/lambda-promtail/Dockerfile from the root of the Loki repository,you can upload this image to your AWS ECR and use via Lambda. clean will remove the built Go binary.

Packaging and deployment

The easiest way to deploy to AWS Lambda using the Golang runtime is to use the lambda-promtail image by uploading it to your ECR.

Alternatively you can build the Go binary and upload it to Lambda as a zip:

GOOS=linux CGO_ENABLED=0 go build main.go
zip function.zip main

To deploy your application for the first time, first make sure you've set the following value in the Terraform file:

  • WRITE_ADDRESS

This is the Loki Write API compatible endpoint that you want to write logs to, either promtail or Loki.

The lambda-promtail code picks this value up via an environment variable.

Also, if your deployment requires a VPC configuration, make sure to edit the vpc_config field in main.tf manually. Additonal documentation for the Lambda specific Terraform configuration is here. If you want to link kinesis data stream to Lambda as event source, see here.

lambda-promtail supports authentication either using HTTP Basic Auth or using Bearer Token.
For development purposes you can set the environment variable SKIP_TLS_VERIFY to true, so you can use self signed certificates, but this is not recommended in production. Default is false.

Then use Terraform to deploy:

## use cloudwatch log group
terraform apply -var "<ecr-repo>:<tag>" -var "write_address=https://your-loki-url/loki/api/v1/push" -var "password=<basic-auth-pw>" -var "username=<basic-auth-username>" -var 'bearer_token=<bearer-token>' -var 'log_group_names=["log-group-01", "log-group-02"]' -var 'extra_labels="name1,value1,name2,value2"' -var "tenant_id=<value>" -var 'skip_tls_verify="false"'
## use kinesis data stream
terraform apply -var "<ecr-repo>:<tag>" -var "write_address=https://your-loki-url/loki/api/v1/push" -var "password=<basic-auth-pw>" -var "username=<basic-auth-username>" -var 'kinesis_stream_name=["kinesis-stream-01", "kinesis-stream-02"]' -var 'extra_labels="name1,value1,name2,value2"' -var "tenant_id=<value>" -var 'skip_tls_verify="false"'

or CloudFormation:

aws cloudformation create-stack --stack-name lambda-promtail-stack --template-body file://template.yaml --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region us-east-2 --parameters ParameterKey=WriteAddress,ParameterValue=https://your-loki-url/loki/api/v1/push ParameterKey=Username,ParameterValue=<basic-auth-username> ParameterKey=Password,ParameterValue=<basic-auth-pw> ParameterKey=BearerToken,ParameterValue=<bearer-token> ParameterKey=LambdaPromtailImage,ParameterValue=<ecr-repo>:<tag> ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2" ParameterKey=TenantID,ParameterValue=<value> ParameterKey=SkipTlsVerify,ParameterValue="false"

Appendix

Golang installation

Please ensure Go 1.x (where 'x' is the latest version) is installed as per the instructions on the official golang website: https://golang.org/doc/install

For example:

GO_VERSION=go1.16.6.linux-amd64.tar.gz

rm -rf /usr/local/bin/go*
rm -rf /usr/local/go
curl -O https://storage.googleapis.com/golang/$GO_VERSION
tar -zxvf $GO_VERSION
sudo mv go /usr/local/
rm $GO_VERSION
ln -s /usr/local/go/bin/* /usr/local/bin/

A quickstart way would be to use Homebrew, chocolatey or your Linux package manager.

Homebrew (Mac)

Issue the following command from the terminal:

brew install golang

If it's already installed, run the following command to ensure it's the latest version:

brew update
brew upgrade golang

Chocolatey (Windows)

Issue the following command from the powershell:

choco install golang

If it's already installed, run the following command to ensure it's the latest version:

choco upgrade golang

CloudFormation and S3 events

Lambda-promtail lets you send logs from different services that use S3 as their logs destination (ALB, VPC Flow, CloudFront access logs, etc.). For this, you need to configure S3 bucket notifications to trigger the lambda-promtail deployment. However, when using CloudFormation to encode infrastructure, there is a known issue when configuring AWS::S3::BucketNotification and the resource that will be triggered by the notification in the same stack.

To manage the issue, AWS introduced S3 event notifications with Event Bridge. In that way, when an object gets created in a S3 bucket, this sends an event to an EventBridge bus, and you can create a rule to send those events to Lambda-promtail.

The template-eventbridge.yaml CloudFormation template configures Lambda-promtail with EventBridge, for the use case mentioned above:

aws cloudformation create-stack --stack-name lambda-promtail-stack --template-body file://template-eventbridge.yaml --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region us-east-2 --parameters ParameterKey=WriteAddress,ParameterValue=https://your-loki-url/loki/api/v1/push ParameterKey=Username,ParameterValue=<basic-auth-username> ParameterKey=Password,ParameterValue=<basic-auth-pw> ParameterKey=BearerToken,ParameterValue=<bearer-token> ParameterKey=LambdaPromtailImage,ParameterValue=<ecr-repo>:<tag> ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2" ParameterKey=TenantID,ParameterValue=<value> ParameterKey=SkipTlsVerify,ParameterValue="false" ParameterKey=EventSourceS3Bucket,ParameterValue="alb-logs-bucket-name"

Limitations

  • Error handling: If promtail is unresponsive, lambda-promtail will drop logs after retry_count, which defaults to 2.
  • AWS CloudWatch quotas state that the event size is limited to 256kb. 256 KB (maximum). This quota can't be changed.