apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Christophe Collot	1c7f791a1e	lambda-promtail: add terraform code to process s3 logs through SQS (#8700 ) ## Add Terraform sample code to process AWS S3 log files through an SQS queue Lambda-promtail supports processing AWS s3 logs files through an SQS queue since #8231 As explained in the [documentation section of lambda-promtail](https://github.com/grafana/loki/blob/main/docs/sources/clients/lambda-promtail/_index.md#triggering-lambda-promtail-via-sqs:~:text=Triggering%20Lambda%2DPromtail,DLQ%20redrive%20feature.), this can be leveraged to re-process logs that lambda-promtail failed to process (or send to loki) using a main SQS queue and a secondary SQS dead-letter queue. AWS has a feature called `SQS redrive`, which enables routing messages pending in the DLQ back to the main (source) queue. This PR demonstrates how to spin up this architecture using terraform.	3 years ago
Mitch Hulscher	257225b486	refactor(lambda-promtail): apply terraform best practices (#8750 ) What this PR does / why we need it: I would like to offer this PR as a suggestion to improve the lambda-promtail terraform-module. I forked it to be able to deploy it more than once in an AWS account. I also applied terraform best-practices. I was hoping that perhaps these changes could be merged into upstream as well. Unlike https://github.com/grafana/loki/pull/8549 , I unfortunately did not end up making a separate commit for each change. If you would like me to create one or more issue(s) to address the points below, I'd be happy to do that as well. List of improvements: 1. Added `var.name` (defaults to lambda-promtail) so that this module can be deployed multiple times in the same AWS account. This allows us to define unique, non-conflicting names for: * the Lambda function * the CloudWatch log-group * the IAM role 2. Split IAM role policies per component; only assign permissions when required 3. Scope down permissions of the IAM role policies 4. During terraform-destroy, ensure CloudWatch log-group is removed after the lambda-function. An accidental invocation of the function could re-create an already destroyed log-group, leaving an orphaned log-group List of style changes: 1. Rename resources to `this` when there is only one instance of this resource-type 2. Add newline after `count\|before_each` and before `depends_on` 3. Group resources together and add a section comment 4. Add missing(?) statement-id to S3 AWS lambda permission Misc. 1. I added a `moves.tf` file to facilitate moving renamed resources in existing terraform statefiles. This prevents some resources from recreated. Can also be removed. These changes are backwards compatible, even though some resources will end up being re-created. A `terraform apply` should succeed (it did for me). Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) Signed-off-by: Mitch Hulscher <mitch.hulscher@lib.io>	3 years ago
James Callahan	6d59b4144e	lambda-promtail: Various Terraform fixes (#8549 ) * lambda-promtail: omit_extra_labels_prefix is a boolean This can be changed without a migration as it hasn't been in a release yet * lambda-promtail: fix typo * lambda-promtail: declare variables as a set * lambda-promtail: add a versions file with provider requirements * lambda-promtail: remove provider block - There's no reason to specify a region here - Empty provider blocks are not needed in terraform for quite some time now (and infact they end up producing a warning about their deprecation) * lambda-promtail: bucket_names can be the empty array Without this, you get an error that `resources` cannot be the empty list * lambda-promtail: convert inline policy to a aws_iam_policy_document	3 years ago
Christian Haudum	20696522fe	lambda-promtail: Add option to omit extra labels prefix `__extra_` (#8548 ) If you want to add an extra label env, lambda-promtail would add the __extra_ prefix, so that the resulting label is __extra_env. However, if you already have the env variable in other stream and you want to have consistent label naming, you would want to omit the prefix so the resulting label is env. Closes https://github.com/grafana/loki/issues/8471 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
chriskuchin	4d5678aa17	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868 ) This add support to the S3 parsing logic for AWS VPC Flow logs. It also makes a small change to allow not printing log lines during processing. It leaves the default of printing the log lines in place. Signed-off-by: Thomas Belian <thomas.belian@bt909.de> Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>	3 years ago
Thomas Belián	6fcc4a2712	Lambda-promtail: Add skip tls verify option (#8059 ) What this PR does / why we need it: This PR sets a possibility to use lambda-promtail with self signed certificates sometimes used in development systems. This setting is configurable via environment variable. Which issue(s) this PR fixes: Fixes #8013 Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	3 years ago
loanshark	3f59fa96d8	lambda-promtail: Add kinesis data stream to use in terraform (#7632 ) What this PR does / why we need it: https://github.com/grafana/loki/pull/5977 With the addition of the kinesis data stream function add kinesis data stream to use in terraform	3 years ago
Thomas Belian	b795672bba	feat(lambda-promtail): add bearer token support (#7333 ) It adds bearer token support for lambda-promtail, as it supports only http basic auth. Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	3 years ago
Jonathan	1b6d0bf3a0	feat: add kms ecryption (#6926 )	3 years ago
Conor Evans	3d49c09e34	fix(lambda-promtail): update default lambda_vpc_subnets / lambda_vpc_security_groups values to be appropriate empty list(string) value (#6453 ) Signed-off-by: Conor Evans <coevans@tcd.ie>	4 years ago
Artem Timchenko	ebb39db864	lambda-promtail: Add multi-tenancy support (#6102 ) * lambda-promtail: adding multi-tenancy support * update changelog and fix import order * fix import order * fix import order	4 years ago
Cristian Greco	02876514b3	lambda-promtail: fix typos in CloudFormation and Terraform files. (#5749 ) This fixes a typo in the CloudFormation template file (`Descripton`) which makes it unusable, as well as some other minor typos.	4 years ago
Jonathan Bédard Schami	776d5c41c7	Lambda-promtail: Enhance lambda-promtail to support adding extra labels from an environment variable value (#5359 ) * updated with ability to add extra tags * updated CHANGELOG.md * simplified extra label env variable to a simple comma seperated list in the form of "name1,value1,name2,value2,..." * refactored parsing of extra labels into its own method. Added unit tests. * removed invalid test, exit early on parseExtraLabels, refactored apply extra labels code to utility method, replaced hardcoded string with const global var in module. * updated docs, terraform template and cloud formation template * updated fmt.Println to fmt.Printf with line termination * updated to validate extra labels and added unit test for extra label validation behaviour * removed unused ExtraLabel struct * updated to use the new applyExtraLabels * pull latest * updated lambda promtail docs with instructions and example on how to use extra labels.	4 years ago
Andre Ziviani	699fffe9e5	lambda-promtail: Add ability to ingest logs from S3 (#5065 ) * Add ability to ingest logs from S3 on lambda-promtail * fix ci * fix typo * bump golang and alpine version * update changelog * add s3 permissions on terraform * use for_each instead of count * fix typo * improve function naming * add documentation and an example of a s3 file path * refact logic to identify event type * add missing iam permission to allow lambda to run inside a vpc * fix typo * allow lambda to access only specified s3 buckets * configure a default log retention policy on log group * add missing depends_on to make sure iam role is created before lambda function * update docs * fix label naming convention * fix merge conflict * use new backoff lib and update dependencies * add option to limit batch size * cache s3 client * update docs and terraform * address some feedback on PR * fix typo	4 years ago
Callum Styan	1493278ff2	Add a a parameter to keep/drop the stream label from cloudwatch. (#4494 ) * Add a a parameter to keep/drop the stream label from cloudwatch. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address Cyril's review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Ensure the keep stream value is properly passed in as an environment variable when deploying via CloudFormatioEnsure the keep stream value is properly passed in as an environment variable when deploying via CloudFormationn Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	05bb3cec4c	Rewrite lambda-promtail to use subscription filters. (#4315 ) * Rewrite lambda-promtail to use subscription filters. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update lambda library vendoring. * Fix small terraform comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Allow for lambda-promtail binary to pull basic auth credentials via env vars. Signed-off-by: Callum Styan <callumstyan@gmail.com> * remove mistakenly committed binary and zip Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update terraform to allow passing of variousn configuration values, like the write endpoint and log groups to subscribe to. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update cloudformation template to also use parameters/env vars. Signed-off-by: Callum Styan <callumstyan@gmail.com> * don't show password in template parameters + include examples of deploy commands in readme Signed-off-by: Callum Styan <callumstyan@gmail.com> * We can use terraform vars, with empty defaults, to allow for specifying VPC config for promtail-lambda deployment. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago

16 Commits (bcdc807dd34d7a4e2e1dcfdffecd869da66e6efe)