apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Max Forasteiro	9bf08f7cc0	feat(lambda-promtail): build lambda with zip file (#13787 ) Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	1 year ago
samuelebstein	2d92fff2aa	feat(lambda-promtail): Adding S3 log parser support for AWS GuardDuty (#13148 )	2 years ago
keyolk	8328345a8b	lambda-promtail: fix IAM policy for clouddwatch log stream (#10909 )	2 years ago
Đỗ Trọng Hải	ec54c723eb	feat(lambda/promtail): support dropping labels (#10755 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #10669 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: hainenber <dotronghai96@gmail.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Christophe Collot	1c7f791a1e	lambda-promtail: add terraform code to process s3 logs through SQS (#8700 ) ## Add Terraform sample code to process AWS S3 log files through an SQS queue Lambda-promtail supports processing AWS s3 logs files through an SQS queue since #8231 As explained in the [documentation section of lambda-promtail](https://github.com/grafana/loki/blob/main/docs/sources/clients/lambda-promtail/_index.md#triggering-lambda-promtail-via-sqs:~:text=Triggering%20Lambda%2DPromtail,DLQ%20redrive%20feature.), this can be leveraged to re-process logs that lambda-promtail failed to process (or send to loki) using a main SQS queue and a secondary SQS dead-letter queue. AWS has a feature called `SQS redrive`, which enables routing messages pending in the DLQ back to the main (source) queue. This PR demonstrates how to spin up this architecture using terraform.	3 years ago
Mitch Hulscher	257225b486	refactor(lambda-promtail): apply terraform best practices (#8750 ) What this PR does / why we need it: I would like to offer this PR as a suggestion to improve the lambda-promtail terraform-module. I forked it to be able to deploy it more than once in an AWS account. I also applied terraform best-practices. I was hoping that perhaps these changes could be merged into upstream as well. Unlike https://github.com/grafana/loki/pull/8549 , I unfortunately did not end up making a separate commit for each change. If you would like me to create one or more issue(s) to address the points below, I'd be happy to do that as well. List of improvements: 1. Added `var.name` (defaults to lambda-promtail) so that this module can be deployed multiple times in the same AWS account. This allows us to define unique, non-conflicting names for: * the Lambda function * the CloudWatch log-group * the IAM role 2. Split IAM role policies per component; only assign permissions when required 3. Scope down permissions of the IAM role policies 4. During terraform-destroy, ensure CloudWatch log-group is removed after the lambda-function. An accidental invocation of the function could re-create an already destroyed log-group, leaving an orphaned log-group List of style changes: 1. Rename resources to `this` when there is only one instance of this resource-type 2. Add newline after `count\|before_each` and before `depends_on` 3. Group resources together and add a section comment 4. Add missing(?) statement-id to S3 AWS lambda permission Misc. 1. I added a `moves.tf` file to facilitate moving renamed resources in existing terraform statefiles. This prevents some resources from recreated. Can also be removed. These changes are backwards compatible, even though some resources will end up being re-created. A `terraform apply` should succeed (it did for me). Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) Signed-off-by: Mitch Hulscher <mitch.hulscher@lib.io>	3 years ago
James Callahan	6d59b4144e	lambda-promtail: Various Terraform fixes (#8549 ) * lambda-promtail: omit_extra_labels_prefix is a boolean This can be changed without a migration as it hasn't been in a release yet * lambda-promtail: fix typo * lambda-promtail: declare variables as a set * lambda-promtail: add a versions file with provider requirements * lambda-promtail: remove provider block - There's no reason to specify a region here - Empty provider blocks are not needed in terraform for quite some time now (and infact they end up producing a warning about their deprecation) * lambda-promtail: bucket_names can be the empty array Without this, you get an error that `resources` cannot be the empty list * lambda-promtail: convert inline policy to a aws_iam_policy_document	3 years ago
Christian Haudum	20696522fe	lambda-promtail: Add option to omit extra labels prefix `__extra_` (#8548 ) If you want to add an extra label env, lambda-promtail would add the __extra_ prefix, so that the resulting label is __extra_env. However, if you already have the env variable in other stream and you want to have consistent label naming, you would want to omit the prefix so the resulting label is env. Closes https://github.com/grafana/loki/issues/8471 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christophe Collot	ec33a891df	lambda-promtail: remove duplicate key in terraform config sample (#8137 ) What this PR does / why we need it: This PR removes a duplicate key in the terraform configuration sample Which issue(s) this PR fixes: None Special notes for your reviewer: This is a trivial PR	3 years ago
chriskuchin	4d5678aa17	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868 ) This add support to the S3 parsing logic for AWS VPC Flow logs. It also makes a small change to allow not printing log lines during processing. It leaves the default of printing the log lines in place. Signed-off-by: Thomas Belian <thomas.belian@bt909.de> Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>	3 years ago
Thomas Belián	6fcc4a2712	Lambda-promtail: Add skip tls verify option (#8059 ) What this PR does / why we need it: This PR sets a possibility to use lambda-promtail with self signed certificates sometimes used in development systems. This setting is configurable via environment variable. Which issue(s) this PR fixes: Fixes #8013 Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	3 years ago
loanshark	3f59fa96d8	lambda-promtail: Add kinesis data stream to use in terraform (#7632 ) What this PR does / why we need it: https://github.com/grafana/loki/pull/5977 With the addition of the kinesis data stream function add kinesis data stream to use in terraform	3 years ago
Thomas Belian	b795672bba	feat(lambda-promtail): add bearer token support (#7333 ) It adds bearer token support for lambda-promtail, as it supports only http basic auth. Signed-off-by: Thomas Belian <thomas.belian@bt909.de>	3 years ago
Jonathan	6b9bf0b5f4	feat: add kms and ec2 permission (#7058 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: This PR will help to increase a necessary policy (kms and ec2) to resource log group and lambda. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Jonathan	1b6d0bf3a0	feat: add kms ecryption (#6926 )	3 years ago
Artem Timchenko	ebb39db864	lambda-promtail: Add multi-tenancy support (#6102 ) * lambda-promtail: adding multi-tenancy support * update changelog and fix import order * fix import order * fix import order	4 years ago
Jonathan Bédard Schami	776d5c41c7	Lambda-promtail: Enhance lambda-promtail to support adding extra labels from an environment variable value (#5359 ) * updated with ability to add extra tags * updated CHANGELOG.md * simplified extra label env variable to a simple comma seperated list in the form of "name1,value1,name2,value2,..." * refactored parsing of extra labels into its own method. Added unit tests. * removed invalid test, exit early on parseExtraLabels, refactored apply extra labels code to utility method, replaced hardcoded string with const global var in module. * updated docs, terraform template and cloud formation template * updated fmt.Println to fmt.Printf with line termination * updated to validate extra labels and added unit test for extra label validation behaviour * removed unused ExtraLabel struct * updated to use the new applyExtraLabels * pull latest * updated lambda promtail docs with instructions and example on how to use extra labels.	4 years ago
Andre Ziviani	699fffe9e5	lambda-promtail: Add ability to ingest logs from S3 (#5065 ) * Add ability to ingest logs from S3 on lambda-promtail * fix ci * fix typo * bump golang and alpine version * update changelog * add s3 permissions on terraform * use for_each instead of count * fix typo * improve function naming * add documentation and an example of a s3 file path * refact logic to identify event type * add missing iam permission to allow lambda to run inside a vpc * fix typo * allow lambda to access only specified s3 buckets * configure a default log retention policy on log group * add missing depends_on to make sure iam role is created before lambda function * update docs * fix label naming convention * fix merge conflict * use new backoff lib and update dependencies * add option to limit batch size * cache s3 client * update docs and terraform * address some feedback on PR * fix typo	4 years ago
Callum Styan	1493278ff2	Add a a parameter to keep/drop the stream label from cloudwatch. (#4494 ) * Add a a parameter to keep/drop the stream label from cloudwatch. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address Cyril's review comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Ensure the keep stream value is properly passed in as an environment variable when deploying via CloudFormatioEnsure the keep stream value is properly passed in as an environment variable when deploying via CloudFormationn Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	05bb3cec4c	Rewrite lambda-promtail to use subscription filters. (#4315 ) * Rewrite lambda-promtail to use subscription filters. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update lambda library vendoring. * Fix small terraform comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Allow for lambda-promtail binary to pull basic auth credentials via env vars. Signed-off-by: Callum Styan <callumstyan@gmail.com> * remove mistakenly committed binary and zip Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update terraform to allow passing of variousn configuration values, like the write endpoint and log groups to subscribe to. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update cloudformation template to also use parameters/env vars. Signed-off-by: Callum Styan <callumstyan@gmail.com> * don't show password in template parameters + include examples of deploy commands in readme Signed-off-by: Callum Styan <callumstyan@gmail.com> * We can use terraform vars, with empty defaults, to allow for specifying VPC config for promtail-lambda deployment. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago

20 Commits (bbbd82bc73322d662ba81efeda3884efcdc09708)