apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Justin Roberson	dd9f30fcb0	lambda-promtail: Support SNS events (#9964 ) What this PR does / why we need it: Some use cases (such as S3 bucket notifications) are better suited for SNS rather than SQS when you need to send events to multiple destinations. Adding a little bit of extra parsing logic for SNS events allows directly connecting an SNS topic to the Lambda Promtail function and bypasses the extra complexity of the SQS queue. This lower complexity is very helpful in cross-AWS-account use cases like CloudTrail event forwarding. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: - I just updated the SQS section of the docs to also reference SNS, since the functionality is basically identical. Happy to take suggestions on improvements there. Not sure if `CHANGELOG.md` needs to be updated as well or not. - This is semi-difficult to unit test since there isn't an existing test harness for event matching. Testing in situ on our own infrastructure seems to work. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
J Stickler	7797a98074	TOC Sync: Move configuration files under Configure (#9879 ) Which issue(s) this PR fixes: Fixes #8713 Special notes for your reviewer: Replaces PR [#8817](https://github.com/grafana/loki/pull/8817), which was stale and badly out of date. Part of the Database products table of contents alignment project. Moves configuration files from Configuration to new Configure heading in TOC. Updates front matter for the topic. Fixes links broken by the move. Updates the MakeFile to point to the new Configure DOCS_SOURCES_PATH.	3 years ago
Jack Baldry	8391e557b1	Update doc-validator to v3.0.0 (#9666 ) ## 3.0.0 ### Added - Structured output for use with [`reviewdog`](https://github.com/reviewdog/reviewdog). You can achieve the original error output by piping the output to the following `jq` expression: `jq -r '"ERROR: \(.location.path):\(.location.range.start.line):\(.location.range.start.column): \(.message)"'`. - Suggestions for simple link fixes. In GitHub, [`reviewdog`](https://github.com/reviewdog/reviewdog) comments these suggestions for convenient replacement. - Support for anchors referring to repeated headings. In the case that a page has multiple headings that share the same text, the renderer appends a zero indexed, numbered suffix to the identifier. For example, when there are two headings that are both "Heading text", the first anchor identifier is `heading-text` and the second is `heading-text-1`. - Error when running `doc-validator` on no files. Notes for reviewer: This also updates the `make-docs` procedure. Please ensure you can run `make docs` from the `docs/` directory before approving and merging this PR. --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	3 years ago
Christophe Collot	f2394355d8	feat(lambda-promtail): add cloudfront log file ingestion support (#9573 ) What this PR does / why we need it: This PR enables ingesting logs from Cloudfront log files stored in s3 (batch). The current setup only supports streaming Cloudfront logs through AWS Kinesis, whereas this PR implements the same flow as for VPC Flow logs, Load Balancer logs, and Cloudtrail logs (s3 --> SQS (optional) --> Lambda Promtail --> Loki) Special notes for your reviewer: + The Cloudfront log file format is different from the already implemented services, meaning we had to build yet another regex. AWS never bothered making all services follow the same log file naming convention but the "good" thing is that it's now very unlikely they will change it in the future. + The Cloudfront file name does not have any mention of the AWS account or the time of log it contains, it means we have to infer the log type from the filename format instead of finding the exact string "cloudfront" in the filename. This is why in `getLabels`, if no `type` parameter is found in the regex, we use the key corresponding to the name of the matching parser. + I introduced a new `parser` struct to group together several parameters specific to a type of log (and avoid relying too much on map key string matching and / or if statements for specific use cases) + I've been successfully running this code in several AWS environments for days. + I corrected a typo from my previous PR #9497 (wrong PR number in Changelog.md) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Christophe Collot	b709b32d6a	feat(lambda-promtail): add cloudtrail log ingestion support (#9497 ) What this PR does / why we need it: ### Add support for AWS CloudTrail audit logs ingestion using lambda-promtail Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for security and debugging purposes. However, I've experienced difficulties with it: + The AWS CloudTrail service is not well integrated with Prometheus (no metrics, no alerts) and I don't want to manage alerts in CloudWatch Alerts + The search experience is painful with CloudTrail via the AWS Console (I will not elaborate 😅). This PR allows ingesting CloudTrail audit logs sent to an S3 bucket using the same approach as VPC flow logs or Load Balancer logs. Special notes for your reviewer: + Because the Cloudtrail file format is not text but json, we stream the json CloudTrail records instead of using the already existing scanner. + Because the Cloudtrail filename format is not the same as for the Flow log or the Load balancer log files, we need to split the regexes by service (although many AWS services seem to share the same `defaultFilenameRegex`). + In the `getLabels` function, we expect the `type` parameter to be found in the filename using the Regex. For some log files (ex: Cloudfront log files). The file name has no reference to the service name. This is why, as a default, when no type is found in the name of the file, I set it to use the key of the matching Regex expression. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
bluesky6529	a9cde84c95	Fix typo on lambda-promtail docs (#9472 ) What this PR does / why we need it: fix for doc typo Which issue(s) this PR fixes:N/A Special notes for your reviewer: N/A	3 years ago
René Scheibe	e050d5c494	Docs: Update Loki GitHub URLs (#9080 ) * use `main` instead of `master` branch * fix outdated URLs	3 years ago
Christophe Collot	a013e9f342	Lambda-promtail: Add support for processing SQS messages, add promtailClient Type, add logger, upgrade dependencies and fix unexpected flushing behaviors (#8231 )	3 years ago
Travis Patterson	1c954166b8	Fix documentation linter errors (#8229 )	3 years ago
chriskuchin	4d5678aa17	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868 ) This add support to the S3 parsing logic for AWS VPC Flow logs. It also makes a small change to allow not printing log lines during processing. It leaves the default of printing the log lines in place. Signed-off-by: Thomas Belian <thomas.belian@bt909.de> Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>	3 years ago
loanshark	3f59fa96d8	lambda-promtail: Add kinesis data stream to use in terraform (#7632 ) What this PR does / why we need it: https://github.com/grafana/loki/pull/5977 With the addition of the kinesis data stream function add kinesis data stream to use in terraform	4 years ago
Juha Tiensyrjä	073c62051b	lambda-promtail: Add support for Kinesis data stream events (#5977 ) What this PR does / why we need it: This PR adds support for sending Kinesis data stream events to lambda-promtail. One use case would be e.g. to send CloudFront realtime logs to Loki. Which issue(s) this PR fixes: Fixes #5978 Special notes for your reviewer: n/a Checklist - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	4 years ago
Karen Miller	8ac08e65d9	Docs: improve wording and grammar (#6861 )	4 years ago
Conor Evans	b65e11ec9b	fix(documentation): correct spelling and scrape_config (#6454 ) Signed-off-by: Conor Evans <coevans@tcd.ie>	4 years ago
Artem Timchenko	ebb39db864	lambda-promtail: Add multi-tenancy support (#6102 ) * lambda-promtail: adding multi-tenancy support * update changelog and fix import order * fix import order * fix import order	4 years ago
Rastislav Barlik	3e17245743	Fix wrong indentation in example config (#5828 )	4 years ago
Jonathan Bédard Schami	776d5c41c7	Lambda-promtail: Enhance lambda-promtail to support adding extra labels from an environment variable value (#5359 ) * updated with ability to add extra tags * updated CHANGELOG.md * simplified extra label env variable to a simple comma seperated list in the form of "name1,value1,name2,value2,..." * refactored parsing of extra labels into its own method. Added unit tests. * removed invalid test, exit early on parseExtraLabels, refactored apply extra labels code to utility method, replaced hardcoded string with const global var in module. * updated docs, terraform template and cloud formation template * updated fmt.Println to fmt.Printf with line termination * updated to validate extra labels and added unit test for extra label validation behaviour * removed unused ExtraLabel struct * updated to use the new applyExtraLabels * pull latest * updated lambda promtail docs with instructions and example on how to use extra labels.	4 years ago
Andre Ziviani	699fffe9e5	lambda-promtail: Add ability to ingest logs from S3 (#5065 ) * Add ability to ingest logs from S3 on lambda-promtail * fix ci * fix typo * bump golang and alpine version * update changelog * add s3 permissions on terraform * use for_each instead of count * fix typo * improve function naming * add documentation and an example of a s3 file path * refact logic to identify event type * add missing iam permission to allow lambda to run inside a vpc * fix typo * allow lambda to access only specified s3 buckets * configure a default log retention policy on log group * add missing depends_on to make sure iam role is created before lambda function * update docs * fix label naming convention * fix merge conflict * use new backoff lib and update dependencies * add option to limit batch size * cache s3 client * update docs and terraform * address some feedback on PR * fix typo	4 years ago
Karen Miller	35ebe967ea	Docs: fix broken links (#4770 ) * Docs: fix broken links * Docs: fix another link name	5 years ago
Owen Diehl	6eab7be5b4	a few doc fixes in preparation for 2.4 (#4597 )	5 years ago
Callum Styan	e49e318b42	Improve lambda-promtail docs based on Owens review. (#4548 ) * Improve lambda-promtail docs based on Owens review. Signed-off-by: Callum Styan <callumstyan@gmail.com> * One last lambda-promtail docs improvement. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix bolding in lambda-promtail docs + include link to drop filter promtail pipeline stage.	5 years ago
Callum Styan	aa713f54f4	Modify lambda-promtail docs based on rewrite. (#4539 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
Karen Miller	24accf6a5b	Docs: correctly represent product name (#4416 ) The product name is Grafana Loki. The first use of the product name on each web page should be the full product name. After that, within each page, it is fine to use just Loki. This PR also set up some frontmatter with weights where it was missing on a few files, and correct where "Loki" was not capitalized.	5 years ago
Karen Miller	11a0d28b61	Docs: first draft, Loki accepts out-of-order writes (#4237 ) * Docs: first draft, Loki accepts out-of-order writes * Update docs/sources/architecture/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Karen Miller	475560ba13	Corrected capitalization and formatting of "Promtail" (#3860 )	5 years ago
Robby Milo	37a7189d4e	docs: add title to Lambda Promtail (#3525 )	5 years ago
Douglas Hall	e64409540b	Doc: Vague link text. (#2698 ) * Fixes remove vague link text #2680 * Fix remaining 'docs' links. * Accept suggestion. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Change HTTP API to Ruler API. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Correct typo. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Capitalize Github. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Capitalize Promtail. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com> * Add copy-edit suggestions from @achatterjee-grafana. * Incorporate @owen-d's changes. Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>	6 years ago
Ed Welch	1664a98111	Ran htmltest on the generated site and fixed all the broken links (#2600 )	6 years ago
Owen Diehl	dd7d821b31	Website doc link fixes (#2566 ) * index file links * non _index.md links * path fix * base readme links * fixes drone badge	6 years ago
Owen Diehl	2a596a77d7	Lambda-Promtail (#2282 ) * specs out lambda-promtail * init fn * udpates readme/template * lambda-promtail docs * lambda promtail includes source timestamp and uses context * non markdown links * new doc structure	6 years ago

30 Commits (89985399817dde28fe5b94de990133d904bc3e5b)