apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Christophe Collot	b709b32d6a	feat(lambda-promtail): add cloudtrail log ingestion support (#9497 ) What this PR does / why we need it: ### Add support for AWS CloudTrail audit logs ingestion using lambda-promtail Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for security and debugging purposes. However, I've experienced difficulties with it: + The AWS CloudTrail service is not well integrated with Prometheus (no metrics, no alerts) and I don't want to manage alerts in CloudWatch Alerts + The search experience is painful with CloudTrail via the AWS Console (I will not elaborate 😅). This PR allows ingesting CloudTrail audit logs sent to an S3 bucket using the same approach as VPC flow logs or Load Balancer logs. Special notes for your reviewer: + Because the Cloudtrail file format is not text but json, we stream the json CloudTrail records instead of using the already existing scanner. + Because the Cloudtrail filename format is not the same as for the Flow log or the Load balancer log files, we need to split the regexes by service (although many AWS services seem to share the same `defaultFilenameRegex`). + In the `getLabels` function, we expect the `type` parameter to be found in the filename using the Regex. For some log files (ex: Cloudfront log files). The file name has no reference to the service name. This is why, as a default, when no type is found in the name of the file, I set it to use the key of the matching Regex expression. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
chriskuchin	4d5678aa17	lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868 ) This add support to the S3 parsing logic for AWS VPC Flow logs. It also makes a small change to allow not printing log lines during processing. It leaves the default of printing the log lines in place. Signed-off-by: Thomas Belian <thomas.belian@bt909.de> Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>	3 years ago
Juha Tiensyrjä	073c62051b	lambda-promtail: Add support for Kinesis data stream events (#5977 ) What this PR does / why we need it: This PR adds support for sending Kinesis data stream events to lambda-promtail. One use case would be e.g. to send CloudFront realtime logs to Loki. Which issue(s) this PR fixes: Fixes #5978 Special notes for your reviewer: n/a Checklist - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	4 years ago

Author

SHA1

Message

Date

Christophe Collot

b709b32d6a

feat(lambda-promtail): add cloudtrail log ingestion support (#9497 )

**What this PR does / why we need it**:
### Add support for AWS CloudTrail audit logs ingestion using
lambda-promtail

Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for
security and debugging purposes. However, I've experienced difficulties
with it:
+ The AWS CloudTrail service is not well integrated with Prometheus (no
metrics, no alerts) and I don't want to manage alerts in CloudWatch
Alerts
+ The search experience is painful with CloudTrail via the AWS Console
(I will not elaborate 😅).

This PR allows ingesting CloudTrail audit logs sent to an S3 bucket
using the same approach as VPC flow logs or Load Balancer logs.

**Special notes for your reviewer**:

+ Because the Cloudtrail file format is not text but json, we stream the
json CloudTrail records instead of using the already existing scanner.
+ Because the Cloudtrail filename format is not the same as for the Flow
log or the Load balancer log files, we need to split the regexes by
service (although many AWS services seem to share the same
`defaultFilenameRegex`).
+ In the `getLabels` function, we expect the `type` parameter to be
found in the filename using the Regex. For some log files (ex:
Cloudfront log files). The file name has no reference to the service
name. This is why, as a default, when no type is found in the name of
the file, I set it to use the key of the matching Regex expression.

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Documentation added
- [x] Tests updated
- [x] `CHANGELOG.md` updated
- [x] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`

3 years ago

chriskuchin

4d5678aa17

lambda-promtail: Add support for VPC flow Logs to lambda-promtail (#7868 )

This add support to the S3 parsing logic for AWS VPC Flow logs.

It also makes a small change to allow not printing log lines during
processing. It leaves the default of printing the log lines in place.

Signed-off-by: Thomas Belian <thomas.belian@bt909.de>
Co-authored-by: Thomas Belián <72987757+bt909@users.noreply.github.com>

3 years ago

Juha Tiensyrjä

073c62051b

lambda-promtail: Add support for Kinesis data stream events (#5977 )

**What this PR does / why we need it**:

This PR adds support for sending Kinesis data stream events to lambda-promtail. One use case would be e.g. to send CloudFront realtime logs to Loki.

**Which issue(s) this PR fixes**:

Fixes #5978 

**Special notes for your reviewer**:

n/a

**Checklist**
- [x] Documentation added
- [x] Tests updated
- [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`.
- [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`

4 years ago

3 Commits (415a997fb989fc7abdc4f4a2593bd2f703cfbaed)