apitech/loki - loki - Gitea: Git with a cup of tea

Author	SHA1	Message	Date
Alexandre Steppé	dbfb19be49	feat: lambda-promtail; ensure messages to Kinesis are usable by refactoring parsing of KinesisEvent to match parsing of CWEvents + code cleanup (#13098 )	2 years ago
Karsten Jeschkies	c9a3ff53e4	Set unique timestamp in lambda-promtail. (#9560 )	3 years ago
Christophe Collot	b709b32d6a	feat(lambda-promtail): add cloudtrail log ingestion support (#9497 ) What this PR does / why we need it: ### Add support for AWS CloudTrail audit logs ingestion using lambda-promtail Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for security and debugging purposes. However, I've experienced difficulties with it: + The AWS CloudTrail service is not well integrated with Prometheus (no metrics, no alerts) and I don't want to manage alerts in CloudWatch Alerts + The search experience is painful with CloudTrail via the AWS Console (I will not elaborate 😅). This PR allows ingesting CloudTrail audit logs sent to an S3 bucket using the same approach as VPC flow logs or Load Balancer logs. Special notes for your reviewer: + Because the Cloudtrail file format is not text but json, we stream the json CloudTrail records instead of using the already existing scanner. + Because the Cloudtrail filename format is not the same as for the Flow log or the Load balancer log files, we need to split the regexes by service (although many AWS services seem to share the same `defaultFilenameRegex`). + In the `getLabels` function, we expect the `type` parameter to be found in the filename using the Regex. For some log files (ex: Cloudfront log files). The file name has no reference to the service name. This is why, as a default, when no type is found in the name of the file, I set it to use the key of the matching Regex expression. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago

Author

SHA1

Message

Date

Alexandre Steppé

feat: lambda-promtail; ensure messages to Kinesis are usable by refactoring parsing of KinesisEvent to match parsing of CWEvents + code cleanup (#13098 )

2 years ago

Karsten Jeschkies

c9a3ff53e4

Set unique timestamp in lambda-promtail. (#9560 )

3 years ago

Christophe Collot

b709b32d6a

feat(lambda-promtail): add cloudtrail log ingestion support (#9497 )

**What this PR does / why we need it**:
### Add support for AWS CloudTrail audit logs ingestion using
lambda-promtail

Calls to AWS APIs are logged in AWS Cloudtrail and are helpful for
security and debugging purposes. However, I've experienced difficulties
with it:
+ The AWS CloudTrail service is not well integrated with Prometheus (no
metrics, no alerts) and I don't want to manage alerts in CloudWatch
Alerts
+ The search experience is painful with CloudTrail via the AWS Console
(I will not elaborate 😅).

This PR allows ingesting CloudTrail audit logs sent to an S3 bucket
using the same approach as VPC flow logs or Load Balancer logs.

**Special notes for your reviewer**:

+ Because the Cloudtrail file format is not text but json, we stream the
json CloudTrail records instead of using the already existing scanner.
+ Because the Cloudtrail filename format is not the same as for the Flow
log or the Load balancer log files, we need to split the regexes by
service (although many AWS services seem to share the same
`defaultFilenameRegex`).
+ In the `getLabels` function, we expect the `type` parameter to be
found in the filename using the Regex. For some log files (ex:
Cloudfront log files). The file name has no reference to the service
name. This is why, as a default, when no type is found in the name of
the file, I set it to use the key of the matching Regex expression.

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Documentation added
- [x] Tests updated
- [x] `CHANGELOG.md` updated
- [x] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`

3 years ago

3 Commits (5aa5bbecaf142e6ae98066fcea809fbd0372ea46)