apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Sandeep Sukhani	6069df8f7d	ingestion: native otlp ingestion support (#10727 ) What this PR does / why we need it: Add support for natively supporting logs ingestion in OTLP format. `/otlp/v1/logs` is the new endpoint where users can push logs in OTLP format. It accepts logs serialized in JSON or proto format. Since OTEL format is very different than what Loki storage model, here is how data in OTEL format will be mapped to Loki data model: * Index labels: The Resource Attributes map quite well to Index labels in Loki since both usually identify the source of the logs. The problem however is that Resource attributes in OTLP can have an unbounded number of values while Loki has a default limit of having up to 30 labels. Since Index labels in Loki can largely drive the kind of querying experience the users are going to have, we have chosen select attributes which would be picked as Index Labels. The ones that are not picked up as Index labels would be stored as Structured Metadata with each log entry. * Timestamp: LogRecord.TimeUnixNano * LogLine: LogRecord.Body holds the body of the log. However, since Loki only supports Log body in string format, we will stringify non-string values using [AsString method from OTEL collector lib](`ab3d6c5b64/pdata/pcommon/value.go (L353)`). * Structured Metadata: Anything which can’t be stored in Index labels and LogLine. Here is a non-exhaustive list of what will be stored in Structured Metadata to give a sense of what it will hold: * Resource Attributes not stored as Index labels is replicated and stored with each log entry. * Everything under InstrumentationScope is replicated and stored with each log entry. * Everything under LogRecord except LogRecord.Body, LogRecord.TimeUnixNano and sometimes LogRecord.ObservedTimestamp. NOTES: * Since Loki does not support `.` or any other special characters other than `_` in label names, we replace all non-supported characters with `_`. * Since Loki only supports string in values of Index Labels and Structured Metadata, all the complex types are converted as follows: * Map would be flattened into label keys using `_` as separator, same as how we do it in [json parser in LogQL](https://grafana.com/docs/loki/latest/query/log_queries/#json). * Everything else is stringified using [AsString method from OTEL collector lib](`ab3d6c5b64/pdata/pcommon/value.go (L353)`) Special notes for your reviewer: I will open follow-up PRs for: * Documentation * Make blessed attributes list configurable per tenant. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label	2 years ago
Ashwanth	98551ceb9a	use go 1.21.3 and go-grpc v1.56.3 (#10869 ) What this PR does / why we need it: Fixes CVE-2023-39325 / CVE-2023-44487 https://github.com/grpc/grpc-go/releases/tag/v1.56.3 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Dylan Guedes	e81dd8aed2	Loki: Bump `dskit` to inherit new backoff configs (#9484 ) What this PR does / why we need it: Bump `dskit` to inherit https://github.com/grafana/dskit/pull/296. It will allow us to configure custom backoff timeouts. Which issue(s) this PR fixes: N/A	3 years ago
dependabot[bot]	fd72aa8908	Bump cloud.google.com/go/storage from 1.27.0 to 1.29.0 (#8276 ) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	3 years ago
Cyril Tovena	8f693e77b0	Update Prometheus to latest (#5545 ) * Update Prometheus to latest Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update promql.QueryTracker Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes querier test to use new grafana/regexp Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test with new textparse signature Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Mauro Stettler	4b59556342	Update cortex to newer version (#4478 ) * update cortex and thanos to newer version * pin k8s.io/apimachinery * fixes for newer libs * add missing method to mock wal data collector * fixing test by using different gokit level pkg * make check-mod	4 years ago
Nick Pillitteri	79a7bdbabc	Chore: Use services and modules from grafana/dskit (#4196 ) * Update Cortex vendor to bcf15611345da65e03604a08a0213aaaaac49e54 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Use services and modules packages from grafana/dskit Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Update promtail test to set default weaveworks server network type Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Punctuation Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Arve Knudsen	3e385464c6	Chore: Upgrade Prometheus and Cortex (#4017 ) * Chore: Upgrade Prometheus Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Upgrade Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Upgrade Cortex Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix build errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Undo unnecessary changes Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Michel Hollands	9137ef921b	Update cortex to 1.8 (#3627 ) * Update Cortex to 1.8.1 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Tidy up Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Marco Pracucci	94041307b1	Upgrade Cortex (#3466 ) * Upgrade Cortex Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated vendor/modules.txt Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Peter Štibraný	e4a5360586	Update vendored Cortex to master (1.1+) (#2149 ) * Updated Cortex to master (post 1.1) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use new modules system from Cortex. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Treat query errors reported by chunk store as client errors. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed obsolete tests. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix race for ct variable. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix test. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Owen Diehl	b5c03b0802	Fix duplicate labels (update cortex) (#1280 ) * Update to latest cortex to fix duplicate labels Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Start the licycler in the ingester Required since this cortex change in `00566f61af`\#diff-dcd65a4ff2de424be70a561f141e067bR180 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go mod tidy Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move cortex to a fork for ARM support Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * updates cortex to master Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * go mod tidy/vendor Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	6 years ago
sh0rez	11ad356396	feat(promtail): initContainers (#655 ) * fix(promtail/targets): remove dependency on prometheus/relabel `filetargetmanager.go` had a dependency on prometheus/relabel, a package that has been removed in favor of prometheus/pkg/relabel. This converts the code to the new package, to allow the prometheus vendor be updated to master * chore(vendor): update prometheus vendor Updates prometheus vendor to current master, to add support for InitContainers in kubernetes service discovery * chore(vendor): do dep's homework Gives dep hints on how to resolve the vendor so that it work * fix(promtail/targets): non-nil check Accidentally checked against the wrong labelSet, this one can never get nil	7 years ago
Adam Shannon	eb390523e3	all: update dependencies (#211 ) * all: update dependencies * Update some more dependancies and pin the right versions for Prometheus deps. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	7c1355cedf	update vendor Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	a1d66f42c0	Add vendor stuff.	8 years ago

16 Commits (5ec40636be19aeb4b7af3d9413d33ffa7edea702)