apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
renovate-sh-app[bot]	8317f7e462	fix(deps): update module github.com/prometheus/client_golang to v1.23.2 (main) (#19763 ) Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	6 months ago
renovate-sh-app[bot]	33eeab62c1	fix(deps): update module github.com/oschwald/geoip2-golang to v2 (main) (#19799 ) Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	6 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206 )	7 months ago
Christian Haudum	860af9ec13	chore(test): Fix Promtail structured metadata stage test (#18603 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Sandeep Sukhani	db72d63211	fix: ensure we do not add duplicate structured metadata from stream labels and extracted map (#18523 )	9 months ago
Shantanu Alshi	61e81c5a4c	chore(otlp): OTLP Metrics (#18281 )	10 months ago
Paul Rogers	27431b7e7e	chore: Linting update for new golangci (#16572 )	1 year ago
renovate[bot]	b37eefe24f	fix(deps): update github.com/prometheus/prometheus (main) (#15950 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
renovate[bot]	3abb3b116d	fix(deps): update module github.com/hashicorp/golang-lru to v2 (#14979 ) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	1 year ago
Cyril Tovena	90c5dbf0d3	chore: Revert syntax module (#14831 )	1 year ago
Cyril Tovena	28c9645d10	chore: Introduce logql/syntax go module (#14790 )	1 year ago
chengehe	9ebc70f406	chore: add missing backtick (#14712 )	1 year ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383 )	2 years ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071 )	2 years ago
lencap	95942cbf1b	chore: fix some comments (#13830 ) Signed-off-by: lencap <techgeek@189.cn> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	2 years ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671 ) Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Paulin Todev	39a7181a60	fix: promtail; clean up metrics generated from logs after a config reload. (#11882 )	2 years ago
Ed Welch	824f5aa20a	chore: fix submodule for v3 (#12438 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
stayweek	420169aaab	fix: fix some comments (#12417 ) Signed-off-by: stayweek <jiangtengtao@outlook.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Danny Kopping	e523809216	bug(volume): return 400 error for invalid volume request (#11313 ) What this PR does / why we need it: Making a request to the `/volume{,_range}` endpoints results in a 500 is a LogQL expression is given instead of the expected label matchers. The reason for this is that the error is not wrapped in such a way as to be interpreted by `ClientHTTPStatusAndError` correctly. 5xx responses from the querier are automatically retried, so any invalid request currently results in 5 retries which are unnecessary. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Ashwanth	766f27645d	config: remove flags & YAML settings deprecated in 2.9.x (#11151 ) What this PR does / why we need it: - remove `-ruler.evaluation-delay-duration` - remove `boltdb.shipper.compactor.deletion-mode` - remove compactor flags with prefix`-boltdb.shipper.compactor.` - remove `validation.enforce-metric-name` - remove `log.use-buffered`, `log.use-sync` Special notes for your reviewer: `log.use-buffered`, `log.use-sync` are not mentioned in the upgrade guide since they were hidden fields Checklist* - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [X] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Karsten Jeschkies	4ed3b117d2	Use newest build image 0.31.2 with golangci-lint update. (#11118 ) What this PR does / why we need it: This just updates the Loki build imaage changed in #11114. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	3 years ago
Raul Gonzales	7d5475541c	promtail: enable structured_metadata convert labels (#10752 ) What this PR does / why we need it: There are several labels that can be extracted by the supports service discovery providers, however the structured metadata wouldn't allow to parse labels to it. This PR allows the users to update the configuration to read from labels discovered, down below are some of the metadata from a pod, that could be of interested of anyone to have that pushed to Loki without creating several streams due to either high-carnality or proliferation of idle chunks: ``` __address__="10.0.165.166:3100" __meta_kubernetes_namespace="loki" __meta_kubernetes_pod_annotation_checksum_config="f008ef39c4f3230c5e153c6e9fde51a678e6f0b1087b3be7d91b8142a03dbe7c" __meta_kubernetes_pod_annotation_kubectl_kubernetes_io_default_container="loki" __meta_kubernetes_pod_annotationpresent_checksum_config="true" __meta_kubernetes_pod_annotationpresent_kubectl_kubernetes_io_default_container="true" __meta_kubernetes_pod_container_id="containerd://e6aef5c39c3fe806d39ea536b198c8c39cdee01d23141fed91e8fe1573cc5e08" __meta_kubernetes_pod_container_image="docker.io/grafana/loki:2.9.1" __meta_kubernetes_pod_container_init="false" __meta_kubernetes_pod_container_name="loki" __meta_kubernetes_pod_container_port_name="http-metrics" __meta_kubernetes_pod_container_port_number="3100" __meta_kubernetes_pod_container_port_protocol="TCP" __meta_kubernetes_pod_controller_kind="StatefulSet" __meta_kubernetes_pod_controller_name="loki-write" __meta_kubernetes_pod_host_ip="10.249.129.191" __meta_kubernetes_pod_ip="10.0.165.166" __meta_kubernetes_pod_label_app_kubernetes_io_component="write" __meta_kubernetes_pod_label_app_kubernetes_io_instance="loki" __meta_kubernetes_pod_label_app_kubernetes_io_name="loki" __meta_kubernetes_pod_label_app_kubernetes_io_part_of="memberlist" __meta_kubernetes_pod_label_controller_revision_hash="loki-write-75847dc945" __meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name="loki-write-1" __meta_kubernetes_pod_labelpresent_app_kubernetes_io_component="true" __meta_kubernetes_pod_labelpresent_app_kubernetes_io_instance="true" __meta_kubernetes_pod_labelpresent_app_kubernetes_io_name="true" __meta_kubernetes_pod_labelpresent_app_kubernetes_io_part_of="true" __meta_kubernetes_pod_labelpresent_controller_revision_hash="true" __meta_kubernetes_pod_labelpresent_statefulset_kubernetes_io_pod_name="true" __meta_kubernetes_pod_name="loki-write-1" __meta_kubernetes_pod_node_name="ip-10-249-129-191.ec2.internal" __meta_kubernetes_pod_phase="Running" __meta_kubernetes_pod_ready="false" __meta_kubernetes_pod_uid="b3937321-fe90-4e15-ac94-495c8fdb9202" ``` Which issue(s) this PR fixes: https://github.com/grafana/loki/issues/10766 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Rens Groothuijsen	f51ee849b0	fix(promtail): correctly parse list of drop stage sources from YAML (#10848 ) What this PR does / why we need it: This PR fixes YAML parsing of the `source` field in the drop stage when said field is a list of strings. Because this list is not being recognized in the code as a list of strings, but rather as a list of generic `interface{}` elements, it fails a type check and throws an error. In order to fix this, an `interface{}` list is manually converted to a `string` list. The problem was already reproducible with existing tests, but the test YAML mistakenly referred to the field as `sources` rather than `source`. Which issue(s) this PR fixes: Fixes #10095 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Vladyslav Diachenko	00a3c5b026	renamed `non indexed labels` feature to `structured metadata` (#10432 ) Loki team has made a decision to rename the feature because the new name is more precise. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	3 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Vladyslav Diachenko	0fc934e4ea	non_indexed_labels stage: additional test-case (#10170 ) added additional test-case to cover match and non_indexed_labels stages together. Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
Vladyslav Diachenko	416e01ca8e	[promtail] non_indexed_labels stage (#9986 ) What this PR does / why we need it: added non_indexed_labels stage to copy values from extracted data tonon-indexed-labels field of the Entry. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
Karsten Jeschkies	2dfac72835	Clarify that label filter expressions are not supported in promtail. (#9894 ) What this PR does / why we need it: Some users of Promtail seem to be confused that certain types of filter expressions are not supported in Promtail's `match` stage. This change clarifies what kind of filter expressions are supported. Which issue(s) this PR fixes: Fixes #5209 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Jan Jansen	8abe080fd6	Promtail: improve behavior of partial lines (#9508 ) What this PR does / why we need it: I started to see OOM after upgrading to a promtail version with #8497. So played around to make partial line behavior more stable. You can now truncate partial lines and configure max partial lines buffer. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Jan Jansen <jan.jansen@gdata.de> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
cui fliter	520e434446	fix function names in comments (#9415 )	3 years ago
李国忠	0295fd4698	[new feature] promtail: Add a Promtail stage for probabilistic sampling (#7127 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: The sampling stage can be directly sampled. The implementation of sampling is to use the algorithm in jaeger go client ``` pipeline_stages: - sampling: rate: 0.1 ``` or it can be used with match for precise sampling. ``` pipeline_stages: - json: expressions: app: - match: pipeline_name: "app2" selector: "{app=\"poki\"}" stages: - sampling: rate: 0.1 ``` Which issue(s) this PR fixes: Fixes #6654 Special notes for your reviewer: The promtail 'rate' stage is also used with the 'match' stage for log filtering.This design makes the code very clean. Other log agents vector have log filtering built into the sampling operator, which I think is too complicated https://vector.dev/docs/reference/configuration/transforms/sample/ ``` transforms: my_transform_id: type: sample inputs: - my-source-or-transform-id exclude: null rate: 10 ``` 'rate' stage review suggestions . https://github.com/grafana/loki/pull/5051 ![image](https://user-images.githubusercontent.com/9583245/189461481-6ee4d835-2573-4b8e-8dec-2814620d758a.png) <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Paul Tirk	540380fa94	Promtail: fix GeoIP typo (#8783 ) What this PR does / why we need it: Fixes a typo for GeoIP stage. From `geoip_continet_name` to `geoip_continent_name`.	3 years ago
Danny Kopping	e2ac2d50e2	Changes from linter (#8769 ) What this PR does / why we need it: We had a few outstanding linter issues, and running `make lint` fixes them all automatically. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Aditya C S	b94c8aaf16	feature: geoip stage in promtail (#3493 ) Co-authored-by: Ed Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	43a21eac03	fix(promtail): Make `cri` tags streams aware. (#8497 )	3 years ago
Nick Moore	85b3e5d4ab	Promtail: Add event log message stage (#8382 ) The Windows Event Log includes a field named Message that include a large amount of additional data, stored in a simple key-value format: key-value pairs are separated with Windows-style \r\n newlines, the key and value are separated with : (or just : if the key has no associated value). The key-value data in the Message field is commonly utilised when investigating Windows events, e.g., in security use-cases, but the existing windows_events pipeline stage keeps the Message data as a string, leading to requiring significant effort at the query time to extract that data to be used in field filters. This PR request introduces a new promtail stage that parses a specified Source (defaulting to message used by the windows_events) and extracts all the key-value pairs into the extracted fields, which can then be used in latter stages in the pipeline.	3 years ago
Witold Duranek	ee1aee0af3	feat(promtail): add more complex options of dropping logs at the drop stage (#6371 )	3 years ago
Jake Herbst	02e0b3ae89	Fix logentry drop to more intuitively match numeric fields (fixes #7011 ) (#7292 ) What this PR does / why we need it: 1. Makes the `drop` stage matching for numeric fields more intuitive. 2. Adds tests scenarios to confirm fix Which issue(s) this PR fixes: Fixes #7011	3 years ago
Kaviraj Kanagaraj	d5b68c0257	fix(promtail): Fix cri tags extra new lines. (#7997 )	3 years ago
Maxim Ivanov	b773a7a3cf	promtail: ratelimiting by label (#7597 ) Limit stage can now takes optional `by_label_name` param, which tracks distinct values of that label and ratelimit entries independently. Log lines without expected label are not considered for ratelimiting and passed to the next pipeline stage.	3 years ago
Max Vorobev	c5911d5342	Loki: implement decolorize filter (#7602 ) Implement `decolorize` filter that get rids of ANSI color codes in the log message. Which issue(s) this PR fixes: Fixes #7601	4 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340 ) I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	4 years ago
Danny Kopping	6bf5b5d8c1	Use a line-buffered logger to deamplify write syscalls (#6954 ) We initialise a global logger in `pkg/util/log/log.go` and use it extensively throughout the Loki codebase. Every time we write a log message, a `write` syscall is invoked. Syscalls are problematic because they transition the process from userspace to kernelspace, which means: - a context-switch is incurred, which is inherently expensive ([1-2 microseconds](https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/)) - the goroutine executing the code is blocked - the underlying OS thread (_M_ in the go scheduler model) is also blocked - the goroutine has to be rescheduled once the syscall exits - the go scheduler may need to spawn additional OS threads if all are blocked in syscalls - which can also be expensive This change introduces the use of a line-buffered logger. It has a buffer of [256](https://gist.github.com/dannykopping/0704db32c0b08751d1d2494efaa734c2) entries, and once that buffer is filled it will flush to disk. However, a situation will arise in which that buffer remains somewhat empty for a period of time, so there is a periodic flush mechanism, configured to flush every 100ms. There is also a preallocated bytes slice of 10MB which is reused, to avoid excessive slice resizing & garbage collection. This does mean that we could lose up to 256 log messages in case of an ungraceful termination of the process, but this would need to be precisely timed within the 100ms flushes - in other words, the likelihood is low, and generally we shouldn't `kill -9` any Loki process.	4 years ago
Pablo	42bfc22d23	Fix promtail `metric` stage causing failure when getting extracted metrics (#7216 ) What this PR does / why we need it: Promtail allows ingested log entries to be modified before pushing them to Loki with the use of pipelines. One of the available pipeline "stages" is called `metrics`, and allows extracting metrics from incoming logs. These metrics are labeled with the labels available in the processed `Entry`. There's some edge cases in which if either a entry label is invalid, or it starts with the prometheus internal label prefix, `__`, calling the promtail `/metrics` fails. One way to easily reproduce this is by having the following configuration: ```yaml pipeline_stages: - static_labels: good_label: "1" - tenant: value: "2" - metrics: loki_count: type: Counter description: "should count all entries" config: match_all: true action: inc ``` Which exposes a `promtail_custom_loki_count` metrics that counts processed log entries. In this case, the step before the `metrics` one injects an internal label `__tenant_id__` to be used by the remote write client. Since this label starts with the prometheus reserved prefix, this will make calling the `metrics` endpoint fail with an error like the following: ``` An error has occurred while serving metrics: 1 error(s) occurred: * "__tenant_id__" is not a valid label name for metric "promtail_custom_loki_count" ``` This PR attempts to fix that by, before creating the corresponding metrics, dropping invalid or internal labels. That way promtail will just use labels that are valid. Which issue(s) this PR fixes: Related to https://github.com/grafana/cloud-onboarding/issues/2067 Special notes for your reviewer: Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	4 years ago
Kaviraj Kanagaraj	01481cca0c	Promtail: Support all `cri-o` tags (multiline tags). (#6177 ) * Promtail: Support `cri-o` tags. Hacked something together that works during flight to Grafanafest. * Fix newCri test cases Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add test for real "panic" stacktrace Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add bounded size for partial lines slice Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Leave the promtail config alone Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks: Avoid discarding lines when partial lines reaches upper bound. Instead merge into single full line and print warning. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Jan-Otto Kröpke	a1e0298a57	Support tenant id from value (#6290 ) Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>	4 years ago
Callum Styan	b6ac00e1a3	[Promtail] drop lines with malformed json (#6099 ) * Refactor json pipeline stage to have a Run function Signed-off-by: Callum Styan <callumstyan@gmail.com> * Allow for dropping log lines in json pipeline stage if they are not valid json strings Signed-off-by: Callum Styan <callumstyan@gmail.com> * fix linting Signed-off-by: Callum Styan <callumstyan@gmail.com> * Clean up new malformed json test. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some review feedback. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Refactor json pipeline stage test to use testify assert instead of t.Errorf Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update changelog and docs with new drop malformed option Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs based on Karen's comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago

1 2

65 Commits (79eae2ca25c8faa262c246bafcf913eb4e8fd2c3)