apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Vladyslav Diachenko	00a3c5b026	renamed `non indexed labels` feature to `structured metadata` (#10432 ) Loki team has made a decision to rename the feature because the new name is more precise. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Đurica Yuri Nikolić	37ec68eebb	Remove unnecessary code after dskit log.Interface has been dereleased (#10434 ) This PR removes unnecessary code used to convert the dereleased `dslog.Interface` (see https://github.com/grafana/dskit/pull/359) into `go-kit.Logger`. Additionally, it modifies the signature and implementation of `pkg/util/log/log.go`'s `InitLogger` as follows: - previous signature ``` InitLogger(server.Config, prometheus.Registerer, bool, bool) ``` has been replaced with ``` InitLogger(server.Config, prometheus.Registerer, bool, bool) log.Logger ``` Namely, the new `InitLogger` initialises the global logger and returns it. - the new implementation initializes the global logger, but it does not override the default logger for the server. This is now done by the callers of `InitLogger`. Previously, it was done inside `InitLogger` for convenience, since the types of the 2 loggers were not the same. Signed-off-by: Yuri Nikolic <durica.nikolic@grafana.com>	2 years ago
Vladyslav Diachenko	6387bf765c	[promtail] include the size of non-indexed labels into batch size (#10412 ) What this PR does / why we need it: When we added non-indexed-labels support to the promtail, we forgot to adjust the batch size calculation which leads to the case when promtail thinks that batch size is `X` but when it sends it to Loki, Loki rejects it because the actual size of the payload is `X`+`N`(size of non-indexed labels). Special notes for your reviewer: This feature is not released yet, so, changelog update is not needed. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Shantanu Alsi	586b9042c6	Upgrade Thanos objstore, dskit and other modules (#10366 ) What this PR does / why we need it: Upgrades various common modules including dskit, thanos objstore, prometheus, etc. Fixes errors/ changes resulting due to module upgrades. This is required in order to eventually upgrade BEL dependency in GEL to point to master. Which issue(s) this PR fixes: None Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
William Dumont	a1c1152b79	Promtail: Add additional fields in cloudflare config (#10301 ) What this PR does / why we need it: In this PR the cloudflare config is enhanced by adding a custom fields type and by having a new property "additional_types" to give the customer the ability to specify the fields that he wants. This functionality has been added in the grafana agent and therefore should also be added in promtail. Which issue(s) this PR fixes: Fixes https://github.com/grafana/agent/issues/4860 Special notes for your reviewer: Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md`	2 years ago
Arve Knudsen	a60e0167a8	Upgrade to latest dskit main (#10300 ) What this PR does / why we need it: Upgrade to latest dskit main branch, which mainly has changes to its `log` package, which requires corresponding fixing up in Loki.	2 years ago
Shantanu Alsi	b57d260ddc	Upgrade alpine 3.16.5 -> 3.16.7 (#10188 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #10181 Fixes Open SSL vulnerabilities [CVE-2023-1255](https://security.alpinelinux.org/vuln/CVE-2023-1255), [CVE-2023-2650](https://security.alpinelinux.org/vuln/CVE-2023-2650), [CVE-2023-2975](https://security.alpinelinux.org/vuln/CVE-2023-2975), [CVE-2023-3446](https://security.alpinelinux.org/vuln/CVE-2023-3446), [CVE-2023-3817](https://security.alpinelinux.org/vuln/CVE-2023-3817) Alpine Release Notes: [3.16.7](https://www.alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html) [3.16.6](https://www.alpinelinux.org/posts/Alpine-3.15.9-3.16.6-3.17.4-3.18.2-released.html) Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Vladyslav Diachenko	0fc934e4ea	non_indexed_labels stage: additional test-case (#10170 ) added additional test-case to cover match and non_indexed_labels stages together. Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Michel Hollands	d484d31856	Add support for cross arch for fluent-bit-plugin-loki image (#10081 ) Copied from https://github.com/grafana/loki/pull/9991 which has problems running CI. What this PR does / why we need it: Currently the `fluent-bit-plugin-loki` image is only build and published for amd64. With more use of arm-based processors, it would be good to add support for those as well. I've build the image using `docker buildx` on my M1 Mac and pushed it to Dockerhub, should you wish to try it first: https://hub.docker.com/r/ebosveld/fluent-bit-plugin-loki/ On AWS ECS Graviton based images it works well with the above mentioned image. I've also fixed #9932 as it needed the full bullseye support. Which issue(s) this PR fixes: Fixes #6738 Fixes #9932 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Edwin Bosveld <edwin.bosveld@persgroep.net> Co-authored-by: Travis Patterson <travis.patterson@grafana.com> Co-authored-by: Edwin Bosveld <ebosveld@users.noreply.github.com>	2 years ago
Bruno FERNANDO	a8a349690b	feat(promtail): retrieve BotTags field from cloudflare (#10010 ) What this PR does / why we need it: Hey folks 👋 Little contribution here to add a useful log field for cloudflare users. Indeed I add the [BotTags field](https://developers.cloudflare.com/bots/concepts/cloudflare-bot-tags/) which is pretty useful when debugging some specific traffic handled by cloudflare Don't hesitate to reach me if you have any questions Cheers 😉 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
J Stickler	7b671348a9	Delete incomplete comment. (#10046 ) Which issue(s) this PR fixes: Fixes #9960 Deletes incomplete comment in code that was introduced from Prometheus as part of `fb26baa5b1`	2 years ago
Pablo	5f04eedc03	Refactor Promtail's `client.Manager` to do multi client instantiation (#9908 ) What this PR does / why we need it: This PR refactors Promtails `client.Manager` to include both WAL-enabled client, and WAL disabled. This will make the transition easier towards the agent codebase, only needing to instantiate one primitive to support both WAL enabled/disabled clients under the `loki.write` component. Also, re-uses a bunch of logic shared by both. Which issue(s) this PR fixes: Part of https://github.com/grafana/loki/issues/8197 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Vladyslav Diachenko	416e01ca8e	[promtail] non_indexed_labels stage (#9986 ) What this PR does / why we need it: added non_indexed_labels stage to copy values from extracted data tonon-indexed-labels field of the Entry. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Karsten Jeschkies	2dfac72835	Clarify that label filter expressions are not supported in promtail. (#9894 ) What this PR does / why we need it: Some users of Promtail seem to be confused that certain types of filter expressions are not supported in Promtail's `match` stage. This change clarifies what kind of filter expressions are supported. Which issue(s) this PR fixes: Fixes #5209 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Michel Hollands	bad691b509	Update go build image (#9914 ) Update the go build image --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Michel Hollands	7be9deacf3	Update go version to 1.20.6 (#9913 ) Update the go version. Another PR will follow that uses the new build image --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Ashwanth	de937e72cd	promtail: apply defaults to HTTP client config (#9863 ) What this PR does / why we need it: Apply defaults to HTTPClientConfig and validate it. similar to how it's done [here](`e9644d8186/vendor/github.com/prometheus/prometheus/config/config.go (L524)`) in prometheus. promtail [configuration page](https://grafana.com/docs/loki/latest/clients/promtail/configuration/) incorrectly claims that `follow_redirects` is true, but it is being set to false given that the defaults are not being applied currently Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
René Scheibe	954df433e9	Promtail: Fix file descriptor leak (#8987 ) This reduces the amount of open file descriptors per followed journal file from 3 to 2. It also releases file descriptors for removed files (journal vacuuming). The `defaultJournalEntryFunc` is only called once upon startup. It creates a `Journal` instance to only read a single entry to determine the age of the entry referenced in the positions file. Right after, this instance is not required anymore. The actually used `Journal` instance used to follow the journal is created inside `journalTargetWithReader`.	2 years ago
Jan Jansen	8abe080fd6	Promtail: improve behavior of partial lines (#9508 ) What this PR does / why we need it: I started to see OOM after upgrading to a promtail version with #8497. So played around to make partial line behavior more stable. You can now truncate partial lines and configure max partial lines buffer. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Jan Jansen <jan.jansen@gdata.de> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	2 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588 ) For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	2 years ago
Bruno FERNANDO	1b410dbc01	feat(promtail): add CF ClientRequestSource field (#9669 ) What this PR does / why we need it: Hey folks 👋 Little contribution here to add a useful log field for cloudflare users. Indeed I add the [ClientRequestSource field](https://developers.cloudflare.com/logs/reference/clientrequestsource/ ) which is pretty useful when debugging some specific traffic handled by cloudflare Extra: Since I was on the documentation I fixed an indentation issue that I spotted Don't hesitate to reach me if you have any questions Cheers 😉 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Loki rocks 🚀 Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Piotr	ac4a442291	Promtail: Fix goroutine leak in file tailer (#9495 ) What this PR does / why we need it: There's a goroutine leak in promtail file tailer, which has been encountered in https://github.com/grafana/agent/issues/3885. The issue is the following: - There are two goroutines that tailer manages: one for reading lines, the other for updating positions: https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/file/tailer.go#L103 - The tailer can be cleanly stopped & drained via `.Stop()` https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/file/tailer.go#L221 - But if the goroutine reading lines encounters an error from the underlying `tail`, the tailer is stopped without draining using the defer statement here: https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/file/tailer.go#L151 - The above defer statement does not shut down the position updating goroutine and the `.Stop()` won't be called, so we end up with a leaked goroutine. Which issue(s) this PR fixes: Fixes https://github.com/grafana/agent/issues/3885 As can be seen in the [goroutine dump attached to the issue](https://gist.github.com/amseager/e37f31984facd9cbc51ce0f2153d6864), there were 933 update position goroutines, while we only had 21 files being tailed. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Tests updated - [x] `CHANGELOG.md` updated	2 years ago
Pablo	306c96ce8a	Notification based Promtail WAL Watcher polling approach (#9135 ) What this PR does / why we need it: This PR implements a new mechanism for the wal Watcher in Promtail, to know there are new records to be read. It uses a combination of: - https://github.com/prometheus/prometheus/pull/11950 - https://github.com/prometheus/prometheus/pull/11949 The main idea is that the primary mechanism is a notification channel between the `wal.Writer` and `wal.Watcher`. The Watcher subscribes to write events the writer publishes, getting notified if the wal has been written. The same subscriptions design is used for cleanup events. As a backup, the watcher has a timer that implements an exponential backoff strategy, which is constrained by a minimum and maximum that the user can configure. Below the cpu difference is shown of running both main and this branch against the same scrape target. <img width="2496" alt="image" src="https://user-images.githubusercontent.com/2617411/232099483-7e5c36fa-9360-4eb9-8240-687adf46e330.png"> The yellow line is the latest main build from where this branch started, and the green line is this branch. Both promtails tailing docker logs, and using the following metrics to get cpu usage from cadvisor: ``` avg by (name) (rate(container_cpu_usage_seconds_total{job=~".+", instance=~".+", name=~"promtail-wal-test_promtail.+"}[$__rate_interval])) ``` Which issue(s) this PR fixes: Part of https://github.com/grafana/loki/issues/8197 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
cui fliter	520e434446	fix function names in comments (#9415 )	2 years ago
Jan Jansen	c4048a3adf	Promtail: Break on iterate journal failure (#9155 )	2 years ago
Danny Kopping	9495b6cc56	Update loki-build-image with go1.20.4 (#9372 ) Continuation of #9370 Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Danny Kopping	f5c0fdf028	Upgrade to latest security release: go1.20.4 (#9370 ) What this PR does / why we need it: https://groups.google.com/g/golang-announce/c/MEb0UyuSMsU Also updated the build image. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Jeff Cantrill	3d7060bf73	fluent-plugin-grapha-loki: Add config to support tls: ciphers, min_version (#9289 )	2 years ago
Trevor Whitney	a11921bbb8	Upgrade alpine 3.16.4 -> 3.16.5 (#9264 ) What this PR does / why we need it: Upgrade alpine version to address [openssl CVEs](https://www.alpinelinux.org/posts/Alpine-3.17.3-released.html) Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205 )	2 years ago
Labesse Kévin	0215a17148	gcplog: improve formatter (#9044 ) What this PR does / why we need it: Add two improvement to the target `gcplog` * capability to ignore the `textPayload` from the log line -> don't trunk important metadata * add new source labels (severity and root labels of log entry) -> more relabel_config possibilities Special notes for your reviewer: I'm not sure about the naming of the option `useFullLine`, feel free to suggest any name, but I guess this feature must be disable by default for compatibility? Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Kevin Labesse <kevin@labesse.me> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Robert Fratto	7bec727c6d	promtail: support configurable polling interval (#8874 ) Currently, Promtail always polls files every 250ms. This PR allows the polling period to be customized, starting at a minimum polling period and exponentially increasing the polling frequency to a bounded maximum if the file has no changes. When file changes are detected, the polling frequency resets to the minimum. The default behavior is the existing behavior; both the minimum and maximum polling intervals are set to 250ms. I'm opening this in draft to get early feedback. There's a few open questions: * Where can I add validations to ensure that the min/max polling intervals are configured properly? * Is this the right approach for implementation? Do I need to make any changes? * Does this need tests? I'll update the CHANGELOG and Documentation after we're agreed on an approach.	2 years ago
Trevor Whitney	38c9c6c460	Bump go from 1.20.1 -> 1.20.3 (#9106 ) What this PR does / why we need it: Bumps our go dependency from 1.20.1 to 1.20.3 to address [security fixes](https://groups.google.com/g/golang-announce/c/Xdv6JL9ENs8) Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Dylan Guedes	6dfcd59c1b	Promtail: Add `decompression` configuration (#8994 ) What this PR does / why we need it: Adds a new `compression` configuration to Promtail to customize decompression behavior. As of now, two options are supported: `enabled`: false by default but if truthy, Promtail will infer a compression algorithm to use based on the file extension `initialDelay`: 0 by default but if higher, Decompressor will sleep for the given duration before trying to decompress. This is to avoid scenarios where the Decompressor starts to work on a file that is still being compressed/generated. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/8784	2 years ago
René Scheibe	46af2cdcf1	Promtail: Prevent logging errors on normal shutdown (#8988 ) What this PR does / why we need it: Since PR #4066 two messages are logged for the errors `ErrExpired` and `EOF`, whereas nothing has been logged beforehand. As `ErrExpired` is returned by the `JournalReader` after sending a message to the `until` channel to signal a normal stop, this should not be logged as an error.	2 years ago
Pablo	89996c9714	Promtail WAL support: Implement reader side (#8302 ) What this PR does / why we need it: This PR is the second in a series that will implement WAL support into Promtail. The main objective of this PR is implement the reader side of WAL. That is, using the previously instantiated `client.Manager` to somehow read entries from the WAL instead of receiving them from a channel. Pending work: - [x] Implement mechanism so that `WriteTo` targets of each `wal.Watcher` can cleanup un-used series - [x] Write a full fledged test-case that tests a WAL enabled Promtail (maybe in a follow up) Which issue(s) this PR fixes: Part of #8197 Special notes for your reviewer: The line count might be big since this PR implements two fully fledged test cases of Promtail with WAL enabled. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Andrii Kushch	17e05d28e4	Promtail: Add a new target for the Azure Event Hub (#8787 ) What this PR does / why we need it: We want to allow receiving logs from the Azure Cloud. To solve this problem, we need to be able to configure Promtail to consume logs from the Azure Events Hub. To achieve this goal, we need to add a new configuration option to the scrape configuration in Promtail and implement the component that, given this configuration, can connect to the Azure Event Hub, parse received data, and prepare it to be stored in Loki. Which issue(s) this PR fixes: Fix [#8788](https://github.com/grafana/loki/issues/8788) # Special notes for your reviewer: With the current implementation, I would like to leverage Azure Events Hub's compatibility with a Kafka protocol and reuse code for a Kafka target. ### Why I decided to create a new target instead of just reusing the Kafka target: 1. Configuration for Kafka in this scenario requires fewer parameters. For example, version and auth should have some specific values. 2. Message handling is different for Azure Logs. In addition to what we are doing for the Kafka target, we want to fix JSON and split incoming messages into multiple log lines before propagating it further. ### From the implementation perspective, there are two goals: - Extract interfaces that are not Kafka configuration specific from the Kafka target module. Splitting Kafka config parsing and target creation. - To make the message handler/parser injectable or configurable: we would like to fix JSON and split a message into multiple messages. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Pablo <2617411+thepalbi@users.noreply.github.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
René Scheibe	3bed7eef55	chore: Fix typo in "tcp" (#8842 )	2 years ago
李国忠	0295fd4698	[new feature] promtail: Add a Promtail stage for probabilistic sampling (#7127 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: The sampling stage can be directly sampled. The implementation of sampling is to use the algorithm in jaeger go client ``` pipeline_stages: - sampling: rate: 0.1 ``` or it can be used with match for precise sampling. ``` pipeline_stages: - json: expressions: app: - match: pipeline_name: "app2" selector: "{app=\"poki\"}" stages: - sampling: rate: 0.1 ``` Which issue(s) this PR fixes: Fixes #6654 Special notes for your reviewer: The promtail 'rate' stage is also used with the 'match' stage for log filtering.This design makes the code very clean. Other log agents vector have log filtering built into the sampling operator, which I think is too complicated https://vector.dev/docs/reference/configuration/transforms/sample/ ``` transforms: my_transform_id: type: sample inputs: - my-source-or-transform-id exclude: null rate: 10 ``` 'rate' stage review suggestions . https://github.com/grafana/loki/pull/5051 ![image](https://user-images.githubusercontent.com/9583245/189461481-6ee4d835-2573-4b8e-8dec-2814620d758a.png) <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Paul Tirk	540380fa94	Promtail: fix GeoIP typo (#8783 ) What this PR does / why we need it: Fixes a typo for GeoIP stage. From `geoip_continet_name` to `geoip_continent_name`.	2 years ago
Danny Kopping	e2ac2d50e2	Changes from linter (#8769 ) What this PR does / why we need it: We had a few outstanding linter issues, and running `make lint` fixes them all automatically. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Dylan Guedes	656ad800fe	Promtail: Stop ticker to avoid leaking (#8695 ) What this PR does / why we need it: Call `Stop` to positions ticker to release associated resources/avoid leaking.	2 years ago
Bryan Boreham	08ac5336d5	WAL: store pointers in record pool (#8667 ) What this PR does / why we need it: Storing a slice in `sync.Pool` wastes 24 bytes every time it is returned to the heap; see https://staticcheck.io/docs/checks#SA6002 Storing a pointer to a slice allows the Pool to work without waste. This came up in discussion about the benchmark in #8489 - with this change the memory usage comes down: ``` name old time/op new time/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 7.37ms ±13% 7.45ms ± 9% ~ (p=0.841 n=5+5) name old alloc/op new alloc/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 727kB ± 0% 679kB ± 0% -6.63% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 12.0k ± 0% 10.0k ± 0% -16.68% (p=0.000 n=5+4) ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Pablo	a15365202d	Optimize Promtail wal writer to batch both series and entries record (#8489 )	2 years ago
Bryan Boreham	6fd4b5e89b	Update prometheus/prometheus from 2.41 to 2.42 (#8571 ) What this PR does / why we need it: Brings in the latest updates from upstream. These open up some opportunities for optimisations in TSDB indexing. Dependencies updated: * github.com/Azure/go-autorest/autorest/adal v0.9.21 -> v0.9.22 (comment-only change) * github.com/docker/docker v20.10.21 -> v20.10.23 (fixing filters bug) * golang.org/x/exp fae10dda9338 -> d38c7dcee874 (optimisations in `BinarySearch` function) Indirect dependencies also updated: * github.com/digitalocean/godo v1.91.1 -> v1.95.0 (nothing alarming in [release notes](https://github.com/digitalocean/godo/releases)) * github.com/google/pprof aee1124e3a93 -> 76d1ae5aea2b (no changes pulled into vendor) * golang.org/x/tools v0.4.0 -> v0.5.0 (relating to Go compiler utilities) * google.golang.org/genproto 76db0878b65f -> 31e0e69b6fc2 * k8s.io/api v0.26.0 -> v0.26.1 (comment-only change) * k8s.io/apimachinery v0.26.0 -> v0.26.1 (no changes pulled into vendor) * k8s.io/client-go v0.26.0 -> v0.26.1 (small fixes) Special notes for your reviewer: A couple of interfaces changed; these have required matching changes in Loki code. Those changes are split into separate commits. I also note that most calls to `relabel` ignore when the rule says "drop". Maybe this is wrong?	2 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583 ) What this PR does / why we need it: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	2 years ago
Aditya C S	b94c8aaf16	feature: geoip stage in promtail (#3493 ) Co-authored-by: Ed Welch <edward.welch@grafana.com>	2 years ago
Yasmine El wahbi	c6542e61fa	Promtail: Add ready endoint to loki-push-api server (#8552 ) When trying to expose loki_push_api service (in our case using GKE ingress), we need a health check target endpoint that would return a 200 response code. This PR contains a simple addition for a HTTP handler that handles the /ready path, along with the necessary tests. Fixes #8551 Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

1 2 3 4 5 ...

275 Commits (7f6c9104e28ae75fb119254773204a0ff0f55bd0)