120 Commits (0dfd2cbbcc15d888466b520ee9345be9b7c8ee8a)

Author	SHA1	Message	Date
Christian Haudum	7fff660144	Add UDP protocol support to Promtail's syslog target (#5790) ... Until now, the syslog target in Promtail did only support TCP. This PR adds support for sending syslog messages to Promtail (via ng-syslog or rsyslog) via UDP. UPD for syslogs can be used if you prefer speed over guaranteed delivery. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Paschalis Tsilias	179d741c5f	promtail: Add support for exclusion patterns in `static_config` (#5943) ... * PR base Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Add CHANGELOG entry; updated documentation for new label Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Follow logic from t.path in exlusion case; always handle matching Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Make error message unique Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Only run if t.pathExclude is set Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Revert "Only run if t.pathExclude is set" This reverts commit 17abb006ce141071c4bca4b7d133bea2aa314594.	4 years ago
Alex Ferm	d7f03be212	Update libsystemd-dev from bullseye-backports for arm32 docker images, fixes (#5938) (#6012)	4 years ago
Vladyslav Diachenko	4a91fdcc7f	migrated gelfTarget to go-gelf/v2 library that contains fix for chunked messages (#5992)	4 years ago
Callum Styan	8f62334a58	[promtail] Add a log message stating that client level stream lag labels are deprecated (#5850) ... * Add a log message stating that client level stream lag labels are deprecated and will not be used. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Error on startup if a client level stream lag labels value is set. Signed-off-by: Callum Styan <callumstyan@gmail.com> * fix lint error Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Christian Simon	9d0fd3a7e3	Upgrade golang version to v1.17.9 (#5899) ... ``` git ls-files '*[Dd]ockerfile*' | grep -v ^vendor/ | xargs sed -i "s/golang:[0-9\\.]*/golang:1.17.9/g" | ag 'golang:' ```	4 years ago
thepheer	81021c920f	remove excessive underscore from `dockerLabelLogStream` (#5890)	4 years ago
Karsten Jeschkies	c9f3868287	Update dskit to v0.0.0-20220331160727-49faf69f72ca. (#5881)	4 years ago
Callum Styan	800b7c6345	[promtail] Mark promtail single client config flags and config file block as deprecated (#5842) ... * Mark promtail single client config flags and config file block as deprecated. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update some promtail docs examples that still use `client` instead of `clients`. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix imports. Signed-off-by: Callum Styan <callumstyan@gmail.com> * properly format url as an array item since we changed to clients from client Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Cyril Tovena	dd7200564b	Improve kafka discovery test. (#5860) ... The `Test_TopicDiscovery` expections were wrong, we only need them to be true at some point since this is reconciliation loop. Fixes #5858 Nice catch ! Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Marc Tudurí	0315102c34	promtail: Add ActiveTargets method (#5839) ... * Add ActiveTargets method to promtail * Update CHANGELOG	4 years ago
Bryan Boreham	1329ea165c	TestPromtail: avoid hanging when test fails midway (#5837) ... * Allow Promtail.Shutdown to be called again I want to do this in a test * TestPromtail: avoid hanging when test fails midway Also call Shutdown via defer so we clean up those resources when the test fails	4 years ago
Christian Simon	3d941ccccb	Update alpine images to 3.15.4 (#5780)	4 years ago
Christian Haudum	ece1fb5a34	Allow promtail to push RFC5424 formatted syslog messages (#5715) ... * Allow promtail to push RFC5424 formatted syslog messages This PR introduces a new flag `use_rfc5424_message` for syslog scrape targets, which allows to push the full [RFC5424](https://datatracker.ietf.org/doc/html/rfc5424) formatted syslog message received by Promtail to be pushed to Loki, instead of only the message content. Storing the full message allows users to parse the message in Loki instead of relabelling in Promtail. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add test for pushing RFC5424 messages Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Allow promtail to push RFC5424 formatted syslog messages Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karen Miller	c860fc6442	Docs: Document Promtail global rate limiting (#5737) ... * Docs: Document Promtail global rate limiting * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Jennifer Villa <jvilla2013@gmail.com> * Revise strings that explain the new configuration parameters * For rate limiting, limit_config becomes limits_config Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>	4 years ago
Ed Welch	a9ac01bd0f	use an empty registry when setting up the logger for the push server (#5761) ... Signed-off-by: Ed Welch <edward.welch@grafana.com>	4 years ago
李国忠	098b5a2d37	[new] promtail pipeline: Promtail Rate Limit stage #5048 (#5051) ... * [new] promtail pipeline: Promtail Rate Limit stage #5048 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update clients/pkg/logentry/stages/limit.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update docs/sources/clients/promtail/stages/limit.md * Update docs/sources/clients/promtail/stages/limit.md * fix import package * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karsten Jeschkies	a5921f791a	Revert changes to generated assets_vfsdata.go. (#5728)	4 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724)	4 years ago
Paul Ryan	4e3f3b7f7e	Promtail: Fix retry/stop when erroring for out of cloudflare retention range (e.g. over 168 hours old) (#5698) ... * fix: 5696 don't retry when out of range for cloudflare	4 years ago
Franz Wong	d67306b657	Promtail: Rename config name limit_config to limits_config (#5707) ... * Rename promtail config name 'limit_config' to 'limits_config' * Add changelog entry * Fix formatting	4 years ago
Susana Ferreira	1d1461824d	Move promtail StreamLagLabels config to upper level config.Config (#5686) ... * Move promtail StreamLagLabels config to upper level config.Config * Fix promtail config load test * Fix promtail configuration reference * Add options promtail config section * Kick GitHub CI Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Callum Styan	2ebbecfcbd	Require unique client configs when there are multiple clients. (#5688) ... * Require unique client configs when there are multiple clients. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fixing linter error Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Callum Styan	2c2578e84c	Fix bug with new promtail stream lag metric where labels might not be (#5675) ... applied if they don't exist on the source stream Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Callum Styan	01d63c8ae8	[promtail] Refactor promtail client metrics so that we can't have duplicate metrics collected for the lag metric. (#5521) ... * Refactor promtail client metrics so that we can't have duplicate metrics collected for the lag metric. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Set up new metrics passing for other client types. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix lint issues. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update changelog and docs, also rename nested client configs struct to configs. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	8f693e77b0	Update Prometheus to latest (#5545) ... * Update Prometheus to latest Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update promql.QueryTracker Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes querier test to use new grafana/regexp Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test with new textparse signature Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	c8bdb37ef3	update loki go version to 1.17.8 (#5543) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Travis Patterson	c654910f70	Fix orphaned metrics in the file tailer (#5497)	4 years ago
Holger Hees	f359a8280f	Update out_loki.rb (#5512) ... this will fix "UndefinedConversionError error="\"\\xC2\" from ASCII-8BIT to UTF-8"" for messages of systemd journal	4 years ago
Salva Corts	5237a348fd	Fix test (#5509)	4 years ago
Leo	6c84304430	promtail/targets/syslog: Enable best effort parsing for Syslog messages (#5409) ... * promtail/targets/syslog: enable best effort parsing * promtail/targets/syslog: fmt	4 years ago
Ed Welch	91b0bcf36a	remove promtail_log_entries_bytes_bucket (#5377) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361) ... * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	4 years ago
Susana Ferreira	82fac990a1	Update promtail to support duration string formats (#5290) ... * Update promtail to support duration string formats * Address comments	4 years ago
Travis Patterson	de1faa0671	Fix transitive dependency error (#5308) ... * Fix transitive dependency error - Don't update the gems that come with the logstash image * ci bump * ci bump	4 years ago
Travis Patterson	1925d77f65	cleanup: consolidate util strings contain functions (#5304) ... * Consolidate util strings contain functions * fix test	4 years ago
Christian Haudum	3fb572abda	Fix Promtail watching deadlock (#5283) ... * Reuse channel from fileEventWatcher map if already exists Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove sync() call in FileTarget constructor The constructor `NewFileTarget` is called from within a locked code block by the FileTargetManager. However, the constructor also calls `sync()` which starts and stops watching new directories. These methods `startWatching` and `stopWatching` both send to the `targetEventHandler` channel, causing the `FileTargetManager` to receive the event in a goroutine (`watch()` function) running in the background. Handling the event calls `sendFileCreateEvent` on the syncers, which subsequently tries to write to the `fileEventWatcher` channel which is never read because of the locked mutex. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Karsten Jeschkies	b92b544ca2	Write Promtail position file atomically. (#5266) ... Fixes #5218	4 years ago
Karsten Jeschkies	54bc2dc40e	Restart Docker target if stopped. (#5280) ... Summary: The Docker target would not reconnect to a container if it restarted and kept the same ID. This should introduces a `startIfNotRunning` method that will be called to restart scraping the container's logs. Fixes #5259	4 years ago
Ed Welch	44ba39055d	logstash: re-add bundle update and update bundler (#5284) ... * update to latest logstash and bundler Signed-off-by: Edward Welch <edward.welch@grafana.com> * re-add bundle update Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Kaviraj Kanagaraj	858f8ea54a	Fix transitive dependency failure while building logstash plugin (#5268) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Lydia Lee	a4cfab4b84	feat(promtail): fix TargetManager.run() not exit (#5238)	4 years ago
Ed Welch	09a5a2cee3	Logstash: Add config option which allows seting up an allowlist for labels to be mapped to Loki (#5244) ... * add a config called `include_fields` which lets you allowlist a specific set of fields which will be mapped to labels and sent to Loki. Signed-off-by: Edward Welch <edward.welch@grafana.com> * update docs Signed-off-by: Edward Welch <edward.welch@grafana.com> * restore path for Dockerfile so it works in CI Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257) ... * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Cyril Tovena	cde9a711f3	fluent-bit: Attempt to unmarshal nested json. (#5223) ... * fluent-bit: Attempt to unmarshal nested json. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Also support nested json array. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Roger Steneteg	bc88f298cb	skip glob search if path is an existing file and not a directory (#5246) ... Signed-off-by: Roger Steneteg <rsteneteg@ea.com>	4 years ago
Cyril Tovena	fa25e54e26	Bump logstash plugin gem (#5230) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karsten Jeschkies	4716eea0db	Provide Docker target and discovery in Promtail. (#4911) ... **What this PR does / why we need it**: This patch adds support to fetch Docker container logs through the Docker daemon API. This should be more robust than the Loki Docker driver or scraping the logs files. The new Docker target will also collect meta information of the scraped containers. **Which issue(s) this PR fixes**: Addresses #2361 Closes #4703 **Special notes for your reviewer**: **Checklist** - [x] Documentation added - [x] Tests updated - [x] Add an entry in the `CHANGELOG.md` about the changes.	4 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187) ... * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Christian Haudum	86ed98c1fa	Fix promtail deadlock (#5170) ... * Remove unused clientConfigs argument from NewTargetsManagers * Do not create targetScrapeConfigs if stdin target is used * fixup! Remove unused clientConfigs argument from NewTargetsManagers * Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup: add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago