apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Franz Wong	d67306b657	Promtail: Rename config name limit_config to limits_config (#5707 ) * Rename promtail config name 'limit_config' to 'limits_config' * Add changelog entry * Fix formatting	3 years ago
Susana Ferreira	1d1461824d	Move promtail StreamLagLabels config to upper level config.Config (#5686 ) * Move promtail StreamLagLabels config to upper level config.Config * Fix promtail config load test * Fix promtail configuration reference * Add options promtail config section * Kick GitHub CI Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Callum Styan	2ebbecfcbd	Require unique client configs when there are multiple clients. (#5688 ) * Require unique client configs when there are multiple clients. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fixing linter error Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Callum Styan	2c2578e84c	Fix bug with new promtail stream lag metric where labels might not be (#5675 ) applied if they don't exist on the source stream Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Callum Styan	01d63c8ae8	[promtail] Refactor promtail client metrics so that we can't have duplicate metrics collected for the lag metric. (#5521 ) * Refactor promtail client metrics so that we can't have duplicate metrics collected for the lag metric. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Set up new metrics passing for other client types. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix lint issues. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update changelog and docs, also rename nested client configs struct to configs. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Cyril Tovena	8f693e77b0	Update Prometheus to latest (#5545 ) * Update Prometheus to latest Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update promql.QueryTracker Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes querier test to use new grafana/regexp Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes test with new textparse signature Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	c8bdb37ef3	update loki go version to 1.17.8 (#5543 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Travis Patterson	c654910f70	Fix orphaned metrics in the file tailer (#5497 )	3 years ago
Holger Hees	f359a8280f	Update out_loki.rb (#5512 ) this will fix "UndefinedConversionError error="\"\\xC2\" from ASCII-8BIT to UTF-8"" for messages of systemd journal	3 years ago
Salva Corts	5237a348fd	Fix test (#5509 )	3 years ago
Leo	6c84304430	promtail/targets/syslog: Enable best effort parsing for Syslog messages (#5409 ) * promtail/targets/syslog: enable best effort parsing * promtail/targets/syslog: fmt	3 years ago
Ed Welch	91b0bcf36a	remove promtail_log_entries_bytes_bucket (#5377 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361 ) * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Susana Ferreira	82fac990a1	Update promtail to support duration string formats (#5290 ) * Update promtail to support duration string formats * Address comments	3 years ago
Travis Patterson	de1faa0671	Fix transitive dependency error (#5308 ) * Fix transitive dependency error - Don't update the gems that come with the logstash image * ci bump * ci bump	3 years ago
Travis Patterson	1925d77f65	cleanup: consolidate util strings contain functions (#5304 ) * Consolidate util strings contain functions * fix test	3 years ago
Christian Haudum	3fb572abda	Fix Promtail watching deadlock (#5283 ) * Reuse channel from fileEventWatcher map if already exists Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove sync() call in FileTarget constructor The constructor `NewFileTarget` is called from within a locked code block by the FileTargetManager. However, the constructor also calls `sync()` which starts and stops watching new directories. These methods `startWatching` and `stopWatching` both send to the `targetEventHandler` channel, causing the `FileTargetManager` to receive the event in a goroutine (`watch()` function) running in the background. Handling the event calls `sendFileCreateEvent` on the syncers, which subsequently tries to write to the `fileEventWatcher` channel which is never read because of the locked mutex. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Karsten Jeschkies	b92b544ca2	Write Promtail position file atomically. (#5266 ) Fixes #5218	3 years ago
Karsten Jeschkies	54bc2dc40e	Restart Docker target if stopped. (#5280 ) Summary: The Docker target would not reconnect to a container if it restarted and kept the same ID. This should introduces a `startIfNotRunning` method that will be called to restart scraping the container's logs. Fixes #5259	3 years ago
Ed Welch	44ba39055d	logstash: re-add bundle update and update bundler (#5284 ) * update to latest logstash and bundler Signed-off-by: Edward Welch <edward.welch@grafana.com> * re-add bundle update Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	858f8ea54a	Fix transitive dependency failure while building logstash plugin (#5268 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Lydia Lee	a4cfab4b84	feat(promtail): fix TargetManager.run() not exit (#5238 )	3 years ago
Ed Welch	09a5a2cee3	Logstash: Add config option which allows seting up an allowlist for labels to be mapped to Loki (#5244 ) * add a config called `include_fields` which lets you allowlist a specific set of fields which will be mapped to labels and sent to Loki. Signed-off-by: Edward Welch <edward.welch@grafana.com> * update docs Signed-off-by: Edward Welch <edward.welch@grafana.com> * restore path for Dockerfile so it works in CI Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257 ) * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Cyril Tovena	cde9a711f3	fluent-bit: Attempt to unmarshal nested json. (#5223 ) * fluent-bit: Attempt to unmarshal nested json. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Also support nested json array. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Roger Steneteg	bc88f298cb	skip glob search if path is an existing file and not a directory (#5246 ) Signed-off-by: Roger Steneteg <rsteneteg@ea.com>	3 years ago
Cyril Tovena	fa25e54e26	Bump logstash plugin gem (#5230 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Karsten Jeschkies	4716eea0db	Provide Docker target and discovery in Promtail. (#4911 ) What this PR does / why we need it: This patch adds support to fetch Docker container logs through the Docker daemon API. This should be more robust than the Loki Docker driver or scraping the logs files. The new Docker target will also collect meta information of the scraped containers. Which issue(s) this PR fixes: Addresses #2361 Closes #4703 Special notes for your reviewer: Checklist - [x] Documentation added - [x] Tests updated - [x] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Christian Haudum	86ed98c1fa	Fix promtail deadlock (#5170 ) * Remove unused clientConfigs argument from NewTargetsManagers * Do not create targetScrapeConfigs if stdin target is used * fixup! Remove unused clientConfigs argument from NewTargetsManagers * Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup: add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	985e965796	Fix encoding error in fluentd client (#5163 ) * Fix encoding error in fluentd client This change fixes a bug that is caused by incorrectly trying to encode any values other than string. This bug was a regression introduced with #5107 and caused `NoMethodError` errors. Fixes #5099 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Bump fluent-plugin-grafana-loki to version 1.2.18 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Cyril Tovena	144bee11af	Retry pulling Cloudflare logs after a stream error (#5158 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Christian Haudum	52fa2114df	fluentd: Fix bug that caused lines to be dropped when containing non utf-8 characters (#5107 ) * fluentd: Remove non utf-8 characters from log lines When a log line in hash format contains non UTF-8 characters fluentd would drop the complete line because it failed to convert the line in key-value format. By forcing UTF-8 encoding and replacing non UTF-8 characters with empty strings the log line will not be dropped but only contain the valid UTF-8 characters. Fixes #5099 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Bump fluent-plugin-grafana-loki to version 1.2.17 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fluentd: Remove non utf-8 characters from log lines Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fluentd: Remove non utf-8 characters from log lines Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
李国忠	6b377b40d5	[new] promtail: add readline rate limit (#5031 ) * [new] promtail: add readline rate limit * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * Update clients/pkg/promtail/limit/config.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Update clients/pkg/promtail/limit/config.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Ed Welch	ba91a0e48a	lock the mutex before working with the `fileEventWatchers` (#5043 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944 )	3 years ago
Christian Haudum	4687b416db	Update logstash dependency in logstash-output-loki plugin (#4930 ) Use `logstash:7.16.1` as base image for the `grafana/logstash-output-loki` Docker image. This logstash version uses `log4j` 2.15.0 which mitigates CVE-2021-44228. - https://www.elastic.co/guide/en/logstash/current/logstash-7-16-1.html - https://nvd.nist.gov/vuln/detail/CVE-2021-44228 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Alan Hollis	1e0b2707cd	Fix replace missing adjacent capture groups (#4874 )	4 years ago
Cyril Tovena	7f71c5f051	Promtail pull cloudflare logs (#4813 )	4 years ago
Hiroki Sakamoto	77dcb16b4e	Fix duplicate registry.MustRegister call in Promtail Kafka (#4866 ) * Move pipeline creation to new targetsyncer func * registry.MustRegister is called every newPipeline so ensure it only at once * add changelog entry	4 years ago
Hiroki Sakamoto	d7c35951c3	Use http prefix path correctly in promtail (#4832 ) * Use http prefix path correctly in promtail * Update changelog entry	4 years ago
Hiroki Sakamoto	11bf736c90	Refactor tests of Promtail filetarget (#4802 ) * Refactor tests of Promtail filetarget * Move tests depend on inotify to filetargetmanager test * Add handling file creation event via channel case in filetarget test * Use Eventually function for waiting	4 years ago
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830 ) * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Hiroki Sakamoto	8a914b5c2f	Aggregate inotify watcher to file target manager (#4794 ) * Aggregate inotify watcher * rename to target event handler * Add changelog entry * refactor: make event handler testable * Use context to handle done * doc: add description to sendFileCreateEvent	4 years ago
Hiroki Sakamoto	d61dd1872a	Expose Kafka message key in labels (#4745 ) * expose kafka message key in labels * add changelog entry * improve performance * goimported * fix label name in test * add empty message key test case * pass actual value in message key	4 years ago
James Callahan	21556efa87	Promtail: improve GCP labels (#4628 ) * Promtail: GCP resource label prefix is now '__gcp_resource_labels_' * Promtail: turn GCP 'resource_type' into the internal label '__gcp_resource_type' * Promtail: add internal label with logName * Add upgrade notes for gcplog changes	4 years ago
Cyril Tovena	81e67f89b3	Add GELF support for Promtail. (#4744 ) * Add Gelf support for Promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Changelog Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move change to MAIN Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update graylog to grafana remote. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test and check mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add log messages around startup and shutdown Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Cyril Tovena	2f1e2fc3d2	Fixes Promtail User-Agent. (#4716 ) This was because promtail client was still user the old version.Version which is initialized at run-time via init. But if we use it also at init time there's no guarantee that the version will be set. So intead I'm using the original version variable stamped in the binary and not at runtime. Fixes #4686 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Hiroki Sakamoto	1557dab34f	Add SASL&mTLS authentication support for Kafka in Promtail (#4663 ) * Add xdg-go module * Add mTLS, SASL/PLAIN, SASL/SCRAM authentication support * docs: Add authentication configuration * Add test tools for kafka authentication * Add test stacks for SSL, SASL/PLAIN, SASL/SCRAM, SASL over TLS authentication * Fix shelllint	4 years ago
Ed Welch	2ac409c23e	Build: simplify how protos are built (#4639 ) * we always seem to be chasing our tails with how protos are generated and Makes use of timestamps to determine if files should be recompiled. Instead of touching files and altering timestamps always delete the compiled proto files when calling `protos` to make sure they are compiled every time. Removed targets to build protos, yaccs, and ragel files when trying to build loki or the canary. This isn't necesary, if you are changing these files you would know you need to build them and the `check-generated-files` should catch any changes to them not committed. * rm -rf * we have a race between check-generated-files and our other steps, so let that run first. also removing `check-generated-files` from the `all` target because its redundant with a separate step and could also race with the parallel lint * remove TOUCH_PROTOS * more cleanup of TOUCH_PROTOS	4 years ago

1 2

100 Commits (b3a2cec7c010bf1f5341a515e64c287ab3297e0e)