apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	bbaef790db	Add usage report into Loki. (#5361 ) * Adds leader election process Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fluke Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes the kv typecheck * wire up the http client * Hooking into loki services, hit a bug * Add stats variable. * re-vendor dskit and improve to never fail service * Intrument Loki with the package * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes compactor test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add configuration documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Add boundary check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add log for success report. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/usagestats/reporter.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Susana Ferreira	82fac990a1	Update promtail to support duration string formats (#5290 ) * Update promtail to support duration string formats * Address comments	3 years ago
Travis Patterson	de1faa0671	Fix transitive dependency error (#5308 ) * Fix transitive dependency error - Don't update the gems that come with the logstash image * ci bump * ci bump	3 years ago
Travis Patterson	1925d77f65	cleanup: consolidate util strings contain functions (#5304 ) * Consolidate util strings contain functions * fix test	3 years ago
Christian Haudum	3fb572abda	Fix Promtail watching deadlock (#5283 ) * Reuse channel from fileEventWatcher map if already exists Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove sync() call in FileTarget constructor The constructor `NewFileTarget` is called from within a locked code block by the FileTargetManager. However, the constructor also calls `sync()` which starts and stops watching new directories. These methods `startWatching` and `stopWatching` both send to the `targetEventHandler` channel, causing the `FileTargetManager` to receive the event in a goroutine (`watch()` function) running in the background. Handling the event calls `sendFileCreateEvent` on the syncers, which subsequently tries to write to the `fileEventWatcher` channel which is never read because of the locked mutex. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Karsten Jeschkies	b92b544ca2	Write Promtail position file atomically. (#5266 ) Fixes #5218	3 years ago
Karsten Jeschkies	54bc2dc40e	Restart Docker target if stopped. (#5280 ) Summary: The Docker target would not reconnect to a container if it restarted and kept the same ID. This should introduces a `startIfNotRunning` method that will be called to restart scraping the container's logs. Fixes #5259	3 years ago
Ed Welch	44ba39055d	logstash: re-add bundle update and update bundler (#5284 ) * update to latest logstash and bundler Signed-off-by: Edward Welch <edward.welch@grafana.com> * re-add bundle update Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	858f8ea54a	Fix transitive dependency failure while building logstash plugin (#5268 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Lydia Lee	a4cfab4b84	feat(promtail): fix TargetManager.run() not exit (#5238 )	3 years ago
Ed Welch	09a5a2cee3	Logstash: Add config option which allows seting up an allowlist for labels to be mapped to Loki (#5244 ) * add a config called `include_fields` which lets you allowlist a specific set of fields which will be mapped to labels and sent to Loki. Signed-off-by: Edward Welch <edward.welch@grafana.com> * update docs Signed-off-by: Edward Welch <edward.welch@grafana.com> * restore path for Dockerfile so it works in CI Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257 ) * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Cyril Tovena	cde9a711f3	fluent-bit: Attempt to unmarshal nested json. (#5223 ) * fluent-bit: Attempt to unmarshal nested json. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Also support nested json array. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Roger Steneteg	bc88f298cb	skip glob search if path is an existing file and not a directory (#5246 ) Signed-off-by: Roger Steneteg <rsteneteg@ea.com>	3 years ago
Cyril Tovena	fa25e54e26	Bump logstash plugin gem (#5230 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Karsten Jeschkies	4716eea0db	Provide Docker target and discovery in Promtail. (#4911 ) What this PR does / why we need it: This patch adds support to fetch Docker container logs through the Docker daemon API. This should be more robust than the Loki Docker driver or scraping the logs files. The new Docker target will also collect meta information of the scraped containers. Which issue(s) this PR fixes: Addresses #2361 Closes #4703 Special notes for your reviewer: Checklist - [x] Documentation added - [x] Tests updated - [x] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Christian Haudum	86ed98c1fa	Fix promtail deadlock (#5170 ) * Remove unused clientConfigs argument from NewTargetsManagers * Do not create targetScrapeConfigs if stdin target is used * fixup! Remove unused clientConfigs argument from NewTargetsManagers * Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup: add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Close file watcher when target is removed Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	985e965796	Fix encoding error in fluentd client (#5163 ) * Fix encoding error in fluentd client This change fixes a bug that is caused by incorrectly trying to encode any values other than string. This bug was a regression introduced with #5107 and caused `NoMethodError` errors. Fixes #5099 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Bump fluent-plugin-grafana-loki to version 1.2.18 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Cyril Tovena	144bee11af	Retry pulling Cloudflare logs after a stream error (#5158 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Christian Haudum	52fa2114df	fluentd: Fix bug that caused lines to be dropped when containing non utf-8 characters (#5107 ) * fluentd: Remove non utf-8 characters from log lines When a log line in hash format contains non UTF-8 characters fluentd would drop the complete line because it failed to convert the line in key-value format. By forcing UTF-8 encoding and replacing non UTF-8 characters with empty strings the log line will not be dropped but only contain the valid UTF-8 characters. Fixes #5099 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Bump fluent-plugin-grafana-loki to version 1.2.17 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fluentd: Remove non utf-8 characters from log lines Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fluentd: Remove non utf-8 characters from log lines Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
李国忠	6b377b40d5	[new] promtail: add readline rate limit (#5031 ) * [new] promtail: add readline rate limit * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * Update clients/pkg/promtail/limit/config.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Update clients/pkg/promtail/limit/config.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Ed Welch	ba91a0e48a	lock the mutex before working with the `fileEventWatchers` (#5043 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944 )	3 years ago
Christian Haudum	4687b416db	Update logstash dependency in logstash-output-loki plugin (#4930 ) Use `logstash:7.16.1` as base image for the `grafana/logstash-output-loki` Docker image. This logstash version uses `log4j` 2.15.0 which mitigates CVE-2021-44228. - https://www.elastic.co/guide/en/logstash/current/logstash-7-16-1.html - https://nvd.nist.gov/vuln/detail/CVE-2021-44228 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Alan Hollis	1e0b2707cd	Fix replace missing adjacent capture groups (#4874 )	4 years ago
Cyril Tovena	7f71c5f051	Promtail pull cloudflare logs (#4813 )	4 years ago
Hiroki Sakamoto	77dcb16b4e	Fix duplicate registry.MustRegister call in Promtail Kafka (#4866 ) * Move pipeline creation to new targetsyncer func * registry.MustRegister is called every newPipeline so ensure it only at once * add changelog entry	4 years ago
Hiroki Sakamoto	d7c35951c3	Use http prefix path correctly in promtail (#4832 ) * Use http prefix path correctly in promtail * Update changelog entry	4 years ago
Hiroki Sakamoto	11bf736c90	Refactor tests of Promtail filetarget (#4802 ) * Refactor tests of Promtail filetarget * Move tests depend on inotify to filetargetmanager test * Add handling file creation event via channel case in filetarget test * Use Eventually function for waiting	4 years ago
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830 ) * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Hiroki Sakamoto	8a914b5c2f	Aggregate inotify watcher to file target manager (#4794 ) * Aggregate inotify watcher * rename to target event handler * Add changelog entry * refactor: make event handler testable * Use context to handle done * doc: add description to sendFileCreateEvent	4 years ago
Hiroki Sakamoto	d61dd1872a	Expose Kafka message key in labels (#4745 ) * expose kafka message key in labels * add changelog entry * improve performance * goimported * fix label name in test * add empty message key test case * pass actual value in message key	4 years ago
James Callahan	21556efa87	Promtail: improve GCP labels (#4628 ) * Promtail: GCP resource label prefix is now '__gcp_resource_labels_' * Promtail: turn GCP 'resource_type' into the internal label '__gcp_resource_type' * Promtail: add internal label with logName * Add upgrade notes for gcplog changes	4 years ago
Cyril Tovena	81e67f89b3	Add GELF support for Promtail. (#4744 ) * Add Gelf support for Promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Changelog Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move change to MAIN Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update graylog to grafana remote. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test and check mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add log messages around startup and shutdown Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Cyril Tovena	2f1e2fc3d2	Fixes Promtail User-Agent. (#4716 ) This was because promtail client was still user the old version.Version which is initialized at run-time via init. But if we use it also at init time there's no guarantee that the version will be set. So intead I'm using the original version variable stamped in the binary and not at runtime. Fixes #4686 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Hiroki Sakamoto	1557dab34f	Add SASL&mTLS authentication support for Kafka in Promtail (#4663 ) * Add xdg-go module * Add mTLS, SASL/PLAIN, SASL/SCRAM authentication support * docs: Add authentication configuration * Add test tools for kafka authentication * Add test stacks for SSL, SASL/PLAIN, SASL/SCRAM, SASL over TLS authentication * Fix shelllint	4 years ago
Ed Welch	2ac409c23e	Build: simplify how protos are built (#4639 ) * we always seem to be chasing our tails with how protos are generated and Makes use of timestamps to determine if files should be recompiled. Instead of touching files and altering timestamps always delete the compiled proto files when calling `protos` to make sure they are compiled every time. Removed targets to build protos, yaccs, and ragel files when trying to build loki or the canary. This isn't necesary, if you are changing these files you would know you need to build them and the `check-generated-files` should catch any changes to them not committed. * rm -rf * we have a race between check-generated-files and our other steps, so let that run first. also removing `check-generated-files` from the `all` target because its redundant with a separate step and could also race with the parallel lint * remove TOUCH_PROTOS * more cleanup of TOUCH_PROTOS	4 years ago
Cyril Tovena	c8f3df33cf	Promtail Kafka target (#4568 ) * Adds a kafka target manager in promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add validations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar test for the fanout client. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * final adjustement Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding topics discovery. Still needs to finish tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing off testing it. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Revert config changes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add group id as discovered label Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * linter Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add tools for running kafka and testing locally. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted shell Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update sarama to compile in ARM. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for kafka target. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve code comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add a few s's * Better cancellation support. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * adds `__meta_kafka_` suffix to discovered labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Roger Steneteg	3b66d94e0f	resolve issue with promtail not scraping target if only path changed in a simpler way that dont need mutex to sync threads (#4599 ) Signed-off-by: Roger Steneteg <rsteneteg@ea.com>	4 years ago
Ed Welch	de4993f921	explicitly define allowed HTTP methods on HTTP endpoints (#4627 )	4 years ago
James Callahan	a92541bd84	Promtail: no need for GCP promtail_instance label now that loki supports out-of-order writes (#4556 ) Closes #4474	4 years ago
Owen Diehl	853260cfae	regenerates assets from current vfsgen dependency (#4588 )	4 years ago
lingpeng0314	76e2a14029	add group_{left,right} to LogQL (#4510 ) * Add group_left/group_right support * Update docs/sources/logql/_index.md * Minor change for a test case * removes CardManyToMany * removes now-unused IsSetOperator helper Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	93a0b716d4	Refactor client configs in Promtail. (#4567 ) This refactor a big how we mutate the original config for backward compatibility. Also add tests on untested code path. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sankalp Rangare	9e2acd656d	add static labels in stages (#4495 ) Signed-off-by: Sankalp Rangare <sankalprangare786@gmail.com>	4 years ago
Lize	69819c2e07	update promtail base image to debian:bullseye-slim (#4516 )	4 years ago
Danny Kopping	703d68bc76	Promtail: allow for customisable stream lag labels (#4507 ) * Allow for customisable stream lag labels Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating test Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Allowing for multiple labels Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding field to docs Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Danny Kopping	cc1d478bb2	Clarify error message (#4524 ) Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Roger Steneteg	0ae0d4da12	fix filetarget to not be stuck if no files was detected on startup (#4461 ) Signed-off-by: Roger Steneteg <rsteneteg@ea.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago

1 2

88 Commits (7378b697f8dc05762e45d4a243005dad27d93d35)