apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Kaviraj Kanagaraj	2597429c89	promtail: Add `max-line-size` limit to drop on client side (#8153 )	2 years ago
Karsten Jeschkies	4504ed2701	Cross build Promtail for ARM with journald support. (#8131 ) This is the follow up to #8111 and adds journald support for Promatil ARM and ARM64 builds. I've tested it locally with my Raspberry Pi and it works. Fixes #1459	2 years ago
Michel Hollands	1e143ace11	Use new build image everywhere (#8063 ) Use the latest build image everywhere. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Rutger	17517bfea8	Promtail: Fix broken promtail-debug docker image (#7852 ) What this PR does / why we need it: The 'grafana/loki-build-image' version was not specified for the promtail-debug docker image, resulting in a ~6 month old 'latest' image as a build image. Updating this resulted in one more change: the newer version has a different location for the 'dlv' binary. The promtail-debug image would also hang waiting for a connection when ran, this adds the same flag, `--continue`, as added to the loki debug file in #1372. Special notes for your reviewer: I was interested in using the `-server.profiling_enabled=true` flag, to enable pprof endpoints, and scrape with phlare. This flag is not available in the non-debug version. Therefore I do not really care for attaching a debugger right away. Open to removing that change though. Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Robert Fratto	bd58352439	Update build image references to 0.25.0 (#7848 ) cc @slim-bean	3 years ago
Paul Wheeler	85d6794d2b	fluent-bit: Update the Fluent Bit plugin README with correct golang minimum version (#7302 ) What this PR does / why we need it: The Loki plugin for Fluent Bit will not build with golang < 1.17 because of the use of net.IP.IsPrivate in the referenced version of github.com/grafana/dskit/netutil/netutil.go Which issue(s) this PR fixes: No issue exists (this is a minor documentation issue). Special notes for your reviewer: Checklist - [X] Reviewed the `CONTRIBUTING.md` guide - [N/A] Documentation added - [N/A] Tests updated - [N/A] `CHANGELOG.md` updated - [N/A] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Danny Kopping	6c8d04fe78	Completing step 2 of updating loki-build-image (#7796 )	3 years ago
Christian Haudum	d5a86e0ebe	Bump new version 1.2.19 of fluentd-plugin-grafana-loki Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	e04fb0073b	Add make target for building and pushing the fluentd plugin The fluentd plugin can now easily be published using the `make fluentd-plugin-push` make target. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Irina	bdb0aa0bf4	Promtail: Add go build tag promtail_journal_enabled (#7587 ) What this PR does / why we need it: This PR introduces the new go build tag `promtail_journal_enabled`. This tag is passed to go build if the environment variable `PROMTAIL_JOURNAL_ENABLED=true` is present. It is useful for users with Linux OS and CGO enabled. Currently, if the user has Linux and CGO enabled he has to install `libsystemd-dev` library (`sudo apt install -y libsystemd-dev`) even when he doesn't need journald support.	3 years ago
Furkan Türkal	28d6a97991	fluent-bit: clarify docs, use 1.9.9 (#6464 ) What this PR does / why we need it: * Pins digests for security enhancements * Pass `-ldflags` using `--build-args` * Use fluent-bit image `1.9.9` * Use golang image `1.19.2` * Clarify fluent-bit readme Signed-off-by: Furkan <furkan.turkal@trendyol.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Christian Haudum	0f5dd2bceb	Revert "fluentd: Add un-escaping of control characters in JSON" (#7468 ) Reverts grafana/loki#7396 according to discussing on PR	3 years ago
Pablo	5f665651d5	Promtail: Add basic tracing support (#7414 ) We've been using Promtail's HTTP/gRPC server-based targets with real workloads for some weeks now. The server that these spin out is based on [weaveworks commons](https://github.com/weaveworks/common) HTTP/gRPC servers. These include by default some middlewares that if enabled, inject tracing support. We found this useful when tracking down request handling time, or even more, if the caller also support Jaeger tracing support, cross-correlated requests between the upstream server and Promtail's. This PR adds support for enabling tracing the same way Loki does. At the moment, this will just generate traces for the before-mentioned targets. In the future, the quality and completeness of tracing support can grow organically to enrich spans.	3 years ago
Richard Treu	5fe714e466	fluentd: Add un-escaping of control characters in JSON (#7396 )	3 years ago
Danny Kopping	33bf36aa21	Promtail: flush logs on exit (#7400 ) Promtail is using the line-buffered logger (https://github.com/grafana/loki/pull/6954), and log messages were not being printed on exit	3 years ago
李国忠	fb26baa5b1	[new feature] promtail: Add config reload endoint / signal to promtail (#7247 )	3 years ago
Hemanth Krishna	0e54477309	area(promtail): add -check-syntax functionality (#7363 ) Adds `-check-syntax` to promtail which will exit just by validating the config file. Fixes #6862. Signed-off-by: Hemanth Krishna <hkpdev008@gmail.com>	3 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340 ) I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	3 years ago
Danny Kopping	355175ff89	Upgrade to Go 1.19 (#7243 ) Upgrade Loki and all related components to Go 1.19.2.	3 years ago
Danny Kopping	6bf5b5d8c1	Use a line-buffered logger to deamplify write syscalls (#6954 ) We initialise a global logger in `pkg/util/log/log.go` and use it extensively throughout the Loki codebase. Every time we write a log message, a `write` syscall is invoked. Syscalls are problematic because they transition the process from userspace to kernelspace, which means: - a context-switch is incurred, which is inherently expensive ([1-2 microseconds](https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/)) - the goroutine executing the code is blocked - the underlying OS thread (_M_ in the go scheduler model) is also blocked - the goroutine has to be rescheduled once the syscall exits - the go scheduler may need to spawn additional OS threads if all are blocked in syscalls - which can also be expensive This change introduces the use of a line-buffered logger. It has a buffer of [256](https://gist.github.com/dannykopping/0704db32c0b08751d1d2494efaa734c2) entries, and once that buffer is filled it will flush to disk. However, a situation will arise in which that buffer remains somewhat empty for a period of time, so there is a periodic flush mechanism, configured to flush every 100ms. There is also a preallocated bytes slice of 10MB which is reused, to avoid excessive slice resizing & garbage collection. This does mean that we could lose up to 256 log messages in case of an ungraceful termination of the process, but this would need to be precisely timed within the 100ms flushes - in other words, the likelihood is low, and generally we shouldn't `kill -9` any Loki process.	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212 ) The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
李国忠	a290549a59	Promtail: Add support for max stream limit (#7101 ) What this PR does / why we need it: Add Promtail support to a new feature/configuration named `max_stream_limit` * This new feature limits the max number of streams Promtail can materialize. The new feature is also useful to avoid OOMing Promtail Which issue(s) this PR fixes: Fixes #6968	3 years ago
Callum Styan	4a21923913	update to alpine:3.6.2 (#6979 ) Updates the alpine image version we use to 3.16.2 and will also build/publish a new loki build image Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Christian Simon	88e2705034	Update alpine and golang base images (#6897 ) * Upgrade base images to alpine 3.15.6 everywhere https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html * Update golang images to 1.18.5	3 years ago
Kaviraj Kanagaraj	0d99d80acd	upgrade `go:1.17.9` to `go:1.18.4` (#6726 )	3 years ago
Rutger	1794a76613	Promtail: Add metrics for journal target (#6105 ) * Add metrics for journal target There are no metrics for the journal target yet, this adds two basic ones: - Total number of lines processed - Total number of errors processing lines (with a reason) Because of the way the journal works, tailing one file with many possible process writing to it, I chose note to expose counts per unit. * Add changelog entry * Fix compilation of (unsupported) journalmanager target on windows * Extract possible errors to constants & add tests	3 years ago
Trevor Wood	23cc93862d	chore(fluentd): resolve rubocop failures (#6359 ) Signed-off-by: Trevor Wood <Trevor.G.Wood@gmail.com>	3 years ago
Trevor Wood	0470e4e7d3	feat(fluentd): allow fluentd_thread label to be configurable (#6240 ) Signed-off-by: Trevor Wood <Trevor.G.Wood@gmail.com>	3 years ago
Alex Ferm	d7f03be212	Update libsystemd-dev from bullseye-backports for arm32 docker images, fixes (#5938 ) (#6012 )	3 years ago
Christian Simon	9d0fd3a7e3	Upgrade golang version to v1.17.9 (#5899 ) ``` git ls-files '[Dd]ockerfile' \| grep -v ^vendor/ \| xargs sed -i "s/golang:[0-9\\.]*/golang:1.17.9/g" \| ag 'golang:' ```	3 years ago
Karsten Jeschkies	c9f3868287	Update dskit to v0.0.0-20220331160727-49faf69f72ca. (#5881 )	3 years ago
Christian Simon	3d941ccccb	Update alpine images to 3.15.4 (#5780 )	3 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724 )	3 years ago
Franz Wong	d67306b657	Promtail: Rename config name limit_config to limits_config (#5707 ) * Rename promtail config name 'limit_config' to 'limits_config' * Add changelog entry * Fix formatting	3 years ago
Susana Ferreira	1d1461824d	Move promtail StreamLagLabels config to upper level config.Config (#5686 ) * Move promtail StreamLagLabels config to upper level config.Config * Fix promtail config load test * Fix promtail configuration reference * Add options promtail config section * Kick GitHub CI Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Callum Styan	2ebbecfcbd	Require unique client configs when there are multiple clients. (#5688 ) * Require unique client configs when there are multiple clients. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fixing linter error Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Callum Styan	01d63c8ae8	[promtail] Refactor promtail client metrics so that we can't have duplicate metrics collected for the lag metric. (#5521 ) * Refactor promtail client metrics so that we can't have duplicate metrics collected for the lag metric. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Set up new metrics passing for other client types. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix lint issues. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update changelog and docs, also rename nested client configs struct to configs. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Cyril Tovena	c8bdb37ef3	update loki go version to 1.17.8 (#5543 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Holger Hees	f359a8280f	Update out_loki.rb (#5512 ) this will fix "UndefinedConversionError error="\"\\xC2\" from ASCII-8BIT to UTF-8"" for messages of systemd journal	3 years ago
Travis Patterson	de1faa0671	Fix transitive dependency error (#5308 ) * Fix transitive dependency error - Don't update the gems that come with the logstash image * ci bump * ci bump	3 years ago
Ed Welch	44ba39055d	logstash: re-add bundle update and update bundler (#5284 ) * update to latest logstash and bundler Signed-off-by: Edward Welch <edward.welch@grafana.com> * re-add bundle update Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Kaviraj Kanagaraj	858f8ea54a	Fix transitive dependency failure while building logstash plugin (#5268 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Ed Welch	09a5a2cee3	Logstash: Add config option which allows seting up an allowlist for labels to be mapped to Loki (#5244 ) * add a config called `include_fields` which lets you allowlist a specific set of fields which will be mapped to labels and sent to Loki. Signed-off-by: Edward Welch <edward.welch@grafana.com> * update docs Signed-off-by: Edward Welch <edward.welch@grafana.com> * restore path for Dockerfile so it works in CI Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Cyril Tovena	cde9a711f3	fluent-bit: Attempt to unmarshal nested json. (#5223 ) * fluent-bit: Attempt to unmarshal nested json. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Also support nested json array. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	fa25e54e26	Bump logstash plugin gem (#5230 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Christian Haudum	985e965796	Fix encoding error in fluentd client (#5163 ) * Fix encoding error in fluentd client This change fixes a bug that is caused by incorrectly trying to encode any values other than string. This bug was a regression introduced with #5107 and caused `NoMethodError` errors. Fixes #5099 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Bump fluent-plugin-grafana-loki to version 1.2.18 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	52fa2114df	fluentd: Fix bug that caused lines to be dropped when containing non utf-8 characters (#5107 ) * fluentd: Remove non utf-8 characters from log lines When a log line in hash format contains non UTF-8 characters fluentd would drop the complete line because it failed to convert the line in key-value format. By forcing UTF-8 encoding and replacing non UTF-8 characters with empty strings the log line will not be dropped but only contain the valid UTF-8 characters. Fixes #5099 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Bump fluent-plugin-grafana-loki to version 1.2.17 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fluentd: Remove non utf-8 characters from log lines Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fluentd: Remove non utf-8 characters from log lines Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
李国忠	6b377b40d5	[new] promtail: add readline rate limit (#5031 ) * [new] promtail: add readline rate limit * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * [new] promtail: add readline rate limit #5031 * Update clients/pkg/promtail/limit/config.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Update clients/pkg/promtail/limit/config.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Christian Haudum	4687b416db	Update logstash dependency in logstash-output-loki plugin (#4930 ) Use `logstash:7.16.1` as base image for the `grafana/logstash-output-loki` Docker image. This logstash version uses `log4j` 2.15.0 which mitigates CVE-2021-44228. - https://www.elastic.co/guide/en/logstash/current/logstash-7-16-1.html - https://nvd.nist.gov/vuln/detail/CVE-2021-44228 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago

1 2 3

121 Commits (cd64d6ddbd58981c4d3328a50a59179b0b33c696)