apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Chris Hodges	eb6ba615f4	Promtail: Add configuration to drop batches when rate limited by Loki (#7973 ) This change will allow for more deterministic behavior by multitenant Promtail instances where one or more tenants are being heavily throttled due to limits exceeded within Loki. Specifically, it will reduce the performance impact of having exponential backoff cause HOL blocking for rate limit response codes while other tenants are ready to send batches. Co-authored-by: chodges15 <chris.hodges@gmail.org>	2 years ago
Michel Hollands	fcf198e701	Add drone env vars to test output to distinguish the 2 unit test runs. (#8093 ) What this PR does / why we need it: Add the drone step name and branch to the test output. These are used to distinguish between the two unit tests runs in a drone build. One unit test run is for the current branch, the other is for the branch to be merged to. This change helps in not reporting the unit tests results twice. There is also a fix for the TestPromtail test. The missing newline made the test reporting fail. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Ed Welch	f8e582a716	Promtail: Remove noisy error message when cleaning up filesystem watches which are already removed. (#8086 ) What this PR does / why we need it: Promtail will emit an error message when trying to remove filesystem watches which do not exist, this PR squelches those error messages thanks to some updates in the underlying fsnotify lib which allow checking for this exact error. Which issue(s) this PR fixes: Fixes #3874 Special notes for your reviewer: Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Michel Hollands	1e143ace11	Use new build image everywhere (#8063 ) Use the latest build image everywhere. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Kaviraj Kanagaraj	1375fa0654	chore(promtail): Make `batchwait` and `batchsize` config explicit with yaml tags. (#8027 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> What this PR does / why we need it: These configs can be used via `batchwait` and `batchsize` keys in yaml. This PR just makes it explicit via yaml tags. Which issue(s) this PR fixes: Fixes #NA Special notes for your reviewer: We already have test to cover those keys. https://github.com/grafana/loki/blob/kavirajk%2Fbatch-configs-make-yaml-explicit/clients/pkg/promtail/client/config_test.go#L28-L29 Also we have config pages for those keys. https://grafana.com/docs/loki/latest/clients/promtail/configuration/#clients Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] `CHANGELOG.md` updated Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Christian Haudum	089ec1b05f	Fix various linter errors These changes have been generated using `make lint`. ```console $ golangci-lint --version golangci-lint has version 1.50.0 built from 704109c6 on 2022-10-04T10:25:07Z ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Kaviraj Kanagaraj	d5b68c0257	fix(promtail): Fix cri tags extra new lines. (#7997 )	2 years ago
Irina	8fe386fbb3	Move to fork github.com/grafana/tail (#7931 )	2 years ago
Maxim Ivanov	b773a7a3cf	promtail: ratelimiting by label (#7597 ) Limit stage can now takes optional `by_label_name` param, which tracks distinct values of that label and ratelimit entries independently. Log lines without expected label are not considered for ratelimiting and passed to the next pipeline stage.	2 years ago
Rutger	17517bfea8	Promtail: Fix broken promtail-debug docker image (#7852 ) What this PR does / why we need it: The 'grafana/loki-build-image' version was not specified for the promtail-debug docker image, resulting in a ~6 month old 'latest' image as a build image. Updating this resulted in one more change: the newer version has a different location for the 'dlv' binary. The promtail-debug image would also hang waiting for a connection when ran, this adds the same flag, `--continue`, as added to the loki debug file in #1372. Special notes for your reviewer: I was interested in using the `-server.profiling_enabled=true` flag, to enable pprof endpoints, and scrape with phlare. This flag is not available in the non-debug version. Therefore I do not really care for attaching a debugger right away. Open to removing that change though. Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Robert Fratto	bd58352439	Update build image references to 0.25.0 (#7848 ) cc @slim-bean	3 years ago
cadrake	ea9ad336dc	Loki: Modified heroku drain target to make any url query parameters available as labels (#7619 ) What this PR does / why we need it: This PR reads the url query parameters included in Heroku drain POST request and creates `__param_<name>` labels similar to how the parameters work with prometheus metrics and metric endpoints as a way to allow setting the true application name when ingesting logs from a Heroku drain. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: The existing `app` field coming from Heroku only specifies application source (either the application itself or Heroku) and thus is insufficient for uniquely identifying a log lines source. Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Sandeep Sukhani	89d81020ce	fix lint issues from PR 7804 (#7814 ) What this PR does / why we need it: I had enabled auto-merge in PR #7804, but somehow it still merged the PR without all the checks passing. This PR fixes the failing lint and tests.	3 years ago
George Tsilias	d3615f7e9e	promtail: Handle nil error on target Details() call (#7771 ) What this PR does / why we need it: This PR addresses issue [#6930](https://github.com/grafana/loki/issues/6930). Promtail should not panic when the Details() method is called for a target that has no errors. Which issue(s) this PR fixes: Fixes #6930	3 years ago
Paul Wheeler	85d6794d2b	fluent-bit: Update the Fluent Bit plugin README with correct golang minimum version (#7302 ) What this PR does / why we need it: The Loki plugin for Fluent Bit will not build with golang < 1.17 because of the use of net.IP.IsPrivate in the referenced version of github.com/grafana/dskit/netutil/netutil.go Which issue(s) this PR fixes: No issue exists (this is a minor documentation issue). Special notes for your reviewer: Checklist - [X] Reviewed the `CONTRIBUTING.md` guide - [N/A] Documentation added - [N/A] Tests updated - [N/A] `CHANGELOG.md` updated - [N/A] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Danny Kopping	6c8d04fe78	Completing step 2 of updating loki-build-image (#7796 )	3 years ago
Chris Hodges	e80b4deb43	Promtail: Add tenant label to client drop metrics and logs (#7593 ) What this PR does / why we need it: Provides better visibility for per-tenant drops/retries in Promtail Which issue(s) this PR fixes: Fixes #7570 Special notes for your reviewer: Does anything depend on these metrics only having 1 label that this change would break? I decided against a reason label due to cardinality concerns given how widely promtail is often deployed. Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
MarNicGit	1f1dd81555	Promtail: Exclude event message (#7462 ) What this PR does / why we need it: Windows Event Logs have an Event Message field that is intended for human eyes, and often contains data that already present in the event data XML. Omitting this field the same way we can already omit `user_data` and `event_data` can easily save a lot of bytes of data per event - Event ID 4264 alone has ~2KB of just text that is already present in `event_data`. Which issue(s) this PR fixes: Fixes #7395 Special notes for your reviewer: I also took the liberty to improve upon the existing test '`Test_renderEntries` by using unique values for each field rather than 10's everywhere. I expect this to conflict with my other PR, #7461.	3 years ago
Christian Haudum	d5a86e0ebe	Bump new version 1.2.19 of fluentd-plugin-grafana-loki Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Christian Haudum	e04fb0073b	Add make target for building and pushing the fluentd plugin The fluentd plugin can now easily be published using the `make fluentd-plugin-push` make target. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Irina	bdb0aa0bf4	Promtail: Add go build tag promtail_journal_enabled (#7587 ) What this PR does / why we need it: This PR introduces the new go build tag `promtail_journal_enabled`. This tag is passed to go build if the environment variable `PROMTAIL_JOURNAL_ENABLED=true` is present. It is useful for users with Linux OS and CGO enabled. Currently, if the user has Linux and CGO enabled he has to install `libsystemd-dev` library (`sudo apt install -y libsystemd-dev`) even when he doesn't need journald support.	3 years ago
Benoît Knecht	489898f529	promtail/server: Disable profiling by default (#5400 ) What this PR does / why we need it: Add a `server.debug` option (that defaults to `false`) to enable the `/debug/fgprof` endpoint in Promtail. Which issue(s) this PR fixes: Fixes #5005 Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
李国忠	fe49e6642e	[bugfix] add Promtail cloudflare close (#7394 ) What this PR does / why we need it: If the http server closes the connection, the client may also return a non-empty reader io.ReadCloser. Which issue(s) this PR fixes: Fixes #6150 Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
gelo22	8b89343b29	batchRetries to countersWithHost (#7392 ) What this PR does / why we need it: Metric promtail_batch_retries_total is absent when has value == 0 Which issue(s) this PR fixes: Fixes #7261	3 years ago
Furkan Türkal	28d6a97991	fluent-bit: clarify docs, use 1.9.9 (#6464 ) What this PR does / why we need it: * Pins digests for security enhancements * Pass `-ldflags` using `--build-args` * Use fluent-bit image `1.9.9` * Use golang image `1.19.2` * Clarify fluent-bit readme Signed-off-by: Furkan <furkan.turkal@trendyol.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Max Vorobev	c5911d5342	Loki: implement decolorize filter (#7602 ) Implement `decolorize` filter that get rids of ANSI color codes in the log message. Which issue(s) this PR fixes: Fixes #7601	3 years ago
MarNicGit	17c36d60f3	Promtail: Fix collecting userdata field from Windows Event Log (#7461 ) What this PR does / why we need it: The windows event log target for Promtail had a typo that led to `EventData` being assigned to `UserData`. This PR fixes this typo. Which issue(s) this PR fixes: Fixes #6167	3 years ago
Robert Fratto	3808fb6ac3	promtail: register reload metrics after running options (#7561 ) What this PR does / why we need it: This allows the reload metrics to be sent to a custom registerer, which is required for Grafana Agent to be able to use the Promtail code. Which issue(s) this PR fixes: N/A Special notes for your reviewer: Is it possible for us to cherry-pick this into k121? That's the version we're trying to update Grafana Agent to. Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Christian Haudum	0f5dd2bceb	Revert "fluentd: Add un-escaping of control characters in JSON" (#7468 ) Reverts grafana/loki#7396 according to discussing on PR	3 years ago
BenjaminVouillaumeJT	2d649a7fdd	Promtail: Add additional fields for cloudflare (#7463 )	3 years ago
Pablo	5f665651d5	Promtail: Add basic tracing support (#7414 ) We've been using Promtail's HTTP/gRPC server-based targets with real workloads for some weeks now. The server that these spin out is based on [weaveworks commons](https://github.com/weaveworks/common) HTTP/gRPC servers. These include by default some middlewares that if enabled, inject tracing support. We found this useful when tracking down request handling time, or even more, if the caller also support Jaeger tracing support, cross-correlated requests between the upstream server and Promtail's. This PR adds support for enabling tracing the same way Loki does. At the moment, this will just generate traces for the before-mentioned targets. In the future, the quality and completeness of tracing support can grow organically to enrich spans.	3 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425 )	3 years ago
Richard Treu	5fe714e466	fluentd: Add un-escaping of control characters in JSON (#7396 )	3 years ago
Danny Kopping	2caf5f7c5d	Promtail: correctly hashes the config on reload (#7421 )	3 years ago
Pablo	e73584a3cd	Promtail: Add handler timeout for GCP Logs Push target (#7401 ) What this PR does / why we need it: After been running the GCP Logs Push target in actual workloads, we've noticed that GCP sends a lot of traffic, and adjusts that send rate according to the rate at which the receiver can process. To perform that adjustment, GCP considers as a NACK [non 2xx HTTP responses](https://cloud.google.com/pubsub/docs/push#receive_push). This PR adds a target handler timeout (that includes as well the `api.Entry` channel send) to allow the user to configure this maximum processing time, therefore giving GCP notice that the sending rate is too high. This new timeout is optional, and disabled by default. Which issue(s) this PR fixes: Related to https://github.com/grafana/cloud-onboarding/issues/2067 Special notes for your reviewer: Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	61bbe15171	[promtail] log: simplify log, only print hash (#7405 ) Compute and print MD5 hash of config	3 years ago
Vijal Patel	5f7543d047	Fix promtail scrape config job name validation (#7402 )	3 years ago
Danny Kopping	33bf36aa21	Promtail: flush logs on exit (#7400 ) Promtail is using the line-buffered logger (https://github.com/grafana/loki/pull/6954), and log messages were not being printed on exit	3 years ago
李国忠	fdbb06d123	add promtail reload changelog and doc (#7386 )	3 years ago
李国忠	fb26baa5b1	[new feature] promtail: Add config reload endoint / signal to promtail (#7247 )	3 years ago
Brandon W Maister	381735d3d4	logging: Change Docker target discovery log level from Error to Info (#7312 ) What this PR does / why we need it: Discovering a new node is not an error, so having it send errors to dashboards is confusing. Special notes for your reviewer: Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Hemanth Krishna	0e54477309	area(promtail): add -check-syntax functionality (#7363 ) Adds `-check-syntax` to promtail which will exit just by validating the config file. Fixes #6862. Signed-off-by: Hemanth Krishna <hkpdev008@gmail.com>	3 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340 ) I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	3 years ago
Danny Kopping	355175ff89	Upgrade to Go 1.19 (#7243 ) Upgrade Loki and all related components to Go 1.19.2.	3 years ago
Danny Kopping	6bf5b5d8c1	Use a line-buffered logger to deamplify write syscalls (#6954 ) We initialise a global logger in `pkg/util/log/log.go` and use it extensively throughout the Loki codebase. Every time we write a log message, a `write` syscall is invoked. Syscalls are problematic because they transition the process from userspace to kernelspace, which means: - a context-switch is incurred, which is inherently expensive ([1-2 microseconds](https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/)) - the goroutine executing the code is blocked - the underlying OS thread (_M_ in the go scheduler model) is also blocked - the goroutine has to be rescheduled once the syscall exits - the go scheduler may need to spawn additional OS threads if all are blocked in syscalls - which can also be expensive This change introduces the use of a line-buffered logger. It has a buffer of [256](https://gist.github.com/dannykopping/0704db32c0b08751d1d2494efaa734c2) entries, and once that buffer is filled it will flush to disk. However, a situation will arise in which that buffer remains somewhat empty for a period of time, so there is a periodic flush mechanism, configured to flush every 100ms. There is also a preallocated bytes slice of 10MB which is reused, to avoid excessive slice resizing & garbage collection. This does mean that we could lose up to 256 log messages in case of an ungraceful termination of the process, but this would need to be precisely timed within the 100ms flushes - in other words, the likelihood is low, and generally we shouldn't `kill -9` any Loki process.	3 years ago
Irina	d1a996afce	Added mapstructure tag to promtail configs structs (#7320 ) Added mapstructure tags to configs structs to promtail client. It is needed for [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib). I working on adding promtail receiver component there. Opentelemetry collector decodes configs with mapstructure	3 years ago
Dylan Guedes	f571282ffc	Tests: Improve several tests behavior (#7283 ) What this PR does / why we need it: Follow-up of https://github.com/grafana/loki/pull/7273 * Tune a filetarget test to remove created files to guarantee that all tailers are interrupted/fred * Call pubsub close for our GCP `pull target` * Append testName to the job names used in GCP tests. This will help us to identify which tests are struggling.	3 years ago
Dylan Guedes	2dc5a71a67	Tests: Reduce flakiness of GCPLog and integration tests (#7273 ) What this PR does / why we need it: - Reduce flakiness of GCPLog tests by reusing context and stopping GCP topic - Reduce flakiness of integration tests by: - using port as they're assigned - reusing context - adding timeout to calls (hence, if something get stuck, tests fail earlier)	3 years ago
Dylan Guedes	f3b72c8b59	Promtail: Change max support compressed line size to be 2MB (#7268 ) What this PR does / why we need it: Change Promtail compression support max line size to 2MB instead of 4096 bytes (~4KB). This however doesn't change the default buffer size. 4096 bytes means we can't ever have a buffer size bigger than ~4KB but since the default mem page size is 4096 ([source](https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space)), we end up ignoring all lines.	3 years ago
Pablo	64fe0af9dd	Expose tenant ID through relabel in push promtail targets (#7250 ) What this PR does / why we need it: Some Promtail targets that support pushing logs through HTTP, allow using a specific header for writing logs for multiple tenants. This log is propagated through a Promtail internal label, that is then consumed by the remote write client. If Promtail is used in a multi-tenant manner, it'd be nice to somehow be able to track how much is each tenant's share from the logs received. This PR exposes the received tenant id value through relabeling. This allows for example, keeping it as a label and exposing a metric though the `metrics` pipeline stage, which can be segregated by tenant. Which issue(s) this PR fixes: Related to https://github.com/grafana/cloud-onboarding/issues/2067	3 years ago

1 2 3 4

199 Commits (3f9b9cc9611105be4271a14e9c0f36dbd31bcbed)