apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
李国忠	0295fd4698	[new feature] promtail: Add a Promtail stage for probabilistic sampling (#7127 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: The sampling stage can be directly sampled. The implementation of sampling is to use the algorithm in jaeger go client ``` pipeline_stages: - sampling: rate: 0.1 ``` or it can be used with match for precise sampling. ``` pipeline_stages: - json: expressions: app: - match: pipeline_name: "app2" selector: "{app=\"poki\"}" stages: - sampling: rate: 0.1 ``` Which issue(s) this PR fixes: Fixes #6654 Special notes for your reviewer: The promtail 'rate' stage is also used with the 'match' stage for log filtering.This design makes the code very clean. Other log agents vector have log filtering built into the sampling operator, which I think is too complicated https://vector.dev/docs/reference/configuration/transforms/sample/ ``` transforms: my_transform_id: type: sample inputs: - my-source-or-transform-id exclude: null rate: 10 ``` 'rate' stage review suggestions . https://github.com/grafana/loki/pull/5051 ![image](https://user-images.githubusercontent.com/9583245/189461481-6ee4d835-2573-4b8e-8dec-2814620d758a.png) <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Paul Tirk	540380fa94	Promtail: fix GeoIP typo (#8783 ) What this PR does / why we need it: Fixes a typo for GeoIP stage. From `geoip_continet_name` to `geoip_continent_name`.	2 years ago
Danny Kopping	e2ac2d50e2	Changes from linter (#8769 ) What this PR does / why we need it: We had a few outstanding linter issues, and running `make lint` fixes them all automatically. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Dylan Guedes	656ad800fe	Promtail: Stop ticker to avoid leaking (#8695 ) What this PR does / why we need it: Call `Stop` to positions ticker to release associated resources/avoid leaking.	2 years ago
Bryan Boreham	08ac5336d5	WAL: store pointers in record pool (#8667 ) What this PR does / why we need it: Storing a slice in `sync.Pool` wastes 24 bytes every time it is returned to the heap; see https://staticcheck.io/docs/checks#SA6002 Storing a pointer to a slice allows the Pool to work without waste. This came up in discussion about the benchmark in #8489 - with this change the memory usage comes down: ``` name old time/op new time/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 7.37ms ±13% 7.45ms ± 9% ~ (p=0.841 n=5+5) name old alloc/op new alloc/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 727kB ± 0% 679kB ± 0% -6.63% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 12.0k ± 0% 10.0k ± 0% -16.68% (p=0.000 n=5+4) ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Pablo	a15365202d	Optimize Promtail wal writer to batch both series and entries record (#8489 )	2 years ago
Bryan Boreham	6fd4b5e89b	Update prometheus/prometheus from 2.41 to 2.42 (#8571 ) What this PR does / why we need it: Brings in the latest updates from upstream. These open up some opportunities for optimisations in TSDB indexing. Dependencies updated: * github.com/Azure/go-autorest/autorest/adal v0.9.21 -> v0.9.22 (comment-only change) * github.com/docker/docker v20.10.21 -> v20.10.23 (fixing filters bug) * golang.org/x/exp fae10dda9338 -> d38c7dcee874 (optimisations in `BinarySearch` function) Indirect dependencies also updated: * github.com/digitalocean/godo v1.91.1 -> v1.95.0 (nothing alarming in [release notes](https://github.com/digitalocean/godo/releases)) * github.com/google/pprof aee1124e3a93 -> 76d1ae5aea2b (no changes pulled into vendor) * golang.org/x/tools v0.4.0 -> v0.5.0 (relating to Go compiler utilities) * google.golang.org/genproto 76db0878b65f -> 31e0e69b6fc2 * k8s.io/api v0.26.0 -> v0.26.1 (comment-only change) * k8s.io/apimachinery v0.26.0 -> v0.26.1 (no changes pulled into vendor) * k8s.io/client-go v0.26.0 -> v0.26.1 (small fixes) Special notes for your reviewer: A couple of interfaces changed; these have required matching changes in Loki code. Those changes are split into separate commits. I also note that most calls to `relabel` ignore when the rule says "drop". Maybe this is wrong?	2 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583 ) What this PR does / why we need it: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	2 years ago
Aditya C S	b94c8aaf16	feature: geoip stage in promtail (#3493 ) Co-authored-by: Ed Welch <edward.welch@grafana.com>	2 years ago
Yasmine El wahbi	c6542e61fa	Promtail: Add ready endoint to loki-push-api server (#8552 ) When trying to expose loki_push_api service (in our case using GKE ingress), we need a health check target endpoint that would return a 200 response code. This PR contains a simple addition for a HTTP handler that handles the /ready path, along with the necessary tests. Fixes #8551 Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Kaviraj Kanagaraj	43a21eac03	fix(promtail): Make `cri` tags streams aware. (#8497 )	2 years ago
Simon Eisenmann	a9268338c3	Fix reading logs from remote Docker daemon with TLS (#6909 )	2 years ago
Nick Moore	85b3e5d4ab	Promtail: Add event log message stage (#8382 ) The Windows Event Log includes a field named Message that include a large amount of additional data, stored in a simple key-value format: key-value pairs are separated with Windows-style \r\n newlines, the key and value are separated with : (or just : if the key has no associated value). The key-value data in the Message field is commonly utilised when investigating Windows events, e.g., in security use-cases, but the existing windows_events pipeline stage keeps the Message data as a string, leading to requiring significant effort at the query time to extract that data to be used in field filters. This PR request introduces a new promtail stage that parses a specified Source (defaulting to message used by the windows_events) and extracts all the key-value pairs into the extracted fields, which can then be used in latter stages in the pipeline.	2 years ago
Pablo	b5ed39c599	Promtail WAL support: Implement writer side (#8267 ) Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Christian Haudum	f52ae115fe	Bump version of `fluent-plugin-grafana-loki` to 1.2.20 (#8426 )	2 years ago
Christian Haudum	1cbedec0b6	Update fluent-plugin-grafana-loki to Ruby 3.2 (#8421 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Etran-H	5447541e2e	Fix open "/tmp/*" the system cannot find the path specified on windows (#8170 ) Instead of hard-coding `/tmp` directory, use the OS default directory for temporary files	2 years ago
Jack King	d434e80877	Promtail: Add custom headers on push requests (#7470 ) (#7472 )	2 years ago
Witold Duranek	ee1aee0af3	feat(promtail): add more complex options of dropping logs at the drop stage (#6371 )	2 years ago
Bryan Boreham	c3c9ac7740	Cosmetic lint warning fixes (#8363 )	2 years ago
Travis Patterson	078a040794	Retry getting logs from pubsub forever (#8283 )	2 years ago
Christian Haudum	1728d09cba	Use Loki build image 0.27.1 in CI (#8279 ) Second part of updating the Loki build image Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Travis Patterson	71979f0f42	Fix pubsub pull target (#8281 )	2 years ago
Christian Haudum	58e29de988	Update golang to 1.19.5 (#8278 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
nicoche	80ea621ff3	promtail: Add `max-line-size-truncate` (#8233 )	2 years ago
Travis Patterson	fad028b4c3	Change tests to bump sarama dependency (#8249 )	2 years ago
Danny Kopping	d0fd4e8ad1	Do not exit when using `stream_lag_labels` setting (#8243 )	2 years ago
Kaviraj Kanagaraj	86461b3faa	promtail: Add doc for `max-line-size` flag. (#8202 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> What this PR does / why we need it: Follow up from #8153 Changes: * Change type of the flag from `int` to `flagext.ByteSize` (to be consistent with similar flag in `distributor`) * Add doc to promtail config page * Fix changelog entries Which issue(s) this PR fixes: Fixes #NA Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] `CHANGELOG.md` updated Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Callum Styan	399eb097fd	Remove the stream lag metric. (#6324 ) Removes the stream lag metric since it's not possible to get a correct value and also not cause issues with metrics registration or collection until we have the WAL Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Kaviraj Kanagaraj	2597429c89	promtail: Add `max-line-size` limit to drop on client side (#8153 )	2 years ago
Irina	5eba028656	Move to fork github.com/grafana/cloudflare-go (#8133 ) What this PR does / why we need it: The PR https://github.com/cloudflare/cloudflare-go/pull/743 to the upstream repo https://github.com/cloudflare/cloudflare-go was closed with the resolution: > We've spoken to the Grafana Loki team and they'll be building out an integration with Logpush so I don't think this is needed from their side anymore. I [spoke](https://raintank-corp.slack.com/archives/C9T1FLN9K/p1666293494583359) to Loki team and found out that moving to Logpush is not on the shortlist plan right now. I suggest moving to [github.com/grafana/cloudflare-go](https://github.com/grafana/cloudflare-go) fork as the main dependency to get rid of replacement directive. `02e0b3ae89/go.mod (L320)` Another point is replace directives complicate using loki as a dependency in other packages because replace directives should be moved to the upstream package as well. In general, replace was designed as a temporary solution to test code. It shouldn't be used permanently. Which issue(s) this PR fixes: Fixes https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16722, https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16453 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](../CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
youngpablo	ce09aac46d	resolves #5950 (input header error) (#7829 ) this PR fix error while parsing logs from containers with tty: true https://pkg.go.dev/github.com/docker/docker/client#Client.ContainerLogs like in docs: >if the container is using a TTY, there is only a single stream (stdout), and data is copied directly from the container output stream, no extra multiplexing or headers. cause of it we dont need to use StdCopy and can just write to StdOut Fixes #5950 thanks for help @scripter-v Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Karsten Jeschkies	4504ed2701	Cross build Promtail for ARM with journald support. (#8131 ) This is the follow up to #8111 and adds journald support for Promatil ARM and ARM64 builds. I've tested it locally with my Raspberry Pi and it works. Fixes #1459	2 years ago
Jake Herbst	02e0b3ae89	Fix logentry drop to more intuitively match numeric fields (fixes #7011 ) (#7292 ) What this PR does / why we need it: 1. Makes the `drop` stage matching for numeric fields more intuitive. 2. Adds tests scenarios to confirm fix Which issue(s) this PR fixes: Fixes #7011	2 years ago
Rutger	7a1fcab465	Promtail: Improve LabelsMap CPU performance (#7882 ) What this PR does / why we need it: The labelsMapToString function takes a relative high amount of CPU in a simple promtail pipeline. By replacing the fmt.Sprintf with a string. Builder we achieve a 50+% improvement in the function. In my local promtail installation I get about a 10% global cpu reduction. ``` name old time/op new time/op delta LabelsMapToString-8 1.10µs ± 3% 0.30µs ± 4% -72.40% (p=0.008 n=5+5) name old alloc/op new alloc/op delta LabelsMapToString-8 344B ± 0% 184B ± 0% -46.51% (p=0.008 n=5+5) name old allocs/op new allocs/op delta LabelsMapToString-8 14.0 ± 0% 6.0 ± 0% -57.14% (p=0.008 n=5+5) ``` Special notes for your reviewer: The code does get more complex, and this might be a trade-off. Considering promtail is running on many machines (ie. kubernetes nodes) I think the gain is worth it. Two alternatives: - Only replace the `lstrs = append(lstrs, fmt.Sprintf("%s=%q", l, v))` call, as it takes most of the time - Get rid of the explicit `b.Grow` calls and the tracking of size. This only adds a slight improvement Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Michel Hollands	ff57f7249c	Fix json stage fields in example (#7654 ) What this PR does / why we need it: Fix the json section of a scrape config used in a unit test. The unit test doesn't fail as the unmarshaling is very generic. The json stage is unit tested in a different place. This was mentioned as a comment on another PR: https://github.com/grafana/loki/pull/7653. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Chris Hodges	eb6ba615f4	Promtail: Add configuration to drop batches when rate limited by Loki (#7973 ) This change will allow for more deterministic behavior by multitenant Promtail instances where one or more tenants are being heavily throttled due to limits exceeded within Loki. Specifically, it will reduce the performance impact of having exponential backoff cause HOL blocking for rate limit response codes while other tenants are ready to send batches. Co-authored-by: chodges15 <chris.hodges@gmail.org>	2 years ago
Michel Hollands	fcf198e701	Add drone env vars to test output to distinguish the 2 unit test runs. (#8093 ) What this PR does / why we need it: Add the drone step name and branch to the test output. These are used to distinguish between the two unit tests runs in a drone build. One unit test run is for the current branch, the other is for the branch to be merged to. This change helps in not reporting the unit tests results twice. There is also a fix for the TestPromtail test. The missing newline made the test reporting fail. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Ed Welch	f8e582a716	Promtail: Remove noisy error message when cleaning up filesystem watches which are already removed. (#8086 ) What this PR does / why we need it: Promtail will emit an error message when trying to remove filesystem watches which do not exist, this PR squelches those error messages thanks to some updates in the underlying fsnotify lib which allow checking for this exact error. Which issue(s) this PR fixes: Fixes #3874 Special notes for your reviewer: Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Michel Hollands	1e143ace11	Use new build image everywhere (#8063 ) Use the latest build image everywhere. Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Kaviraj Kanagaraj	1375fa0654	chore(promtail): Make `batchwait` and `batchsize` config explicit with yaml tags. (#8027 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> What this PR does / why we need it: These configs can be used via `batchwait` and `batchsize` keys in yaml. This PR just makes it explicit via yaml tags. Which issue(s) this PR fixes: Fixes #NA Special notes for your reviewer: We already have test to cover those keys. https://github.com/grafana/loki/blob/kavirajk%2Fbatch-configs-make-yaml-explicit/clients/pkg/promtail/client/config_test.go#L28-L29 Also we have config pages for those keys. https://grafana.com/docs/loki/latest/clients/promtail/configuration/#clients Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] `CHANGELOG.md` updated Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Christian Haudum	089ec1b05f	Fix various linter errors These changes have been generated using `make lint`. ```console $ golangci-lint --version golangci-lint has version 1.50.0 built from 704109c6 on 2022-10-04T10:25:07Z ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Kaviraj Kanagaraj	d5b68c0257	fix(promtail): Fix cri tags extra new lines. (#7997 )	2 years ago
Irina	8fe386fbb3	Move to fork github.com/grafana/tail (#7931 )	2 years ago
Maxim Ivanov	b773a7a3cf	promtail: ratelimiting by label (#7597 ) Limit stage can now takes optional `by_label_name` param, which tracks distinct values of that label and ratelimit entries independently. Log lines without expected label are not considered for ratelimiting and passed to the next pipeline stage.	2 years ago
Rutger	17517bfea8	Promtail: Fix broken promtail-debug docker image (#7852 ) What this PR does / why we need it: The 'grafana/loki-build-image' version was not specified for the promtail-debug docker image, resulting in a ~6 month old 'latest' image as a build image. Updating this resulted in one more change: the newer version has a different location for the 'dlv' binary. The promtail-debug image would also hang waiting for a connection when ran, this adds the same flag, `--continue`, as added to the loki debug file in #1372. Special notes for your reviewer: I was interested in using the `-server.profiling_enabled=true` flag, to enable pprof endpoints, and scrape with phlare. This flag is not available in the non-debug version. Therefore I do not really care for attaching a debugger right away. Open to removing that change though. Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Robert Fratto	bd58352439	Update build image references to 0.25.0 (#7848 ) cc @slim-bean	2 years ago
cadrake	ea9ad336dc	Loki: Modified heroku drain target to make any url query parameters available as labels (#7619 ) What this PR does / why we need it: This PR reads the url query parameters included in Heroku drain POST request and creates `__param_<name>` labels similar to how the parameters work with prometheus metrics and metric endpoints as a way to allow setting the true application name when ingesting logs from a Heroku drain. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: The existing `app` field coming from Heroku only specifies application source (either the application itself or Heroku) and thus is insufficient for uniquely identifying a log lines source. Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Sandeep Sukhani	89d81020ce	fix lint issues from PR 7804 (#7814 ) What this PR does / why we need it: I had enabled auto-merge in PR #7804, but somehow it still merged the PR without all the checks passing. This PR fixes the failing lint and tests.	3 years ago
George Tsilias	d3615f7e9e	promtail: Handle nil error on target Details() call (#7771 ) What this PR does / why we need it: This PR addresses issue [#6930](https://github.com/grafana/loki/issues/6930). Promtail should not panic when the Details() method is called for a target that has no errors. Which issue(s) this PR fixes: Fixes #6930	3 years ago

1 2 3 4 5 ...

285 Commits (26fcedb3d39a84afcf76f8b3a134657d218f947e)