213 Commits (04ce1b87213d18f414f56a74e02a33e7b65fc8f1)

Author	SHA1	Message	Date
Maxim Ivanov	b773a7a3cf	promtail: ratelimiting by label (#7597) ... Limit stage can now takes optional `by_label_name` param, which tracks distinct values of that label and ratelimit entries independently. Log lines without expected label are not considered for ratelimiting and passed to the next pipeline stage.	3 years ago
Chris James	527ee12220	Fixing a typo in configuration.md (#7878) ... **What this PR does / why we need it**: Fixes a typo in configuration.md, improves reader clarify and confidence **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
cadrake	ea9ad336dc	Loki: Modified heroku drain target to make any url query parameters available as labels (#7619) ... **What this PR does / why we need it**: This PR reads the url query parameters included in Heroku drain POST request and creates `__param_<name>` labels similar to how the parameters work with prometheus metrics and metric endpoints as a way to allow setting the true application name when ingesting logs from a Heroku drain. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: The existing `app` field coming from Heroku only specifies application source (either the application itself or Heroku) and thus is insufficient for uniquely identifying a log lines source. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
MarNicGit	1f1dd81555	Promtail: Exclude event message (#7462) ... **What this PR does / why we need it**: Windows Event Logs have an Event Message field that is intended for human eyes, and often contains data that already present in the event data XML. Omitting this field the same way we can already omit `user_data` and `event_data` can easily save a lot of bytes of data per event - Event ID 4264 alone has ~2KB of just text that is already present in `event_data`. **Which issue(s) this PR fixes**: Fixes #7395 **Special notes for your reviewer**: I also took the liberty to improve upon the existing test '`Test_renderEntries` by using unique values for each field rather than 10's everywhere. I expect this to conflict with my other PR, #7461.	3 years ago
Matthias Kunnen	aa0bec4786	Promtail docs: Fix docker stage code example (#7653) ... The current, 2.6.x, documentation on [configuring promtail _pipeline_stages_](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#pipeline_stages) contains a [docker stage](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#docker) which wraps the json, labels, timestamp, and output stage. The unwrapped configuration is shown as example. However, this example is outdated as can be seen by the lack of `expressions`. I have stumbled across this as I tried the example in my own config as a starter to modify to my own needs. It took me some time to realize what `failed to make file target manager: invalid json stage config: JMES expression is required` meant. This PR updates the example to match the current code at <a6df09bc97/clients/pkg/logentry/stages/extensions.go (L16-L42)> Going back in history points to #576 as the origin of the divergence.	3 years ago
loanshark	3f59fa96d8	lambda-promtail: Add kinesis data stream to use in terraform (#7632) ... **What this PR does / why we need it**: https://github.com/grafana/loki/pull/5977 With the addition of the kinesis data stream function add kinesis data stream to use in terraform	3 years ago
Marin Florin	9517e18bdb	Documentation: Add a more complex template example (#7634) ... **What this PR does / why we need it**: It adds a more complex example for the template stage.	3 years ago
Benoît Knecht	489898f529	promtail/server: Disable profiling by default (#5400) ... **What this PR does / why we need it**: Add a `server.debug` option (that defaults to `false`) to enable the `/debug/fgprof` endpoint in Promtail. **Which issue(s) this PR fixes**: Fixes #5005 Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
Max Vorobev	c5911d5342	Loki: implement decolorize filter (#7602) ... Implement `decolorize` filter that get rids of ANSI color codes in the log message. **Which issue(s) this PR fixes**: Fixes #7601	3 years ago
Dan Cech	fa781e0488	Improve CRI format documentation (#7582) ... The documentation doesn't mention the flags/tags component in a CRI log line. I went with flags since that's the name it's given in our regex.	3 years ago
Pablo	db47e38157	Fix Promtail GCP Logs diagram route (#7577) ... **What this PR does / why we need it**: Some of the latest changes to the GCP Logs Promtial docs include an architecture diagram. It seems the link to this diagram is broken. This PR attempts to fix that by removing a relative routing the image tag path, since the image is deployed in [here](https://grafana.com/docs/loki/next/clients/promtail/gcp-logs-diagram.png), but was being fetched in this path: `https://grafana.com/docs/loki/next/clients/promtail/gcplog-cloud/gcp-logs-diagram.png`. The img is deployed in the same directory as the site, so should use a `../` parent directory relative routing, as in [here](3d6210e424/docs/sources/clients/promtail/troubleshooting.md (L48)). **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Pablo	8b66e1c3fc	Add terraform snippets to Promtail GCP Logs documentation (#7526) ... **What this PR does / why we need it**: This PR is a follow up to the documentation added (https://github.com/grafana/loki/pull/7511) for configuring the infrastructure resources for shipping logs to the new-ish GCP Logs Push Promtail target. Terraform snippets are added for writing the same infrastructure in an as-code manner, which works better for infra folks. **TODO** - [ ] Add snippet on how to setup the pubsub push caveat with terraform (project wide policy) **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: kgeckhart <kgeckhart@users.noreply.github.com>	3 years ago
Pablo	1df72ee209	Add GCP Logs push target Promtail docs (#7511) ... **What this PR does / why we need it**: Some months ago, a new Promtail target was added to support receiving GCP Logs in a push manner. This PR updates the setting up GCP guide, for adding the new required push subscription. Also, adds some notes on details of the setup procedure. This PR will have a follow up expanding on the terraform code needed to setup the GCP resources in a IaaC way. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Dylan Guedes	67de0334e9	Docs: Add max_streams to Promtail configuration doc (#7489) ... **What this PR does / why we need it**: Document `max_streams` configuration, introduced in 2.7. Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>	3 years ago
BenjaminVouillaumeJT	2d649a7fdd	Promtail: Add additional fields for cloudflare (#7463)	3 years ago
Pablo	5f665651d5	Promtail: Add basic tracing support (#7414) ... We've been using Promtail's HTTP/gRPC server-based targets with real workloads for some weeks now. The server that these spin out is based on [weaveworks commons](https://github.com/weaveworks/common) HTTP/gRPC servers. These include by default some middlewares that if enabled, inject tracing support. We found this useful when tracking down request handling time, or even more, if the caller also support Jaeger tracing support, cross-correlated requests between the upstream server and Promtail's. This PR adds support for enabling tracing the same way Loki does. At the moment, this will just generate traces for the before-mentioned targets. In the future, the quality and completeness of tracing support can grow organically to enrich spans.	3 years ago
Pablo	e73584a3cd	Promtail: Add handler timeout for GCP Logs Push target (#7401) ... **What this PR does / why we need it**: After been running the GCP Logs Push target in actual workloads, we've noticed that GCP sends a lot of traffic, and adjusts that send rate according to the rate at which the receiver can process. To perform that adjustment, GCP considers as a NACK [non 2xx HTTP responses](https://cloud.google.com/pubsub/docs/push#receive_push). This PR adds a target handler timeout (that includes as well the `api.Entry` channel send) to allow the user to configure this maximum processing time, therefore giving GCP notice that the sending rate is too high. This new timeout is optional, and disabled by default. **Which issue(s) this PR fixes**: Related to https://github.com/grafana/cloud-onboarding/issues/2067 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
MarNicGit	82c1d1276c	Update installation.md (#7388) ... **What this PR does / why we need it**: This PR attempts to clarify where someone can find the latest binary release of Promtail. I spent a good hour trying to find a download for Promtail before realizing it is included in the assets of each Loki release. Opening the releases page barely shows any mention of Promtail until you fold out the assets list. **Special notes for your reviewer**: This is a documentation change. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	fdbb06d123	add promtail reload changelog and doc (#7386)	3 years ago
Paul Tobias	fe17feeda5	promtail/scraping.md: Fix 404 (#7317) ... **What this PR does / why we need it**: Currently the `Cloudflare` link refers to: https://grafana.com/docs/loki/latest/clients/configuration/#cloudflare But it should be: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#cloudflare **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Hemanth Krishna	0e54477309	area(promtail): add -check-syntax functionality (#7363) ... Adds `-check-syntax` to promtail which will exit just by validating the config file. Fixes #6862. Signed-off-by: Hemanth Krishna <hkpdev008@gmail.com>	3 years ago
Dylan Guedes	f3b72c8b59	Promtail: Change max support compressed line size to be 2MB (#7268) ... **What this PR does / why we need it**: Change Promtail compression support max line size to 2MB instead of 4096 bytes (~4KB). This however doesn't change the default buffer size. 4096 bytes means we can't ever have a buffer size bigger than ~4KB but since the default mem page size is 4096 ([source](https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space)), we end up ignoring all lines.	3 years ago
Dylan Guedes	73bea7e4fa	Promtail: Add compressed files support (#6708) ... **What this PR does / why we need it**: Adds to Promtail the ability to read compressed files. It works by: 1. Infer which compression format to use based on the file extension 2. Uncompress the file with the native `golang/compress` packages 3. Iterate over uncompressed lines and send them to Loki Its usage is the same as our current file tailing. **Which issue(s) this PR fixes**: Fixes #5956 Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
nniehoff	a5c1557d03	docs: Change AWS EKS example eventrouter rbac resources from v1beta1 apis to v1 (#6483) ... RBAC Resource apiVersion `rbac.authorization.k8s.io/v1beta1` was deprecated with Kubernetes 1.22 (see the [deprecation notice](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#rbac-resources-v122)). Now that EKS supports 1.22 the docs need to be updated.	3 years ago
Jeremy Wu	161fcac646	support custom label selector in kubernetes_sd_configs (#5920) ... make promtail scrape logs of pod with specific field or label selector, so that it will watch less logs and make fewer calculations and get a better performance in some scenarios. Signed-off-by: Zhimin Wu <hoozecn@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Pablo	76c1198d58	Improvements to Promtail GCP push target (#7057) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: This PR includes some pending improvements to newish promtail GCP push target: - Pending low hanging optimization for replacing a regex that gets evaluated on every incoming request from GCP, for a simpler solution. [Related comments](https://github.com/grafana/loki/pull/6777#discussion_r935804383) - Some small refactors. See: [this](https://github.com/grafana/loki/pull/6777#pullrequestreview-1059064564) - Add some validations over the payload the target receives. Payload details are [here](https://cloud.google.com/pubsub/docs/push). This PR adds just validation over the log line, the incoming GCP message ID to add some traceability if needed, and the subscription name because it can be used as label. - Add `subscription` GCP payload field as label, which includes the PubSub subscription identifier. - Add `cause` label to error metrics. Since there's multiple failure point this might help debugging / notice strange behaviours. Also, by running the push target in some dev workloads, we noticed it has some labels different to the ones added in the pull target. This PR also refactors the logic to re-use the same GCP log entry parsing logic in both targets. That way the labels exposed to the user in the push target are $Labels_{pull} \cup Labels_{push}$. **Which issue(s) this PR fixes**: Related to: https://github.com/grafana/cloud-onboarding/issues/2067 **Special notes for your reviewer**: This PR is a follow up [this other one](https://github.com/grafana/loki/pull/6777). It works on some comments made after merge, and adds some changes we saw necessary upon running this target in real environments. <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Callum Styan	da6fd01444	[docs] document logfmt pipeline stage (#6238) ... I noticed earlier while writing a design doc that the `logfmt` pipeline stage is not documented. It was originally added here: https://github.com/grafana/loki/pull/4346 I've also fixed a minor typo in the json pipeline stage docs. Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Juha Tiensyrjä	073c62051b	lambda-promtail: Add support for Kinesis data stream events (#5977) ... **What this PR does / why we need it**: This PR adds support for sending Kinesis data stream events to lambda-promtail. One use case would be e.g. to send CloudFront realtime logs to Loki. **Which issue(s) this PR fixes**: Fixes #5978 **Special notes for your reviewer**: n/a **Checklist** - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Alex Close	288e644894	docs: Fixed examples within Loki and Promtail docs (#6591) ... * Updated the promtail install example to both fix the Daemonset yaml file and provide a more concrete example for deploying to kubernetes * Removed the sidecar documentation for promtail install as it was both broken and didn't do what it said it did * Updated the simple scalable chart docs to include an example of using a overrides values file and to point to a working example of deploying promtail in a k8s cluster * Updated promtail and loki SSD install docs based on PR feedback * Update docs/sources/clients/promtail/installation.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/installation/simple-scalable-helm.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/installation/simple-scalable-helm.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/installation/simple-scalable-helm.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/installation/simple-scalable-helm.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
carlospeon	171a09b822	Add support for journal matches (only conjuntions) (#6656)	3 years ago
Karen Miller	8ac08e65d9	Docs: improve wording and grammar (#6861)	3 years ago
Alexandre de Sá	6b47aab2de	promtail/client: add bot fields (#6828) ... * chore: add bot fields * Update CHANGELOG.md * chore: update cloudflare configuration section URI * docs: include BotScore and BotScoreSrc to all set of fields * Update CHANGELOG.md	3 years ago
Pablo	c9ef0621d1	Promtail: Add new target for receiving GCP PubSub Push messages (#6777)	3 years ago
Adam Brodziak	210b9ec461	Fix YAML syntax and ConfigMap reference (#6735)	3 years ago
Pablo	0e28452f1f	Targets: Add Heroku HTTPS drain target (#6448)	3 years ago
Dylan Guedes	f17d3d768c	Promtail: Add text encoding conversion to file targets (#6395) ... Adds support for text encoding conversion for file targets. To use it, add `encoding: <encoding_name>` into the scrapeconfig.	3 years ago
Benjamin P. Jung	56bf3a49a2	Fix documentation of environment variable defaults (#6536) ... For some reason, the `-` character was missing here in the documentation. After struggling for some time, I finally figured out that -- quite obviously -- envsubst default behavior applies and thus the documentation was wrong. :-)	4 years ago
Conor Evans	b65e11ec9b	fix(documentation): correct spelling and scrape_config (#6454) ... Signed-off-by: Conor Evans <coevans@tcd.ie>	4 years ago
Karen Miller	ba7e36afd1	Docs: Edits for several PRs that did not have a docs review (#6398)	4 years ago
Michał Jaroń	57d543eff7	Update _index.md - an unofficial Loki client added (#6312) ... I would like to add mjaron-tinyloki-java to unofficial Loki clients. It is standalone small library without any dependencies. It may be an alternative to other libraries which use many dependencies and require more configuration to send logs.	4 years ago
Jan-Otto Kröpke	a1e0298a57	Support tenant id from value (#6290) ... Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>	4 years ago
Trevor Wood	0470e4e7d3	feat(fluentd): allow fluentd_thread label to be configurable (#6240) ... Signed-off-by: Trevor Wood <Trevor.G.Wood@gmail.com>	4 years ago
Christian Haudum	b2b4fdec71	Update docs for xk6-loki log formats (#6134) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Nicklas	6922190cde	Documentation: Explain --config.expand-env=true double slash with slash substitution (#6139) ... * docs: Promtail configuration explain expand-env slash pitfall * docs: link to correct envsubst go package Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	4 years ago
Callum Styan	b6ac00e1a3	[Promtail] drop lines with malformed json (#6099) ... * Refactor json pipeline stage to have a Run function Signed-off-by: Callum Styan <callumstyan@gmail.com> * Allow for dropping log lines in json pipeline stage if they are not valid json strings Signed-off-by: Callum Styan <callumstyan@gmail.com> * fix linting Signed-off-by: Callum Styan <callumstyan@gmail.com> * Clean up new malformed json test. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Address some review feedback. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Refactor json pipeline stage test to use testify assert instead of t.Errorf Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update changelog and docs with new drop malformed option Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs based on Karen's comments. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Christian Haudum	d5c337c4d3	docs: Fix broken link from unpack parser to pack stage (#6207) ... and make hyperlinks more obvious Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Artem Timchenko	ebb39db864	lambda-promtail: Add multi-tenancy support (#6102) ... * lambda-promtail: adding multi-tenancy support * update changelog and fix import order * fix import order * fix import order	4 years ago
Christian Haudum	7fff660144	Add UDP protocol support to Promtail's syslog target (#5790) ... Until now, the syslog target in Promtail did only support TCP. This PR adds support for sending syslog messages to Promtail (via ng-syslog or rsyslog) via UDP. UPD for syslogs can be used if you prefer speed over guaranteed delivery. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Corentin Altepe	d9327ecd0c	Added NLog-Targets-Loki as unofficial C# client to Loki (#6061) ... Added a link to the HTTP C# client [NLog-Targets-Loki](https://github.com/corentinaltepe/nlog.loki) in Loki's clients documentation in the unofficial clients sections, right next to its Serilog counterpart.	4 years ago
Paschalis Tsilias	179d741c5f	promtail: Add support for exclusion patterns in `static_config` (#5943) ... * PR base Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Add CHANGELOG entry; updated documentation for new label Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Follow logic from t.path in exlusion case; always handle matching Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Make error message unique Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Only run if t.pathExclude is set Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Revert "Only run if t.pathExclude is set" This reverts commit 17abb006ce141071c4bca4b7d133bea2aa314594.	4 years ago