apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
William Dumont	a1c1152b79	Promtail: Add additional fields in cloudflare config (#10301 ) What this PR does / why we need it: In this PR the cloudflare config is enhanced by adding a custom fields type and by having a new property "additional_types" to give the customer the ability to specify the fields that he wants. This functionality has been added in the grafana agent and therefore should also be added in promtail. Which issue(s) this PR fixes: Fixes https://github.com/grafana/agent/issues/4860 Special notes for your reviewer: Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md`	2 years ago
J Stickler	d0545bff2d	[docs] Create top level Send data section, part 2 (#10247 ) Part of the database information architecture Epic # 8710 Which issue(s) this PR fixes: Second half of work for issue # 8741 that was started in PR #10192 When closed, fixes #8741 Move the following files under Send data: ./sources/clients/promtail/_index.md ./sources/clients/promtail/configuration.md ./sources/clients/promtail/gcplog-cloud.md ./sources/clients/promtail/installation.md ./sources/clients/promtail/logrotation/_index.md ./sources/clients/promtail/pipelines.md ./sources/clients/promtail/scraping.md ./sources/clients/promtail/stages/_index.md ./sources/clients/promtail/stages/cri.md ./sources/clients/promtail/stages/decolorize.md ./sources/clients/promtail/stages/docker.md ./sources/clients/promtail/stages/drop.md ./sources/clients/promtail/stages/json.md ./sources/clients/promtail/stages/labelallow.md ./sources/clients/promtail/stages/labeldrop.md ./sources/clients/promtail/stages/labels.md ./sources/clients/promtail/stages/limit.md ./sources/clients/promtail/stages/logfmt.md ./sources/clients/promtail/stages/match.md ./sources/clients/promtail/stages/metrics.md ./sources/clients/promtail/stages/multiline.md ./sources/clients/promtail/stages/output.md ./sources/clients/promtail/stages/pack.md ./sources/clients/promtail/stages/regex.md ./sources/clients/promtail/stages/replace.md ./sources/clients/promtail/stages/static_labels.md ./sources/clients/promtail/stages/template.md ./sources/clients/promtail/stages/tenant.md ./sources/clients/promtail/stages/timestamp.md ./sources/clients/promtail/troubleshooting/_index.md This PR also - Revises the Clients landing page to clarify supported clients. - Updates the metadata (descriptions, weights) - Adds aliases to redirect from the old URLs. - Updates cross-references broken by the move/renaming. - A few other small fixes (headings, typos, etc.) Special notes for your reviewer: Please review the updates to the Clients landing page (now called Send Data) as I've made some extensive edits to try to clarify recommended/supported clients. The file is docs/sources/send-data/_index.md --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
J Stickler	1951eef914	[Docs] Create top level Send data section, part 1 (#10192 ) Part of the database information architecture Epic # 8710 Which issue(s) this PR fixes: First half of work for issue # 8741 Move the following files under Send data: /sources/clients/_index.md ./sources/clients/aws/_index.md ./sources/clients/aws/ec2/_index.md ./sources/clients/aws/ecs/_index.md ./sources/clients/aws/eks/_index.md ./sources/clients/docker-driver/_index.md ./sources/clients/docker-driver/configuration.md ./sources/clients/fluentbit/_index.md ./sources/clients/fluentd/_index.md ./sources/clients/k6/_index.md ./sources/clients/k6/log-generation.md ./sources/clients/k6/query-scenario.md ./sources/clients/k6/write-scenario.md ./sources/clients/lambda-promtail/_index.md ./sources/clients/logstash/_index.md Updates the metadata (descriptions, weights) Adds aliases to redirect from the old URLs. Updates cross-references broken by the move/renaming. A few other small fixes (headings, typos, etc.) NOTE:The promtail files will move in a second PR.	2 years ago
Bruno FERNANDO	a8a349690b	feat(promtail): retrieve BotTags field from cloudflare (#10010 ) What this PR does / why we need it: Hey folks 👋 Little contribution here to add a useful log field for cloudflare users. Indeed I add the [BotTags field](https://developers.cloudflare.com/bots/concepts/cloudflare-bot-tags/) which is pretty useful when debugging some specific traffic handled by cloudflare Don't hesitate to reach me if you have any questions Cheers 😉 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
J Stickler	11e2375e6d	Removing 'please' from the docs. (#9658 ) Which issue(s) this PR fixes: Fixes #8922 Special notes for your reviewer: This is to conform with our Style Guide. https://developers.google.com/style/tone#politeness-and-use-of-please	3 years ago
Jack Baldry	8391e557b1	Update doc-validator to v3.0.0 (#9666 ) ## 3.0.0 ### Added - Structured output for use with [`reviewdog`](https://github.com/reviewdog/reviewdog). You can achieve the original error output by piping the output to the following `jq` expression: `jq -r '"ERROR: \(.location.path):\(.location.range.start.line):\(.location.range.start.column): \(.message)"'`. - Suggestions for simple link fixes. In GitHub, [`reviewdog`](https://github.com/reviewdog/reviewdog) comments these suggestions for convenient replacement. - Support for anchors referring to repeated headings. In the case that a page has multiple headings that share the same text, the renderer appends a zero indexed, numbered suffix to the identifier. For example, when there are two headings that are both "Heading text", the first anchor identifier is `heading-text` and the second is `heading-text-1`. - Error when running `doc-validator` on no files. Notes for reviewer: This also updates the `make-docs` procedure. Please ensure you can run `make docs` from the `docs/` directory before approving and merging this PR. --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	3 years ago
Bruno FERNANDO	1b410dbc01	feat(promtail): add CF ClientRequestSource field (#9669 ) What this PR does / why we need it: Hey folks 👋 Little contribution here to add a useful log field for cloudflare users. Indeed I add the [ClientRequestSource field](https://developers.cloudflare.com/logs/reference/clientrequestsource/ ) which is pretty useful when debugging some specific traffic handled by cloudflare Extra: Since I was on the documentation I fixed an indentation issue that I spotted Don't hesitate to reach me if you have any questions Cheers 😉 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Loki rocks 🚀 Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
BlakeFernandez	905d14bee7	Add Prometheus Link for scrape_configs (#9270 ) What this PR does / why we need it: Adds some context to the scrape configs section in Promtail Config Docs to ensure clarity around the connection to Prometheus Here is a link to a slack thread where the customer was confused on this: https://raintank-corp.slack.com/archives/C04KY6NJSCE/p1681761335754199 --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Alexandre de Verteuil	e38ccb4418	Uniformize "Available Labels" headers in Promtail config reference doc (#9436 ) What this PR does / why we need it: There were various formats for the "Available Labels" headings. This PR makes them all the same heading level and spelling.	3 years ago
J Stickler	1bb489c438	TOC Sync: Move LogQL and LogCLI files under Query. (#9200 )	3 years ago
J Stickler	8ee9a2c3c7	TOC Sync: Move API docs under Reference. (#9203 ) Which issue(s) this PR fixes: Fixes #8719 Special notes for your reviewer: Part of the Database products table of contents alignment project. Moves API doc under Reference heading in TOC. Updates front matter for the topic. Fixes links broken by the move. Adds aliases for redirects.	3 years ago
Labesse Kévin	0215a17148	gcplog: improve formatter (#9044 ) What this PR does / why we need it: Add two improvement to the target `gcplog` * capability to ignore the `textPayload` from the log line -> don't trunk important metadata * add new source labels (severity and root labels of log entry) -> more relabel_config possibilities Special notes for your reviewer: I'm not sure about the naming of the option `useFullLine`, feel free to suggest any name, but I guess this feature must be disable by default for compatibility? Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Kevin Labesse <kevin@labesse.me> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
nikuljain	b5946df30e	Update configuration.md (#9085 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Mitsuru Kariya	926a4e148e	Docs: add missing configuration option (#9092 ) What this PR does / why we need it: Promtail can set the tenant id from a label with PR 6290 (commit `a1e0298a5`). However, while docs/sources/clients/promtail/stages/tenant.md was appropriately updated, docs/sources/clients/promtail/configuration.md was left untouched. Which issue(s) this PR fixes: Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
René Scheibe	e050d5c494	Docs: Update Loki GitHub URLs (#9080 ) * use `main` instead of `master` branch * fix outdated URLs	3 years ago
Robert Fratto	7bec727c6d	promtail: support configurable polling interval (#8874 ) Currently, Promtail always polls files every 250ms. This PR allows the polling period to be customized, starting at a minimum polling period and exponentially increasing the polling frequency to a bounded maximum if the file has no changes. When file changes are detected, the polling frequency resets to the minimum. The default behavior is the existing behavior; both the minimum and maximum polling intervals are set to 250ms. I'm opening this in draft to get early feedback. There's a few open questions: * Where can I add validations to ensure that the min/max polling intervals are configured properly? * Is this the right approach for implementation? Do I need to make any changes? * Does this need tests? I'll update the CHANGELOG and Documentation after we're agreed on an approach.	3 years ago
Dylan Guedes	6dfcd59c1b	Promtail: Add `decompression` configuration (#8994 ) What this PR does / why we need it: Adds a new `compression` configuration to Promtail to customize decompression behavior. As of now, two options are supported: `enabled`: false by default but if truthy, Promtail will infer a compression algorithm to use based on the file extension `initialDelay`: 0 by default but if higher, Decompressor will sleep for the given duration before trying to decompress. This is to avoid scenarios where the Decompressor starts to work on a file that is still being compressed/generated. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/8784	3 years ago
Erik Sjölund	2cccc1bc68	Docs: Fix typos and grammar (#8966 ) What this PR does / why we need it: Fixes typos and grammar. Most of them were occurrences of `it's` that should be `its`. Which issue(s) this PR fixes: None Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>	3 years ago
Andrii Kushch	17e05d28e4	Promtail: Add a new target for the Azure Event Hub (#8787 ) What this PR does / why we need it: We want to allow receiving logs from the Azure Cloud. To solve this problem, we need to be able to configure Promtail to consume logs from the Azure Events Hub. To achieve this goal, we need to add a new configuration option to the scrape configuration in Promtail and implement the component that, given this configuration, can connect to the Azure Event Hub, parse received data, and prepare it to be stored in Loki. Which issue(s) this PR fixes: Fix [#8788](https://github.com/grafana/loki/issues/8788) # Special notes for your reviewer: With the current implementation, I would like to leverage Azure Events Hub's compatibility with a Kafka protocol and reuse code for a Kafka target. ### Why I decided to create a new target instead of just reusing the Kafka target: 1. Configuration for Kafka in this scenario requires fewer parameters. For example, version and auth should have some specific values. 2. Message handling is different for Azure Logs. In addition to what we are doing for the Kafka target, we want to fix JSON and split incoming messages into multiple log lines before propagating it further. ### From the implementation perspective, there are two goals: - Extract interfaces that are not Kafka configuration specific from the Kafka target module. Splitting Kafka config parsing and target creation. - To make the message handler/parser injectable or configurable: we would like to fix JSON and split a message into multiple messages. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Pablo <2617411+thepalbi@users.noreply.github.com> Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Langdi	9b5bbf3fb0	Fix word confusions (#8778 ) Fixes two word confusion in the docs: * in multiple occasions `\` is referred to as slash instead of backslash * "most be convertible [...]" probably should be "must be convertible [...]"	3 years ago
Yasmine El wahbi	c6542e61fa	Promtail: Add ready endoint to loki-push-api server (#8552 ) When trying to expose loki_push_api service (in our case using GKE ingress), we need a health check target endpoint that would return a 200 response code. This PR contains a simple addition for a HTTP handler that handles the /ready path, along with the necessary tests. Fixes #8551 Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Jack King	d434e80877	Promtail: Add custom headers on push requests (#7470 ) (#7472 )	3 years ago
nicoche	80ea621ff3	promtail: Add `max-line-size-truncate` (#8233 )	3 years ago
Danny Kopping	d0fd4e8ad1	Do not exit when using `stream_lag_labels` setting (#8243 )	3 years ago
Travis Patterson	1c954166b8	Fix documentation linter errors (#8229 )	3 years ago
Kaviraj Kanagaraj	86461b3faa	promtail: Add doc for `max-line-size` flag. (#8202 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> What this PR does / why we need it: Follow up from #8153 Changes: * Change type of the flag from `int` to `flagext.ByteSize` (to be consistent with similar flag in `distributor`) * Add doc to promtail config page * Fix changelog entries Which issue(s) this PR fixes: Fixes #NA Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] `CHANGELOG.md` updated Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Chris Hodges	eb6ba615f4	Promtail: Add configuration to drop batches when rate limited by Loki (#7973 ) This change will allow for more deterministic behavior by multitenant Promtail instances where one or more tenants are being heavily throttled due to limits exceeded within Loki. Specifically, it will reduce the performance impact of having exponential backoff cause HOL blocking for rate limit response codes while other tenants are ready to send batches. Co-authored-by: chodges15 <chris.hodges@gmail.org>	3 years ago
Kim Nylander	4d97dc0cbb	[DOC] Fix broken links in docs (#8065 ) What this PR does / why we need it: This PR fixes several broken links in the Promtail, operations, and install sections of the doc.	3 years ago
Chris James	527ee12220	Fixing a typo in configuration.md (#7878 ) What this PR does / why we need it: Fixes a typo in configuration.md, improves reader clarify and confidence Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
cadrake	ea9ad336dc	Loki: Modified heroku drain target to make any url query parameters available as labels (#7619 ) What this PR does / why we need it: This PR reads the url query parameters included in Heroku drain POST request and creates `__param_<name>` labels similar to how the parameters work with prometheus metrics and metric endpoints as a way to allow setting the true application name when ingesting logs from a Heroku drain. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: The existing `app` field coming from Heroku only specifies application source (either the application itself or Heroku) and thus is insufficient for uniquely identifying a log lines source. Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
MarNicGit	1f1dd81555	Promtail: Exclude event message (#7462 ) What this PR does / why we need it: Windows Event Logs have an Event Message field that is intended for human eyes, and often contains data that already present in the event data XML. Omitting this field the same way we can already omit `user_data` and `event_data` can easily save a lot of bytes of data per event - Event ID 4264 alone has ~2KB of just text that is already present in `event_data`. Which issue(s) this PR fixes: Fixes #7395 Special notes for your reviewer: I also took the liberty to improve upon the existing test '`Test_renderEntries` by using unique values for each field rather than 10's everywhere. I expect this to conflict with my other PR, #7461.	3 years ago
Matthias Kunnen	aa0bec4786	Promtail docs: Fix docker stage code example (#7653 ) The current, 2.6.x, documentation on [configuring promtail _pipeline_stages_](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#pipeline_stages) contains a [docker stage](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#docker) which wraps the json, labels, timestamp, and output stage. The unwrapped configuration is shown as example. However, this example is outdated as can be seen by the lack of `expressions`. I have stumbled across this as I tried the example in my own config as a starter to modify to my own needs. It took me some time to realize what `failed to make file target manager: invalid json stage config: JMES expression is required` meant. This PR updates the example to match the current code at <`a6df09bc97/clients/pkg/logentry/stages/extensions.go (L16-L42)`> Going back in history points to #576 as the origin of the divergence.	3 years ago
Benoît Knecht	489898f529	promtail/server: Disable profiling by default (#5400 ) What this PR does / why we need it: Add a `server.debug` option (that defaults to `false`) to enable the `/debug/fgprof` endpoint in Promtail. Which issue(s) this PR fixes: Fixes #5005 Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
Dylan Guedes	67de0334e9	Docs: Add max_streams to Promtail configuration doc (#7489 ) What this PR does / why we need it: Document `max_streams` configuration, introduced in 2.7. Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>	3 years ago
BenjaminVouillaumeJT	2d649a7fdd	Promtail: Add additional fields for cloudflare (#7463 )	3 years ago
Pablo	5f665651d5	Promtail: Add basic tracing support (#7414 ) We've been using Promtail's HTTP/gRPC server-based targets with real workloads for some weeks now. The server that these spin out is based on [weaveworks commons](https://github.com/weaveworks/common) HTTP/gRPC servers. These include by default some middlewares that if enabled, inject tracing support. We found this useful when tracking down request handling time, or even more, if the caller also support Jaeger tracing support, cross-correlated requests between the upstream server and Promtail's. This PR adds support for enabling tracing the same way Loki does. At the moment, this will just generate traces for the before-mentioned targets. In the future, the quality and completeness of tracing support can grow organically to enrich spans.	3 years ago
Pablo	e73584a3cd	Promtail: Add handler timeout for GCP Logs Push target (#7401 ) What this PR does / why we need it: After been running the GCP Logs Push target in actual workloads, we've noticed that GCP sends a lot of traffic, and adjusts that send rate according to the rate at which the receiver can process. To perform that adjustment, GCP considers as a NACK [non 2xx HTTP responses](https://cloud.google.com/pubsub/docs/push#receive_push). This PR adds a target handler timeout (that includes as well the `api.Entry` channel send) to allow the user to configure this maximum processing time, therefore giving GCP notice that the sending rate is too high. This new timeout is optional, and disabled by default. Which issue(s) this PR fixes: Related to https://github.com/grafana/cloud-onboarding/issues/2067 Special notes for your reviewer: Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	fdbb06d123	add promtail reload changelog and doc (#7386 )	3 years ago
Jeremy Wu	161fcac646	support custom label selector in kubernetes_sd_configs (#5920 ) make promtail scrape logs of pod with specific field or label selector, so that it will watch less logs and make fewer calculations and get a better performance in some scenarios. Signed-off-by: Zhimin Wu <hoozecn@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Pablo	76c1198d58	Improvements to Promtail GCP push target (#7057 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: This PR includes some pending improvements to newish promtail GCP push target: - Pending low hanging optimization for replacing a regex that gets evaluated on every incoming request from GCP, for a simpler solution. [Related comments](https://github.com/grafana/loki/pull/6777#discussion_r935804383) - Some small refactors. See: [this](https://github.com/grafana/loki/pull/6777#pullrequestreview-1059064564) - Add some validations over the payload the target receives. Payload details are [here](https://cloud.google.com/pubsub/docs/push). This PR adds just validation over the log line, the incoming GCP message ID to add some traceability if needed, and the subscription name because it can be used as label. - Add `subscription` GCP payload field as label, which includes the PubSub subscription identifier. - Add `cause` label to error metrics. Since there's multiple failure point this might help debugging / notice strange behaviours. Also, by running the push target in some dev workloads, we noticed it has some labels different to the ones added in the pull target. This PR also refactors the logic to re-use the same GCP log entry parsing logic in both targets. That way the labels exposed to the user in the push target are $Labels_{pull} \cup Labels_{push}$. Which issue(s) this PR fixes: Related to: https://github.com/grafana/cloud-onboarding/issues/2067 Special notes for your reviewer: This PR is a follow up [this other one](https://github.com/grafana/loki/pull/6777). It works on some comments made after merge, and adds some changes we saw necessary upon running this target in real environments. <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Alexandre de Sá	6b47aab2de	promtail/client: add bot fields (#6828 ) * chore: add bot fields * Update CHANGELOG.md * chore: update cloudflare configuration section URI * docs: include BotScore and BotScoreSrc to all set of fields * Update CHANGELOG.md	3 years ago
Pablo	c9ef0621d1	Promtail: Add new target for receiving GCP PubSub Push messages (#6777 )	3 years ago
Pablo	0e28452f1f	Targets: Add Heroku HTTPS drain target (#6448 )	4 years ago
Dylan Guedes	f17d3d768c	Promtail: Add text encoding conversion to file targets (#6395 ) Adds support for text encoding conversion for file targets. To use it, add `encoding: <encoding_name>` into the scrapeconfig.	4 years ago
Benjamin P. Jung	56bf3a49a2	Fix documentation of environment variable defaults (#6536 ) For some reason, the `-` character was missing here in the documentation. After struggling for some time, I finally figured out that -- quite obviously -- envsubst default behavior applies and thus the documentation was wrong. :-)	4 years ago
Karen Miller	ba7e36afd1	Docs: Edits for several PRs that did not have a docs review (#6398 )	4 years ago
Nicklas	6922190cde	Documentation: Explain --config.expand-env=true double slash with slash substitution (#6139 ) * docs: Promtail configuration explain expand-env slash pitfall * docs: link to correct envsubst go package Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	4 years ago
Paschalis Tsilias	179d741c5f	promtail: Add support for exclusion patterns in `static_config` (#5943 ) * PR base Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Add CHANGELOG entry; updated documentation for new label Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Follow logic from t.path in exlusion case; always handle matching Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Make error message unique Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Only run if t.pathExclude is set Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> * Revert "Only run if t.pathExclude is set" This reverts commit 17abb006ce141071c4bca4b7d133bea2aa314594.	4 years ago
Callum Styan	800b7c6345	[promtail] Mark promtail single client config flags and config file block as deprecated (#5842 ) * Mark promtail single client config flags and config file block as deprecated. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update some promtail docs examples that still use `client` instead of `clients`. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix imports. Signed-off-by: Callum Styan <callumstyan@gmail.com> * properly format url as an array item since we changed to clients from client Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Karen Miller	c860fc6442	Docs: Document Promtail global rate limiting (#5737 ) * Docs: Document Promtail global rate limiting * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Jennifer Villa <jvilla2013@gmail.com> * Revise strings that explain the new configuration parameters * For rate limiting, limit_config becomes limits_config Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>	4 years ago

2 Commits (ca19e8d7941bfac86355b2853ef2ea110f32075e)