693 Commits (daad5bb8129c088d57e6d70bf9e44c5874150f45)

Author	SHA1	Message	Date
Ursula Kallio	06bcecfebe	Remove CTA per @grafanawriter. (#7557)	3 years ago
Pablo	8b66e1c3fc	Add terraform snippets to Promtail GCP Logs documentation (#7526) ... **What this PR does / why we need it**: This PR is a follow up to the documentation added (https://github.com/grafana/loki/pull/7511) for configuring the infrastructure resources for shipping logs to the new-ish GCP Logs Push Promtail target. Terraform snippets are added for writing the same infrastructure in an as-code manner, which works better for infra folks. **TODO** - [ ] Add snippet on how to setup the pubsub push caveat with terraform (project wide policy) **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: kgeckhart <kgeckhart@users.noreply.github.com>	3 years ago
Ursula Kallio	9cc0f3c3a5	Remove CTA per @grafanawriter. (#7558)	3 years ago
Karsten Jeschkies	b16ad8cb38	Document new Helm Charts (#7034) ... **What this PR does / why we need it**: This change describes various Helm related tasks such as installing Loki and configuring meta monitoring. It also explains different concepts related to the new Helm chart and provides a reference of the new Helm values file. **Which issue(s) this PR fixes**: Fixes #7035	3 years ago
Pablo	1df72ee209	Add GCP Logs push target Promtail docs (#7511) ... **What this PR does / why we need it**: Some months ago, a new Promtail target was added to support receiving GCP Logs in a push manner. This PR updates the setting up GCP guide, for adding the new required push subscription. Also, adds some notes on details of the setup procedure. This PR will have a follow up expanding on the terraform code needed to setup the GCP resources in a IaaC way. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Trevor Whitney	f83988a061	Add 2.7.0 section to upgrading doc (#7457) ... Add 2.7.0 section to Upgrading doc. Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>	3 years ago
Michel Hollands	fd93a6c999	Add how to connect to Azure using Service Principal (#7485) ... Add documentation for https://github.com/grafana/loki/pull/7179 which added authentication using a Service Principal for Azure.	3 years ago
Travis Patterson	b4ed595461	Remove groupcache (#7126) ... As discussed during the caching retro, this PR removes the experimental distributed cache based on Mailgun's fork of groupcache. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	67de0334e9	Docs: Add max_streams to Promtail configuration doc (#7489) ... **What this PR does / why we need it**: Document `max_streams` configuration, introduced in 2.7. Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>	3 years ago
Mohamed-Amine Bouqsimi	7debe7c235	Ruler: enable Loki to configure Alertmanagers per tenant (#7359)	3 years ago
Ursula Kallio	7fa3e500c3	Change `master` to `main` and include `.md` ext.	3 years ago
BenjaminVouillaumeJT	2d649a7fdd	Promtail: Add additional fields for cloudflare (#7463)	3 years ago
Pablo	5f665651d5	Promtail: Add basic tracing support (#7414) ... We've been using Promtail's HTTP/gRPC server-based targets with real workloads for some weeks now. The server that these spin out is based on [weaveworks commons](https://github.com/weaveworks/common) HTTP/gRPC servers. These include by default some middlewares that if enabled, inject tracing support. We found this useful when tracking down request handling time, or even more, if the caller also support Jaeger tracing support, cross-correlated requests between the upstream server and Promtail's. This PR adds support for enabling tracing the same way Loki does. At the moment, this will just generate traces for the before-mentioned targets. In the future, the quality and completeness of tracing support can grow organically to enrich spans.	3 years ago
Michel Hollands	109b9dce9a	Add remote schema option (#7442) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Danny Kopping	c6b2294bbf	Add Li Guozhong as a maintainer (#7440) ... A formal vote has taken place and `liguozhong` has been accepted! 🎉	3 years ago
Pablo	e73584a3cd	Promtail: Add handler timeout for GCP Logs Push target (#7401) ... **What this PR does / why we need it**: After been running the GCP Logs Push target in actual workloads, we've noticed that GCP sends a lot of traffic, and adjusts that send rate according to the rate at which the receiver can process. To perform that adjustment, GCP considers as a NACK [non 2xx HTTP responses](https://cloud.google.com/pubsub/docs/push#receive_push). This PR adds a target handler timeout (that includes as well the `api.Entry` channel send) to allow the user to configure this maximum processing time, therefore giving GCP notice that the sending rate is too high. This new timeout is optional, and disabled by default. **Which issue(s) this PR fixes**: Related to https://github.com/grafana/cloud-onboarding/issues/2067 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Abder	cc9ba40be0	promtail: Fix exposed metric documentation for request_duration_seconds (#7420) ... **What this PR does / why we need it**: Fixing the documentation about exposed promtail metrics for promtail_request_duration_seconds **Which issue(s) this PR fixes**: No issue related **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Karsten Jeschkies	85e524788e	Provide example policy for S3 storage. (#7407) ... **What this PR does / why we need it**: This should make it easier for users to set the proper policies for their S3 storage. **Which issue(s) this PR fixes**: Relates to #7403 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Joel Verezhak	46d3dec010	[loki-canary] Allow insecure TLS connections (#7398) ... **What this PR does / why we need it**: This change allows client certificates signed by a self-signed certificate authority to be used by the Loki canary. **Which issue(s) this PR fixes**: Fixes #4366 **Special notes for your reviewer**: This has been tested on linux amd64 with self-signed certificates. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Vladyslav Diachenko	57c0c2cde2	[docs: logql-analyzer] changed url for main branch (#7378) ... pointed unreleased docs(`next`) to logql-analyzer that is built from main branch	3 years ago
MarNicGit	82c1d1276c	Update installation.md (#7388) ... **What this PR does / why we need it**: This PR attempts to clarify where someone can find the latest binary release of Promtail. I spent a good hour trying to find a download for Promtail before realizing it is included in the assets of each Loki release. Opening the releases page barely shows any mention of Promtail until you fold out the assets list. **Special notes for your reviewer**: This is a documentation change. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Dennis Szczepanski	4e61c60296	Loki: Renames metric 'loki_log_messages_total' to 'loki_internal_log_messages_total' (#7361) ... **What this PR does / why we need it**: Renames the metric 'loki_log_messages_total' to a more descriptive name. Keeps and deprecates the previously existing metric to not break functionality. **Which issue(s) this PR fixes**: Fixes #7088 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	fdbb06d123	add promtail reload changelog and doc (#7386)	3 years ago
Brandon W Maister	381735d3d4	logging: Change Docker target discovery log level from Error to Info (#7312) ... **What this PR does / why we need it**: Discovering a new node is not an error, so having it send errors to dashboards is confusing. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Paul Tobias	fe17feeda5	promtail/scraping.md: Fix 404 (#7317) ... **What this PR does / why we need it**: Currently the `Cloudflare` link refers to: https://grafana.com/docs/loki/latest/clients/configuration/#cloudflare But it should be: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#cloudflare **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Hemanth Krishna	0e54477309	area(promtail): add -check-syntax functionality (#7363) ... Adds `-check-syntax` to promtail which will exit just by validating the config file. Fixes #6862. Signed-off-by: Hemanth Krishna <hkpdev008@gmail.com>	3 years ago
ebCrypto	a7c7f075e2	documentation: correct yaml file (#7375) ... **What this PR does / why we need it**: correct indentation of `relabel_configs` to make it valid and consistent with https://grafana.com/docs/loki/latest/clients/promtail/scraping/#syslog-receiver **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Travis Patterson	7979cfbe07	Implement Distributor Rate Store (#7349) ... The `RateStore` concurrently queries all ingester `GetStreamRate` APIs and stores the results. `RateStore` deduplicates replicas and combines sharded streams so a caller of `RateFor` gets an unsharded view of a stream's rate.	3 years ago
Dylan Guedes	b1d4efab12	Loki: Per-tenant stream sharding (#7311) ... **What this PR does / why we need it**: - Move stream sharding configuration to its own package to avoid cyclic imports - Change stream sharding to be a per-tenant configuration - Change ingesters to reject whole streams due to rate-limit based on per-tenant stream sharding - Change stream sharding flags prefix from `distributor.shard-stream` to `shard-stream`	3 years ago
Mostafa Moradian	f07e562a9d	Clarify how and where to download the Loki config file from (#7346) ... **What this PR does / why we need it**: The config file URL mentioned in the docs contain config options that doesn't match the latest version of Loki, which, as of now, is v2.6.1. **Which issue(s) this PR fixes**: Fixes #6915 **Special notes for your reviewer**: Disclaimer: I am a Grafanista from k6. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
David Birks	5465aaf3e0	docs: Document setting up command completion for logcli (#7325) ... This documents the existing command completion that kingpin provides (docs [here](https://github.com/alecthomas/kingpin#bashzsh-shell-completion) on that). **Which issue(s) this PR fixes**: Fixes #2949	3 years ago
Danny Kopping	c2ccb25390	Updating loki-build-image to Go 1.19 (#7334) ... **What this PR does / why we need it**: Updated the `loki-build-image` to Go 1.19.2 as a precursor for https://github.com/grafana/loki/pull/7243. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
wilfriedroset	8886800bf6	Add support for username to redis cache configuration (#7270) ... **What this PR does / why we need it**: Add support for username to redis cache configuration **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>	3 years ago
Christian Haudum	c34078719c	[docs] Fix docs for `rate_counter()` ... `rate_counter` behaves like `rate` and therefore does not support grouping. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Quentin DUPUY	c5ec1ac27f	fix/docs: Fixed a typo in the ingestor section (#7285)	3 years ago
Gerard Vanloo	bcdf17b0a5	Configurable TLS MinVersion and CipherSuites (#7227)	3 years ago
Dylan Guedes	f3b72c8b59	Promtail: Change max support compressed line size to be 2MB (#7268) ... **What this PR does / why we need it**: Change Promtail compression support max line size to 2MB instead of 4096 bytes (~4KB). This however doesn't change the default buffer size. 4096 bytes means we can't ever have a buffer size bigger than ~4KB but since the default mem page size is 4096 ([source](https://unix.stackexchange.com/questions/128213/how-is-page-size-determined-in-virtual-address-space)), we end up ignoring all lines.	3 years ago
Dylan Guedes	73bea7e4fa	Promtail: Add compressed files support (#6708) ... **What this PR does / why we need it**: Adds to Promtail the ability to read compressed files. It works by: 1. Infer which compression format to use based on the file extension 2. Uncompress the file with the native `golang/compress` packages 3. Iterate over uncompressed lines and send them to Loki Its usage is the same as our current file tailing. **Which issue(s) this PR fixes**: Fixes #5956 Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Dylan Guedes	dcfba366de	Loki: Shard streams based on desired rate (#7199) ... **What this PR does / why we need it**: Modify our sharding mechanism to be dictated by the desired rate and ingestion rate. Once the number of shards is calculated, the stream entries will be distributed across the shards. If the number of entries is lower than number of shards, we report a metric and use number of entries as the number of shards.	3 years ago
tonistuulNG	277d8261eb	Documentation: Fix broken link (#7136) ... There is currently a broken link leading to 404 in Loki documentation.	3 years ago
Yahya	6bb73ef97e	Docs: remove redundant section in Template functions (#6458) ... Pipelines are already defined in L18-19	3 years ago
Garrett Guillotte	7eadddbcfc	Docs: Populate empty index pages (#7200) ... Three docs index pages are dead ends - they are either blank or have no links. In lieu of more detailed content, this PR populates these pages with `{{< section >}}` shortcodes that automatically generate TOC-like links to subpages.	3 years ago
Danny Kopping	72aec6a38b	Document validation & rate-limit errors (#7145) ... This PR adds documentation for the various validation & rate-limit errors that can occur when sending logs to Loki.	3 years ago
nniehoff	a5c1557d03	docs: Change AWS EKS example eventrouter rbac resources from v1beta1 apis to v1 (#6483) ... RBAC Resource apiVersion `rbac.authorization.k8s.io/v1beta1` was deprecated with Kubernetes 1.22 (see the [deprecation notice](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#rbac-resources-v122)). Now that EKS supports 1.22 the docs need to be updated.	3 years ago
Jeremy Wu	161fcac646	support custom label selector in kubernetes_sd_configs (#5920) ... make promtail scrape logs of pod with specific field or label selector, so that it will watch less logs and make fewer calculations and get a better performance in some scenarios. Signed-off-by: Zhimin Wu <hoozecn@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	3 years ago
Salva Corts	df4461d712	Evenly spread memcached across nodes (#7183) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: This PR enables the [TopologySpreadConstraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) for Memcached chunks cache introduced at https://github.com/grafana/jsonnet-libs/pull/832. Now the chunks cache will be evenly spread across nodes but more than one cache can run on the same node. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Brian Evans	e7acb5356b	Docs: Fix template function examples (#7148) ... Some examples only had a single open brace, but require two. I added the missing braces. **What this PR does / why we need it**: Improves documentation, helps users who are reading docs about template functions.	3 years ago
Travis Patterson	ef47e0ee8a	Fix errors loading deletes after force-removing partially-completed delete (#7125) ... When sharded deletes that are partially completed are force-removed, everything is removed except the shards that are currently being deleted. The shards being deleted aren't necessarily in numerical order so there's no way to know assume which shards have and haven't been deleted. The Delete store assumed it could look up requests in sequential order and returned empty requests where there were gaps in the stored data. This change takes the `sequence number` from the stored key of the delete requests it finds and uses that to look up details.	3 years ago
Michel Hollands	ebdb64b3f5	Small updates to the deletion documentation (#7112) ... **What this PR does / why we need it**: - Fix a typo - Fix the links in for the deletion API **Checklist** - [X] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Kaviraj Kanagaraj	2c9fa05c29	feat(loki-canary): Add support to push logs directly to Loki. (#7063) ... Add additional `push` mode to loki-canary, which pushes the logs directly to given Loki URL as it generates logs. The real function of Loki Canary is to act like a tenant and help us know the whether Loki installation is working as perceived by a real tenant. Main rationale for this additional push mode is to make canary more standalone without needing for `promtail` (or `grafana-agent`) to scrape it's logs and send to loki, with this change, Loki canary happily tests Loki behavior without needing any other dependencies. **NOTES**: 1. If you run Loki behind any proxy that has different authorization policies to READ or WRITE to Loki, then important change that canary operator need to be aware of it, now the user credentials they pass via `-user` and `-pass` to access loki endpoints need to have both `read' and `write` permissions (previously canary just query the logs where as promtail is the one pushes the logs, so just READ permissions was sufficient). 2. There will be follow up PR(s) to cleanup, particularly `reader` and `comparitor` component in terms of logging with proper logger. Rationale is to keep the changes small per PR to make it easy to review. 3. This changes were tested it in one of the internal Loki dev cell. 4. **This PR is a no-op if this new `push` flag is disabled (it's disabled by default)** Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago