4138 Commits (ceb09efdf8aab7ebddbdb285b6f5e7d8fac2c951)

 

All Branches   

Search

Author	SHA1	Message	Date
Kaviraj Kanagaraj	ceb09efdf8	fix(loki-canary): Send to Loki after updating `totalEntries`. (#7211) ... **What this PR does / why we need it**: Fixes: #7142 (take a look for more details about the problem there) After trying few things to avoid this race and discussin it with Ed, This small changes seems to be more accurate and seems to fix the issue. This is unblocking rolling out new canary in our internal cells Update: For some reason I thought the fix didn't fix the original issue when I tested it with internal loki dev cell, almost a month ago. But after testing it for more than 2hr on same cell. I realized it is working fine (will leave it for a day like this just to confirm) Verified following: ``` {.... container="loki-canary"} |= "websocket missing" ``` and noticed it's happening no longer after the fix. Also verified the metrics ``` sum(increase(loki_canary_websocket_missing_entries_total{...}[$__range])) ``` It dropped to zero.   Update 2: After running it for whole day, looks there was two log entries missed on the websocket. My guess is since websocket is long-live connection and something can interrupt the connection loosing this message.  I also quickly checked how often this happens on other bigger envs (say ops) and looks like it's not that uncommon.  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/7142 **Special notes for your reviewer**: Tested it on internal loki dev cells **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	4 years ago
dmorozov001	2881c52da4	Fixing a typo in CONTRIBUTING.md (#7547) ... **What this PR does / why we need it**: fixing minor typos in Contributing.md . This would make documents cleaner and easy to read **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	4 years ago
Ursula Kallio	06bcecfebe	Remove CTA per @grafanawriter. (#7557)	4 years ago
Owen Diehl	fd3a6dfaf1	Adds extra docs to flehs out ParseLabels implementation (#7562) ... Adding some extra comments for archaeological purposes	4 years ago
Pablo	8b66e1c3fc	Add terraform snippets to Promtail GCP Logs documentation (#7526) ... **What this PR does / why we need it**: This PR is a follow up to the documentation added (https://github.com/grafana/loki/pull/7511) for configuring the infrastructure resources for shipping logs to the new-ish GCP Logs Push Promtail target. Terraform snippets are added for writing the same infrastructure in an as-code manner, which works better for infra folks. **TODO** - [ ] Add snippet on how to setup the pubsub push caveat with terraform (project wide policy) **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: kgeckhart <kgeckhart@users.noreply.github.com>	4 years ago
Ursula Kallio	9cc0f3c3a5	Remove CTA per @grafanawriter. (#7558)	4 years ago
Travis Patterson	a65da7d78e	Shard streams round-robin rather in batches. (#7560) ... To avoid hot spots, keep track of shard numbers across pushes. When tracking shard numbers across pushes, it's no longer necessary to randomize shards, in the case where the desired number of shards is greater than the number of entries. Instead, we just partition the entries among the shards in round robin fashion. Over time, this should give us a more even distribution of logs in shards, particularly for large streams that are made of small individual pushes.	4 years ago
Michel Hollands	16761723f4	Add way to override userId for caching (#7542) ... **What this PR does / why we need it**: Add a way to change the userId used in caching. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Owen Diehl	6b4563cd53	Prevent IndexShipper from leaking mutex accesses. (#7548) ... # What Inverts control in the `index-shipper` so consumers don't manage accesses to mutexes. This prevents deadlocks and no longer requires consumers have internal knowledge of the `index-shipper` itself. This PR reverts a few previous attempts. Look at the last commit to see the net-additions. # Background: TSDB previously didn't use mmap, which meant we could freely pass around TSDBs as `[]byte`. Since they were immutable, we didn't need to know when an underlying index was cleaned up/closed; this coordination was unnecessary and the garbage collector would clean up dangling memory for us. After introducing TSDB, we needed to control access to avoid use-after-free bugs. A few attempts at controlling this became increasingly complex & error prone. Instead of requiring _consumers_ to prevent deadlocks on index-shipper internals, it makes more sense that references to the index-shipper's resources should not leak outside the `ForEach` functions that give temporary access to them. This _inverts control_ from the consumer (code which queries the indices) to the producer (the index-shipper). Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	4 years ago
Robert Fratto	3808fb6ac3	promtail: register reload metrics after running options (#7561) ... **What this PR does / why we need it**: This allows the reload metrics to be sent to a custom registerer, which is required for Grafana Agent to be able to use the Promtail code. **Which issue(s) this PR fixes**: N/A **Special notes for your reviewer**: Is it possible for us to cherry-pick this into k121? That's the version we're trying to update Grafana Agent to. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	4 years ago
Travis Patterson	0a654e2357	Ensure labels on sharded streams are sorted before they're sent anywhere (#7556) ... Pushing streams with unsorted labels to ingesters results in a bug where ingesters duplicate chunks for streams. This causes a memory leak and eventual OOM crash when ingesters recover from the WAL because one of the duplicate streams is orphaned. This PR ensures that sharded streams have sorted labels to avoid this bug.	4 years ago
Trevor Whitney	b523ff336e	[Helm] Fix invalid ruler config when filesystem storage is used (#7528) ... **What this PR does / why we need it**: When using filesystem storage, there's a bug in the configmap helm template that sets `ruler.storage` to `null`. This fixes that bug. **Which issue(s) this PR fixes**: Fixes #7267	4 years ago
Callum Styan	46b40a5582	Hardcode ring.WriteNoExtend for distributor push operations (#7517) ... With zone awareness enabled we allow restarting multiple ingesters within a single zone at the same time, this means that for many more write requests there's potentially only 2 of the 3 replicas for a stream up at the time the write request is received. This lead to a large increase in 500s during rollouts. My suspicion is that the 500s on rollouts was always related to the following; the replica set ring code has something referred to as "extending the replica set". This essentially is the adding of ingesters in the ring that are not in an ACTIVE state (for example, are in LEAVING) to the set of replicas considered valid for an operation. Adding of the instance in a state other than ACTIVE can be controlled by using a different operation type, which Mimir was using but we were not. Here are the relevant functions, [ring.Get](8d6d914ef6/ring/ring.go (L335-L403)) (particularly these lines [1](8d6d914ef6/ring/ring.go (L381-L389)) and [2](8d6d914ef6/ring/ring.go (L394-L397))) and [replicationStrategy.Filter](789ec0ca4a/ring/replication_strategy.go (L22-L71)) (particularly these lines [1](789ec0ca4a/ring/replication_strategy.go (L32-L36)) and [2](789ec0ca4a/ring/replication_strategy.go (L54-L67))) The tl; dr of the above code snippets is that we seem to always have required a minSuccess from replicationFactor/2 + 1 except we overwrite replicationFactor to be the # of instances which is likely always going to be more than just 3 ingester replicas because of the replica set extension. At least for zone aware rollouts this seems to be the case. This is because many ingesters will be in the LEAVING state. We can avoid this by just using `WriteNoExtend` operations instead of `Write` operations ([here](789ec0ca4a/ring/ring.go (L83-L93))) In this PR I've hardcoded our call to the ring in `distributor.Push` to always use `ring.WriteNoExtend`. IMO this is the better option to allowing `ring.WriteNoExtend` to be used optionally and defaulting to `ring.Write`. Extending the replication set on writes feels like a footgun for write outages and there's even some feelings internally that allowing that extension is a bug. For more background information see: https://github.com/grafana/mimir/issues/1854 Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Karsten Jeschkies	b16ad8cb38	Document new Helm Charts (#7034) ... **What this PR does / why we need it**: This change describes various Helm related tasks such as installing Loki and configuring meta monitoring. It also explains different concepts related to the new Helm chart and provides a reference of the new Helm values file. **Which issue(s) this PR fixes**: Fixes #7035	4 years ago
Cyril Tovena	7b01864a91	Improves performance of distributor validations (#7534) ... Most of the time we don't need to create those strings which create an alloc per entry.	4 years ago
Pablo	1df72ee209	Add GCP Logs push target Promtail docs (#7511) ... **What this PR does / why we need it**: Some months ago, a new Promtail target was added to support receiving GCP Logs in a push manner. This PR updates the setting up GCP guide, for adding the new required push subscription. Also, adds some notes on details of the setup procedure. This PR will have a follow up expanding on the terraform code needed to setup the GCP resources in a IaaC way. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	4 years ago
Trevor Whitney	705e16fbea	[helm] Fix included rules (#7430) ... **What this PR does / why we need it**: Adds the correct cluster label to our recording rules so they work correctly in the included dashboards. Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	4 years ago
Ed Welch	3ecfe2478c	Fix drone signature (#7512) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Ed Welch	45caba4459	Loki: Remove the bypass for "limited" queries (#7510) ... **What this PR does / why we need it**: Limited queries are queries which don't have a filter expression. Now all log queries will be handled by the LogFilterTripperware which will result in them being split by time (they previously were not) Signed-off-by: Edward Welch <edward.welch@grafana.com> **Which issue(s) this PR fixes**: We've found that very large timeframe `limited` queries will be sent to queriers and then ingesters and can stall out the read path because of the large time ranges. Splitting these by time avoids this problem by keeping any subquery limited in length to the `split_by_interval`. It's important to note that this will likely increase some extra work done by limited queries as more work for these will be parallelized and can result in extra data being processed. This will be the same as how filter queries are handled now, so it will be no worse than that. In fact this was an optimization based on the premise that it's advantageous not to split/shard limited queries but we are seeing this not be the case when limited queries are run over very large time ranges matching streams with huge volumes and combined with parsers like `| json` **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Trevor Whitney	f83988a061	Add 2.7.0 section to upgrading doc (#7457) ... Add 2.7.0 section to Upgrading doc. Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>	4 years ago
Trevor Whitney	57c3a704b8	Few small changelog changes (#7460) ... A few small fixes to changelog. Removed one entry that was just about updating docs, and reorganized a few others to the correct section.	4 years ago
Trevor Whitney	6681923a64	Make 2.7.0 section (#7456) ... Update the 2.7 changelog to move unreleased section into 2.7.	4 years ago
Jan-Otto Kröpke	77225e8a7c	Remove prometheus CRDs (#7499) ... **What this PR does / why we need it**: Remove Prometheus CRDs from loki chart. Signed-off-by: Jan-Otto Kröpke <jok@cloudeteer.de> Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>	4 years ago
Trevor Whitney	6c53113be1	[helm] Add a loki canary test to the helm chart (#7229) ... **What this PR does / why we need it**: Add a helm chart test that utilizes the Loki canary to confirm the Loki cluster is functioning correctly. **Which issue(s) this PR fixes**: Fixes #7228 Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	4 years ago
Karsten Jeschkies	9bc6e85604	Helm: Allow users to define remote write for metrics. (#7399)	4 years ago
Goutham Veeramachaneni	1260adc756	[mixin] Add cortex-gw-internal to dashboards (#7481)	4 years ago
Sandeep Sukhani	bb2927172a	tsdb post compaction cleanup (#7502) ... **What this PR does / why we need it**: Previously when we were reading the whole tsdb index in memory, there was not much cleanup required after we were done with the index. Now that we are mmaping the tsdb index, we need to close the mmap. I had missed calling `index.Close()` on the compacted-away multi-tenant index. This causes the compactor to retain the files on disk, leading to the compactor running away from disk space. This PR takes care of it. While doing the changes, I noticed that some of the tests failed due to `LabelValues` tsdb implementation returning strings valid only until the underlying byte slice stays valid. Since the underlying byte slice could become invalid when we close the tsdb, we should return a copy instead of yolostring. This can also happen on the query path when a sync closes the index before we get to return the response. This PR also fixes that.	4 years ago
Ed Welch	23cc448881	Loki: log when a query starts in the logql engine (#7469) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: There are cases where a query cannot finish (OOM of a querier/ruler) and most of our logs are emitted upon completion of a query. This can help us find queries which are OOM crashing a component. Note: It only covers queries executed by the engine, so only instant and range queries. Another PR will be necessary to cover labels/series endpoints. However this approach seemed the best to be able to get this information from ruler executed queries. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Owen Diehl	70f6bf2e45	Index gw lock bugfix (#7497) ... While I'm still in favor of a longer refactor to address the complexity described in https://github.com/grafana/loki/pull/7490, I'm opening this PR as a shorter, more immediate alternative that should fix the deadlocks. This also reduces the number of total lock calls in many cases as we lock to act over entire groups rather than individual items.	4 years ago
Travis Patterson	45d59ba571	Improve rate store refresh latency (#7492) ... Rather than iterating streams and getting rates per stream, keep all necessary information in the rate calculator and just hand back the current rates when asked. This change took p99 latencies from ~1.8s to ~400ms and p50 latences from ~1s to 6ms.	4 years ago
Michel Hollands	fd93a6c999	Add how to connect to Azure using Service Principal (#7485) ... Add documentation for https://github.com/grafana/loki/pull/7179 which added authentication using a Service Principal for Azure.	4 years ago
Travis Patterson	b4ed595461	Remove groupcache (#7126) ... As discussed during the caching retro, this PR removes the experimental distributed cache based on Mailgun's fork of groupcache. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Trevor Whitney	ae2a6f31ae	[Helm] Add flags needed for migration from distributed (#7331) ... **What this PR does / why we need it**: Adds flags to the helm charts that are needed when doing a zero-downtime migration from a `loki-distributed` helm chart deployment. The flags allow for an additional memberlist join member to be specified. This allows the new `grafana/loki` cluster to join the old `grafana/loki-distributed` cluster, forming one big cluster. The ingesters in the `grafana/loki-distributed` cluster can then be taken down 1-by-1, allowing the whole `grafana/loki-distributed` cluster to be deleted once all the ingesters have scaled down. **Which issue(s) this PR fixes**: Fixes #7330 Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	4 years ago
Dylan Guedes	da5378c927	Loki: Remove RatelimitedStream proto (#7235) ... **What this PR does / why we need it**: Remove an unused proto (RateLimitedStream). It was only used for an older approach to deal with rate-limited streams but since we changed approaches this isn't necessary anymore.	4 years ago
Dylan Guedes	67de0334e9	Docs: Add max_streams to Promtail configuration doc (#7489) ... **What this PR does / why we need it**: Document `max_streams` configuration, introduced in 2.7. Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>	4 years ago
Trevor Whitney	16862b1edc	[helm] Mimic loki service account with canary account (#7236) ... **What this PR does / why we need it**: Bring the Loki Canary service account in line with the Loki service account, removing the post install hook, and sharing the top level properties for `automountServiceAccountToken` and `imagePullSecrets`. I'm open to creating canary specific values for these setting if needed, but want to start by sharing the top level values to reduce config if possible. Also default canary to Appversion to be in sync with default Loki version **Which issue(s) this PR fixes**: Fixes two issues raised in comments from @towolf (thanks!) on this [PR](https://github.com/grafana/loki/pull/7173#issuecomment-1256410181). Fixes #7237	4 years ago
Tim Schwenke	ff406bd4a9	Helm: Replace incorrect key loki.authEnabled in example override values (#7432) ... **What this PR does**: Replace `loki.authEnabled` in one of the example override YAMLs with `loki.auth_enabled` to match the actual key name. Also adds missing newline at end of file.	4 years ago
Mohamed-Amine Bouqsimi	7debe7c235	Ruler: enable Loki to configure Alertmanagers per tenant (#7359)	4 years ago
Christian Haudum	0f5dd2bceb	Revert "fluentd: Add un-escaping of control characters in JSON" (#7468) ... Reverts grafana/loki#7396 according to discussing on PR	4 years ago
Michel Hollands	488e64e3ff	Update helm chart version (#7484) ... Update the Loki helm chart version after the changes made in https://github.com/grafana/loki/pull/7479 Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Michel Hollands	ec541d70e9	Add envFrom to the tokengen job (#7479) ... Add an envFrom section to the tokengen job. Fixes https://github.com/grafana/loki/issues/7411 Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Mohamed-Amine Bouqsimi	7e7436d482	operator: Expose and migrate query_timeout in limits config (#7471)	4 years ago
Dylan Guedes	28e192f22e	Loki: Implement random entries assignment when sharding (#7427) ... **What this PR does / why we need it**: Modify sharding to assign random shards when the number of shards is bigger than the number of entries. This helps us to avoid hot-spotting the first shards. By randomizing the entries assignment we help spread the load between all shards.	4 years ago
Ursula Kallio	709c8508da	Update MAINTAINERS.md (#7467) ... This change removes removes Karen Miller, and includes at least one alternative, interim technical writer. Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Ursula Kallio	7fa3e500c3	Change `master` to `main` and include `.md` ext.	4 years ago
BenjaminVouillaumeJT	2d649a7fdd	Promtail: Add additional fields for cloudflare (#7463)	4 years ago
Periklis Tsirakidis	f8604ba0a7	operator: Upgrade operator-sdk to v1.24.1 (#7449)	4 years ago
Pablo	5f665651d5	Promtail: Add basic tracing support (#7414) ... We've been using Promtail's HTTP/gRPC server-based targets with real workloads for some weeks now. The server that these spin out is based on [weaveworks commons](https://github.com/weaveworks/common) HTTP/gRPC servers. These include by default some middlewares that if enabled, inject tracing support. We found this useful when tracking down request handling time, or even more, if the caller also support Jaeger tracing support, cross-correlated requests between the upstream server and Promtail's. This PR adds support for enabling tracing the same way Loki does. At the moment, this will just generate traces for the before-mentioned targets. In the future, the quality and completeness of tracing support can grow organically to enrich spans.	4 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	4 years ago
Periklis Tsirakidis	4176de17d1	Add compactor delete client tls client config (#7426)	4 years ago