Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Callum Styan 46b40a5582 Hardcode ring.WriteNoExtend for distributor push operations (#7517) ... With zone awareness enabled we allow restarting multiple ingesters within a single zone at the same time, this means that for many more write requests there's potentially only 2 of the 3 replicas for a stream up at the time the write request is received. This lead to a large increase in 500s during rollouts. My suspicion is that the 500s on rollouts was always related to the following; the replica set ring code has something referred to as "extending the replica set". This essentially is the adding of ingesters in the ring that are not in an ACTIVE state (for example, are in LEAVING) to the set of replicas considered valid for an operation. Adding of the instance in a state other than ACTIVE can be controlled by using a different operation type, which Mimir was using but we were not. Here are the relevant functions, [ring.Get](8d6d914ef6/ring/ring.go (L335-L403)) (particularly these lines [1](8d6d914ef6/ring/ring.go (L381-L389)) and [2](8d6d914ef6/ring/ring.go (L394-L397))) and [replicationStrategy.Filter](789ec0ca4a/ring/replication_strategy.go (L22-L71)) (particularly these lines [1](789ec0ca4a/ring/replication_strategy.go (L32-L36)) and [2](789ec0ca4a/ring/replication_strategy.go (L54-L67))) The tl; dr of the above code snippets is that we seem to always have required a minSuccess from replicationFactor/2 + 1 except we overwrite replicationFactor to be the # of instances which is likely always going to be more than just 3 ingester replicas because of the replica set extension. At least for zone aware rollouts this seems to be the case. This is because many ingesters will be in the LEAVING state. We can avoid this by just using `WriteNoExtend` operations instead of `Write` operations ([here](789ec0ca4a/ring/ring.go (L83-L93))) In this PR I've hardcoded our call to the ring in `distributor.Push` to always use `ring.WriteNoExtend`. IMO this is the better option to allowing `ring.WriteNoExtend` to be used optionally and defaulting to `ring.Write`. Extending the replication set on writes feels like a footgun for write outages and there's even some feelings internally that allowing that extension is a bug. For more background information see: https://github.com/grafana/mimir/issues/1854 Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>		3 years ago
..
clientpool	Fix two remote_timeout configs in ingester_client block (#5418)	3 years ago
shardstreams	Loki: Per-tenant stream sharding (#7311)	3 years ago
distributor.go	Hardcode ring.WriteNoExtend for distributor push operations (#7517)	3 years ago
distributor_ring.go	correct heartbeat timeout to distributor lifecycler for ring page (#7253)	3 years ago
distributor_test.go	Loki: Implement random entries assignment when sharding (#7427)	3 years ago
http.go	Loki: Remove ring client from the distributor (#6622)	3 years ago
http_test.go	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212)	3 years ago
ingestion_rate_strategy.go	Chore: Use dskit/limiter (#4327)	4 years ago
ingestion_rate_strategy_test.go	Fix inconsistent coverage in the distributor package (#6645)	3 years ago
limits.go	Measure Per-Stream rates (#7370)	3 years ago
ratestore.go	Improve rate store refresh latency (#7492)	3 years ago
ratestore_metrics.go	Fix stream sharding (#7393)	3 years ago
ratestore_test.go	Measure Per-Stream rates (#7370)	3 years ago
streamsharder_test.go	Loki: Shard streams based on desired rate (#7199)	3 years ago
validator.go	Improves performance of distributor validations (#7534)	3 years ago
validator_test.go	Refactor LogQL syntax into its own package (#5539)	3 years ago