From 02027e442c0faecabc9a96649cca3e47c6e908f2 Mon Sep 17 00:00:00 2001 From: Karsten Jeschkies Date: Mon, 10 Jul 2023 23:48:19 +0200 Subject: [PATCH] Document deadlocked Docker daemon workaround. (#9896) **What this PR does / why we need it**: This pulls @Pandry's [workaround](https://github.com/grafana/loki/issues/2361#issuecomment-718024318) for the seemingly deadlocked Docker daemon into the documentation. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213) --- docs/sources/clients/docker-driver/_index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/sources/clients/docker-driver/_index.md b/docs/sources/clients/docker-driver/_index.md index aa7a0f4237..1bab0341c7 100644 --- a/docs/sources/clients/docker-driver/_index.md +++ b/docs/sources/clients/docker-driver/_index.md @@ -65,8 +65,10 @@ docker plugin disable loki --force docker plugin rm loki ``` -## Known Issues +## Known Issue: Deadlocked Docker Daemon The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of `max_retries` has been exceeded. To avoid the dropping of log entries, setting `max_retries` to zero allows unlimited retries; the driver will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck. +The wait time can be lowered by setting `loki-retries=2`, `loki-max-backoff_800ms`, `loki-timeout=1s` and `keep-file=true`. This way the daemon will be locked only for a short time and the logs will be persisted locally when the Loki client is unable to re-connect. + To avoid this issue, use the Promtail [Docker target]({{< relref "../promtail/configuration#docker" >}}) or [Docker service discovery]({{< relref "../promtail/configuration#docker_sd_config" >}}).