Document deadlocked Docker daemon workaround. (#9896)

**What this PR does / why we need it**:
This pulls @Pandry's
[workaround](https://github.com/grafana/loki/issues/2361#issuecomment-718024318)
for the seemingly deadlocked Docker daemon into the documentation.

**Special notes for your reviewer**:

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e3ec)
pull/9836/head
Karsten Jeschkies 2 years ago committed by GitHub
parent f986492bb7
commit 02027e442c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 4
      docs/sources/clients/docker-driver/_index.md

@ -65,8 +65,10 @@ docker plugin disable loki --force
docker plugin rm loki
```
## Known Issues
## Known Issue: Deadlocked Docker Daemon
The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of `max_retries` has been exceeded. To avoid the dropping of log entries, setting `max_retries` to zero allows unlimited retries; the driver will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck.
The wait time can be lowered by setting `loki-retries=2`, `loki-max-backoff_800ms`, `loki-timeout=1s` and `keep-file=true`. This way the daemon will be locked only for a short time and the logs will be persisted locally when the Loki client is unable to re-connect.
To avoid this issue, use the Promtail [Docker target]({{< relref "../promtail/configuration#docker" >}}) or [Docker service discovery]({{< relref "../promtail/configuration#docker_sd_config" >}}).

Loading…
Cancel
Save