apitech
/
loki
mirror of https://github.com/grafana/loki
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1316 Commits
1054 Branches
354 Tags
708 MiB
Tag: Branch: Tree: 6841c418ef
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '6841c418ef'
${ noResults }
loki/docs/getting-started/troubleshooting.md

127 lines
4.8 KiB
Raw Normal View History Unescape

Documentation Rewrite (#982) * docs: create structure of docs overhaul This commit removes all old docs and lays out the table of contents and framework for how the new documentation will be intended to be read. * docs: add design docs back in * docs: add community documentation * docs: add LogQL docs * docs: port existing operations documentation * docs: add new placeholder file for promtail configuration docs * docs: add TOC for operations/storage * docs: add Loki API documentation * docs: port troubleshooting document * docs: add docker-driver documentation * docs: link to configuration from main docker-driver document * docs: update API for new paths * docs: fix broken links in api.md and remove json marker from examples * docs: incorporate api changes from #1009 * docs: port promtail documentation * docs: add TOC to promtail configuration reference * docs: fix promtail spelling errors * docs: add loki configuration reference * docs: add TOC to configuration * docs: add loki configuration example * docs: add Loki overview with brief explanation about each component * docs: add comparisons document * docs: add info on table manager and update storage/README.md * docs: add getting started * docs: incorporate config yaml changes from #755 * docs: fix typo in releases url for promtail * docs: add installation instructions * docs: add more configuration examples * docs: add information on fluentd client fluent-bit has been temporarily removed until the PR for it is merged. * docs: PR review feedback * docs: add architecture document * docs: add missing information from old docs * `localy` typo Co-Authored-By: Ed Welch <ed@oqqer.com> * docs: s/ran/run/g * Typo * Typo * Tyop * Typo * docs: fixed typo * docs: PR feedback * docs: @cyriltovena PR feedback * docs: add more details to promtail url config option * docs: expand promtail's pipelines document with extra detail * docs: remove reference to Stage interface in pipelines.md * docs: fixed some spelling * docs: clarify promtail configuration and scraping * docs: attempt #2 at explaining promtail's usage of machine hostname * docs: spelling fixes * docs: add reference to promtail custom metrics and fix silly typo * docs: cognizant -> aware * docs: typo * docs: typos * docs: add which components expose which API endpoints in microservices mode * docs: change ksonnet installation to tanka * docs: address most @pracucci feedback * docs: fix all spelling errors so reviewers don't have to keep finding them :) * docs: incorporate changes to API endpoints made in #1022 * docs: add missing loki metrics * docs: add missing promtail metrics * docs: @pstribrany feedback * docs: more @pracucci feedback * docs: move metrics into a table * docs: update push path references to /loki/api/v1/push * docs: add detail to further explain limitations of monolithic mode * docs: add alternative names to modes_of_operation diagram * docs: add log ordering requirement * docs: add procedure for updating docs with latest version * docs: separate out stages documentation into one document per stage * docs: list supported stores in storage documentation * docs: add info on duplicate log lines in pipelines * docs: add line_format as key feature to fluentd * docs: hopefully final commit :)
6 years ago
# Troubleshooting Loki
## "Loki: Bad Gateway. 502"
This error can appear in Grafana when Loki is added as a
datasource, indicating that Grafana in unable to connect to Loki. There may
one of many root causes:
- If Loki is deployed with Docker, and Grafana and Loki are not running in the
same node, check your firewall to make sure the nodes can connect.
- If Loki is deployed with Kubernetes:
- If Grafana and Loki are in the same namespace, set the Loki URL as
`http://$LOKI_SERVICE_NAME:$LOKI_PORT`
- Otherwise, set the Loki URL as
`http://$LOKI_SERVICE_NAME.$LOKI_NAMESPACE:$LOKI_PORT`
## "Data source connected, but no labels received. Verify that Loki and Promtail is configured properly."
This error can appear in Grafana when Loki is added as a datasource, indicating
that although Grafana has connected to Loki, Loki hasn't received any logs from
Promtail yet. There may be one of many root causes:
- Promtail is running and collecting logs but is unable to connect to Loki to
send the logs. Check Promtail's output.
- Promtail started sending logs to Loki before Loki was ready. This can
happen in test environment where Promtail has already read all logs and sent
them off. Here is what you can do:
- Start Promtail after Loki, e.g., 60 seconds later.
- To force Promtail to re-send log messages, delete the positions file
(default location `/tmp/positions.yaml`).
- Promtail is ignoring targets and isn't reading any logs because of a
configuration issue.
- This can be detected by turning on debug logging in Promtail and looking
for `dropping target, no labels` or `ignoring target` messages.
- Promtail cannot find the location of your log files. Check that the
`scrape_configs` contains valid path settings for finding the logs on your
worker nodes.
- Your pods are running with different labels than the ones Promtail is
configured to read. Check `scrape_configs` to validate.
## Troubleshooting targets
Promtail exposes two web pages that can be used to understand how its service
discovery works.
The service discovery page (`/service-discovery`) shows all
discovered targets with their labels before and after relabeling as well as
the reason why the target has been dropped.
The targets page (`/targets`) displays only targets that are being actively
scraped and their respective labels, files, and positions.
On Kubernetes, you can access those two pages by port-forwarding the Promtail
port (`9080` or `3101` if using Helm) locally:
```bash
$ kubectl port-forward loki-promtail-jrfg7 9080
# Then, in a web browser, visit http://localhost:9080/service-discovery
```
## Debug output
Both `loki` and `promtail` support a log level flag on the command-line:
```bash
$ loki —log.level=debug
$ promtail -log.level=debug
```
## Failed to create target, `ioutil.ReadDir: readdirent: not a directory`
The Promtail configuration contains a `__path__` entry to a directory that
Promtail cannot find.
## Connecting to a Promtail pod to troubleshoot
First check [Troubleshooting targets](#troubleshooting-targets) section above.
If that doesn't help answer your questions, you can connect to the Promtail pod
to investigate further.
If you are running Promtail as a DaemonSet in your cluster, you will have a
Promtail pod on each node, so figure out which Promtail you need to debug first:
```shell
$ kubectl get pods --all-namespaces -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
...
nginx-7b6fb56fb8-cw2cm 1/1 Running 0 41d 10.56.4.12 node-ckgc <none>
...
promtail-bth9q 1/1 Running 0 3h 10.56.4.217 node-ckgc <none>
```
That output is truncated to highlight just the two pods we are interested in,
you can see with the `-o wide` flag the NODE on which they are running.
You'll want to match the node for the pod you are interested in, in this example
NGINX, to the Promtail running on the same node.
To debug you can connect to the Promtail pod:
```shell
kubectl exec -it promtail-bth9q -- /bin/sh
```
Once connected, verify the config in `/etc/promtail/promtail.yml` has the
contents you expect.
Also check `/var/log/positions.yaml` (`/run/promtail/positions.yaml` when
deployed by Helm or whatever value is specified for `positions.file`) and make
sure Promtail is tailing the logs you would expect.
You can check the Promtail log by looking in `/var/log/containers` at the
Promtail container log.
## Enable tracing for Loki
Loki can be traced using [Jaeger](https://www.jaegertracing.io/) by setting
the environment variable `JAEGER_AGENT_HOST` to the hostname and port where
Loki is running.
If you deploy with Helm, use the following command:
```bash
$ helm upgrade --install loki loki/loki --set "loki.tracing.jaegerAgentHost=YOUR_JAEGER_AGENT_HOST"
```