description: Learn how to configure tracing so that you can troubleshoot Grafana.
description: Learn how to configure profiling and tracing so that you can troubleshoot Grafana.
keywords:
- grafana
- troubleshooting
- documentation
- guide
labels:
products:
- enterprise
- oss
menuTitle: Configure tracing
title: Configure tracing to troubleshoot Grafana
menuTitle: Configure profiling and tracing
title: Configure profiling and tracing to troubleshoot Grafana
weight: 200
---
# Configure tracing to troubleshoot Grafana
# Configure profiling and tracing to troubleshoot Grafana
You can set up the `grafana-server` process to enable certain diagnostics when it starts. This can be useful
when investigating certain performance problems. It's _not_ recommended to have these enabled by default.
## Turn on profiling
## Turn on profiling and collect profiles
The `grafana-server` can be started with the command-line option `-profile` to enable profiling, `-profile-addr` to override the default HTTP address (`localhost`), and
`-profile-port` to override the default HTTP port (`6060`) where the `pprof` debugging endpoints are available. For example:
`-profile-port` to override the default HTTP port (`6060`) where the `pprof` debugging endpoints are available. Further, [`-profile-block-rate`](https://pkg.go.dev/runtime#SetBlockProfileRate) controls the fraction of goroutine blocking events that are reported in the blocking profile, default `1` (i.e. track every event) for backward compatibility reasons, and [`-profile-mutex-rate`](https://pkg.go.dev/runtime#SetMutexProfileFraction) controls the fraction of mutex contention events that are reported in the mutex profile, default `0` (i.e. track no events). The higher the fraction (that is, the smaller this value) the more overhead it adds to normal operations.
Running Grafana with profiling enabled and without block and mutex profiling enabled should only add a fraction of overhead and is suitable for [continuous profiling](https://grafana.com/oss/pyroscope/). Adding a small fraction of block and mutex profiling, such as 10-5 (10%-20%) should in general be fine.
Enable profiling:
```bash
./grafana server -profile -profile-addr=0.0.0.0 -profile-port=8080
```
Note that `pprof` debugging endpoints are served on a different port than the Grafana HTTP server.
Enable profiling with block and mutex profiling enabled with a fraction of 20%:
```bash
./grafana server -profile -profile-addr=0.0.0.0 -profile-port=8080 -profile-block-rate=5 -profile-mutex-rate=5
```
Note that `pprof` debugging endpoints are served on a different port than the Grafana HTTP server. Check what debugging endpoints are available by browsing `http://<profile-addr><profile-port>/debug/pprof`.
There are some additional [godeltaprof](https://github.com/grafana/pyroscope-go/tree/main/godeltaprof) endpoints available which are more suitable in a continuous profiling scenario. These endpoints are `/debug/pprof/delta_heap`, `/debug/pprof/delta_block`, `/debug/pprof/delta_mutex`.
You can configure or override profiling settings using environment variables:
@ -34,9 +51,41 @@ You can configure or override profiling settings using environment variables:
export GF_DIAGNOSTICS_PROFILING_ENABLED=true
export GF_DIAGNOSTICS_PROFILING_ADDR=0.0.0.0
export GF_DIAGNOSTICS_PROFILING_PORT=8080
export GF_DIAGNOSTICS_PROFILING_BLOCK_RATE=5
export GF_DIAGNOSTICS_PROFILING_MUTEX_RATE=5
```
In general, you use the [Go command pprof](https://golang.org/cmd/pprof/) to both collect and analyze profiling data. You can also use [curl](https://curl.se/) or similar to collect profiles which could be convenient in environments where you don't have the Go/pprof command available. Next, some usage examples of using curl and pprof to collect and analyze memory and CPU profiles.
**Analyzing high memory usage/memory leaks:**
When experiencing high memory usage or potential memory leaks it's useful to collect several heap profiles and later when analyzing, compare them. It's a good idea to wait some time, e.g. 30 seconds, between collecting each profile to allow memory consumption to increase.
If you're experiencing backend performance problems, such as high memory or CPU usage, please refer to [Configure profiling and tracing to troubleshoot Grafana]({{< relref "../setup-grafana/configure-grafana/configure-tracing/index.md" >}}).
## More help
Check out the [Grafana Community](https://community.grafana.com/) for more troubleshooting help (you must be logged in to post or comment).
Usage:"Controls the fraction of goroutine blocking events that are reported in the blocking profile. The profiler aims to sample an average of one blocking event per rate nanoseconds spent blocked. To turn off profiling entirely, use 0",
Destination:&ProfileBlockRate,
},
&cli.IntFlag{
Name:"profile-mutex-rate",
Value:runtime.SetMutexProfileFraction(-1),
Usage:"Controls the fraction of mutex contention events that are reported in the mutex profile. On average 1/rate events are reported. To turn off mutex profiling entirely, use 0",
Marcus Efraimsson
12 months ago
committed by