The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
grafana/docs/sources/setup-grafana/set-up-grafana-monitoring.md

160 lines
5.9 KiB

---
aliases:
- ../admin/metrics/
- ../administration/jaeger-instrumentation/
- ../administration/view-server/internal-metrics/
description: Jaeger traces emitted and propagation by Grafana
keywords:
- grafana
- jaeger
- tracing
Explicitly set all front matter labels in the source files (#71548) * Set every page to have defaults of 'Enterprise' and 'Open source' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration pages to have of 'Cloud', 'Enterprise', and 'Open source' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/enterprise-licensing pages to have 'Enterprise' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/organization-management pages to have 'Enterprise' and 'Open source' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/provisioning pages to have 'Enterprise' and 'Open source' labels Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/recorded-queries pages to have labels cloud,enterprise * Set administration/roles-and-permissions/access-control pages to have labels cloud,enterprise Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set administration/stats-and-license pages to have labels cloud,enterprise * Set alerting pages to have labels cloud,enterprise,oss * Set breaking-changes pages to have labels cloud,enterprise,oss * Set dashboards pages to have labels cloud,enterprise,oss * Set datasources pages to have labels cloud,enterprise,oss * Set explore pages to have labels cloud,enterprise,oss * Set fundamentals pages to have labels cloud,enterprise,oss * Set introduction/grafana-cloud pages to have labels cloud Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix introduction pages products Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set panels-visualizations pages to have labels cloud,enterprise,oss * Set release-notes pages to have labels cloud,enterprise,oss * Set search pages to have labels cloud,enterprise,oss * Set setup-grafana/configure-security/audit-grafana pages to have labels cloud,enterprise Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set setup-grafana/configure-security/configure-authentication pages to have labels cloud,enterprise,oss * Set setup-grafana/configure-security/configure-authentication/enhanced-ldap pages to have labels cloud,enterprise * Set setup-grafana/configure-security/configure-authentication/saml pages to have labels cloud,enterprise * Set setup-grafana/configure-security/configure-database-encryption/encrypt-secrets-using-hashicorp-key-vault pages to have labels cloud,enterprise * Set setup-grafana/configure-security/configure-request-security pages to have labels cloud,enterprise,oss Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set setup-grafana/configure-security/configure-team-sync pages to have labels cloud,enterprise Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set setup-grafana/configure-security/export-logs pages to have labels cloud,enterprise Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Set troubleshooting pages to have labels cloud,enterprise,oss * Set whatsnew pages to have labels cloud,enterprise,oss * Apply updated labels from review Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> Co-authored-by: Isabel <76437239+imatwawana@users.noreply.github.com> --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> Co-authored-by: Isabel <76437239+imatwawana@users.noreply.github.com>
2 years ago
labels:
products:
- enterprise
- oss
title: Set up Grafana monitoring
weight: 800
---
# Set up Grafana monitoring
Grafana supports tracing.
Grafana can emit Jaeger or OpenTelemetry Protocol (OTLP) traces for its HTTP API endpoints and propagate Jaeger and [w3c Trace Context](https://www.w3.org/TR/trace-context/) trace information to compatible data sources.
All HTTP endpoints are logged evenly (annotations, dashboard, tags, and so on).
When a trace ID is propagated, it is reported with operation 'HTTP /datasources/proxy/:id/\*'.
Enable doc-validator for specific directories (#63524) * Enable doc-validator for specific directories Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix one linting error to trigger CI Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update doc-validator to latest release Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update make-docs procedure Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Use doc-validator version from CI in local make target Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Revert to 1.11.0 Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * adds missing descriptions * Fix titles and headings Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix link formats Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix easy to resolve anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Remove broken anchor link This anchor appears to have been broken for a long time. Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Use doc-validator image with support for numbered anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update make-docs procedure to support doc-validator 2.0.x Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix a bunch of broken anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Ignore old whatsnew content Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update doc-validator to v2.0.x and use reviewdog to report errors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * removes broken links --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: Chris Moyer <chris.moyer@grafana.com>
2 years ago
Refer to [Configuration's OpenTelemetry section]({{< relref "./configure-grafana#tracingopentelemetry" >}}) for a reference of tracing options available in Grafana.
## View Grafana internal metrics
Grafana collects some metrics about itself internally. Grafana supports pushing metrics to Graphite or exposing them to be scraped by Prometheus.
Enable doc-validator for specific directories (#63524) * Enable doc-validator for specific directories Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix one linting error to trigger CI Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update doc-validator to latest release Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update make-docs procedure Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Use doc-validator version from CI in local make target Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Revert to 1.11.0 Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * adds missing descriptions * Fix titles and headings Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix link formats Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix easy to resolve anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Remove broken anchor link This anchor appears to have been broken for a long time. Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Use doc-validator image with support for numbered anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update make-docs procedure to support doc-validator 2.0.x Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix a bunch of broken anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Ignore old whatsnew content Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update doc-validator to v2.0.x and use reviewdog to report errors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * removes broken links --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: Chris Moyer <chris.moyer@grafana.com>
2 years ago
For more information about configuration options related to Grafana metrics, refer to [metrics]({{< relref "./configure-grafana#metrics" >}}) and [metrics.graphite]({{< relref "./configure-grafana#metricsgraphite" >}}) in [Configuration]({{< relref "./configure-grafana" >}}).
### Available metrics
When enabled, Grafana exposes a number of metrics, including:
- Active Grafana instances
- Number of dashboards, users, and playlists
- HTTP status codes
- Requests by routing group
- Grafana active alerts
- Grafana performance
### Pull metrics from Grafana into Prometheus
These instructions assume you have already added Prometheus as a data source in Grafana.
1. Enable Prometheus to scrape metrics from Grafana. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system) remove the semicolon to enable the following configuration options:
```
# Metrics available at HTTP URL /metrics and /metrics/plugins/:pluginId
[metrics]
# Disable / Enable internal metrics
enabled = true
# Disable total stats (stat_totals_*) metrics to be generated
disable_total_stats = false
```
1. (optional) If you want to require authorization to view the metrics endpoints, then uncomment and set the following options:
```
basic_auth_username =
basic_auth_password =
```
1. Restart Grafana. Grafana now exposes metrics at http://localhost:3000/metrics.
1. Add the job to your prometheus.yml file.
Example:
```
- job_name: 'grafana_metrics'
scrape_interval: 15s
scrape_timeout: 5s
static_configs:
- targets: ['localhost:3000']
```
1. Restart Prometheus. Your new job should appear on the Targets tab.
1. In Grafana, hover your mouse over the **Configuration** (gear) icon on the left sidebar and then click **Data Sources**.
1. Select the **Prometheus** data source.
1. On the Dashboards tab, **Import** the Grafana metrics dashboard. All scraped Grafana metrics are available in the dashboard.
### View Grafana metrics in Graphite
These instructions assume you have already added Graphite as a data source in Grafana.
1. Enable sending metrics to Graphite. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system) remove the semicolon to enable the following configuration options:
```
# Metrics available at HTTP API Url /metrics
[metrics]
# Disable / Enable internal metrics
enabled = true
# Disable total stats (stat_totals_*) metrics to be generated
disable_total_stats = false
```
1. Enable [metrics.graphite] options:
```
# Send internal metrics to Graphite
[metrics.graphite]
# Enable by setting the address setting (ex localhost:2003)
address = <hostname or ip>:<port#>
prefix = prod.grafana.%(instance_name)s.
```
1. Restart Grafana. Grafana now exposes metrics at http://localhost:3000/metrics and sends them to the Graphite location you specified.
### Pull metrics from Grafana backend plugin into Prometheus
Docs: Remove old plugin docs, fix URLs to go to new dev portal (#75325) * Remove old plugin docs, add redirects to new dev portal * Restore plugin.schema.json file * Update docs/sources/administration/plugin-management/index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * More updates * Cleanup links * Update docs/sources/alerting/fundamentals/evaluate-grafana-alerts.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * fix codeowners * Change reference links to inline links to avoid 404s * Update docs/sources/datasources/_index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Update docs/sources/datasources/mysql/_index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Update docs/sources/datasources/postgres/_index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Update docs/sources/fundamentals/timeseries-dimensions/index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Update docs/sources/fundamentals/dashboards-overview/index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Update docs/sources/fundamentals/dashboards-overview/index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Update docs/sources/panels-visualizations/configure-standard-options/index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Update docs/sources/panels-visualizations/query-transform-data/transform-data/index.md Co-authored-by: Jack Baldry <jack.baldry@grafana.com> * Remove grafana.com from 2 URLs * Fix 1 more reference link * Prettier fix --------- Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
2 years ago
Any installed [backend plugin](/developers/plugin-tools/introduction/backend-plugins) exposes a metrics endpoint through Grafana that you can configure Prometheus to scrape.
These instructions assume you have already added Prometheus as a data source in Grafana.
1. Enable Prometheus to scrape backend plugin metrics from Grafana. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system) remove the semicolon to enable the following configuration options:
```
# Metrics available at HTTP URL /metrics and /metrics/plugins/:pluginId
[metrics]
# Disable / Enable internal metrics
enabled = true
# Disable total stats (stat_totals_*) metrics to be generated
disable_total_stats = false
```
1. (optional) If you want to require authorization to view the metrics endpoints, then uncomment and set the following options:
```
basic_auth_username =
basic_auth_password =
```
Enable doc-validator for specific directories (#63524) * Enable doc-validator for specific directories Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix one linting error to trigger CI Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update doc-validator to latest release Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update make-docs procedure Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Use doc-validator version from CI in local make target Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Revert to 1.11.0 Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * adds missing descriptions * Fix titles and headings Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix link formats Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix easy to resolve anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Remove broken anchor link This anchor appears to have been broken for a long time. Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Use doc-validator image with support for numbered anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update make-docs procedure to support doc-validator 2.0.x Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix a bunch of broken anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Ignore old whatsnew content Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update doc-validator to v2.0.x and use reviewdog to report errors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * removes broken links --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: Chris Moyer <chris.moyer@grafana.com>
2 years ago
1. Restart Grafana. Grafana now exposes metrics at `http://localhost:3000/metrics/plugins/<plugin id>`, e.g. http://localhost:3000/metrics/plugins/grafana-github-datasource if you have the [Grafana GitHub datasource](/grafana/plugins/grafana-github-datasource/) installed.
1. Add the job to your prometheus.yml file.
Example:
```
- job_name: 'grafana_github_datasource'
scrape_interval: 15s
scrape_timeout: 5s
metrics_path: /metrics/plugins/grafana-test-datasource
static_configs:
- targets: ['localhost:3000']
```
1. Restart Prometheus. Your new job should appear on the Targets tab.
1. In Grafana, hover your mouse over the **Configuration** (gear) icon on the left sidebar and then click **Data Sources**.
1. Select the **Prometheus** data source.
Enable doc-validator for specific directories (#63524) * Enable doc-validator for specific directories Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix one linting error to trigger CI Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update doc-validator to latest release Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update make-docs procedure Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Use doc-validator version from CI in local make target Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Revert to 1.11.0 Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * adds missing descriptions * Fix titles and headings Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix link formats Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix easy to resolve anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Remove broken anchor link This anchor appears to have been broken for a long time. Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Use doc-validator image with support for numbered anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update make-docs procedure to support doc-validator 2.0.x Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Fix a bunch of broken anchors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Ignore old whatsnew content Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * Update doc-validator to v2.0.x and use reviewdog to report errors Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * removes broken links --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: Chris Moyer <chris.moyer@grafana.com>
2 years ago
1. Import a Golang application metrics dashboard - for example [Go Processes](/grafana/dashboards/6671).