mirror of https://github.com/grafana/loki
Loki cloud integration instructions (and necessary mixin changes) (#8492)
**What this PR does / why we need it**: This PR is a first stab at documentation for setting up the Grafana Cloud Loki Integration to monitor a local Loki cluster, installed using the helm chart, using Grafana Cloud. In addition to instructions on how to collect the necessary kubernetes metrics, there were also some things with the Loki mixin (which provides dashboards, alerts, and rules to the integration) that needed to be fixed. For example, our dashboards rely on there being a cluster label on our recording rules, yet our mixin does not include this? Also, we are still including `cortex-gw` panels even when the mixin is compile with `internal_components: false`. **Special notes for your reviewer**: The changes to the mixin will not be reflected in the current version of the integration. We need to merge this PR first then update the integration. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>pull/8636/head
Trevor Whitney
3 years ago
committed by
GitHub
parent
9f8aa4b98a
commit
dac3b84d08
@ -0,0 +1,19 @@ |
||||
--- |
||||
title: Monitoring |
||||
description: monitoring |
||||
weight: 200 |
||||
aliases: |
||||
- /docs/installation/helm/monitor-and-alert |
||||
keywords: |
||||
- helm |
||||
- scalable |
||||
- simple-scalable |
||||
- monitor |
||||
--- |
||||
|
||||
# Monitoring |
||||
|
||||
There are two common ways to monitor Loki: |
||||
|
||||
- [Monitor using Grafana Cloud (recommended)]({{<relref "with-grafana-cloud">}}) |
||||
- [Monitor using using Local Monitoring]({{<relref "with-local-monitoring">}}) |
||||
@ -0,0 +1,100 @@ |
||||
--- |
||||
title: Configure Monitoring and Alerting of Loki Using Grafana Cloud |
||||
menuTitle: Monitor Loki with Grafana Cloud |
||||
description: setup monitoring and alerts for Loki using Grafana Cloud |
||||
aliases: |
||||
- /docs/installation/helm/monitoring/with-grafana-cloud |
||||
weight: 100 |
||||
keywords: |
||||
- monitoring |
||||
- alert |
||||
- alerting |
||||
- grafana cloud |
||||
--- |
||||
|
||||
# Configure Monitoring and Alerting of Loki Using Grafana Cloud |
||||
|
||||
This topic will walk you through using Grafana Cloud to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a Grafana Cloud Logs instance. This approach also does not require the installation of the Prometheus Operator and instead sends metrics to a Grafana Cloud Metrics instance. Using Grafana Cloud to monitor Loki has the added benefit of being able to troubleshoot problems with Loki when the Helm installed Loki is down, as the logs will still be available in the Grafana Cloud Logs instance. |
||||
|
||||
**Before you begin:** |
||||
|
||||
- Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). |
||||
- A Grafana Cloud account and stack (including Cloud Grafana, Cloud Metrics, and Cloud Logs) |
||||
- [Grafana Kubernetes Monitoring using Agent](/docs/grafana-cloud/kubernetes-monitoring/configuration/config-k8s-agent-guide/) configured for the Kubernetes cluster |
||||
- A running Loki deployment installed in that Kubernetes cluster via the Helm chart |
||||
|
||||
**Prequisites for Monitoring Loki:** |
||||
|
||||
You must setup the Grafana Kubernetes Integration following the instructions in [Grafana Kubernetes Monitoring using Agent](/docs/grafana-cloud/kubernetes-monitoring/configuration/config-k8s-agent-guide/) as this will install necessary components for collecting metrics about your Kubernetes cluster and sending them to Grafana Cloud. Many of the dashboards installed as a part of the Loki integration rely on these metrics. |
||||
|
||||
Walking through this installation will create two Grafana Agent configurations, one for metrics and one for logs, that will add the external label `cluster: cloud`. In order for the Dashboards in the self-hosted Grafana Loki integration to work, the cluster name needs to match your Helm installation name. If you installed Loki using the command `helm install best-loki-cluster grafana/loki`, you would need to change the `cluster` value in both Grafana Agent configurations from `cloud` to `best-loki-cluster` when setting up the Grafana Kubernetes integration. |
||||
|
||||
**To set up the Loki integration in Grafana Cloud:** |
||||
|
||||
1. Get valid Push credentials for your Cloud Metrics and Cloud Logs instances. |
||||
1. Create a secret in the same namespace as Loki to store your Cloud Logs credentials. |
||||
|
||||
```bash |
||||
cat <<'EOF' | NAMESPACE=loki /bin/sh -c 'kubectl apply -n $NAMESPACE -f -' |
||||
apiVersion: v1 |
||||
data: |
||||
password: <BASE64_ENCODED_CLOUD_LOGS_PASSWORD> |
||||
username: <BASE64_ENCODED_CLOUD_LOGS_USERNAME> |
||||
kind: Secret |
||||
metadata: |
||||
name: grafana-cloud-logs-credentials |
||||
type: Opaque |
||||
EOF |
||||
``` |
||||
|
||||
1. Create a secret to store your Cloud Metrics credentials. |
||||
|
||||
```bash |
||||
cat <<'EOF' | NAMESPACE=loki /bin/sh -c 'kubectl apply -n $NAMESPACE -f -' |
||||
apiVersion: v1 |
||||
data: |
||||
password: <BASE64_ENCODED_CLOUD_METRICS_PASSWORD> |
||||
username: <BASE64_ENCODED_CLOUD_METRICS_USERNAME> |
||||
kind: Secret |
||||
metadata: |
||||
name: grafana-cloud-metrics-credentials |
||||
type: Opaque |
||||
EOF |
||||
``` |
||||
|
||||
1. Enable monitoring metrics and logs for the Loki installation to be sent your cloud database instances by adding the following to your Helm `values.yaml` file: |
||||
|
||||
```yaml |
||||
--- |
||||
monitoring: |
||||
dashboards: |
||||
enabled: false |
||||
rules: |
||||
enabled: false |
||||
selfMonitoring: |
||||
logsInstance: |
||||
clients: |
||||
- url: <CLOUD_METRICS_URL> |
||||
basicAuth: |
||||
username: |
||||
name: grafana-cloud-logs-credentials |
||||
key: username |
||||
password: |
||||
name: grafana-cloud-logs-credentials |
||||
key: password |
||||
serviceMonitor: |
||||
metricsInstance: |
||||
remoteWrite: |
||||
- url: <CLOUD_LOGS_URL> |
||||
basicAuth: |
||||
username: |
||||
name: grafana-cloud-metrics-credentials |
||||
key: username |
||||
password: |
||||
name: grafana-cloud-metrics-credentials |
||||
key: password |
||||
``` |
||||
|
||||
1. Install the self-hosted Grafana Loki integration by going to your hosted Grafana instance, clicking the lightning bolt icon labeled **Integrations and Connections**, then search for and install the **Self-hosted Grafana Loki** integration. |
||||
|
||||
1. Once the self-hosted Grafana Loki integration is installed, click the **View Dashboards** button to see the installed dashboards. |
||||
@ -0,0 +1,34 @@ |
||||
local k = import 'github.com/grafana/jsonnet-libs/ksonnet-util/kausal.libsonnet'; |
||||
local tanka = import 'github.com/grafana/jsonnet-libs/tanka-util/main.libsonnet'; |
||||
local configMap = k.core.v1.configMap; |
||||
|
||||
local spec = (import './spec.json').spec; |
||||
|
||||
{ |
||||
_config+:: { |
||||
namespace: spec.namespace, |
||||
}, |
||||
|
||||
lokiNamespace: k.core.v1.namespace.new('loki'), |
||||
|
||||
gelLicenseSecret: k.core.v1.secret.new('gel-license', {}, type='Opaque') |
||||
+ k.core.v1.secret.withStringData({ |
||||
'license.jwt': importstr '../../secrets/gel.jwt', |
||||
}) |
||||
+ k.core.v1.secret.metadata.withNamespace('loki'), |
||||
local grafanaCloudCredentials = import '../../secrets/grafana-cloud-credentials.json', |
||||
grafanaCloudMetricsCredentials: k.core.v1.secret.new('grafana-cloud-metrics-credentials', {}, type='Opaque') |
||||
+ k.core.v1.secret.withStringData({ |
||||
username: '%d' % grafanaCloudCredentials.metrics.username, |
||||
password: grafanaCloudCredentials.metrics.password, |
||||
}) |
||||
+ k.core.v1.secret.metadata.withNamespace('loki'), |
||||
grafanaCloudLogsCredentials: k.core.v1.secret.new('grafana-cloud-logs-credentials', {}, type='Opaque') |
||||
+ k.core.v1.secret.withStringData({ |
||||
username: '%d' % grafanaCloudCredentials.logs.username, |
||||
password: grafanaCloudCredentials.logs.password, |
||||
}) |
||||
+ k.core.v1.secret.metadata.withNamespace('loki'), |
||||
|
||||
|
||||
} |
||||
@ -0,0 +1,43 @@ |
||||
--- |
||||
loki: |
||||
querier: |
||||
multi_tenant_queries_enabled: true |
||||
enterprise: |
||||
enabled: true |
||||
adminToken: |
||||
secret: "gel-admin-token" |
||||
useExternalLicense: true |
||||
externalLicenseName: gel-license |
||||
provisioner: |
||||
provisionedSecretPrefix: "provisioned-secret" |
||||
monitoring: |
||||
dashboards: |
||||
enabled: false |
||||
rules: |
||||
enabled: false |
||||
selfMonitoring: |
||||
tenant: |
||||
name: loki |
||||
logsInstance: |
||||
clients: |
||||
- url: https://logs-prod-us-central1.grafana.net/loki/api/v1/push |
||||
basicAuth: |
||||
username: |
||||
name: grafana-cloud-logs-credentials |
||||
key: username |
||||
password: |
||||
name: grafana-cloud-logs-credentials |
||||
key: password |
||||
serviceMonitor: |
||||
metricsInstance: |
||||
remoteWrite: |
||||
- url: https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push |
||||
basicAuth: |
||||
username: |
||||
name: grafana-cloud-metrics-credentials |
||||
key: username |
||||
password: |
||||
name: grafana-cloud-metrics-credentials |
||||
key: password |
||||
minio: |
||||
enabled: true |
||||
Loading…
Reference in new issue