operator: Document how to connect Grafana to gateway (#6250)

4 years ago · e61f8dccdd
parent 9b2786bde3
commit e61f8dccdd
3 changed files with 702 additions and 0 deletions
--- a/operator/docs/howto_connect_grafana.md
+++ b/operator/docs/howto_connect_grafana.md
@ -0,0 +1,102 @@
+# How-To: Connect Grafana to an in-cluster LokiStack
+
+## Introduction
+
+[Grafana](https://grafana.com/grafana/) is an established web-based dashboarding and visualization tool for interacting with Loki. As such, it is also intended to be usable as a front-end for a LokiStack.
+
+This document gives instructions on how to set up a Grafana instance in the same Kubernetes cluster where the components of the LokiStack are running.
+
+Depending on how the LokiStack and the cluster are set up it should also be possible to set up a Grafana instance to communicate with a LokiStack hosted in a different cluster, but this is not in the scope of this document.
+
+Generally it is possible to access the reading side of the LokiStack deployment in two ways:
+
+- Through an authenticated gateway
+- Using the query-frontend service directly, bypassing authentication / tenancy
+
+The gateway component should be available on most deployments except for testing purposes and is the preferred way to access the LokiStack.
+
+## Pre-Requisites
+
+All the following instructions assume that a working LokiStack deployment already exists in your cluster and that you have log-forwarders connected to it. There is more documentation on how to set up the deployment and connect the forwarders in other documents.
+
+The instructions also assume that you have access to the Kubernetes cluster as an administrator.
+
+If your LokiStack deployment has the gateway enabled, use one of the first two options of configuring Grafana, depending on whether you are able to configure proper authentication. Should you not be ablet to use the gateway, there's a final variant as a fall-back.
+
+## Deploying Grafana
+
+### Using the Gateway With OpenShift-based Authentication
+
+The preferred option for accessing the data stored in Loki managed by loki-operator when running on OpenShift with the default OpenShift tenancy model is to go through the LokiStack gateway and do proper authentication against the authentication service included in OpenShift.
+
+An example configuration authenticating to the gateway in this manner is available in  [`addon_grafana_gateway_ocp_oauth.yaml`](../hack/addon_grafana_gateway_ocp_oauth.yaml).
+
+The configuration uses `oauth-proxy` to authenticate the user to the Grafana instance and forwards the token through Grafana to LokiStack's gateway service. This enables the configuration to fully take advantage of the tenancy model, so that users can only see the logs of their applications and only admins can view infrastructure and audit logs.
+
+As the open-source version of Grafana does not support to limit datasources to certain groups of users, all three datasources ("application", "infrastructure" and "audit") will be visible to all users. The infrastructure and audit datasources will not yield any data for non-admin users.
+
+### Using the Gateway With a Static Token
+
+Similar to the above configuration this variant makes use of `oauth-proxy` to authenticate the user to Grafana. The difference is that a statically-configured token is used for communicating with LokiStack's gateway, so that each user has access to the logs available to the static token.
+
+As this configuration does not provide any tenancy it should only be used for testing or debugging a LokiStack. It does not completely bypass authentication though, so no public access of the data stored in Loki is possible.
+
+An example configuration using this technique is available in [`addon_grafana_gateway_ocp.yaml`](../hack/addon_grafana_gateway_ocp.yaml).
+
+### Accessing the Query-Frontend Directly
+
+This is the simplest variant but should only be used when running the LokiStack in a testing configuration as it bypasses the authentication.
+
+Note that this configuration also works when the gateway component is available in the LokiStack deployment. It is just not recommended to be used.
+
+For this example we will assume that the LokiStack and Grafana will be deployed to the same namespace `default`, which will be reflected in some host-names in the configuration. If your deployment is in a different namespace, adjust the configuration accordingly.
+
+To create the datasource used for accessing Loki inside Grafana we use a configuration file that will be loaded by Grafana as part of its [provisioning facilities](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources):
+
+```yaml
+apiVersion: 1
+datasources:
+- name: Loki (${LOKI_TENANT_ID})
+  isDefault: true
+  type: loki
+  access: proxy
+  url: http://${LOKISTACK_NAME}-query-frontend-http.default.svc:3100/
+  jsonData:
+    httpHeaderName1: X-Scope-OrgID
+  secureJsonData:
+    httpHeaderValue1: ${LOKI_TENANT_ID}
+```
+
+If the operator was started with the `--with-tls-service-monitors` option, then the protocol used to access the service needs to be set to `https` and, depending on the used certificate another option needs to be added to the `jsonData`: `tlsSkipVerify: true`
+
+The values for the variables used in the configuration file depend on the Lokistack deployment and which Loki tenant needs to be accessed.
+
+The above configuration file can, for example, be added to a ConfigMap and mounted into a Deployment (based on the [example here](https://grafana.com/docs/grafana/latest/installation/kubernetes/)) or StatefulSet:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-datasources
+data:
+  loki.yaml: |
+    -- datasource configuration file contents --
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {}
+spec:
+  template:
+    metadata: {}
+    spec:
+      volumes:
+      - name: grafana-datasources
+        configMap:
+          name: grafana-datasources
+      containers:
+      - name: grafana
+        volumeMounts:
+        - name: grafana-datasources
+          mountPath: /etc/grafana/provisioning/datasources
+          readOnly: true
+```
--- a/operator/hack/addon_grafana_gateway_ocp.yaml
+++ b/operator/hack/addon_grafana_gateway_ocp.yaml
@ -0,0 +1,327 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  annotations:
+    serviceaccounts.openshift.io/oauth-redirectreference.grafana: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"grafana"}}'
+  name: grafana
+  namespace: openshift-logging
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: logging-logs-reader
+rules:
+- apiGroups:
+  - loki.grafana.com
+  resourceNames:
+  - logs
+  resources:
+  - application
+  - infrastructure
+  - audit
+  verbs:
+  - get
+  - create
+- apiGroups:
+  - ""
+  resources:
+  - namespaces
+  verbs:
+  - get
+  - list
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: logging-grafana-auth-delegator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: system:auth-delegator
+subjects:
+- kind: ServiceAccount
+  name: grafana
+  namespace: openshift-logging
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: logging-grafana-logs-reader
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: logging-logs-reader
+subjects:
+- kind: ServiceAccount
+  name: grafana
+  namespace: openshift-logging
+---
+apiVersion: v1
+data:
+  config.ini: |
+    [analytics]
+    check_for_updates = false
+    reporting_enabled = false
+    [auth]
+    disable_login_form = true
+    disable_signout_menu = true
+    [auth.basic]
+    enabled = false
+    [auth.proxy]
+    auto_sign_up = true
+    enabled = true
+    header_name = X-Forwarded-User
+    [security]
+    admin_user = system:does-not-exist
+    cookie_secure = true
+    [users]
+    viewers_can_edit = true
+    default_theme = light
+    [log]
+    mode = console
+kind: ConfigMap
+metadata:
+  name: grafana-config-mbkktkdhtm
+  namespace: openshift-logging
+---
+apiVersion: v1
+data:
+  loki.yaml: |
+    apiVersion: 1
+    datasources:
+      - name: Loki (Application)
+        isDefault: true
+        type: loki
+        access: proxy
+        url: http://${GATEWAY_ADDRESS}/api/logs/v1/application/
+        jsonData:
+          httpHeaderName1: Authorization
+        secureJsonData:
+          httpHeaderValue1: Bearer ${GATEWAY_BEARER_TOKEN}
+      - name: Loki (Infrastructure)
+        type: loki
+        access: proxy
+        url: http://${GATEWAY_ADDRESS}/api/logs/v1/infrastructure/
+        jsonData:
+          httpHeaderName1: Authorization
+        secureJsonData:
+          httpHeaderValue1: Bearer ${GATEWAY_BEARER_TOKEN}
+      - name: Loki (Audit)
+        type: loki
+        access: proxy
+        url: http://${GATEWAY_ADDRESS}/api/logs/v1/audit/
+        jsonData:
+          httpHeaderName1: Authorization
+        secureJsonData:
+          httpHeaderValue1: Bearer ${GATEWAY_BEARER_TOKEN}
+kind: ConfigMap
+metadata:
+  name: grafana-datasources-hkh56t48tg
+  namespace: openshift-logging
+---
+apiVersion: v1
+data:
+  prepare-token.sh: |
+    #!/usr/bin/env bash
+
+    set -e -u -o pipefail
+
+    bearer_token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+    sed "s/\${GATEWAY_BEARER_TOKEN}/$bearer_token/g" /input/loki.yaml > /output/loki.yaml
+kind: ConfigMap
+metadata:
+  name: grafana-init-scripts-6mt2bg5dd8
+  namespace: openshift-logging
+---
+apiVersion: v1
+kind: Service
+metadata:
+  annotations:
+    service.alpha.openshift.io/serving-cert-secret-name: grafana-tls
+  labels:
+    app: grafana
+  name: grafana
+  namespace: openshift-logging
+spec:
+  ports:
+  - name: https
+    port: 8443
+    protocol: TCP
+    targetPort: https
+  - name: http-grafana
+    port: 3000
+    protocol: TCP
+    targetPort: http-grafana
+  selector:
+    app: grafana
+  sessionAffinity: None
+  type: ClusterIP
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  labels:
+    app: grafana
+  name: grafana
+  namespace: openshift-logging
+spec:
+  selector:
+    matchLabels:
+      app: grafana
+  serviceName: grafana
+  template:
+    metadata:
+      labels:
+        app: grafana
+    spec:
+      containers:
+      - args:
+        - -config=/etc/grafana/config.ini
+        env:
+        - name: GATEWAY_ADDRESS
+          value: lokistack-dev-gateway-http.openshift-logging.svc:8080
+        - name: GF_PATHS_PROVISIONING
+          value: /var/lib/provisioning
+        - name: GF_SECURITY_ADMIN_USER
+          value: kube:admin
+        image: docker.io/grafana/grafana:8.5.2
+        imagePullPolicy: IfNotPresent
+        livenessProbe:
+          failureThreshold: 3
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          successThreshold: 1
+          tcpSocket:
+            port: 3000
+          timeoutSeconds: 1
+        name: grafana
+        ports:
+        - containerPort: 3000
+          name: http-grafana
+          protocol: TCP
+        readinessProbe:
+          failureThreshold: 3
+          httpGet:
+            path: /robots.txt
+            port: 3000
+            scheme: HTTP
+          initialDelaySeconds: 10
+          periodSeconds: 30
+          successThreshold: 1
+          timeoutSeconds: 2
+        resources:
+          requests:
+            cpu: 250m
+            memory: 256Mi
+        volumeMounts:
+        - mountPath: /etc/grafana
+          name: grafana-config
+        - mountPath: /var/lib/provisioning/datasources
+          name: patched-datasources
+        - mountPath: /var/lib/grafana
+          name: grafana
+        - mountPath: /etc/grafana/provisioning/datasources
+          name: grafana-datasources
+      - args:
+        - -provider=openshift
+        - -https-address=:8443
+        - -http-address=
+        - -upstream=http://localhost:3000
+        - -tls-cert=/etc/tls/private/tls.crt
+        - -tls-key=/etc/tls/private/tls.key
+        - -client-id=system:serviceaccount:openshift-logging:grafana
+        - -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
+        - -cookie-secret=ZXhhbXBsZS1jb29raWUtc2VjcmV0enp6
+        - -cookie-expire=24h
+        - -skip-provider-button
+        - -scope=user:info user:check-access user:list-projects
+        - -pass-access-token
+        env:
+        - name: HTTP_PROXY
+        - name: HTTPS_PROXY
+        - name: NO_PROXY
+        image: quay.io/openshift/origin-oauth-proxy:4.10
+        imagePullPolicy: IfNotPresent
+        name: grafana-proxy
+        ports:
+        - containerPort: 8443
+          name: https
+          protocol: TCP
+        readinessProbe:
+          failureThreshold: 3
+          httpGet:
+            path: /oauth/healthz
+            port: https
+            scheme: HTTPS
+          periodSeconds: 10
+          successThreshold: 1
+          timeoutSeconds: 1
+        resources:
+          requests:
+            cpu: 1m
+            memory: 20Mi
+        terminationMessagePath: /dev/termination-log
+        terminationMessagePolicy: File
+        volumeMounts:
+        - mountPath: /etc/tls/private
+          name: secret-grafana-tls
+      initContainers:
+      - args:
+        - /usr/bin/bash
+        - -c
+        - /scripts/prepare-token.sh
+        image: docker.io/library/fedora:35
+        name: write-bearer-token
+        volumeMounts:
+        - mountPath: /scripts
+          name: grafana-init-scripts
+        - mountPath: /input
+          name: grafana-datasources
+        - mountPath: /output
+          name: patched-datasources
+      serviceAccountName: grafana
+      volumes:
+      - configMap:
+          name: grafana-config-mbkktkdhtm
+        name: grafana-config
+      - name: secret-grafana-tls
+        secret:
+          defaultMode: 420
+          secretName: grafana-tls
+      - configMap:
+          defaultMode: 493
+          name: grafana-init-scripts-6mt2bg5dd8
+        name: grafana-init-scripts
+      - emptyDir: {}
+        name: patched-datasources
+      - configMap:
+          name: grafana-datasources-hkh56t48tg
+        name: grafana-datasources
+  volumeClaimTemplates:
+  - metadata:
+      labels:
+        app: grafana
+      name: grafana
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      resources:
+        requests:
+          storage: 1Gi
+---
+apiVersion: route.openshift.io/v1
+kind: Route
+metadata:
+  name: grafana
+  namespace: openshift-logging
+spec:
+  port:
+    targetPort: https
+  tls:
+    insecureEdgeTerminationPolicy: Redirect
+    termination: reencrypt
+  to:
+    kind: Service
+    name: grafana
+    weight: 100
+  wildcardPolicy: None
--- a/operator/hack/addon_grafana_gateway_ocp_oauth.yaml
+++ b/operator/hack/addon_grafana_gateway_ocp_oauth.yaml
@ -0,0 +1,273 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  annotations:
+    serviceaccounts.openshift.io/oauth-redirectreference.grafana: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"grafana"}}'
+  name: grafana
+  namespace: openshift-logging
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: logging-application-logs-reader
+rules:
+- apiGroups:
+  - loki.grafana.com
+  resourceNames:
+  - logs
+  resources:
+  - application
+  verbs:
+  - get
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: logging-grafana-auth-delegator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: system:auth-delegator
+subjects:
+- kind: ServiceAccount
+  name: grafana
+  namespace: openshift-logging
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: logging-users-application-logs-reader
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: logging-application-logs-reader
+subjects:
+- apiGroup: rbac.authorization.k8s.io
+  kind: Group
+  name: system:authenticated
+---
+apiVersion: v1
+data:
+  config.ini: |
+    [analytics]
+    check_for_updates = false
+    reporting_enabled = false
+    [auth]
+    disable_login_form = true
+    disable_signout_menu = true
+    [auth.basic]
+    enabled = false
+    [auth.proxy]
+    auto_sign_up = true
+    enabled = true
+    header_name = X-Forwarded-User
+    [paths]
+    data = /var/lib/grafana
+    logs = /var/lib/grafana/logs
+    plugins = /var/lib/grafana/plugins
+    provisioning = /etc/grafana/provisioning
+    [security]
+    admin_user = system:does-not-exist
+    cookie_secure = true
+    [users]
+    viewers_can_edit = true
+    default_theme = light
+    [log]
+    mode = console
+kind: ConfigMap
+metadata:
+  name: grafana-config-5kt2h4545b
+  namespace: openshift-logging
+---
+apiVersion: v1
+data:
+  loki.yaml: |
+    apiVersion: 1
+    datasources:
+      - name: Loki (Application)
+        isDefault: true
+        type: loki
+        access: proxy
+        url: http://${GATEWAY_ADDRESS}/api/logs/v1/application/
+      - name: Loki (Infrastructure)
+        type: loki
+        access: proxy
+        url: http://${GATEWAY_ADDRESS}/api/logs/v1/infrastructure/
+      - name: Loki (Audit)
+        type: loki
+        access: proxy
+        url: http://${GATEWAY_ADDRESS}/api/logs/v1/audit/
+kind: ConfigMap
+metadata:
+  name: grafana-datasources-d9f6t65c72
+  namespace: openshift-logging
+---
+apiVersion: v1
+kind: Service
+metadata:
+  annotations:
+    service.alpha.openshift.io/serving-cert-secret-name: grafana-tls
+  labels:
+    app: grafana
+  name: grafana
+  namespace: openshift-logging
+spec:
+  ports:
+  - name: https
+    port: 8443
+    protocol: TCP
+    targetPort: https
+  - name: http-grafana
+    port: 3000
+    protocol: TCP
+    targetPort: http-grafana
+  selector:
+    app: grafana
+  sessionAffinity: None
+  type: ClusterIP
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  labels:
+    app: grafana
+  name: grafana
+  namespace: openshift-logging
+spec:
+  selector:
+    matchLabels:
+      app: grafana
+  serviceName: grafana
+  template:
+    metadata:
+      labels:
+        app: grafana
+    spec:
+      containers:
+      - args:
+        - -config=/etc/grafana/config.ini
+        env:
+        - name: GATEWAY_ADDRESS
+          value: lokistack-dev-gateway-http.openshift-logging.svc:8080
+        - name: GF_SECURITY_ADMIN_USER
+          value: kube:admin
+        image: docker.io/grafana/grafana:8.5.2
+        imagePullPolicy: IfNotPresent
+        livenessProbe:
+          failureThreshold: 3
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          successThreshold: 1
+          tcpSocket:
+            port: 3000
+          timeoutSeconds: 1
+        name: grafana
+        ports:
+        - containerPort: 3000
+          name: http-grafana
+          protocol: TCP
+        readinessProbe:
+          failureThreshold: 3
+          httpGet:
+            path: /robots.txt
+            port: 3000
+            scheme: HTTP
+          initialDelaySeconds: 10
+          periodSeconds: 30
+          successThreshold: 1
+          timeoutSeconds: 2
+        resources:
+          requests:
+            cpu: 250m
+            memory: 256Mi
+        volumeMounts:
+        - mountPath: /etc/grafana
+          name: grafana-config
+        - mountPath: /var/lib/grafana
+          name: grafana
+        - mountPath: /etc/grafana/provisioning/datasources
+          name: grafana-datasources
+      - args:
+        - -provider=openshift
+        - -https-address=:8443
+        - -http-address=
+        - -upstream=http://localhost:3000
+        - -tls-cert=/etc/tls/private/tls.crt
+        - -tls-key=/etc/tls/private/tls.key
+        - -client-id=system:serviceaccount:openshift-logging:grafana
+        - -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
+        - -cookie-secret=ZXhhbXBsZS1jb29raWUtc2VjcmV0enp6
+        - -cookie-expire=24h
+        - -skip-provider-button
+        - -scope=user:info user:check-access user:list-projects
+        - -pass-access-token
+        - -pass-basic-auth=false
+        env:
+        - name: HTTP_PROXY
+        - name: HTTPS_PROXY
+        - name: NO_PROXY
+        image: quay.io/openshift/origin-oauth-proxy:4.10
+        imagePullPolicy: IfNotPresent
+        name: grafana-proxy
+        ports:
+        - containerPort: 8443
+          name: https
+          protocol: TCP
+        readinessProbe:
+          failureThreshold: 3
+          httpGet:
+            path: /oauth/healthz
+            port: https
+            scheme: HTTPS
+          periodSeconds: 10
+          successThreshold: 1
+          timeoutSeconds: 1
+        resources:
+          requests:
+            cpu: 1m
+            memory: 20Mi
+        terminationMessagePath: /dev/termination-log
+        terminationMessagePolicy: File
+        volumeMounts:
+        - mountPath: /etc/tls/private
+          name: secret-grafana-tls
+      serviceAccountName: grafana
+      volumes:
+      - configMap:
+          name: grafana-config-5kt2h4545b
+        name: grafana-config
+      - name: secret-grafana-tls
+        secret:
+          defaultMode: 420
+          secretName: grafana-tls
+      - configMap:
+          name: grafana-datasources-d9f6t65c72
+        name: grafana-datasources
+  volumeClaimTemplates:
+  - metadata:
+      labels:
+        app: grafana
+      name: grafana
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      resources:
+        requests:
+          storage: 1Gi
+---
+apiVersion: route.openshift.io/v1
+kind: Route
+metadata:
+  name: grafana
+  namespace: openshift-logging
+spec:
+  port:
+    targetPort: https
+  tls:
+    insecureEdgeTerminationPolicy: Redirect
+    termination: reencrypt
+  to:
+    kind: Service
+    name: grafana
+    weight: 100
+  wildcardPolicy: None