Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
loki/operator/docs/forwarding_logs_to_gateway.md

8.0 KiB

Forwarding Logs to LokiStack

This document will describe how to send application, infrastructure, and audit logs to the Lokistack Gateway as different tenants using Promtail or Fluentd. The built-in gateway provides secure access to the distributor (and query-frontend) via consulting an OAuth/OIDC endpoint for the request subject.

Please read the hacking guide before proceeding with the following instructions.

Note: While this document will only give instructions for two methods of log forwarding into the gateway, the examples given in the Promtail and Fluentd sections can be extrapolated to other log forwarders.

OpenShift Logging

OpenShift Logging supports forwarding logs to an external Loki instance. This can also be used to forward logs to LokiStack gateway.

  • Deploy the Loki Operator and an lokistack instance with the gateway flag enabled.

  • Deploy the OpenShift Logging Operator from the Operator Hub or using the following command locally:

    make deploy-image deploy-catalog install
    
  • Create a Cluster Logging instance in the openshift-logging namespace with only collection defined.

    apiVersion: logging.openshift.io/v1
    kind: ClusterLogging
    metadata:
      name: instance
      namespace: openshift-logging
    spec:
      collection:
        logs:
          type: fluentd
          fluentd: {}
    
  • The LokiStack Gateway requires a bearer token for communication with fluentd. Therefore, create a secret with token key and the path to the file.

    kubectl -n openshift-logging create secret generic lokistack-gateway-bearer-token \
    --from-literal=token="/var/run/secrets/kubernetes.io/serviceaccount/token"
    
  • Create the following ClusterRole and ClusterRoleBinding which will allow the cluster to authenticate the user(s) submitting the logs:

    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: lokistack-dev-tenant-logs
    rules:
    - apiGroups:
      - 'loki.grafana.com'
      resources:
      - application
      - infrastructure
      - audit
      resourceNames:
      - logs
      verbs:
      - 'create'
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: lokistack-dev-tenant-logs
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: lokistack-dev-tenant-logs
    subjects:
    - kind: ServiceAccount
      name: logcollector
      namespace: openshift-logging
    
  • Now create a ClusterLogForwarder CR to forward logs to LokiStack:

    apiVersion: logging.openshift.io/v1
    kind: ClusterLogForwarder
    metadata:
      name: instance
      namespace: openshift-logging
    spec:
      outputs:
       - name: loki-app
         type: loki
         url: http://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/application
         secret:
           name: lokistack-gateway-bearer-token
       - name: loki-infra
         type: loki
         url: http://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/infrastructure
         secret:
           name: lokistack-gateway-bearer-token
       - name: loki-audit
         type: loki
         url: http://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit
         secret:
           name: lokistack-gateway-bearer-token
      pipelines:
       - name: send-app-logs
         inputRefs:
         - application
         outputRefs:
         - loki-app
       - name: send-infra-logs
         inputRefs:
         - infrastructure
         outputRefs:
         - loki-infra
       - name: send-audit-logs
         inputRefs:
         - audit
         outputRefs:
         - loki-audit
    

    Note: You can add/remove any pipeline from the ClusterLogForwarder spec in case if you want to limit the logs being sent.

Forwarding Clients

In order to enable communication between the client(s) and the gateway, follow these steps:

  1. Deploy the Loki Operator and an lokistack instance with the gateway flag enabled.

  2. Create a ServiceAccount to generate the Secret which will be used to authorize the forwarder.

kubectl -n openshift-logging create serviceaccount <SERVICE_ACCOUNT_NAME>
  1. Configure the forwarder and deploy it to the openshift-logging namespace.

  2. Create the following ClusterRole and ClusterRoleBinding which will allow the cluster to authenticate the user(s) submitting the logs:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lokistack-dev-tenant-logs-role
rules:
- apiGroups:
  - 'loki.grafana.com'
  resources:
  - application
  - infrastructure
  - audit
  resourceNames:
  - logs
  verbs:
  - 'get'
  - 'create'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lokistack-dev-tenant-logs-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: lokistack-dev-tenant-logs-role
subjects:
- kind: ServiceAccount
  name: "<SERVICE_ACCOUNT_NAME>"
  namespace: openshift-logging

Promtail

Promtail is an agent managed by Grafana which forwards logs to a Loki instance. The Grafana documentation can be consulted for configuring and deploying an instance of Promtail in a Kubernetes cluster.

To configure Promtail to send application, audit, and infrastructure logs, add the following clients to the Promtail configuration

clients:
  - # ...
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    url: http://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push
  - # ...
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    url: http://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/application/loki/api/v1/push
  - # ...
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    url: http://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/infrastructure/loki/api/v1/push

The rest of the configuration can be configured to the developer's desire.

Fluentd

Loki can receive logs from Fluentd via the Grafana plugin.

The Fluentd configuration can be overrided to target the application endpoint to send those log types.

<match **>
  @type loki
  # ...
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  url: http://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/application
</match>

Troubleshooting

Log Entries Out of Order

If the forwarder is configured to send too much data in a short span of time, Loki will back-pressure the forwarder and respond to the POST requests with 429 errors. In order to alleviate this, a few changes could be made to the spec:

  • Consider moving up a t-shirt size. This will bring in addition resources and have a higher ingestion rate.
kubectl -n openshift-logging edit lokistack
size: 1x.medium
  • Manually change the ingestion rate (global or tenant) can be changed via configuration changes to lokistack:
kubectl -n openshift-logging edit lokistack
limits:
    tenants:
        4a5bb098-7caf-42ec-9b1a-8e1d979bfb95:
            IngestionLimits:
                IngestionRate: 15