Allow configuring more options for output configuration (#2089)

* Allow configuring more options for output configuration

The `TenantID`, `BatchWait`, `BatchSize` and `AutoKubernetesLabels` options in the fluent-bit loki output plugin were not configurable. This change allows configuring those options via the helm chart.

* typo in production/helm/fluent-bit/README.md

Co-authored-by: Joseph Petersen <josephp90@gmail.com>

Co-authored-by: Joseph Petersen <josephp90@gmail.com>
Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>

production/helm/fluent-bit/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: "v1"
 name: fluent-bit
-version: 0.1.2
+version: 0.1.3
 appVersion: v1.5.0
 kubeVersion: "^1.10.0-0"
 description: "Uses fluent-bit Loki go plugin for gathering logs and sending them to Loki"

production/helm/fluent-bit/README.md

@@ -77,11 +77,15 @@ For more details, read the [Fluent Bit documentation](../../../cmd/fluent-bit/RE
 | `loki.user`              | The http basic auth username to access the Loki service.                                           |                                  |
 | `loki.password`          | The http basic auth password to access the Loki service.                                           |                                  |
 | `config.port`            | the Fluent Bit port to listen. (This is mainly used to serve metrics)                              | `2020`                           |
+| `config.tenantID`        | The tenantID used by default to push logs to Loki                                                  | `''`                             |
+| `config.batchWait`       | Time to wait before send a log batch to Loki, full or not. (unit: secs)                            | `1`                              |
+| `config.batchSize`       | Log batch size to send a log batch to Loki. (unit: bytes)                                          | `10240` (10KiB)                  |
 | `config.loglevel`        | the Fluent Bit log level (debug,info,warn,error).                                                  | `warn`                           |
 | `config.lineFormat`      | The line format to use to send a record (json/key_value)                                           | `json`                           |
 | `config.k8sLoggingParser`| Allow Kubernetes Pods to suggest a pre-defined Parser. See [Official Fluent Bit documentation](https://docs.fluentbit.io/manual/filter/kubernetes#kubernetes-annotations).                                                                                      | `Off`                           |
 | `config.removeKeys`      | The list of key to remove from each record                                                         | `[removeKeys,stream]`            |
 | `config.labels`          | A set of labels to send for every log                                                              | `'{job="fluent-bit"}'`           |
+| `config.autoKubernetesLabels` | If set to true, it will add all Kubernetes labels to Loki labels                                   | `false`                          |
 | `config.labelMap`        | Mapping of labels from a record. See [Fluent Bit documentation](../../../cmd/fluent-bit/README.md) |                                  |
 | `config.parsers`         | Definition of extras fluent bit parsers. See [Official Fluent Bit documentation](https://docs.fluentbit.io/manual/filter/parser). The format is a sequence of mappings where each key is the same as the one in the [PARSER] section of parsers.conf file       | `[]`                            |
 | `config.extraOutputs`    | Definition of extras fluent bit outputs. See [Official Fluent Bit documentation](https://docs.fluentbit.io/manual/pipeline/outputs/). The format is a sequence of mappings where each key is the same as the one in the [OUTPUT]                                | `[]`                            |

production/helm/fluent-bit/templates/configmap.yaml

@@ -39,8 +39,12 @@ data:
           {{- else }}
         Url {{ .Values.loki.serviceScheme }}://{{ include "loki.serviceName" . }}:{{ .Values.loki.servicePort }}{{ .Values.loki.servicePath }}
         {{- end }}
+        TenantID {{ .Values.config.tenantID }}
+        BatchWait {{ .Values.config.batchWait }}
+        BatchSize {{ .Values.config.batchSize }}
         Labels {{ .Values.config.labels }}
         RemoveKeys {{ include "helm-toolkit.utils.joinListWithComma" .Values.config.removeKeys }}
+        AutoKubernetesLabels {{ .Values.config.autoKubernetesLabels }}
         LabelMapPath /fluent-bit/etc/labelmap.json
         LineFormat {{ .Values.config.lineFormat }}
         LogLevel {{ .Values.config.loglevel }}

production/helm/fluent-bit/values.yaml

@@ -8,12 +8,16 @@ loki:
   # password: pass
 config:
   port: 2020
+  tenantID: '""'
+  batchWait: 1
+  batchSize: 10240
   loglevel: warn
   lineFormat: json
   k8sLoggingParser: "Off"
   removeKeys:
     - kubernetes
     - stream
+  autoKubernetesLabels: false
   labels: '{job="fluent-bit"}'
   labelMap:
     kubernetes: