mirror of https://github.com/grafana/loki
Unite docs for fluentd plugin (#1634)
* Use original README.md for fluentd plugin explanations `doc/client/fluentd.m` includes obsolete explanations and it confuses new users. For example, label_keys option was removed at https://github.com/grafana/loki/pull/1186 but `fluentd.md` has the explanation. It's simple to maitain `fluentd/fluent-plugin-grafana-loki/README.md` for the explantion of the output plugin and use the link when we share how to use it. * Remove `label_keys` option and simplify usage for `drop_single_key` * Fix environment variable: FLUENT_CONFIG -> FLUENTD_CONFIG `docker-compose up` fails due to the wrong environment variable specifying the fluentd config. As a result this causes the following error ``` docker-compose up --build WARNING: The LOKI_USERNAME variable is not set. Defaulting to a blank string. WARNING: The LOKI_PASSWORD variable is not set. Defaulting to a blank string. Building fluentd Step 1/17 : FROM fluent/fluentd:v1.3.2-debian ---> 4790aaf4d1e5 Step 2/17 : USER root ---> Using cache ---> 7b54da6cec9e Step 3/17 : WORKDIR /home/fluent ---> Using cache ---> a161d2fcf64b Step 4/17 : ENV PATH /fluentd/vendor/bundle/ruby/2.3.0/bin:$PATH ---> Using cache ---> b67529aad0b5 Step 5/17 : ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.3.0 ---> Using cache ---> 3a8b05beb185 Step 6/17 : ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.3.0 ---> Using cache ---> d68a94345cb4 Step 7/17 : ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1 ---> Using cache ---> 70f9bbf646e8 Step 8/17 : COPY docker/Gemfile* /fluentd/ ---> Using cache ---> 5f8b2f6ffe1a Step 9/17 : RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" && apt-get update && apt-get install -y --no-install-recommends $buildDeps libsystemd0 net-tools libjemalloc1 && gem install bundler --version 1.16.2 && bundle config silence_root_warning true && bundle install --gemfile=/fluentd/Gemfile --path=/fluentd/vendor/bundle && sudo gem sources --clear-all && SUDO_FORCE_REMOVE=yes apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $buildDeps && rm -rf /var/lib/apt/lists/* /home/fluent/.gem/ruby/2.3.0/cache/*.gem /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem ---> Using cache ---> 36c960a53c2c Step 10/17 : COPY docker/entrypoint.sh /fluentd/entrypoint.sh ---> Using cache ---> 7f8d72ae63ca Step 11/17 : COPY lib/fluent/plugin/out_loki.rb /fluentd/plugins/out_loki.rb ---> Using cache ---> d09473ee4f25 Step 12/17 : COPY docker/conf/ /fluentd/etc/loki/ ---> Using cache ---> 00d0439ef3c8 Step 13/17 : ENV FLUENTD_CONF="/fluentd/etc/loki/fluentd.conf" ---> Using cache ---> 2d33a002baaf Step 14/17 : ENV FLUENTD_OPT="" ---> Using cache ---> c314df064756 Step 15/17 : ENV LOKI_URL "https://logs-us-west1.grafana.net" ---> Using cache ---> df78c829d416 Step 16/17 : ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so.1" ---> Using cache ---> 3d7d14cf37f5 Step 17/17 : ENTRYPOINT ["/fluentd/entrypoint.sh"] ---> Using cache ---> 82c70c2354ac Successfully built 82c70c2354ac Successfully tagged fluentd:loki Starting docker_fluentd_1 ... done Attaching to docker_fluentd_1 fluentd_1 | /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in `initialize': No such file or directory @ rb_sysopen - /fluentd/etc/loki/fluentd.conf (Errno::ENOENT) fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in `open' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in `read_config' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:479:in `run_supervisor' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/command/fluentd.rb:314:in `<top (required)>' fluentd_1 | from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' fluentd_1 | from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/bin/fluentd:8:in `<top (required)>' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in `load' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in `<main>' docker_fluentd_1 exited with code 1 ``` * Use docs/clients/fluentd/README.md for fluent-plugin-grafana-loki explanation * Arrange list order based on number * Revert "Fix environment variable: FLUENT_CONFIG -> FLUENTD_CONFIG" This reverts commit 8ed58ac97a82d5536321fb9a76120fed766ae506. * Elaborate the way to use docker image * Fix grammar: remove unnecessary with * Add disclaimer for multi-worker usagepull/1664/head
Takayuki Watanabe
5 years ago
committed by
GitHub
parent
e18810949f
commit
973b96d094
@ -1,179 +0,0 @@ |
||||
# Fluentd |
||||
|
||||
Loki has a [Fluentd](https://www.fluentd.org/) output plugin called |
||||
`fluent-plugin-grafana-loki` that enables shipping logs to a private Loki |
||||
instance or [Grafana Cloud](https://grafana.com/oss/loki). |
||||
|
||||
The plugin offers two line formats and uses protobuf to send compressed data to |
||||
Loki. |
||||
|
||||
Key features: |
||||
|
||||
* `extra_labels`: Labels to be added to every line of a log file, useful for |
||||
designating environments |
||||
* `label_keys`: Customizable list of keys for stream labels |
||||
* `line_format`: Format to use when flattening the record to a log line (`json` |
||||
or `key_value`). |
||||
|
||||
## Installation |
||||
|
||||
```bash |
||||
$ gem install fluent-plugin-grafana-loki |
||||
``` |
||||
|
||||
## Usage |
||||
|
||||
In your Fluentd configuration, use `@type loki`. Additional configuration is |
||||
optional. Default values look like this: |
||||
|
||||
``` |
||||
<match **> |
||||
@type loki |
||||
url "https://logs-us-west1.grafana.net" |
||||
username "#{ENV['LOKI_USERNAME']}" |
||||
password "#{ENV['LOKI_PASSWORD']}" |
||||
extra_labels {"env":"dev"} |
||||
flush_interval 10s |
||||
flush_at_shutdown true |
||||
buffer_chunk_limit 1m |
||||
</match> |
||||
``` |
||||
|
||||
### Multi-worker usage |
||||
|
||||
Loki doesn't currently support out-of-order inserts - if you try to insert a log |
||||
entry with an earlier timestamp after a log entry with with identical labels but |
||||
a later timestamp, the insert will fail with the message |
||||
`HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of |
||||
order`. Therefore, in order to use this plugin in a multi-worker Fluentd setup, |
||||
you'll need to include the worker ID in the labels sent to Loki. |
||||
|
||||
For example, using |
||||
[fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier): |
||||
|
||||
``` |
||||
<filter mytag> |
||||
@type record_modifier |
||||
<record> |
||||
fluentd_worker "#{worker_id}" |
||||
</record> |
||||
</filter> |
||||
|
||||
<match mytag> |
||||
@type loki |
||||
# ... |
||||
label_keys "fluentd_worker" |
||||
# ... |
||||
</match> |
||||
``` |
||||
|
||||
## Docker Image |
||||
|
||||
There is a Docker image `grafana/fluent-plugin-grafana-loki:master` which |
||||
contains default configuration files to git log information |
||||
a host's `/var/log` directory, and from the host's journald. To use it, you can set |
||||
the `LOKI_URL`, `LOKI_USERNAME`, and `LOKI_PASSWORD` environment variables |
||||
(`LOKI_USERNAME` and `LOKI_PASSWORD` can be left blank if Loki is not protected |
||||
behind an authenticating proxy). |
||||
|
||||
An example Docker Swarm Compose configuration looks like: |
||||
|
||||
```yaml |
||||
services: |
||||
fluentd: |
||||
image: grafana/fluent-plugin-grafana-loki:master |
||||
command: |
||||
- "fluentd" |
||||
- "-v" |
||||
- "-p" |
||||
- "/fluentd/plugins" |
||||
environment: |
||||
LOKI_URL: http://loki:3100 |
||||
LOKI_USERNAME: |
||||
LOKI_PASSWORD: |
||||
deploy: |
||||
mode: global |
||||
configs: |
||||
- source: loki_config |
||||
target: /fluentd/etc/loki/loki.conf |
||||
networks: |
||||
- loki |
||||
volumes: |
||||
- host_logs:/var/log |
||||
# Needed for journald log ingestion: |
||||
- /etc/machine-id:/etc/machine-id |
||||
- /dev/log:/dev/log |
||||
- /var/run/systemd/journal/:/var/run/systemd/journal/ |
||||
logging: |
||||
options: |
||||
tag: infra.monitoring |
||||
``` |
||||
|
||||
## Configuration |
||||
|
||||
### Proxy Support |
||||
|
||||
Starting with version 0.8.0, this gem uses `excon`, which supports proxy with |
||||
environment variables - https://github.com/excon/excon#proxy-support |
||||
|
||||
### `url` |
||||
|
||||
The URL of the Loki server to send logs to. When sending data the publish path |
||||
(`/loki/api/v1/push`) will automatically be appended. By default the URL is set to |
||||
`https://logs-us-west1.grafana.net`, the URL of the Grafana Labs [hosted |
||||
Loki](https://grafana.com/loki) service. |
||||
|
||||
### `username` / `password` |
||||
|
||||
Specify a username and password if the Loki server requires authentication. |
||||
If using the Grafana Labs' hosted Loki, the username needs to be set to your |
||||
instanceId and the password should be a grafana.com API Key. |
||||
|
||||
### `tenant` |
||||
|
||||
Loki is a multi-tenant log storage platform and all requests sent must include a |
||||
tenant. For some installations (like Hosted Loki) the tenant will be set |
||||
automatically by an authenticating proxy. Otherwise you can define a tenant to |
||||
be passed through. The tenant can be any string value. |
||||
|
||||
### output format |
||||
|
||||
Loki is intended to index and group log streams using only a small set of |
||||
labels and is not intended for full-text indexing. When sending logs to Loki, |
||||
the majority of log message will be sent as a single log "line". |
||||
|
||||
There are few configurations settings to control the output format: |
||||
|
||||
- `extra_labels`: (default: nil) set of labels to include with every Loki |
||||
stream. (e.g., `{"env":"dev", "datacenter": "dc1"}`) |
||||
|
||||
- `remove_keys`: (default: nil) comma separated list of record keys to |
||||
remove. All other keys will be placed into the log line. |
||||
|
||||
- `label_keys`: (default: "job,instance") comma separated list of keys to use as |
||||
stream labels. |
||||
|
||||
- `line_format`: format to use when flattening the record to a log line. Valid |
||||
values are `json` or `key_value`. If set to `json` the log line sent to Loki |
||||
will be the fluentd record (excluding any keys extracted out as labels) dumped |
||||
as json. If set to `key_value`, the log line will be each item in the record |
||||
concatenated together (separated by a single space) in the format |
||||
`<key>=<value>`. |
||||
|
||||
- `drop_single_key`: if set to true, when the set of extracted `label_keys` |
||||
after dropping with `remove_keys`, the log line sent to Loki will just be |
||||
the value of the single remaining record key. |
||||
|
||||
### Buffer options |
||||
|
||||
`fluentd-plugin-loki` extends [Fluentd's builtin Output |
||||
plugin](https://docs.fluentd.org/v1.0/articles/output-plugin-overview) and uses |
||||
the `compat_parameters` plugin helper. It adds the following options: |
||||
|
||||
``` |
||||
buffer_type memory |
||||
flush_interval 10s |
||||
retry_limit 17 |
||||
retry_wait 1.0 |
||||
num_threads 1 |
||||
``` |
||||
@ -0,0 +1,211 @@ |
||||
# Fluentd |
||||
|
||||
Loki has a [Fluentd](https://www.fluentd.org/) output plugin called |
||||
`fluent-plugin-grafana-loki` that enables shipping logs to a private Loki |
||||
instance or [Grafana Cloud](https://grafana.com/oss/loki). |
||||
|
||||
## Installation |
||||
|
||||
``` |
||||
$ gem install fluent-plugin-grafana-loki |
||||
``` |
||||
|
||||
## Usage |
||||
In your Fluentd configuration, use `@type loki`. Additional configuration is optional, default values would look like this: |
||||
``` |
||||
<match **> |
||||
@type loki |
||||
url "https://logs-us-west1.grafana.net" |
||||
username "#{ENV['LOKI_USERNAME']}" |
||||
password "#{ENV['LOKI_PASSWORD']}" |
||||
extra_labels {"env":"dev"} |
||||
flush_interval 10s |
||||
flush_at_shutdown true |
||||
buffer_chunk_limit 1m |
||||
</match> |
||||
``` |
||||
|
||||
### Using labels |
||||
|
||||
Simple label from top level attribute |
||||
``` |
||||
<match mytag> |
||||
@type loki |
||||
# ... |
||||
<label> |
||||
fluentd_worker |
||||
</label> |
||||
# ... |
||||
</match> |
||||
``` |
||||
|
||||
You can rewrite the label keys as well as the following |
||||
|
||||
``` |
||||
<match mytag> |
||||
@type loki |
||||
# ... |
||||
<label> |
||||
worker fluentd_worker |
||||
</label> |
||||
# ... |
||||
</match> |
||||
``` |
||||
|
||||
You can use record accessor syntax for nested field. https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax |
||||
|
||||
``` |
||||
<match mytag> |
||||
@type loki |
||||
# ... |
||||
<label> |
||||
container $.kubernetes.container |
||||
</label> |
||||
# ... |
||||
</match> |
||||
``` |
||||
|
||||
### Extracting Kubernetes labels |
||||
|
||||
As Kubernetes labels are a list of nested key-value pairs there is a separate option to extract them. |
||||
Note that special characters like "`. - /`" will be overwritten with `_`. |
||||
Use with the `remove_keys kubernetes` option to eliminate metadata from the log. |
||||
``` |
||||
<match mytag> |
||||
@type loki |
||||
# ... |
||||
extract_kubernetes_labels true |
||||
remove_keys kubernetes |
||||
<label> |
||||
container $.kubernetes.container |
||||
</label> |
||||
# ... |
||||
</match> |
||||
``` |
||||
|
||||
### Multi-worker usage |
||||
|
||||
Loki doesn't currently support out-of-order inserts - if you try to insert a log entry an earlier timestamp after a log entry with identical labels but a later timestamp, the insert will fail with `HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of order`. Therefore, in order to use this plugin in a multi worker Fluentd setup, you'll need to include the worker ID in the labels or otherwise [ensure log streams are always sent to the same worker](https://docs.fluentd.org/deployment/multi-process-workers#less-than-worker-n-greater-than-directive). |
||||
|
||||
For example, using [fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier): |
||||
``` |
||||
<filter mytag> |
||||
@type record_modifier |
||||
<record> |
||||
fluentd_worker "#{worker_id}" |
||||
</record> |
||||
</filter> |
||||
|
||||
<match mytag> |
||||
@type loki |
||||
# ... |
||||
<label> |
||||
fluentd_worker |
||||
</label> |
||||
# ... |
||||
</match> |
||||
``` |
||||
|
||||
### Using multiple buffer flush threads |
||||
|
||||
Similarly, when using `flush_thread_count` > 1 in the [`buffer`](https://docs.fluentd.org/configuration/buffer-section#flushing-parameters) |
||||
section, a thread identifier must be added as a label to ensure that log chunks flushed in parallel to loki by fluentd always have increasing |
||||
times for their unique label sets. |
||||
|
||||
This plugin automatically adds a `fluentd_thread` label with the name of the buffer flush thread when `flush_thread_count` > 1. |
||||
|
||||
## Docker Image |
||||
|
||||
There is a Docker image `grafana/fluent-plugin-grafana-loki:master` which contains [default configuration files](https://github.com/grafana/loki/tree/master/fluentd/fluent-plugin-grafana-loki/docker/conf). By default, fluentd containers use the configurations but you can also specify your `fluentd.conf` with `FLUENTD_CONF` environment variable. |
||||
|
||||
This image also uses `LOKI_URL`, `LOKI_USERNAME`, and `LOKI_PASSWORD` environment variables to specify the Loki's endpoint, user, and password (you can leave the USERNAME and PASSWORD blank if they're not used). |
||||
|
||||
### Example |
||||
|
||||
A Docker Compose configuration that will work looks like: |
||||
|
||||
``` |
||||
services: |
||||
fluentd: |
||||
image: grafana/fluent-plugin-grafana-loki:master |
||||
command: |
||||
- "fluentd" |
||||
- "-v" |
||||
- "-p" |
||||
- "/fluentd/plugins" |
||||
environment: |
||||
LOKI_URL: http://loki:3100 |
||||
LOKI_USERNAME: |
||||
LOKI_PASSWORD: |
||||
deploy: |
||||
mode: global |
||||
configs: |
||||
- source: loki_config |
||||
target: /fluentd/etc/loki/loki.conf |
||||
networks: |
||||
- loki |
||||
volumes: |
||||
- host_logs:/var/log |
||||
# Needed for journald log ingestion: |
||||
- /etc/machine-id:/etc/machine-id |
||||
- /dev/log:/dev/log |
||||
- /var/run/systemd/journal/:/var/run/systemd/journal/ |
||||
logging: |
||||
options: |
||||
tag: infra.monitoring |
||||
``` |
||||
|
||||
## Configuration |
||||
|
||||
### url |
||||
The url of the Loki server to send logs to. When sending data the publish path (`/api/prom/push`) will automatically be appended. |
||||
By default the url is set to `https://logs-us-west1.grafana.net`, the url of the Grafana Labs preview (hosted Loki)[https://grafana.com/loki] service. |
||||
|
||||
#### Proxy Support |
||||
|
||||
Starting with version 0.8.0, this gem uses excon, which supports proxy with environment variables - https://github.com/excon/excon#proxy-support |
||||
|
||||
### username / password |
||||
Specify a username and password if the Loki server requires authentication. |
||||
If using the GrafanaLab's hosted Loki, the username needs to be set to your instanceId and the password should be a Grafana.com api key. |
||||
|
||||
### tenant |
||||
Loki is a multi-tenant log storage platform and all requests sent must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through. The tenant can be any string value. |
||||
|
||||
### client certificate verification |
||||
Specify a pair of client certificate and private key with `cert` and `key` if a reverse proxy with client certificate verification is configured in front of Loki. `ca_cert` can also be specified if the server uses custom certificate authority. |
||||
|
||||
``` |
||||
<match **> |
||||
@type loki |
||||
|
||||
url "https://loki" |
||||
|
||||
cert /path/to/certificate.pem |
||||
key /path/to/key.key |
||||
ca_cert /path/to/ca.pem |
||||
|
||||
... |
||||
</match> |
||||
``` |
||||
|
||||
### output format |
||||
Loki is intended to index and group log streams using only a small set of labels. It is not intended for full-text indexing. When sending logs to Loki the majority of log message will be sent as a single log "line". |
||||
|
||||
There are few configurations settings to control the output format. |
||||
- extra_labels: (default: nil) set of labels to include with every Loki stream. eg `{"env":"dev", "datacenter": "dc1"}` |
||||
- remove_keys: (default: nil) comma separated list of needless record keys to remove. All other keys will be placed into the log line. You can use [record_accessor syntax](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax). |
||||
- line_format: format to use when flattening the record to a log line. Valid values are "json" or "key_value". If set to "json" the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to "key_value", the log line will be each item in the record concatenated together (separated by a single space) in the format `<key>=<value>`. |
||||
- drop_single_key: if set to true and a record only has 1 key after extracting `<label></label>` blocks, set the log line to the value and discard the key. |
||||
|
||||
### Buffer options |
||||
|
||||
`fluentd-plugin-loki` extends [Fluentd's builtin Output plugin](https://docs.fluentd.org/v1.0/articles/output-plugin-overview) and use `compat_parameters` plugin helper. It adds the following options: |
||||
|
||||
``` |
||||
buffer_type memory |
||||
flush_interval 10s |
||||
retry_limit 17 |
||||
retry_wait 1.0 |
||||
num_threads 1 |
||||
``` |
||||
Loading…
Reference in new issue