mirror of https://github.com/grafana/loki
Unite docs for fluentd plugin (#1634)
* Use original README.md for fluentd plugin explanations `doc/client/fluentd.m` includes obsolete explanations and it confuses new users. For example, label_keys option was removed at https://github.com/grafana/loki/pull/1186 but `fluentd.md` has the explanation. It's simple to maitain `fluentd/fluent-plugin-grafana-loki/README.md` for the explantion of the output plugin and use the link when we share how to use it. * Remove `label_keys` option and simplify usage for `drop_single_key` * Fix environment variable: FLUENT_CONFIG -> FLUENTD_CONFIG `docker-compose up` fails due to the wrong environment variable specifying the fluentd config. As a result this causes the following error ``` docker-compose up --build WARNING: The LOKI_USERNAME variable is not set. Defaulting to a blank string. WARNING: The LOKI_PASSWORD variable is not set. Defaulting to a blank string. Building fluentd Step 1/17 : FROM fluent/fluentd:v1.3.2-debian ---> 4790aaf4d1e5 Step 2/17 : USER root ---> Using cache ---> 7b54da6cec9e Step 3/17 : WORKDIR /home/fluent ---> Using cache ---> a161d2fcf64b Step 4/17 : ENV PATH /fluentd/vendor/bundle/ruby/2.3.0/bin:$PATH ---> Using cache ---> b67529aad0b5 Step 5/17 : ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.3.0 ---> Using cache ---> 3a8b05beb185 Step 6/17 : ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.3.0 ---> Using cache ---> d68a94345cb4 Step 7/17 : ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1 ---> Using cache ---> 70f9bbf646e8 Step 8/17 : COPY docker/Gemfile* /fluentd/ ---> Using cache ---> 5f8b2f6ffe1a Step 9/17 : RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" && apt-get update && apt-get install -y --no-install-recommends $buildDeps libsystemd0 net-tools libjemalloc1 && gem install bundler --version 1.16.2 && bundle config silence_root_warning true && bundle install --gemfile=/fluentd/Gemfile --path=/fluentd/vendor/bundle && sudo gem sources --clear-all && SUDO_FORCE_REMOVE=yes apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $buildDeps && rm -rf /var/lib/apt/lists/* /home/fluent/.gem/ruby/2.3.0/cache/*.gem /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem ---> Using cache ---> 36c960a53c2c Step 10/17 : COPY docker/entrypoint.sh /fluentd/entrypoint.sh ---> Using cache ---> 7f8d72ae63ca Step 11/17 : COPY lib/fluent/plugin/out_loki.rb /fluentd/plugins/out_loki.rb ---> Using cache ---> d09473ee4f25 Step 12/17 : COPY docker/conf/ /fluentd/etc/loki/ ---> Using cache ---> 00d0439ef3c8 Step 13/17 : ENV FLUENTD_CONF="/fluentd/etc/loki/fluentd.conf" ---> Using cache ---> 2d33a002baaf Step 14/17 : ENV FLUENTD_OPT="" ---> Using cache ---> c314df064756 Step 15/17 : ENV LOKI_URL "https://logs-us-west1.grafana.net" ---> Using cache ---> df78c829d416 Step 16/17 : ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so.1" ---> Using cache ---> 3d7d14cf37f5 Step 17/17 : ENTRYPOINT ["/fluentd/entrypoint.sh"] ---> Using cache ---> 82c70c2354ac Successfully built 82c70c2354ac Successfully tagged fluentd:loki Starting docker_fluentd_1 ... done Attaching to docker_fluentd_1 fluentd_1 | /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in `initialize': No such file or directory @ rb_sysopen - /fluentd/etc/loki/fluentd.conf (Errno::ENOENT) fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in `open' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:769:in `read_config' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:479:in `run_supervisor' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/lib/fluent/command/fluentd.rb:314:in `<top (required)>' fluentd_1 | from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' fluentd_1 | from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.6.2/bin/fluentd:8:in `<top (required)>' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in `load' fluentd_1 | from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in `<main>' docker_fluentd_1 exited with code 1 ``` * Use docs/clients/fluentd/README.md for fluent-plugin-grafana-loki explanation * Arrange list order based on number * Revert "Fix environment variable: FLUENT_CONFIG -> FLUENTD_CONFIG" This reverts commit 8ed58ac97a82d5536321fb9a76120fed766ae506. * Elaborate the way to use docker image * Fix grammar: remove unnecessary with * Add disclaimer for multi-worker usagepull/1664/head
Takayuki Watanabe
5 years ago
committed by
GitHub
parent
e18810949f
commit
973b96d094
@ -1,179 +0,0 @@ |
|||||||
# Fluentd |
|
||||||
|
|
||||||
Loki has a [Fluentd](https://www.fluentd.org/) output plugin called |
|
||||||
`fluent-plugin-grafana-loki` that enables shipping logs to a private Loki |
|
||||||
instance or [Grafana Cloud](https://grafana.com/oss/loki). |
|
||||||
|
|
||||||
The plugin offers two line formats and uses protobuf to send compressed data to |
|
||||||
Loki. |
|
||||||
|
|
||||||
Key features: |
|
||||||
|
|
||||||
* `extra_labels`: Labels to be added to every line of a log file, useful for |
|
||||||
designating environments |
|
||||||
* `label_keys`: Customizable list of keys for stream labels |
|
||||||
* `line_format`: Format to use when flattening the record to a log line (`json` |
|
||||||
or `key_value`). |
|
||||||
|
|
||||||
## Installation |
|
||||||
|
|
||||||
```bash |
|
||||||
$ gem install fluent-plugin-grafana-loki |
|
||||||
``` |
|
||||||
|
|
||||||
## Usage |
|
||||||
|
|
||||||
In your Fluentd configuration, use `@type loki`. Additional configuration is |
|
||||||
optional. Default values look like this: |
|
||||||
|
|
||||||
``` |
|
||||||
<match **> |
|
||||||
@type loki |
|
||||||
url "https://logs-us-west1.grafana.net" |
|
||||||
username "#{ENV['LOKI_USERNAME']}" |
|
||||||
password "#{ENV['LOKI_PASSWORD']}" |
|
||||||
extra_labels {"env":"dev"} |
|
||||||
flush_interval 10s |
|
||||||
flush_at_shutdown true |
|
||||||
buffer_chunk_limit 1m |
|
||||||
</match> |
|
||||||
``` |
|
||||||
|
|
||||||
### Multi-worker usage |
|
||||||
|
|
||||||
Loki doesn't currently support out-of-order inserts - if you try to insert a log |
|
||||||
entry with an earlier timestamp after a log entry with with identical labels but |
|
||||||
a later timestamp, the insert will fail with the message |
|
||||||
`HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of |
|
||||||
order`. Therefore, in order to use this plugin in a multi-worker Fluentd setup, |
|
||||||
you'll need to include the worker ID in the labels sent to Loki. |
|
||||||
|
|
||||||
For example, using |
|
||||||
[fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier): |
|
||||||
|
|
||||||
``` |
|
||||||
<filter mytag> |
|
||||||
@type record_modifier |
|
||||||
<record> |
|
||||||
fluentd_worker "#{worker_id}" |
|
||||||
</record> |
|
||||||
</filter> |
|
||||||
|
|
||||||
<match mytag> |
|
||||||
@type loki |
|
||||||
# ... |
|
||||||
label_keys "fluentd_worker" |
|
||||||
# ... |
|
||||||
</match> |
|
||||||
``` |
|
||||||
|
|
||||||
## Docker Image |
|
||||||
|
|
||||||
There is a Docker image `grafana/fluent-plugin-grafana-loki:master` which |
|
||||||
contains default configuration files to git log information |
|
||||||
a host's `/var/log` directory, and from the host's journald. To use it, you can set |
|
||||||
the `LOKI_URL`, `LOKI_USERNAME`, and `LOKI_PASSWORD` environment variables |
|
||||||
(`LOKI_USERNAME` and `LOKI_PASSWORD` can be left blank if Loki is not protected |
|
||||||
behind an authenticating proxy). |
|
||||||
|
|
||||||
An example Docker Swarm Compose configuration looks like: |
|
||||||
|
|
||||||
```yaml |
|
||||||
services: |
|
||||||
fluentd: |
|
||||||
image: grafana/fluent-plugin-grafana-loki:master |
|
||||||
command: |
|
||||||
- "fluentd" |
|
||||||
- "-v" |
|
||||||
- "-p" |
|
||||||
- "/fluentd/plugins" |
|
||||||
environment: |
|
||||||
LOKI_URL: http://loki:3100 |
|
||||||
LOKI_USERNAME: |
|
||||||
LOKI_PASSWORD: |
|
||||||
deploy: |
|
||||||
mode: global |
|
||||||
configs: |
|
||||||
- source: loki_config |
|
||||||
target: /fluentd/etc/loki/loki.conf |
|
||||||
networks: |
|
||||||
- loki |
|
||||||
volumes: |
|
||||||
- host_logs:/var/log |
|
||||||
# Needed for journald log ingestion: |
|
||||||
- /etc/machine-id:/etc/machine-id |
|
||||||
- /dev/log:/dev/log |
|
||||||
- /var/run/systemd/journal/:/var/run/systemd/journal/ |
|
||||||
logging: |
|
||||||
options: |
|
||||||
tag: infra.monitoring |
|
||||||
``` |
|
||||||
|
|
||||||
## Configuration |
|
||||||
|
|
||||||
### Proxy Support |
|
||||||
|
|
||||||
Starting with version 0.8.0, this gem uses `excon`, which supports proxy with |
|
||||||
environment variables - https://github.com/excon/excon#proxy-support |
|
||||||
|
|
||||||
### `url` |
|
||||||
|
|
||||||
The URL of the Loki server to send logs to. When sending data the publish path |
|
||||||
(`/loki/api/v1/push`) will automatically be appended. By default the URL is set to |
|
||||||
`https://logs-us-west1.grafana.net`, the URL of the Grafana Labs [hosted |
|
||||||
Loki](https://grafana.com/loki) service. |
|
||||||
|
|
||||||
### `username` / `password` |
|
||||||
|
|
||||||
Specify a username and password if the Loki server requires authentication. |
|
||||||
If using the Grafana Labs' hosted Loki, the username needs to be set to your |
|
||||||
instanceId and the password should be a grafana.com API Key. |
|
||||||
|
|
||||||
### `tenant` |
|
||||||
|
|
||||||
Loki is a multi-tenant log storage platform and all requests sent must include a |
|
||||||
tenant. For some installations (like Hosted Loki) the tenant will be set |
|
||||||
automatically by an authenticating proxy. Otherwise you can define a tenant to |
|
||||||
be passed through. The tenant can be any string value. |
|
||||||
|
|
||||||
### output format |
|
||||||
|
|
||||||
Loki is intended to index and group log streams using only a small set of |
|
||||||
labels and is not intended for full-text indexing. When sending logs to Loki, |
|
||||||
the majority of log message will be sent as a single log "line". |
|
||||||
|
|
||||||
There are few configurations settings to control the output format: |
|
||||||
|
|
||||||
- `extra_labels`: (default: nil) set of labels to include with every Loki |
|
||||||
stream. (e.g., `{"env":"dev", "datacenter": "dc1"}`) |
|
||||||
|
|
||||||
- `remove_keys`: (default: nil) comma separated list of record keys to |
|
||||||
remove. All other keys will be placed into the log line. |
|
||||||
|
|
||||||
- `label_keys`: (default: "job,instance") comma separated list of keys to use as |
|
||||||
stream labels. |
|
||||||
|
|
||||||
- `line_format`: format to use when flattening the record to a log line. Valid |
|
||||||
values are `json` or `key_value`. If set to `json` the log line sent to Loki |
|
||||||
will be the fluentd record (excluding any keys extracted out as labels) dumped |
|
||||||
as json. If set to `key_value`, the log line will be each item in the record |
|
||||||
concatenated together (separated by a single space) in the format |
|
||||||
`<key>=<value>`. |
|
||||||
|
|
||||||
- `drop_single_key`: if set to true, when the set of extracted `label_keys` |
|
||||||
after dropping with `remove_keys`, the log line sent to Loki will just be |
|
||||||
the value of the single remaining record key. |
|
||||||
|
|
||||||
### Buffer options |
|
||||||
|
|
||||||
`fluentd-plugin-loki` extends [Fluentd's builtin Output |
|
||||||
plugin](https://docs.fluentd.org/v1.0/articles/output-plugin-overview) and uses |
|
||||||
the `compat_parameters` plugin helper. It adds the following options: |
|
||||||
|
|
||||||
``` |
|
||||||
buffer_type memory |
|
||||||
flush_interval 10s |
|
||||||
retry_limit 17 |
|
||||||
retry_wait 1.0 |
|
||||||
num_threads 1 |
|
||||||
``` |
|
||||||
@ -0,0 +1,211 @@ |
|||||||
|
# Fluentd |
||||||
|
|
||||||
|
Loki has a [Fluentd](https://www.fluentd.org/) output plugin called |
||||||
|
`fluent-plugin-grafana-loki` that enables shipping logs to a private Loki |
||||||
|
instance or [Grafana Cloud](https://grafana.com/oss/loki). |
||||||
|
|
||||||
|
## Installation |
||||||
|
|
||||||
|
``` |
||||||
|
$ gem install fluent-plugin-grafana-loki |
||||||
|
``` |
||||||
|
|
||||||
|
## Usage |
||||||
|
In your Fluentd configuration, use `@type loki`. Additional configuration is optional, default values would look like this: |
||||||
|
``` |
||||||
|
<match **> |
||||||
|
@type loki |
||||||
|
url "https://logs-us-west1.grafana.net" |
||||||
|
username "#{ENV['LOKI_USERNAME']}" |
||||||
|
password "#{ENV['LOKI_PASSWORD']}" |
||||||
|
extra_labels {"env":"dev"} |
||||||
|
flush_interval 10s |
||||||
|
flush_at_shutdown true |
||||||
|
buffer_chunk_limit 1m |
||||||
|
</match> |
||||||
|
``` |
||||||
|
|
||||||
|
### Using labels |
||||||
|
|
||||||
|
Simple label from top level attribute |
||||||
|
``` |
||||||
|
<match mytag> |
||||||
|
@type loki |
||||||
|
# ... |
||||||
|
<label> |
||||||
|
fluentd_worker |
||||||
|
</label> |
||||||
|
# ... |
||||||
|
</match> |
||||||
|
``` |
||||||
|
|
||||||
|
You can rewrite the label keys as well as the following |
||||||
|
|
||||||
|
``` |
||||||
|
<match mytag> |
||||||
|
@type loki |
||||||
|
# ... |
||||||
|
<label> |
||||||
|
worker fluentd_worker |
||||||
|
</label> |
||||||
|
# ... |
||||||
|
</match> |
||||||
|
``` |
||||||
|
|
||||||
|
You can use record accessor syntax for nested field. https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax |
||||||
|
|
||||||
|
``` |
||||||
|
<match mytag> |
||||||
|
@type loki |
||||||
|
# ... |
||||||
|
<label> |
||||||
|
container $.kubernetes.container |
||||||
|
</label> |
||||||
|
# ... |
||||||
|
</match> |
||||||
|
``` |
||||||
|
|
||||||
|
### Extracting Kubernetes labels |
||||||
|
|
||||||
|
As Kubernetes labels are a list of nested key-value pairs there is a separate option to extract them. |
||||||
|
Note that special characters like "`. - /`" will be overwritten with `_`. |
||||||
|
Use with the `remove_keys kubernetes` option to eliminate metadata from the log. |
||||||
|
``` |
||||||
|
<match mytag> |
||||||
|
@type loki |
||||||
|
# ... |
||||||
|
extract_kubernetes_labels true |
||||||
|
remove_keys kubernetes |
||||||
|
<label> |
||||||
|
container $.kubernetes.container |
||||||
|
</label> |
||||||
|
# ... |
||||||
|
</match> |
||||||
|
``` |
||||||
|
|
||||||
|
### Multi-worker usage |
||||||
|
|
||||||
|
Loki doesn't currently support out-of-order inserts - if you try to insert a log entry an earlier timestamp after a log entry with identical labels but a later timestamp, the insert will fail with `HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of order`. Therefore, in order to use this plugin in a multi worker Fluentd setup, you'll need to include the worker ID in the labels or otherwise [ensure log streams are always sent to the same worker](https://docs.fluentd.org/deployment/multi-process-workers#less-than-worker-n-greater-than-directive). |
||||||
|
|
||||||
|
For example, using [fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier): |
||||||
|
``` |
||||||
|
<filter mytag> |
||||||
|
@type record_modifier |
||||||
|
<record> |
||||||
|
fluentd_worker "#{worker_id}" |
||||||
|
</record> |
||||||
|
</filter> |
||||||
|
|
||||||
|
<match mytag> |
||||||
|
@type loki |
||||||
|
# ... |
||||||
|
<label> |
||||||
|
fluentd_worker |
||||||
|
</label> |
||||||
|
# ... |
||||||
|
</match> |
||||||
|
``` |
||||||
|
|
||||||
|
### Using multiple buffer flush threads |
||||||
|
|
||||||
|
Similarly, when using `flush_thread_count` > 1 in the [`buffer`](https://docs.fluentd.org/configuration/buffer-section#flushing-parameters) |
||||||
|
section, a thread identifier must be added as a label to ensure that log chunks flushed in parallel to loki by fluentd always have increasing |
||||||
|
times for their unique label sets. |
||||||
|
|
||||||
|
This plugin automatically adds a `fluentd_thread` label with the name of the buffer flush thread when `flush_thread_count` > 1. |
||||||
|
|
||||||
|
## Docker Image |
||||||
|
|
||||||
|
There is a Docker image `grafana/fluent-plugin-grafana-loki:master` which contains [default configuration files](https://github.com/grafana/loki/tree/master/fluentd/fluent-plugin-grafana-loki/docker/conf). By default, fluentd containers use the configurations but you can also specify your `fluentd.conf` with `FLUENTD_CONF` environment variable. |
||||||
|
|
||||||
|
This image also uses `LOKI_URL`, `LOKI_USERNAME`, and `LOKI_PASSWORD` environment variables to specify the Loki's endpoint, user, and password (you can leave the USERNAME and PASSWORD blank if they're not used). |
||||||
|
|
||||||
|
### Example |
||||||
|
|
||||||
|
A Docker Compose configuration that will work looks like: |
||||||
|
|
||||||
|
``` |
||||||
|
services: |
||||||
|
fluentd: |
||||||
|
image: grafana/fluent-plugin-grafana-loki:master |
||||||
|
command: |
||||||
|
- "fluentd" |
||||||
|
- "-v" |
||||||
|
- "-p" |
||||||
|
- "/fluentd/plugins" |
||||||
|
environment: |
||||||
|
LOKI_URL: http://loki:3100 |
||||||
|
LOKI_USERNAME: |
||||||
|
LOKI_PASSWORD: |
||||||
|
deploy: |
||||||
|
mode: global |
||||||
|
configs: |
||||||
|
- source: loki_config |
||||||
|
target: /fluentd/etc/loki/loki.conf |
||||||
|
networks: |
||||||
|
- loki |
||||||
|
volumes: |
||||||
|
- host_logs:/var/log |
||||||
|
# Needed for journald log ingestion: |
||||||
|
- /etc/machine-id:/etc/machine-id |
||||||
|
- /dev/log:/dev/log |
||||||
|
- /var/run/systemd/journal/:/var/run/systemd/journal/ |
||||||
|
logging: |
||||||
|
options: |
||||||
|
tag: infra.monitoring |
||||||
|
``` |
||||||
|
|
||||||
|
## Configuration |
||||||
|
|
||||||
|
### url |
||||||
|
The url of the Loki server to send logs to. When sending data the publish path (`/api/prom/push`) will automatically be appended. |
||||||
|
By default the url is set to `https://logs-us-west1.grafana.net`, the url of the Grafana Labs preview (hosted Loki)[https://grafana.com/loki] service. |
||||||
|
|
||||||
|
#### Proxy Support |
||||||
|
|
||||||
|
Starting with version 0.8.0, this gem uses excon, which supports proxy with environment variables - https://github.com/excon/excon#proxy-support |
||||||
|
|
||||||
|
### username / password |
||||||
|
Specify a username and password if the Loki server requires authentication. |
||||||
|
If using the GrafanaLab's hosted Loki, the username needs to be set to your instanceId and the password should be a Grafana.com api key. |
||||||
|
|
||||||
|
### tenant |
||||||
|
Loki is a multi-tenant log storage platform and all requests sent must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through. The tenant can be any string value. |
||||||
|
|
||||||
|
### client certificate verification |
||||||
|
Specify a pair of client certificate and private key with `cert` and `key` if a reverse proxy with client certificate verification is configured in front of Loki. `ca_cert` can also be specified if the server uses custom certificate authority. |
||||||
|
|
||||||
|
``` |
||||||
|
<match **> |
||||||
|
@type loki |
||||||
|
|
||||||
|
url "https://loki" |
||||||
|
|
||||||
|
cert /path/to/certificate.pem |
||||||
|
key /path/to/key.key |
||||||
|
ca_cert /path/to/ca.pem |
||||||
|
|
||||||
|
... |
||||||
|
</match> |
||||||
|
``` |
||||||
|
|
||||||
|
### output format |
||||||
|
Loki is intended to index and group log streams using only a small set of labels. It is not intended for full-text indexing. When sending logs to Loki the majority of log message will be sent as a single log "line". |
||||||
|
|
||||||
|
There are few configurations settings to control the output format. |
||||||
|
- extra_labels: (default: nil) set of labels to include with every Loki stream. eg `{"env":"dev", "datacenter": "dc1"}` |
||||||
|
- remove_keys: (default: nil) comma separated list of needless record keys to remove. All other keys will be placed into the log line. You can use [record_accessor syntax](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax). |
||||||
|
- line_format: format to use when flattening the record to a log line. Valid values are "json" or "key_value". If set to "json" the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to "key_value", the log line will be each item in the record concatenated together (separated by a single space) in the format `<key>=<value>`. |
||||||
|
- drop_single_key: if set to true and a record only has 1 key after extracting `<label></label>` blocks, set the log line to the value and discard the key. |
||||||
|
|
||||||
|
### Buffer options |
||||||
|
|
||||||
|
`fluentd-plugin-loki` extends [Fluentd's builtin Output plugin](https://docs.fluentd.org/v1.0/articles/output-plugin-overview) and use `compat_parameters` plugin helper. It adds the following options: |
||||||
|
|
||||||
|
``` |
||||||
|
buffer_type memory |
||||||
|
flush_interval 10s |
||||||
|
retry_limit 17 |
||||||
|
retry_wait 1.0 |
||||||
|
num_threads 1 |
||||||
|
``` |
||||||
Loading…
Reference in new issue