apitech
/
loki
mirror of https://github.com/grafana/loki
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Improve fluentd plugin development experience (#2179)

Browse Source 
* Add Gemfile.lock and .rspec_status to .gitignore

* Converge gem definition in gemspec file

* Add missing `bundle exec` to README

* s/FluentD/Fluentd/g
k22
Takayuki Watanabe 5 years ago committed by GitHub
parent c3d3f2ba77
commit 59f312545c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 17 additions and 15 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 8
      CHANGELOG.md
  2. 4
      cmd/fluentd/.gitignore
  3. 6
      cmd/fluentd/Gemfile
  4. 6
      cmd/fluentd/README.md
  5. 6
      cmd/fluentd/fluent-plugin-grafana-loki.gemspec
  6. 2
      docs/best-practices/current-best-practices.md

8
CHANGELOG.md
Unescape View File

@ -183,7 +183,7 @@ We now GPG sign helm packages!
* [1906](https://github.com/grafana/loki/pull/1906) **cyriltovena**: Add no-file and keep-file log option for docker driver.
* [1903](https://github.com/grafana/loki/pull/1903) **cyriltovena**: Log docker driver config map.
#### FluentD
#### Fluentd
* [2074](https://github.com/grafana/loki/pull/2074) **osela**: fluentd plugin: support placeholders in tenant field
* [2006](https://github.com/grafana/loki/pull/2006) **Skeen**: fluent-plugin-loki: Restructuring and CI
* [1909](https://github.com/grafana/loki/pull/1909) **jgehrcke**: fluentd loki plugin README: add note about labels
@ -330,7 +330,7 @@ These two configs let you set the max time a chunk can stay in memory in Loki, t
This isn't a feature but it's an important fix, this is the second time our canaries have tried to DDOS our Loki clusters so you should update to prevent them from trying to attack you. Aggressive little things these canaries...
* [1840](https://github.com/grafana/loki/pull/1840) **slim-bean**: promtail: Retry 429 rate limit errors from Loki, increase default retry limits
* [1845](https://github.com/grafana/loki/pull/1845) **wardbekker**: throw exceptions on HTTPTooManyRequests and HTTPServerError so FluentD will retry
* [1845](https://github.com/grafana/loki/pull/1845) **wardbekker**: throw exceptions on HTTPTooManyRequests and HTTPServerError so Fluentd will retry
These two PR's change how 429 HTTP Response codes are handled (Rate Limiting), previously these responses were dropped, now they will be retried for these clients
@ -442,7 +442,7 @@ There are many other important fixes and improvements to Loki, way too many to c
* [1791](https://github.com/grafana/loki/pull/1791) **cyriltovena**: Improve fluentbit logfmt.
* [1717](https://github.com/grafana/loki/pull/1717) **adityacs**: Fluent-bit: Fix panic error when AutoKubernetesLabels is true
#### FluentD
#### Fluentd
* [1811](https://github.com/grafana/loki/pull/1811) **JamesJJ**: Error handling: Show data stream at "debug" level, not "warn"
* [1728](https://github.com/grafana/loki/pull/1728) **irake99**: docs: fix outdated link to fluentd
* [1703](https://github.com/grafana/loki/pull/1703) **Skeen**: fluent-plugin-grafana-loki: Update fluentd base image to current images (edge)
@ -606,7 +606,7 @@ Once again we can't thank our community and contributors enough for the signific
* [1294](https://github.com/grafana/loki/pull/1294) **JensErat**: fluent-bit: multi-instance support
* [1514](https://github.com/grafana/loki/pull/1514) **shane-axiom**: fluent-plugin-grafana-loki: Add `fluentd_thread` label when `flush_thread_count` > 1
#### FluentD
#### Fluentd
* [1500](https://github.com/grafana/loki/pull/1500) **cyriltovena**: Bump fluentd plugin to 1.2.6.
* [1475](https://github.com/grafana/loki/pull/1475) **Horkyze**: fluentd-plugin: call gsub for strings only

4
cmd/fluentd/.gitignore vendored
Unescape View File

@ -1 +1,3 @@
coverage/
/coverage/
/.rspec_status
/Gemfile.lock

6
cmd/fluentd/Gemfile
Unescape View File

@ -2,9 +2,5 @@
source 'https://rubygems.org'
# Specify your gem's dependencies in fluent-plugin-grafana-loki.gemspec
gemspec
gem 'fluentd', '1.9.3'
gem 'rubocop-rspec'
gem 'simplecov', require: false, group: :test
gem 'test-unit'

6
cmd/fluentd/README.md
Unescape View File

@ -1,4 +1,4 @@
# FluentD output plugin
# Fluentd output plugin
[Fluentd](https://fluentd.org/) is a data collector for unified logging layer, it can be configured with the Loki output plugin, provided in this folder, to ship logs to Loki.
@ -6,7 +6,7 @@ See [docs/client/fluentd/README.md](../../docs/clients/fluentd/README.md) for de
## Development
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
After checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `fluent-plugin-grafana-loki.gemspec`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
@ -44,7 +44,7 @@ The expected output is:
]
```
Start FluentBit + FluentD using:
Start FluentBit + Fluentd using:
```
LOKI_URL=http://{{ IP }}:3100 make fluentd-test
```

6
cmd/fluentd/fluent-plugin-grafana-loki.gemspec
Unescape View File

@ -25,8 +25,12 @@ Gem::Specification.new do |spec|
# spec.test_files = test_files
# spec.require_paths = ['lib']
spec.add_dependency 'fluentd', '1.9.3'
spec.add_development_dependency 'bundler'
spec.add_development_dependency 'rake', '~> 12.0'
spec.add_development_dependency 'rspec', '~> 3.0'
spec.add_runtime_dependency 'fluentd', ['>=1.9.0', '< 2']
spec.add_development_dependency 'rubocop-rspec'
spec.add_development_dependency 'simplecov'
spec.add_development_dependency 'test-unit'
end

2
docs/best-practices/current-best-practices.md
Unescape View File

@ -30,7 +30,7 @@ Try to keep values bounded to as small a set as possible. We don't have perfect
## 4. Be aware of dynamic labels applied by clients
Loki has several client options: [Promtail](https://github.com/grafana/loki/tree/master/docs/clients/promtail) (which also supports systemd journal ingestion and TCP-based syslog ingestion), [FluentD](https://github.com/grafana/loki/tree/master/fluentd/fluent-plugin-grafana-loki), [Fluent Bit](https://github.com/grafana/loki/tree/master/cmd/fluent-bit), a [Docker plugin](https://grafana.com/blog/2019/07/15/lokis-path-to-ga-docker-logging-driver-plugin-support-for-systemd/), and more!
Loki has several client options: [Promtail](https://github.com/grafana/loki/tree/master/docs/clients/promtail) (which also supports systemd journal ingestion and TCP-based syslog ingestion), [Fluentd](https://github.com/grafana/loki/tree/master/fluentd/fluent-plugin-grafana-loki), [Fluent Bit](https://github.com/grafana/loki/tree/master/cmd/fluent-bit), a [Docker plugin](https://grafana.com/blog/2019/07/15/lokis-path-to-ga-docker-logging-driver-plugin-support-for-systemd/), and more!
Each of these come with ways to configure what labels are applied to create log streams. But be aware of what dynamic labels might be applied. Use the Loki series API to get an idea of what your log streams look like and see if there might be ways to reduce streams and cardinality. Details of the Series API can be found [here](https://github.com/grafana/loki/blob/master/docs/api.md#series), or you can use [logcli](https://github.com/grafana/loki/blob/master/docs/getting-started/logcli.md) to query Loki for series information.

Write Preview
Loading…
Cancel
Save