apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Đurica Yuri Nikolić	37ec68eebb	Remove unnecessary code after dskit log.Interface has been dereleased (#10434 ) This PR removes unnecessary code used to convert the dereleased `dslog.Interface` (see https://github.com/grafana/dskit/pull/359) into `go-kit.Logger`. Additionally, it modifies the signature and implementation of `pkg/util/log/log.go`'s `InitLogger` as follows: - previous signature ``` InitLogger(server.Config, prometheus.Registerer, bool, bool) ``` has been replaced with ``` InitLogger(server.Config, prometheus.Registerer, bool, bool) log.Logger ``` Namely, the new `InitLogger` initialises the global logger and returns it. - the new implementation initializes the global logger, but it does not override the default logger for the server. This is now done by the callers of `InitLogger`. Previously, it was done inside `InitLogger` for convenience, since the types of the 2 loggers were not the same. Signed-off-by: Yuri Nikolic <durica.nikolic@grafana.com>	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Bryan Boreham	c3c9ac7740	Cosmetic lint warning fixes (#8363 )	2 years ago
Danny Kopping	d0fd4e8ad1	Do not exit when using `stream_lag_labels` setting (#8243 )	2 years ago
Callum Styan	399eb097fd	Remove the stream lag metric. (#6324 ) Removes the stream lag metric since it's not possible to get a correct value and also not cause issues with metrics registration or collection until we have the WAL Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Pablo	5f665651d5	Promtail: Add basic tracing support (#7414 ) We've been using Promtail's HTTP/gRPC server-based targets with real workloads for some weeks now. The server that these spin out is based on [weaveworks commons](https://github.com/weaveworks/common) HTTP/gRPC servers. These include by default some middlewares that if enabled, inject tracing support. We found this useful when tracking down request handling time, or even more, if the caller also support Jaeger tracing support, cross-correlated requests between the upstream server and Promtail's. This PR adds support for enabling tracing the same way Loki does. At the moment, this will just generate traces for the before-mentioned targets. In the future, the quality and completeness of tracing support can grow organically to enrich spans.	3 years ago
Danny Kopping	33bf36aa21	Promtail: flush logs on exit (#7400 ) Promtail is using the line-buffered logger (https://github.com/grafana/loki/pull/6954), and log messages were not being printed on exit	3 years ago
李国忠	fb26baa5b1	[new feature] promtail: Add config reload endoint / signal to promtail (#7247 )	3 years ago
Hemanth Krishna	0e54477309	area(promtail): add -check-syntax functionality (#7363 ) Adds `-check-syntax` to promtail which will exit just by validating the config file. Fixes #6862. Signed-off-by: Hemanth Krishna <hkpdev008@gmail.com>	3 years ago
Danny Kopping	6bf5b5d8c1	Use a line-buffered logger to deamplify write syscalls (#6954 ) We initialise a global logger in `pkg/util/log/log.go` and use it extensively throughout the Loki codebase. Every time we write a log message, a `write` syscall is invoked. Syscalls are problematic because they transition the process from userspace to kernelspace, which means: - a context-switch is incurred, which is inherently expensive ([1-2 microseconds](https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/)) - the goroutine executing the code is blocked - the underlying OS thread (_M_ in the go scheduler model) is also blocked - the goroutine has to be rescheduled once the syscall exits - the go scheduler may need to spawn additional OS threads if all are blocked in syscalls - which can also be expensive This change introduces the use of a line-buffered logger. It has a buffer of [256](https://gist.github.com/dannykopping/0704db32c0b08751d1d2494efaa734c2) entries, and once that buffer is filled it will flush to disk. However, a situation will arise in which that buffer remains somewhat empty for a period of time, so there is a periodic flush mechanism, configured to flush every 100ms. There is also a preallocated bytes slice of 10MB which is reused, to avoid excessive slice resizing & garbage collection. This does mean that we could lose up to 256 log messages in case of an ungraceful termination of the process, but this would need to be precisely timed within the 100ms flushes - in other words, the likelihood is low, and generally we shouldn't `kill -9` any Loki process.	3 years ago
Karsten Jeschkies	c9f3868287	Update dskit to v0.0.0-20220331160727-49faf69f72ca. (#5881 )	3 years ago
Susana Ferreira	1d1461824d	Move promtail StreamLagLabels config to upper level config.Config (#5686 ) * Move promtail StreamLagLabels config to upper level config.Config * Fix promtail config load test * Fix promtail configuration reference * Add options promtail config section * Kick GitHub CI Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Callum Styan	01d63c8ae8	[promtail] Refactor promtail client metrics so that we can't have duplicate metrics collected for the lag metric. (#5521 ) * Refactor promtail client metrics so that we can't have duplicate metrics collected for the lag metric. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Set up new metrics passing for other client types. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix lint issues. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Update changelog and docs, also rename nested client configs struct to configs. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484 ) * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Ed Welch	11e215d79f	Loki: Common Config (#4347 ) * refactor the pkg/util/cfg package to simplify and make it accessable outside of this package. create a pkg/cfg which handles building the Loki config, this is created separately from pkg/util (which is Apache 2 licensed) to maintain proper separations of AGPL and Apache 2 code Update the main.go for several of the apps with the refactoring * refactor how dynamic/common config is loaded Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Add common config logic for path prefix Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Missed a file, update common config with prefix property Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * fix test, remove loki from args Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Clean up test names Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * fix typo in filename, make linter happy Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Update pkg/loki/config_wrapper.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/loki/config_wrapper.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/loki/config_wrapper_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * remove temp files in test Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Arve Knudsen	cfb4fc1f55	Use flagext from dskit (#4225 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Danny Kopping	ac63ea8500	Promtail: adding pipeline stage inspector (#4011 ) * Adding promtail pipeline stage inspector Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding library Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding docs Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Further appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing old version of go-cmp to pass check-mod step Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Making screenshot narrower and correcting link Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Copy changes Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Copy changes Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Cyril Tovena	bca29c011f	Embed timezone data in Promtail. (#3332 ) Fixes #2995 Fixes #1361 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295 ) * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Keith Byrne	5793c4964e	Configuration: Support environment expansion in configuration (#2837 ) * Support environment expansion in configuration Inspired by https://github.com/cortexproject/cortex/pull/2147 Fixes #2311 Adds -config.expand-env as flag to loki and promtail. * Disable env expansion on logcli * Small nit updates	5 years ago
Owen Diehl	05e0e91d6e	Fix/promtail yaml config (#2471 ) * promtail config cloneable * config parsing compile time checks against cloneable	5 years ago
Ed Welch	0b5996021f	Loki: add -print-config-stderr flag to dump loki's runtime config to stderr (#2336 ) * Loki: add -print-config-stderr flag to dump loki's runtime config to stderr * add ability to print inline config for better readability in grafana * add an option to log the config in reverse order to make viewing in Grafana easier. Also added this capability to Promtail	5 years ago
Cyril Tovena	4210fcef35	--dry-run Promtail. (#1652 ) * --dry-run Promtail. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove changed on gomod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flaky test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Ensure position file is readonly when doing dry-run Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add information about dry run mode in the troubleshooting section. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more spaces. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journaltarget refactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journaltarget refactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journaltarget refactor. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	814cc87eb2	Pipe data to Promtail (#1649 ) * Adds readerTarget that read from io.Reader Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates comment and function names. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds stdin manager Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds default config with labels and support for static configs labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds a test with pipeline stages. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes race on shutdown and adds documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes deadlock on server.run Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/clients/promtail/troubleshooting.md Co-Authored-By: Owen Diehl <ow.diehl@gmail.com> * gofmt Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * gofmt Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * unlock mutex if Promtail.Run is called when Promtail is stopped Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	5 years ago
Peter Štibraný	49d0d0f0da	Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases. (#1095 ) * Extracted -mod=vendor into a variable * Use ?= to set variable only if not set yet. This enables "MOD_VENDOR= make all" use from command line * Enable/disable vendoring with VENDOR=true or false. Enabled by default * Make go generate use -mod=vendor depending on VENDOR flag * Use GOMOD to specify module version to use. GOMOD can be set to empty string, "vendor", "readonly" and in Go 1.14 to "mod". If not defined, defaults to "vendor". Non empty values are passed to -mod parameter to Go tools, and --modules-download-mode parameter for golangci-cli linting tool. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix versioning info when not using vendoring. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated build image to use Golintci-cli v1.21.0 and Go 1.13.4. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated build image to version 0.8.0. Changs: Golintci-cli v1.21.0, Go 1.13.4. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Renamed MOD_VENDOR to MOD_FLAG to better describe its function. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix version info in all command line tools. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use loki-build-image:0.8.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Mstislav Bobakov	149c70a2ca	[promtail] Fix #810 . Set stderr as a default output for klog library (#1180 ) * Fix #810 . Set stderr as a default output for klog library * sync go mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Marco Pracucci	003cf32b9c	promtail: fix error message displaying on invalid config file	6 years ago
sh0rez	00245403d9	feat: configuration source precedence (#980 ) This implements more sophisticated handling of our configuration: - Correct precedence: Flags set on the command line now precede values from YAML, which in turn precede the defaults from the `flag` package - Sticks to the `flagext.Registerer` pattern, as it is partly vendored from Cortex / Weaveworks code we cannot change easily	6 years ago
sh0rez	794d7da845	feat: -version flag (#1036 ) While we were already setting the version on the built binaries, this now also exposes them as a user-facing flag (`-version`). The already used Prometheus package `github.com/prometheus/common/version` is used for generated the printed string.	6 years ago
Edward Welch	a2e2272653	use a global variable to wrap debug log statments in the pipeline to prevent unnecessary allocations when debug logging is not enabled	6 years ago
sh0rez	420a359ff7	fix(loki\|promtail): logger re-init nil config panic (#697 ) When a more or less invalid config (e.g. `null`) is supplied, the log level prop of the config receives an invalid value which causes the logger to panic on re-init. Prevents this by checking for a nil log-level and notifies the user in case	6 years ago
Edward Welch	97e4a122ca	adding a pipeline processing histogram	6 years ago
Cyril Tovena	b8b7caa450	Promtail targets and service discovery pages (#532 ) * add a custom server for loki with assets embedded * adds a targets page for promtail to see discovered files and labels * lint fixes * fix css table size * add service discovery logic * add service discovery template * add promtail pages documentation * ignored gen file in linter	6 years ago
Xiang Dai	00906d2066	promtail: fix wrong config path (#375 ) Signed-off-by: Xiang Dai <764524258@qq.com>	6 years ago
Ed	f6d0857ddf	Adding a sync() function to filetarget (#293 ) * Adding a sync() function to filetarget which runs on a timer to catch any events missed by fsnotify * moving where we mark the final position to avoid a race condition exposed in testing, we want to stop the tailer before shutting down our tailer loop * Fix changes from PR and refactor to remove circular imports and make config consistent * Adding an "integration" type unit test which tests promtail as a whole * fixing a bunch of imports I munged moved tailer to its own file refactoring filetarget to make synclogic easier to follow and reused as much as possible	6 years ago
Christian Simon	31789eafc4	Refactor structure of promtail This allows easier integration of other targets than files for promtail. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Vangelis Katsikaros	22f5968d0d	Correct name in promtail error message (#180 )	7 years ago
Carl Bergquist	5aea1fe408	adds build info to loki and promtail (#172 )	7 years ago
Tom Wilkie	5f73598389	Rename Tempo to Loki. (#36 ) * Rename Tempo to Loki. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use new build image, don't delete generated files on clean. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	600b24101e	Promtail updates (#30 ) * Make promtail read all its config from a file. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use gokit logging everywhere. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Start work to parse normal log lines. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * update common and kuberesolver. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * lint Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Get EntryParser stuff working, so we can tail raw log files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make sure filename label works. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * lint, remove interfacer. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	1fef92bc17	Lint stuff (#27 ) * Check in generated files, to make getting started easier. Add gometalinter, and a lint check for changes to the generated files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make lint pass? Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	d06ed8fb1f	Remove flagext, use cortex's. Allow flags to be loaded from a file. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	0ffb432f30	Update for cortex changes. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	3f62b4ec36	s/logish/tempo/ Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	01ee8313c0	Fix request paths and logging. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	aa44de07bb	Move promtail from kausalco/public, update it for prometheus changes. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago

19 Commits (37ec68eebb533a545765458badd8236033a184f6)