125 Commits (841befaf8eaa58adc953caf10b0b0c3cfb8060b3)

Author	SHA1	Message	Date
Edward Welch	b43039ed5a	update the position and size metrics at the same time to try to end the race between these two metrics and the two timers that care about position and size ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Chance Zibolski	3a2e5d4bbc	pkg/promtail: Initialize counters to 0 when creating client (#1453) ... This ensures metrics are not missing before they're incremented the first time, as recommended by the upstream Prometheus instrumentation guidelines: https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics. Currently, in the case of dropped_entries_total and dropped_bytes_total, the metric is simply missing until the first occurrence of an entry being dropped.	5 years ago
petruha	4f314f3df4	Fileglob parsing fixes (#1485) ... * Update doublestar module to version 1.2.2. * Add absolute path for each element in list. * Remove doublestar test and update vendore module version.	5 years ago
Owen Diehl	0b72ddb6a6	positions.ignore-corruptions (#1472) ... * positions.ignore-corruptions * semantic change: s/ignore-corruptions/ignore-invalid-yaml/ * Update pkg/promtail/positions/positions.go Co-Authored-By: Robert Fratto <robertfratto@gmail.com>	5 years ago
Robert Fratto	874d016bd7	promtail: add support for passing through journal entries as JSON ... This commit adds an extra configuration field (`json`) to the journal target configuration schema. When set to true, messages from the journal target will be passed through as JSON, containing all of the fields from the entry as keys and their values as the value. For backwards compatibility, the value for `json` defaults to false. Closes #1288.	6 years ago
Peter Štibraný	4fea713b1e	pkg/promtail/client: missing URL in client returns error (#1416) ... Previously it would just panic, because URL.Host is used to construct logger. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Sebastian Widmer	f0f6f24926	pkg/promtail: IETF Syslog (RFC5424) Support (#1275) ... * IETF syslog (rfc5254) support * Upgrade go-syslog to correctly released v2.0.1 * Incorporate feedback from new linter * Update documentation with relevant RFC sections. * Incorporate feedback from review * Use context to shutdown server. * Use util.Backoff from cortex * Do not embed mutex into TestLabeledClient * Use strings.HasPrefix * Better naming of connectionsClosed -> openConnections * Incorporate further feedback from review * Callback instead of channel in ParseStream. Removes one running Goroutine per connection. * Improve parse error log level. * Move mutex above field it protects. * Use backoff.Ongoing() * Switch to "nontransparent" parser * Further improvements to the documentation	6 years ago
Cyril Tovena	b34c2b0bd3	Move to jsoniter instead of default json package (#1249) ... * Move to jsoniter instead of default json package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes jsoniter on the logcli output Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Peter Štibraný	49d0d0f0da	Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases. (#1095) ... * Extracted -mod=vendor into a variable * Use ?= to set variable only if not set yet. This enables "MOD_VENDOR= make all" use from command line * Enable/disable vendoring with VENDOR=true or false. Enabled by default * Make go generate use -mod=vendor depending on VENDOR flag * Use GOMOD to specify module version to use. GOMOD can be set to empty string, "vendor", "readonly" and in Go 1.14 to "mod". If not defined, defaults to "vendor". Non empty values are passed to -mod parameter to Go tools, and --modules-download-mode parameter for golangci-cli linting tool. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix versioning info when not using vendoring. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated build image to use Golintci-cli v1.21.0 and Go 1.13.4. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Updated build image to version 0.8.0. Changs: Golintci-cli v1.21.0, Go 1.13.4. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Renamed MOD_VENDOR to MOD_FLAG to better describe its function. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Fix version info in all command line tools. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use loki-build-image:0.8.0 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Cyril Tovena	27f17bc1de	Fixes linter warning from the yacc file. (#1328) ... * Fixes linter warning from the yacc file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves sed to work on POSIX Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Peter Štibraný	125cfbfd5b	pkg/promtail/client: Handle fingerprint hash collisions (#1254) ... * Handle fingerprint hash collisions Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Instead of using fingerprints, use label strings as map keys. Label strings are properly sorted key=value pairs. This solution produces more garbage, but doesn't reinvent its own hashing, and keeps code simple. This seems to be a good tradeoff for promtail. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	9956717383	Include positions filename in the error when YAML unmarshal fails. (#1311) ... * Include positions filename in the error when YAML unmarshal fails. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Include filename when parsing of YAML/JSON file fails. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Robert Fratto	715f215c50	pkg/promtail: remove journal target forced path (#1298) ... * pkg/promtail: remove journal target forced path The forced default path for the journal target (/var/log/journal) did not apply to all Linux systems: systemd actually defaults to reading from _two_ default paths: /var/log/journal and /run/log/journal. Not setting an explicit default will enable both of those paths to be searched for journal logs. The comment mentioning the local machine ID was technically accurate, but is normally dealt with by bind mounting /etc/machine-id. That's the standard approach for running fluent-bit in a container, and should be the standard approach for Promtail as well. Fixes #1286.	6 years ago
Peter Štibraný	fd25e6d180	Write positions to new file first, move to target destination afterwards. (#1304) ... Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Marco Pracucci	9ec3117245	Move ReservedLabelTenantID out from a dedicated file (#1239)	6 years ago
Marco Pracucci	04f58c880f	promtail: add multi-tenant support (#1135) ... * promtail: added tenant_id support to promtail client * promtail: added tenant stage to dinamically override the tenant ID * promtail: documented client's batch struct and improved tenant stage config pointer usage * Added static value support to tenant stage	6 years ago
Srinidhi Kaushik	90b937c743	Update dependencies to use weaveworks/common upstream (#1226)	6 years ago
Cyril Tovena	d438bda94a	Update cortex to bring v11 schema (#1201) ... * Update cortex to bring v11 schema Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes change in ParseProtoReader Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * sync go mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Ed Welch	94b9c4ba16	Improve how we record file size metric to avoid a race in our file lagging alert (#1194) ... * Update the filetarget sync() code to report size based on what the tailer reports for size (or read the file size from the file name if no tailer exists yet) Update vendored (and forked) hpcloud tail library which includes a Size() function which reports size from the file handle and not file name * needed to tidy? * addressing code review comments	6 years ago
Robert Fratto	05f8e564db	Clarify journal warning (#1109)	6 years ago
Joe Elliott	fc6c5c063f	dep => go mod (#1062) ... * go mod files added. dep removed Signed-off-by: Joe Elliott <number101010@gmail.com> * Magically got prometheus version to stick Signed-off-by: Joe Elliott <number101010@gmail.com> * Cortex updated and prometheus updated Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated prometheus dependency Signed-off-by: Joe Elliott <number101010@gmail.com> * Added additional deps Signed-off-by: Joe Elliott <number101010@gmail.com> * Added two replaces from Gopkg.toml. Tests passing Signed-off-by: Joe Elliott <number101010@gmail.com> * Added deps Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated build image to 1.12 Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated golangci-lint to use vendored dependencies Signed-off-by: Joe Elliott <number101010@gmail.com> * Added check-mod makefile step and referenced in drone and circle ci configs Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated loki-build-image to 1.12 Signed-off-by: Joe Elliott <number101010@gmail.com> * Fixed linting error. Force go111module on for linting Signed-off-by: Joe Elliott <number101010@gmail.com> * go mod vendor Signed-off-by: Joe Elliott <number101010@gmail.com> * Forced the inclusion of modtimevfs Signed-off-by: Joe Elliott <number101010@gmail.com> * Pin client-go due to issue with v12 go.mod Signed-off-by: Joe Elliott <number101010@gmail.com> * go mod tidy Signed-off-by: Joe Elliott <number101010@gmail.com> * Added check-mod to drone Signed-off-by: Joe Elliott <number101010@gmail.com> * Re-readded correct golang client Signed-off-by: Joe Elliott <number101010@gmail.com> * go mod tidy Signed-off-by: Joe Elliott <number101010@gmail.com> * Pinned golang/x/net to avoid proxy errors Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed check-mod from all. Not necessary for tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated go.sum to match new pin Signed-off-by: Joe Elliott <number101010@gmail.com> * Upgraded proto to match build image Signed-off-by: Joe Elliott <number101010@gmail.com> * Force check-mod to wait til after test and lint are successful Signed-off-by: Joe Elliott <number101010@gmail.com> * Added mod vendor to go builds to force usage of vendored dependencies Signed-off-by: Joe Elliott <number101010@gmail.com> * Turn on gomodules on all builds Signed-off-by: Joe Elliott <number101010@gmail.com> * Revert "Added mod vendor to go builds to force usage of vendored dependencies" This reverts commit 65865a24c9a23133e0fa52942f2828ead7c22147. * Moved builds out of the gopath to enforce vendor usage Signed-off-by: Joe Elliott <number101010@gmail.com> * Revert "Turn on gomodules on all builds" This reverts commit b5847f0158e928e935e0b3c1b1d4eaba840ca3dc. * Explicitly choose build image for docker driver Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated golang.org/x/sys to match prometheus's version to fix promtail windows compilation Signed-off-by: Joe Elliott <number101010@gmail.com> * Added fluentbit dependency Signed-off-by: Joe Elliott <number101010@gmail.com> * Added dependency management clause Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated go version in contributing Signed-off-by: Joe Elliott <number101010@gmail.com> * Added phony makefile targets Signed-off-by: Joe Elliott <number101010@gmail.com> * Circle CI is increasingly failing linting Signed-off-by: Joe Elliott <number101010@gmail.com> * Force vendored deps on builds Signed-off-by: Joe Elliott <number101010@gmail.com> * Added logcli to gitignore Signed-off-by: Joe Elliott <number101010@gmail.com> * Reduced concurrency to help CircleCI Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved drone builds out of GOPATH to force use of go module Signed-off-by: Joe Elliott <number101010@gmail.com> * Added mod vendor to prevent go clean from redownloading all packages Signed-off-by: Joe Elliott <number101010@gmail.com> * Added to test as well Signed-off-by: Joe Elliott <number101010@gmail.com> * Added mod-vendor to final go commands Signed-off-by: Joe Elliott <number101010@gmail.com> * Moved fluent-bit out of GO_PATH to force go modules Signed-off-by: Joe Elliott <number101010@gmail.com> * Pass mod vendor to the last holdout: go generate Signed-off-by: Joe Elliott <number101010@gmail.com> * Updated ugorji/go to 1.17 to avoid version regression Signed-off-by: Joe Elliott <number101010@gmail.com> * go mod tidy and go mod vendor for ugorji/go Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
Joe Elliott	778c0b6857	Update loki push path (#1049) ... This commit adds a /loki/api/v1/push endpoint to mirror the new v1 API, and updates all references to the old push endpoint to new the one.	6 years ago
Erwin van Eyk	297cc63e5d	Fix externalURL and path prefix issues (#1026)	6 years ago
Marco Pracucci	62d5122a11	Promtail exports metrics on sent and dropped log entries	6 years ago
Cyrill Troxler	6b5180535f	Set default value for BatchWait as ticker does not accept 0	6 years ago
Cyrill Troxler	dca8a4fb17	Use ticker instead of timer for batch wait ... Fixes #280	6 years ago
Marco Pracucci	22a1bd5f45	promtail: do not send the last batch - to ingester - if empty (#934) ... * Do not send the last batch from promtail on graceful shutdown if the batch is empty * Tested promtail Client.Handle()	6 years ago
Robert Fratto	4c418430be	promtail: add "max_age" field to configure cutoff for journal reading (#921) ... The journal scrape config in promtail has been updated to support a "max_age" field. max_age determines the oldest journal entry promtail will read when starting the journal reader. When unspecified, max_age defaults to 7h. Even if a position in the journal is saved in the promtail positions file, that position may be ignored if the entry corresponding to that position is older than the max_age cutoff time. Example promtail config for getting up to the last 12h of journal entries: server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://localhost:3100/api/prom/push scrape_configs: - job_name: journal journal: max_age: 12h path: /var/log/journal labels: job: systemd-journal relabel_configs: - source_labels: ['__journal__systemd_unit'] target_label: 'unit'	6 years ago
Aditya C S	da4964dd70	Add pipeline unit testing to promtail	6 years ago
Robert Fratto	001d36f510	pkg/promtail/positions: remove executable bit from positions file (#893) ... File permissions on creation of positions.yaml will now be 0600 instead of 0700. Fixes #892.	6 years ago
Cyril Tovena	2561a70155	fix tail library logs to use our own log format (#579) ... * fix tail library logs to use our own log format * PR Feedbacks	6 years ago
Robert Fratto	f2bec3b0a5	promtail: restore ability to show target labels in promtail UI ... PR #791 accidentally removed the ability to use the promtail UI by removing methods from the Target interface that were only used within the HTML templates. Along with restoring the methods in the Target interface, this commit also introduces details for the JournalTarget, which currently only provides the position in the journal being tracked.	6 years ago
Edward Welch	7deb02b532	add a Name() method to the stage interface so that debug logging can show you the name of the pipeline stage which just processed the log ... remove some unnecessary logging around fsnotify events we don't care about and saving positions Make the Processing Log Lines doc a first class citizen, I reference this a lot and currently it's hidden behind 3 clicks	6 years ago
Robert Fratto	9eb3098dea	promtail: Add systemd journal support (#730) ... Support for reading systemd journal entries has been added. promtail will look for a job in scrape_configs with a journal key to activate the journal target. If GOOS=linux and CGO_ENABLED=1, promtail will now require libsystemd headers to be available for building. If GOOS is not linux or CGO_ENABLED is not 1, journal support will be unavailable and a log message will be printed warning the user that their config file has journal tailing configured without it being built into promtail. See docs/promtail-examples.md for a concrete example of using journal support. Other structural changes made: 1. Ability for checking if scrape.Config.ServiceDiscoveryConfig is non-zero has been added. This was chosen over making ServiceDiscoveryConfig a pointer as yaml.v2 cannot parse an inline struct into a pointer value. 2. Updated pipeline logger component name to journal_pipeline and file_pipeline for JournalTargetManager and FileTargetManager respectively. 3. The positions file will now store positions as strings instead of integers. Existing positions will be read in properly but written out as strings the next time the positions file is saved. This is done to be able to store the journal cursor, which is a string. The positions API has been updated to support reading in the old integer values and the new string values.	6 years ago
Marco Pracucci	f7cc7b035e	Redirect / to /targets in promtail server	6 years ago
Edward Welch	922b2d02cd	changing the log length histogram to be a normal histogram with only a `path` label, also removing the label values when no longer tailing the file. Following the same pattern we used for readBytes and totalBytes.	6 years ago
Cyril Tovena	be6664892f	fix docker plugin ci (#664) ... * fix docker plugin ci * use binary cache	6 years ago
Cyril Tovena	2909fe54e1	Docker Logging Driver (#663) ... * adds first version of docker driver * without logrus and fixes some linter issue * fix the driver and start a build system * adds swarm label discovery * Add documentation and more targets * make the linter happy ❤️ * indent config.json * with circleci steps for master and branch * Review Feebacks	6 years ago
sh0rez	11ad356396	feat(promtail): initContainers (#655) ... * fix(promtail/targets): remove dependency on prometheus/relabel `filetargetmanager.go` had a dependency on prometheus/relabel, a package that has been removed in favor of prometheus/pkg/relabel. This converts the code to the new package, to allow the prometheus vendor be updated to master * chore(vendor): update prometheus vendor Updates prometheus vendor to current master, to add support for InitContainers in kubernetes service discovery * chore(vendor): do dep's homework Gives dep hints on how to resolve the vendor so that it work * fix(promtail/targets): non-nil check Accidentally checked against the wrong labelSet, this one can never get nil	6 years ago
Xiang Dai	3021d6f29a	Remove 404 link (#637) ... Signed-off-by: Xiang Dai <764524258@qq.com>	6 years ago
Cyril Tovena	2a7a97d052	Remove LogCount	6 years ago
Edward Welch	3a2ac64b7c	cleaning up remaining TODO's, adding tests ... cleaning up GoDoc pipeline_name is now optional for matcher stage	6 years ago
Edward Welch	29eaec1a04	implementing all the functions for the counter and gauge metric types ... metric test is still failing because of missing metrics when the counter is 0 added a lot of TODO's I need to go back and cleanup	6 years ago
Edward Welch	eaecf0ff90	Making the pipeline itself a Stage so that we can use it to better implement the Match stage (and it cleans up the Docker and CRI extensions some too) ... Still needs to fix the metrics test	6 years ago
Cyril Tovena	69d732b4d0	move custom metrics to /metrics and prefix them	6 years ago
Cyril Tovena	a5e47a50e8	adds matchers and metrics to regex and json	6 years ago
Steven Sheehy	c1603363ec	Improve readiness probe output (#556) ... Signed-off-by: Steven Sheehy <ssheehy@firescope.com>	6 years ago
Cyril Peponnet	7942493524	impr/clients: Handle TLS config and MTLS for logcli and promtail (#540) ... * impr/clients: Handle TLS config and MTLS for logcli and promtail * fix/tls: Please gofmt... * impr/clients: use prometheus HTTPClientConfig for logcli and promtail * fix/promtail: Set proper Client config name * impr/promtail: Use prometheus HTTPClientConfig configuration * adapt with master * address review * fix conflicts * address requested changes * remove file Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	6 years ago
Cyril Tovena	0734c9f0a6	adds external labels to be passed via flags (#510)	6 years ago
Edward Welch	87d25fac79	Clean up the metric for pipeline duration to keep it within the pipeline file, also changing unit to seconds and namespace to logentry	6 years ago