4 Commits (de93a1bd541e314f993b79814ebe4296ba516c86)

Author	SHA1	Message	Date
Trevor Whitney	de93a1bd54	Promtail: add consul agent service discovery (#3834) ... * working prototype Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * update test for new responses and metadata labels Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * fix linting issues Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * clarify license/copyright/attribution to prometheus code Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * undo changes to go.sum when adding consul dep Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add documentation around consul and consulagent scrape configs Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/configuration.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Ed Welch <edward.welch@grafana.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Sandeep Sukhani	d765921735	revendor cortex to latest master (#3771)	4 years ago
Owen Diehl	d88f3996ea	revendor compatibility: various prom+k8s+cortex (#3664) ... * revendor compatibility: various prom+k8s+cortex * fix logcli client instantiation * go mod tidy	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623) ... * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Aditya C S	bf4fa6613a	support max_message_length configuration for syslog parser (#3400) ... * support max_message_length configuration for syslog parser * address review comments	4 years ago
Danny Kopping	faa1eab620	Remove default docker pipeline stage - this is undocumented and unexpected behaviour (#3404) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Gregory Brzeski	6c8fdd68aa	fix typo in config variable name: BookmarkPath (#3298)	4 years ago
Cyril Tovena	72b3acb925	Windows events (#3246) ... * First commit for windows event targets. I had to add couple of new go modules and hook the windows log in. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add the ability to save where we are. * Finished setup tests and comments. * nits * adding documentation * go tidy * lint windows * nope * add manager for non windows * don't lint forked package * mod check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/promtail/targets/windows/bookmark.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Use passed-in Prometheus registerer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Apply suggestions from code review Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * testing out windows drone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * type exec. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * docker image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go/go.exe Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * build windows on container. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * trying another image. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo in image Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixed local timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests with timezone. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes timezone tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update logstash since the last image is not working anymore 🤷 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Kaviraj	6cc41f92f8	Gcplog targetmanager (#3083) ... * Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Chance Zibolski	f0d4adc53d	promtail: Add support for using syslog message timestamp (#2914) ... Currently promtail sets the timestamp of incoming syslog messages to the time it was received by promtail. In some cases, it is preferable to use the source timestamp instead. This adds a new `use_message_timestamp` option to the syslog target config, which allows users to opt-in to the behavior of using the timestamp on the message, if one exists.	5 years ago
Owen Diehl	607ccc4d77	Service discovery refactor (#2761) ... * promtail sd loading refactor + tests * flippin bools like flipper * adds back base scrapeconfig test * passes all service discovery variants	5 years ago
Owen Diehl	54c0c5febb	Revendor Cortex (#2755) ... * replaces prometheus & revendors cortex * compat * compat * sd refactor * updates docker * removes prom replace * adds delimiters back to object addrs in boltdb	5 years ago
Kaviraj	690a387cce	fix: Remove depricated `entry_parser` from scrapeconfig (#2752)	5 years ago
Ed Welch	b24a143043	Promtail: Loki Push API (#2296) ... * refactor to put targets in separate packages * rename scrape package to scrapeconfig * Push implementation, working with logging * adding new vendored library * adding tests * adding documentation and an additional config option. * fixing journal test * rename scrape config entry * Update docs/clients/promtail/configuration.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * PR feedback Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Robert Fratto	874d016bd7	promtail: add support for passing through journal entries as JSON ... This commit adds an extra configuration field (`json`) to the journal target configuration schema. When set to true, messages from the journal target will be passed through as JSON, containing all of the fields from the entry as keys and their values as the value. For backwards compatibility, the value for `json` defaults to false. Closes #1288.	6 years ago
Sebastian Widmer	f0f6f24926	pkg/promtail: IETF Syslog (RFC5424) Support (#1275) ... * IETF syslog (rfc5254) support * Upgrade go-syslog to correctly released v2.0.1 * Incorporate feedback from new linter * Update documentation with relevant RFC sections. * Incorporate feedback from review * Use context to shutdown server. * Use util.Backoff from cortex * Do not embed mutex into TestLabeledClient * Use strings.HasPrefix * Better naming of connectionsClosed -> openConnections * Incorporate further feedback from review * Callback instead of channel in ParseStream. Removes one running Goroutine per connection. * Improve parse error log level. * Move mutex above field it protects. * Use backoff.Ongoing() * Switch to "nontransparent" parser * Further improvements to the documentation	6 years ago
Robert Fratto	4c418430be	promtail: add "max_age" field to configure cutoff for journal reading (#921) ... The journal scrape config in promtail has been updated to support a "max_age" field. max_age determines the oldest journal entry promtail will read when starting the journal reader. When unspecified, max_age defaults to 7h. Even if a position in the journal is saved in the promtail positions file, that position may be ignored if the entry corresponding to that position is older than the max_age cutoff time. Example promtail config for getting up to the last 12h of journal entries: server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://localhost:3100/api/prom/push scrape_configs: - job_name: journal journal: max_age: 12h path: /var/log/journal labels: job: systemd-journal relabel_configs: - source_labels: ['__journal__systemd_unit'] target_label: 'unit'	6 years ago
Robert Fratto	9eb3098dea	promtail: Add systemd journal support (#730) ... Support for reading systemd journal entries has been added. promtail will look for a job in scrape_configs with a journal key to activate the journal target. If GOOS=linux and CGO_ENABLED=1, promtail will now require libsystemd headers to be available for building. If GOOS is not linux or CGO_ENABLED is not 1, journal support will be unavailable and a log message will be printed warning the user that their config file has journal tailing configured without it being built into promtail. See docs/promtail-examples.md for a concrete example of using journal support. Other structural changes made: 1. Ability for checking if scrape.Config.ServiceDiscoveryConfig is non-zero has been added. This was chosen over making ServiceDiscoveryConfig a pointer as yaml.v2 cannot parse an inline struct into a pointer value. 2. Updated pipeline logger component name to journal_pipeline and file_pipeline for JournalTargetManager and FileTargetManager respectively. 3. The positions file will now store positions as strings instead of integers. Existing positions will be read in properly but written out as strings the next time the positions file is saved. This is done to be able to store the journal cursor, which is a string. The positions API has been updated to support reading in the old integer values and the new string values.	6 years ago
Edward Welch	eaecf0ff90	Making the pipeline itself a Stage so that we can use it to better implement the Match stage (and it cleans up the Docker and CRI extensions some too) ... Still needs to fix the metrics test	6 years ago
Edward Welch	ebb4f483bf	updating tests ... adding CRI and Docker format pipeline stage extensions adding backwards compatibility with current config	6 years ago
Edward Welch	0fac61801c	adding regex pipeline stage	6 years ago
Cyril Tovena	a9b9e22303	Plug the log entry pipeline into promtail ... Co-authored-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Edward Welch	8dd490d238	beginnings of label extraction pipeline	6 years ago
Ed	f6d0857ddf	Adding a sync() function to filetarget (#293) ... * Adding a sync() function to filetarget which runs on a timer to catch any events missed by fsnotify * moving where we mark the final position to avoid a race condition exposed in testing, we want to stop the tailer before shutting down our tailer loop * Fix changes from PR and refactor to remove circular imports and make config consistent * Adding an "integration" type unit test which tests promtail as a whole * fixing a bunch of imports I munged moved tailer to its own file refactoring filetarget to make synclogic easier to follow and reused as much as possible	6 years ago