17 Commits (f2bec3b0a5291a4b11b0aa0f3fd60ea60abe1e9e)

Author	SHA1	Message	Date
Robert Fratto	f2bec3b0a5	promtail: restore ability to show target labels in promtail UI ... PR #791 accidentally removed the ability to use the promtail UI by removing methods from the Target interface that were only used within the HTML templates. Along with restoring the methods in the Target interface, this commit also introduces details for the JournalTarget, which currently only provides the position in the journal being tracked.	6 years ago
Edward Welch	7deb02b532	add a Name() method to the stage interface so that debug logging can show you the name of the pipeline stage which just processed the log ... remove some unnecessary logging around fsnotify events we don't care about and saving positions Make the Processing Log Lines doc a first class citizen, I reference this a lot and currently it's hidden behind 3 clicks	6 years ago
Robert Fratto	9eb3098dea	promtail: Add systemd journal support (#730) ... Support for reading systemd journal entries has been added. promtail will look for a job in scrape_configs with a journal key to activate the journal target. If GOOS=linux and CGO_ENABLED=1, promtail will now require libsystemd headers to be available for building. If GOOS is not linux or CGO_ENABLED is not 1, journal support will be unavailable and a log message will be printed warning the user that their config file has journal tailing configured without it being built into promtail. See docs/promtail-examples.md for a concrete example of using journal support. Other structural changes made: 1. Ability for checking if scrape.Config.ServiceDiscoveryConfig is non-zero has been added. This was chosen over making ServiceDiscoveryConfig a pointer as yaml.v2 cannot parse an inline struct into a pointer value. 2. Updated pipeline logger component name to journal_pipeline and file_pipeline for JournalTargetManager and FileTargetManager respectively. 3. The positions file will now store positions as strings instead of integers. Existing positions will be read in properly but written out as strings the next time the positions file is saved. This is done to be able to store the journal cursor, which is a string. The positions API has been updated to support reading in the old integer values and the new string values.	6 years ago
Edward Welch	922b2d02cd	changing the log length histogram to be a normal histogram with only a `path` label, also removing the label values when no longer tailing the file. Following the same pattern we used for readBytes and totalBytes.	6 years ago
Edward Welch	bb89b9b585	Fix a bug where we should use the glob matcher on fsnotify new file create. ... Changed the timing of the sync function in the unit test to fix flakiness in the test.	6 years ago
Cyril Tovena	48def27e1e	fix review feedback	6 years ago
Cyril Tovena	9290a5d228	rename __filename__ label to filename	6 years ago
Cyril Tovena	b8b7caa450	Promtail targets and service discovery pages (#532) ... * add a custom server for loki with assets embedded * adds a targets page for promtail to see discovered files and labels * lint fixes * fix css table size * add service discovery logic * add service discovery template * add promtail pages documentation * ignored gen file in linter	6 years ago
Edward Welch	4b8eda64f5	PR feedback	6 years ago
Edward Welch	3f8963ce01	Updating hpcloud tail library to forked version modified for our purposes. ... Changes to the library move the handling of symlinks and re-opening to the tail library. fsnotify events are only used now to start new tailers for files not previously seen.	6 years ago
Steven Sheehy	8c0b3d8099	Add a /ready endpoint to promtail ... Signed-off-by: Steven Sheehy <ssheehy@firescope.com>	6 years ago
Tom Wilkie	b2fafdf360	Export position in file, not bytes read. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Ed	ca1a5be975	Adding metric and alert to notify promtail falling behind (#412) ... * Adding the metric promtail_file_bytes_total which will report on the total size of each file with an active tailer. Added an alert which will fire whenever the total bytes for a file is 100kb or more ahead the read bytes for more than 15mins. Added an additional alert to track client bytes ready to send (encoded bytes) vs actually sent so we can add an alert on promtail send failures and still track stats on encoded bytes being sent. * Moving where we update the total file bytes outside the tailer so it shouldn't get hung up by tailer issues. * Recording file size based on Glob matches rather than what we are tailing to give even better insight	6 years ago
Baokun Lee	e6192a6908	Fix start tailing panic ... Signed-off-by: kun <oiooj@qq.com>	6 years ago
Ed	0fdbf24b79	adding third party lib for Glob pattern matching which supports /**/ style subdirectory matching (#331) ... * adding third party lib for Glob pattern matching which supports ** refactor promtail_test to be easier to read and include more tests	6 years ago
Ed	f6d0857ddf	Adding a sync() function to filetarget (#293) ... * Adding a sync() function to filetarget which runs on a timer to catch any events missed by fsnotify * moving where we mark the final position to avoid a race condition exposed in testing, we want to stop the tailer before shutting down our tailer loop * Fix changes from PR and refactor to remove circular imports and make config consistent * Adding an "integration" type unit test which tests promtail as a whole * fixing a bunch of imports I munged moved tailer to its own file refactoring filetarget to make synclogic easier to follow and reused as much as possible	6 years ago
Tom Wilkie	9d7691a52e	Move stuff from file -> targets to prevent the stutter; move tests about; update broken rebase. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Christian Simon	31789eafc4	Refactor structure of promtail ... This allows easier integration of other targets than files for promtail. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Anthony Woods	df84364788	handle symbolic links correctly (#275) ... * handle symbolic links correctly When tailing a file that is rotated we expect that a fsnotify.Rename event will be sent when the old file is renamed and a fsnotif.Create event will be sent when the replacement file is created. This allows us to close and re-open the file. However if the file we are tailing is a symbolic link (and the target is in another directory) we wont get these events. To address this we need to tail the target file directly. fixes #274 * handle symlinks being re-linked When a symbolic link is re-linked we will only get a fsnotify.Create event. No rename or delete event will have been sent to trigger the tailer to be stoped. When this happens we should stop the current tailer for the path and start a new one. This will allow us to perform a new Readink on the path to get the updated target filename.	6 years ago
Tom Wilkie	29c2213d1a	Add more metrics to promtail. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	eeac68f183	Log filename with parsing errors. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Ed	3e6a75e182	removing incorrect calling of positions.Stop from within a target as it runs single instance within promtail, moved the call to Stop up to promtail.go within the Shutdown() method. Should Fix #249 (#252)	6 years ago
Ed	07f20b0db3	Unit tests for Promtail (#244) ... * Adding a test for target.go Fixing some shutdown issues where tailers were stopped but didn't properly wait for the current position to be saved to file Fixed an issue where an empty directory would lead to nothing being tailed * adding more tests for target.go * renaming quitComplete -> done per feedback making PositionsFile private again, no reason for it to be public removed unnecessary save of empty file on creation of positions file * fixing lint errors * cleaning up some comments removing TODO around directories, added a test to verify behavior some more cleanup per review * fixing imports	6 years ago
Anthony Woods	fed3943596	allow promtail path to be a glob match (#206) ... * allow promtail target path to be a glob match Promtail now expects __path__ to be a glob. All files that match the glob expression will be tailed. If new files are created that match the glob, they will immediately be tailed. * remove entry from positions file when file is deleted * fix syncing the current positions to positions file * handle file renames When a file is renamed a "rename" and a "create" event are sent. We need to treat the rename as a remove. If the new filename matches the path glob, then we will start tailing it when the "create" event is reveived. * lint	6 years ago
Ye Ben	904c190364	add readLines metric (#214)	6 years ago
Egor Kovetskiy	4d78e165dc	Fix duplicating base directory for file create event (#128) ... Signed-off-by: Egor Kovetskiy <e.kovetskiy@gmail.com>	7 years ago
Tom Wilkie	5f73598389	Rename Tempo to Loki. (#36) ... * Rename Tempo to Loki. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use new build image, don't delete generated files on clean. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	600b24101e	Promtail updates (#30) ... * Make promtail read all its config from a file. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use gokit logging everywhere. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Start work to parse normal log lines. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * update common and kuberesolver. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * lint Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Get EntryParser stuff working, so we can tail raw log files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make sure filename label works. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * lint, remove interfacer. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	f6d5ebf513	Document how to get promtail running locally. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	1fef92bc17	Lint stuff (#27) ... * Check in generated files, to make getting started easier. Add gometalinter, and a lint check for changes to the generated files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make lint pass? Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Goutham Veeramachaneni	5233e78048	Add the required label name to the metric (#20) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	6f883b330e	Add metric for number of bytes read from each log. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	9cb87c2c7d	Fix a few NPEs. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	aa44de07bb	Move promtail from kausalco/public, update it for prometheus changes. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago