1735 Commits (6e2b7d25898f321bfd6f94de05d6ce934c785da9)

Author	SHA1	Message	Date
Tom Wilkie	68e1a049a0	lint ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	9d7691a52e	Move stuff from file -> targets to prevent the stutter; move tests about; update broken rebase. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Christian Simon	31789eafc4	Refactor structure of promtail ... This allows easier integration of other targets than files for promtail. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Anthony Woods	df84364788	handle symbolic links correctly (#275) ... * handle symbolic links correctly When tailing a file that is rotated we expect that a fsnotify.Rename event will be sent when the old file is renamed and a fsnotif.Create event will be sent when the replacement file is created. This allows us to close and re-open the file. However if the file we are tailing is a symbolic link (and the target is in another directory) we wont get these events. To address this we need to tail the target file directly. fixes #274 * handle symlinks being re-linked When a symbolic link is re-linked we will only get a fsnotify.Create event. No rename or delete event will have been sent to trigger the tailer to be stoped. When this happens we should stop the current tailer for the path and start a new one. This will allow us to perform a new Readink on the path to get the updated target filename.	7 years ago
Goutham Veeramachaneni	7763e8f3dd	Flush idle chunks properly (#209) ... * Flush idle chunks properly Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Review feedback Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Review feedback. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Check the correct chunk is full before appending a new chunk. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Add the config to helm and ksonnet too. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	a422f394bb	Wait for the right number of streams. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	29c2213d1a	Add more metrics to promtail. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	eeac68f183	Log filename with parsing errors. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	f6d547d32d	Don't log request body. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	ab4c5bebab	Validate labels in the distributor. (#251) ... Also, make ErrOutOfOrder a HTTP 4xx error and continue to append even when we encourter an out of order entry. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Ed	3e6a75e182	removing incorrect calling of positions.Stop from within a target as it runs single instance within promtail, moved the call to Stop up to promtail.go within the Shutdown() method. Should Fix #249 (#252)	7 years ago
Ed	07f20b0db3	Unit tests for Promtail (#244) ... * Adding a test for target.go Fixing some shutdown issues where tailers were stopped but didn't properly wait for the current position to be saved to file Fixed an issue where an empty directory would lead to nothing being tailed * adding more tests for target.go * renaming quitComplete -> done per feedback making PositionsFile private again, no reason for it to be public removed unnecessary save of empty file on creation of positions file * fixing lint errors * cleaning up some comments removing TODO around directories, added a test to verify behavior some more cleanup per review * fixing imports	7 years ago
Anthony Woods	fed3943596	allow promtail path to be a glob match (#206) ... * allow promtail target path to be a glob match Promtail now expects __path__ to be a glob. All files that match the glob expression will be tailed. If new files are created that match the glob, they will immediately be tailed. * remove entry from positions file when file is deleted * fix syncing the current positions to positions file * handle file renames When a file is renamed a "rename" and a "create" event are sent. We need to treat the rename as a remove. If the new filename matches the path glob, then we will start tailing it when the "create" event is reveived. * lint	7 years ago
Ken Haines	99d0f97a5c	Added call to StrorageConfig.RegisterFlags, in main.go for loki. (#216)	7 years ago
Ye Ben	904c190364	add readLines metric (#214)	7 years ago
Adam Shannon	eb390523e3	all: update dependencies (#211) ... * all: update dependencies * Update some more dependancies and pin the right versions for Prometheus deps. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Goutham Veeramachaneni	9b67d6cb7a	Rename instance to tenant to avoid clash with Prom (#164) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Goutham Veeramachaneni	8609fcab74	Cut new blocks once the existing ones are full (#71) ... * Cut new blocks once the existing ones are full Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Address PR comments Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	da27baad2c	Support for JSON, not just protos. (#152) ... * Support for JSON, not just protos. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * s/timestamp/ts/ Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Goutham Veeramachaneni	7bcdf0e0b1	Tenant metrics (#135) ... * Fix stream tracking metric Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Track lines and bytes received per user Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use instance instead of org Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Nuke dead code Usage has been removed as part of perf-optimisation: 3006e3901d (diff-180f56d9aaf5a9aa079e6fc9cfcf1bc8L365) Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	d6e8788eae	Don't forget to apply regexp filters to chunk entries. (#124) ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Egor Kovetskiy	4d78e165dc	Fix duplicating base directory for file create event (#128) ... Signed-off-by: Egor Kovetskiy <e.kovetskiy@gmail.com>	7 years ago
Yeh-lei Wu	251017e41e	Refine ingester locks: eliminate smell & allow concurrent ingester query (#120) ... Signed-off-by: alei <rayingecho@gmail.com> Fix chunk retain period Signed-off-by: alei <rayingecho@gmail.com> Fix undefined symbol Signed-off-by: alei <rayingecho@gmail.com>	7 years ago
Eugen Biegler	1d87a2285e	Add line length to batchSize (#108) ... * Add line length to batchSize Signed-off-by: Eugen Biegler <nupagadi@posteo.de> * Reset batchSize in timer condition too Signed-off-by: Eugen Biegler <nupagadi@posteo.de>	7 years ago
Ye Ben	248c3756a2	fix a typo Eeror -> Error (#66) ... Signed-off-by: yeya24 <ben.ye@daocloud.io>	7 years ago
Tom Wilkie	9f257c591a	[WIP] Persistence (#44) ... Persistence chunks using the Cortex chunk store.	7 years ago
Goutham Veeramachaneni	f2de97c880	Add external labels support (#43) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	5f73598389	Rename Tempo to Loki. (#36) ... * Rename Tempo to Loki. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use new build image, don't delete generated files on clean. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	600b24101e	Promtail updates (#30) ... * Make promtail read all its config from a file. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use gokit logging everywhere. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Start work to parse normal log lines. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * update common and kuberesolver. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * lint Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Get EntryParser stuff working, so we can tail raw log files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make sure filename label works. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * lint, remove interfacer. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	46e67f71a8	Don't ignore the error ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	f6d5ebf513	Document how to get promtail running locally. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	1fef92bc17	Lint stuff (#27) ... * Check in generated files, to make getting started easier. Add gometalinter, and a lint check for changes to the generated files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make lint pass? Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	da18cb343a	Add option to disable auth when running locally. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	d06ed8fb1f	Remove flagext, use cortex's. Allow flags to be loaded from a file. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	4635769b4f	Don't register variables twice. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	de05515248	Init pool config correctly. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	3b69cf127d	Don't repeat client config. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	0ffb432f30	Update for cortex changes. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	9658f3dbd8	Don't commit generated files. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	3f62b4ec36	s/logish/tempo/ ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Goutham Veeramachaneni	8a1f4e190d	Fix broken test. ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	8 years ago
Goutham Veeramachaneni	8f4e12a5ec	Chunking (#10) ... * Add checkenc without serialisation for now. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Simplify interface and add serialisatio` Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Move away from \xFF magic to something simple Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Add serialisation and Deserialisation Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Modify interface to be closer to logish interface. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * chunkenc: Fix race b/w append and iteration. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * chunkenc: Make iterators honour bounds Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * chunkenc: Remove locks as safety is assured externally Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * chunkenc: Add checksums Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Add code quotes around block design. * Split headBlock into it's own type. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Simplify encoding and decoding. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Expose flags. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use the already existing EntryIterator interface Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use existing Chunk interface. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Review feedback Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Integrate the compressed chunk and add metrics Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	8 years ago
Tom Wilkie	9b1dc8062e	Fix heap iterator & make tests a lot better. (#23) ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago
Goutham Veeramachaneni	5233e78048	Add the required label name to the metric (#20) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	8 years ago
Tom Wilkie	96223322ba	Make queries take nanoseconds since epoch. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago
Tom Wilkie	6f883b330e	Add metric for number of bytes read from each log. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago
Tom Wilkie	6b83717339	Don't forget to register the metrics. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago
Tom Wilkie	5ec9dfc5d1	Fix logic. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago
Tom Wilkie	b247a5450d	Command to fetch label names. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago
Tom Wilkie	248f9e25e4	Move iterators to own package; check in protos and parser to make CLI go gettable. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago