267 Commits (f7d30e2cbca9745c152e2ff5be26dcc2e9b280ea)

Author	SHA1	Message	Date
Marco Pracucci	e7fb205827	fix(logcli): timestamp timezone handling (#821) ... Logcli used to inconsistently handle timezones in timestamps. This addresses this by adding a new `--timezone` / `-z` flag, to explicitly set the timezone the timestamps shall be formatted in. Furthermore it refactors the output framework inside of logcli to test it easier. Fixes #798	7 years ago
Cyril Tovena	2561a70155	fix tail library logs to use our own log format (#579) ... * fix tail library logs to use our own log format * PR Feedbacks	7 years ago
Ed	573673a0ab	Simplify our makefile as much as possible (#753) ... refactor(makefile): simplify makefile by removing the autogenerated targets and make building binaries and dockerfiles separate operations.	7 years ago
Robert Fratto	d8ee6b8a5b	ingester: register Ingester service in gRPC on loki start ... Registering the Ingester service to gRPC enables the chunk transfer mechanism to work.	7 years ago
Robert Fratto	f2bec3b0a5	promtail: restore ability to show target labels in promtail UI ... PR #791 accidentally removed the ability to use the promtail UI by removing methods from the Target interface that were only used within the HTML templates. Along with restoring the methods in the Target interface, this commit also introduces details for the JournalTarget, which currently only provides the position in the journal being tracked.	7 years ago
Marco Pracucci	476c3961c2	Added -querier.query_timeout support (#788)	7 years ago
Robert Fratto	e64c7e5faf	ingester: log error if closing client after transfer fails	7 years ago
Robert Fratto	e4e5e3c1f1	ingester: fix feedback from PR review	7 years ago
Robert Fratto	15cbba7242	ingester: clean up chunk transfer code	7 years ago
Edward Welch	7deb02b532	add a Name() method to the stage interface so that debug logging can show you the name of the pipeline stage which just processed the log ... remove some unnecessary logging around fsnotify events we don't care about and saving positions Make the Processing Log Lines doc a first class citizen, I reference this a lot and currently it's hidden behind 3 clicks	7 years ago
Robert Fratto	f8155a1109	ingester: Add test for chunk transfers	7 years ago
Robert Fratto	2eb666b0a8	ingester: fix lint issues for chunk transfers	7 years ago
Robert Fratto	d34b1f7450	ingester: support chunk transfers on ingester shutdown. ... This commit introduces chunk transfers, borrowing the mechanism from Cortex's implementation: when an ingester is shut down with claim on rollout enabled, the ingester will find a pending ingester and transfer all of its chunks to it.	7 years ago
Edward Welch	3d5319e72a	adding ability to supply timezone to timestamp pipeline stage	7 years ago
Cyril Tovena	cb7c8f6233	Use prometheus pool for line buffer. (#790) ... * Use prometheus pool for line buffer.	7 years ago
sh0rez	38cb093358	feat(loki): extended tailing (#764) ... * refactor(querier/ingester): TailRequest Lookback window Moves the specifications of the Lookback Window out of the logproto.QueryRequest into it's own type logproto.Lookback. This is required, because the Lookback Window will be used in the TailRequest as well. * feat(querier): parse Lookback from HTTP Request * feat(logcli): send Lookback Window spec with tail request * feat(querier): include historic entries in tail mode Extends tailing by sending a configurable amount of historic entries with before the live entries. This enables a behaviour that is closer to kubectl logs -f and docker logs -f. It is implemented by running a regular Query before subscribing to the ingesters. * fix: adapt tests to Lookback change * feat(querier): check all errors to make the linter happy * fix(logproto): flatten Lookback window spec Flattens the Lookback window spec into the individual queries * fix(ingester): adapt test to Lookback flatten	7 years ago
Cyril Tovena	4bf77662d3	rounds nanoseconds boundaries to milliseconds (#771) ... * rounds nanoseconds boundaries to milliseconds * convert also store query	7 years ago
Edward Welch	e6f431ad46	finishing up loki-canary move into loki repo	7 years ago
Edward Welch	cd5368a44d	preparing for move into loki repo	7 years ago
Sandeep Sukhani	3a2b569684	Added tail length limit to limit duration of live tailing of logs to 1 hour (#756) ... * Added tail length limit to limit duration of live tailing of logs to 1 hour * Some code refactoring suggested from PR review for live tailing duration limit * Fixed error messages in live tailing of logs * Fixed lint errors	7 years ago
Marco Pracucci	e6f8d6ffad	Added date without year support to timestamp stage (#760)	7 years ago
Cyril Tovena	3346ce1b40	Storage memory improvement (#713) ... * add benchmark for storage queries * improve iterator to load only on next * fix memory retained by lazy chunks * reverse backward lazy iterator	7 years ago
Robert Fratto	9eb3098dea	promtail: Add systemd journal support (#730) ... Support for reading systemd journal entries has been added. promtail will look for a job in scrape_configs with a journal key to activate the journal target. If GOOS=linux and CGO_ENABLED=1, promtail will now require libsystemd headers to be available for building. If GOOS is not linux or CGO_ENABLED is not 1, journal support will be unavailable and a log message will be printed warning the user that their config file has journal tailing configured without it being built into promtail. See docs/promtail-examples.md for a concrete example of using journal support. Other structural changes made: 1. Ability for checking if scrape.Config.ServiceDiscoveryConfig is non-zero has been added. This was chosen over making ServiceDiscoveryConfig a pointer as yaml.v2 cannot parse an inline struct into a pointer value. 2. Updated pipeline logger component name to journal_pipeline and file_pipeline for JournalTargetManager and FileTargetManager respectively. 3. The positions file will now store positions as strings instead of integers. Existing positions will be read in properly but written out as strings the next time the positions file is saved. This is done to be able to store the journal cursor, which is a string. The positions API has been updated to support reading in the old integer values and the new string values.	7 years ago
Marco Pracucci	94c252b91f	Parallelly run regex and json stages pipeline tests	7 years ago
Marco Pracucci	d235e483c4	Added source validation to regex and json stages	7 years ago
Marco Pracucci	0a79e2a96e	Converted source in regex and josn stages to string pointer	7 years ago
Marco Pracucci	812aecc795	Added source support to regex and json stages	7 years ago
Edward Welch	d5688b5eb6	adding a golang Template stage	7 years ago
Edward Welch	1249e92ef3	use strconf.FormatFloat instead of fmt.Sprintf for converting floats to strings, this way we can eliminate non significant trailing zeros such that the float value 1 would be "1" as a string instead of "1.000000"	7 years ago
Marco Pracucci	f7cc7b035e	Redirect / to /targets in promtail server	7 years ago
Edward Welch	de83272b2f	Add support to timestamp stage to parse Unix seconds, milliseconds, and nanosecond timestamps	7 years ago
Edward Welch	eedd3d9fc2	prune interval is configurable ... canary will suspend all operations on SIGINT but not exit, allowing you to shutdown the canary without it being restarted by docker/kubernetes SIGTERM will shutdown everything and end the process	7 years ago
Cyril Tovena	86bc3c0518	Query label values and names are now fetched from the store. (#521) ... * Query label values and names are now fetched from the store. A time range is now required by the /api/prom/label with a sane default (6 hours from now). * fix http querystring and update doc * update vendor * rebased	7 years ago
Marco Pracucci	2bbb464848	Fixed orderedDeps() order stability (#721)	7 years ago
Edward Welch	3b02474597	removing entries.go as we are no longer using the custom Histogram for the log_entries_bytes histogram	7 years ago
Edward Welch	922b2d02cd	changing the log length histogram to be a normal histogram with only a `path` label, also removing the label values when no longer tailing the file. Following the same pattern we used for readBytes and totalBytes.	7 years ago
Sandeep Sukhani	4c7138231f	Removed test for tailer in querier due to issues with synchronization	7 years ago
Sandeep Sukhani	2a8a03795a	Improvements in live tailing of logs (#541) ... * Improvements in live tailing of logs Instead of polling for new logs, a grpc stream is opened between ingester and querier to get live logs Querier reconnects to disconnected or newly added ingesters * Added more comments to code for live log tailing * Some code refactoring in live log tailing * handling delayfor in logcli, max delayfor to be 5 seconds * some changes in tail response in live tailing * Fixed issue with stopping ingesters gracefully when live tailing is being used * Added tests for tailer in querier * Live tailing made logql compatible, some code refactoring suggested in PR	7 years ago
Edward Welch	c3084dbdc7	Removing the pre-allocation of a buffer when serializing blocks, in most of my empirical testing we were using 9-10k out of the 32k allocated leaving about 2/3 of the buffer allocated and unused times the number of blocks per chunk, times the number of chunks kept in memory. This was adding up to a fair amount of allocated but unused space.	7 years ago
Cyril Tovena	be6664892f	fix docker plugin ci (#664) ... * fix docker plugin ci * use binary cache	7 years ago
Cyril Tovena	2909fe54e1	Docker Logging Driver (#663) ... * adds first version of docker driver * without logrus and fixes some linter issue * fix the driver and start a build system * adds swarm label discovery * Add documentation and more targets * make the linter happy ❤️ * indent config.json * with circleci steps for master and branch * Review Feebacks	7 years ago
Edward Welch	7cf05c5dfc	adding a list of received entries (acknowledged) to use for comparison against entries which were not expected so that we can report them as duplicates.	7 years ago
Edward Welch	02aed71a28	refactoring things so that the comparator can use the reader to make a direct query to loki to look for logs not received over the websocket before reporting them as missing	7 years ago
Cyril Tovena	f939e42afd	add support for RFC3339Nano in query timestamps (#656)	7 years ago
sh0rez	11ad356396	feat(promtail): initContainers (#655) ... * fix(promtail/targets): remove dependency on prometheus/relabel `filetargetmanager.go` had a dependency on prometheus/relabel, a package that has been removed in favor of prometheus/pkg/relabel. This converts the code to the new package, to allow the prometheus vendor be updated to master * chore(vendor): update prometheus vendor Updates prometheus vendor to current master, to add support for InitContainers in kubernetes service discovery * chore(vendor): do dep's homework Gives dep hints on how to resolve the vendor so that it work * fix(promtail/targets): non-nil check Accidentally checked against the wrong labelSet, this one can never get nil	7 years ago
Xiang Dai	3021d6f29a	Remove 404 link (#637) ... Signed-off-by: Xiang Dai <764524258@qq.com>	7 years ago
Cyril Tovena	922f1daf5d	Remove label __name__ from store querier (#648)	7 years ago
Edward Welch	1dc130d8cc	PR Feedback	7 years ago
Edward Welch	04facc4d00	updating docs for new pipeline config. ... removed helm entry in scrape_config.sh because there doesn't seem to be anyway to make it work with new pipeline config	7 years ago
Edward Welch	51bdd15237	renaming `metric` stage to `metrics` as it defines multiple metrics, similar to labels stage which is also plural. ... Adding a couple unit tests to regex and json stage to act as examples	7 years ago