75 Commits (c63928e20f74d5a15bac9952058d569376952211)

Author	SHA1	Message	Date
Ed Welch	5aa82ba199	allow configuring a target chunk size in compressed bytes (#1406) ... * allow configuring a target chunk size in compressed bytes Signed-off-by: Edward Welch <edward.welch@grafana.com> * addressing feedback Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Peter Štibraný	e0ec61f986	pkg/ingester: handle labels mapping to the same fast fingerprint. (#1247) ... * pkg/ingester: handle labels mapping to the same fast fingerprint. Uses slightly adapted fpMapper code from Cortex. Fixes issue #898 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * empty commit to force new build Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed empty lines in imports. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Added test that pushes entries concurrently. To be run with -race. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Stream now keeps both original and mapped fingerprint Mapped fingerprint is used in streams map. Original ("rawFP") is preserved when saving chunks. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Vendoring Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Reverted previous commit (keep the test, with updated checks). Preserving raw fingerprints could lead to data loss when doing chunk deduplication while storing chunks. We don't want that. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Joe Elliott	b835e2082a	Fix Loki Chunks Dashboard (#1126) ... This commit cleans up the Loki Chunks dashboard either by renaming incorrectly named series or adding needed metrics. Added new metrics: - loki_ingester_memory_chunks - loki_ingester_memory_streams - loki_ingester_chunk_utilization - loki_ingester_chunk_compression_ratio The following metrics were already being published with a different name. Fixed in the dashboard: - cortex_chunk_store_index_entries_per_chunk - cortex_ingester_flush_queue_length - loki_ingester_chunk_entries	6 years ago
Ward Bekker	3b965105ec	Improved Ingester out-of-order error for faster troubleshooting (#1008)	6 years ago
Robert Fratto	d34b1f7450	ingester: support chunk transfers on ingester shutdown. ... This commit introduces chunk transfers, borrowing the mechanism from Cortex's implementation: when an ingester is shut down with claim on rollout enabled, the ingester will find a pending ingester and transfer all of its chunks to it.	6 years ago
Cyril Tovena	3346ce1b40	Storage memory improvement (#713) ... * add benchmark for storage queries * improve iterator to load only on next * fix memory retained by lazy chunks * reverse backward lazy iterator	6 years ago
Sandeep Sukhani	2a8a03795a	Improvements in live tailing of logs (#541) ... * Improvements in live tailing of logs Instead of polling for new logs, a grpc stream is opened between ingester and querier to get live logs Querier reconnects to disconnected or newly added ingesters * Added more comments to code for live log tailing * Some code refactoring in live log tailing * handling delayfor in logcli, max delayfor to be 5 seconds * some changes in tail response in live tailing * Fixed issue with stopping ingesters gracefully when live tailing is being used * Added tests for tailer in querier * Live tailing made logql compatible, some code refactoring suggested in PR	6 years ago
Ed	fe53924ff8	using the timestamp from the log file might lead to undesired behavior as it could be way in the past or future depending on the clock for the source system generating the logs. Given that it's compared to the servers time.Now() when looking for idle chunks it's probably best to use the server time when setting the value. (#546)	6 years ago
Xiang Dai	aa4684ee52	support config chunk size (#464) ... * support config chunk size Signed-off-by: Xiang Dai <764524258@qq.com> * fix lint Signed-off-by: Xiang Dai <764524258@qq.com> * review feedback	6 years ago
Goutham Veeramachaneni	e54a1b21c3	vendor: update cortex (#512) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	6 years ago
Tom Wilkie	40ae03b48a	Include the stream's labels in OOO error responses. (#304) ... * Include the stream's labels in OOO error responses. Also, read the body of error responses and log them. And retries 500s and connection errors. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Log close errors. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Update pkg/promtail/client/client.go Co-Authored-By: tomwilkie <tomwilkie@users.noreply.github.com> * Log retries at warn, final errors at error. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Goutham Veeramachaneni	7763e8f3dd	Flush idle chunks properly (#209) ... * Flush idle chunks properly Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Review feedback Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Review feedback. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Check the correct chunk is full before appending a new chunk. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Add the config to helm and ksonnet too. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	ab4c5bebab	Validate labels in the distributor. (#251) ... Also, make ErrOutOfOrder a HTTP 4xx error and continue to append even when we encourter an out of order entry. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	9f257c591a	[WIP] Persistence (#44) ... Persistence chunks using the Cortex chunk store.	7 years ago
Tom Wilkie	5f73598389	Rename Tempo to Loki. (#36) ... * Rename Tempo to Loki. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Use new build image, don't delete generated files on clean. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	1fef92bc17	Lint stuff (#27) ... * Check in generated files, to make getting started easier. Add gometalinter, and a lint check for changes to the generated files. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Make lint pass? Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	3f62b4ec36	s/logish/tempo/ ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Goutham Veeramachaneni	8f4e12a5ec	Chunking (#10) ... * Add checkenc without serialisation for now. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Simplify interface and add serialisatio` Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Move away from \xFF magic to something simple Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Add serialisation and Deserialisation Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Modify interface to be closer to logish interface. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * chunkenc: Fix race b/w append and iteration. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * chunkenc: Make iterators honour bounds Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * chunkenc: Remove locks as safety is assured externally Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * chunkenc: Add checksums Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Add code quotes around block design. * Split headBlock into it's own type. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Simplify encoding and decoding. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Expose flags. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use the already existing EntryIterator interface Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Use existing Chunk interface. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Review feedback Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Integrate the compressed chunk and add metrics Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	248f9e25e4	Move iterators to own package; check in protos and parser to make CLI go gettable. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	544eed1b95	Add some unit tests for the iterators. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	00afed1eb3	Add start, end and direction options to querier. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	e33a029c97	Reverse the order of the iterators so they go from end to start. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	bbf4552cc4	Wire up query path with new parsers and iterators.	7 years ago
Tom Wilkie	847ea21230	Add iterator for chunks and streams.	7 years ago
Tom Wilkie	af93e2019c	Really dumb chunk implementation.	7 years ago