103 Commits (768bf914e00f008bc833b87da7fdbdb55152df35)

Author	SHA1	Message	Date
Peter Štibraný	dca244f7aa	pkg/ingester: added sync period flags ... SyncPeriod can be used to tell ingesters to cut chunks early (before they are full) to synchronize ingesters so that they put the same data into the new chunks. If they do, they generate fully equal chunks and then they don't need to store them. To avoid very small chunks, additional parameter is added: -ingester.sync-min-utilization. When set to eg. 0.5, chunk must be at least 50% full (utilization) in order to be cut early for sync purposes. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	93b85f92fa	pkg/ingester: Added possibility to disable transfers. (#1425) ... Previously, with -ingester.max-transfer-retries set to zero, transfers would never stop. Now zero means don't transfer at all. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Cyril Tovena	7654c27c12	Adds configurable compression algorithms for chunks (#1411) ... * Adds L4Z encoding. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds encoding benchmarks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds snappy encoding. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds chunk size test Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds snappy v2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve benchmarks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove chunkenc Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update lz4 to latest master version. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Use temporary buffer in serialise method to avoid allocations when doing string -> byte conversion. It also makes code little more readable. We pool those buffers for reuse. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Added gzip -1 for comparison. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Initialize reader and buffered reader lazily. This helps with reader/buffered reader reuse. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Don't keep entries, extracted generateData function (mostly to get more understandable profile) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Improve test and benchmark to cover all encodings. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds support for a new chunk format with encoding info. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Ingesters now support encoding config. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add support for no compression. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add docs Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove default Gzip for ByteChunk. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes none, snappyv2 and gzip-1 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move log test lines to testdata and add supported encoding stringer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Ed Welch	f9b25968f7	change the chunk size histogram to allow for bigger buckets (#1409) ... * change the chunk size histogram to allow for bigger buckets Signed-off-by: Edward Welch <edward.welch@grafana.com> * changing buckets again Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Edward Welch	591905ed64	forgot to register the new metric ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Ed Welch	5aa82ba199	allow configuring a target chunk size in compressed bytes (#1406) ... * allow configuring a target chunk size in compressed bytes Signed-off-by: Edward Welch <edward.welch@grafana.com> * addressing feedback Signed-off-by: Edward Welch <edward.welch@grafana.com>	6 years ago
Peter Štibraný	dd96fa146c	pkg/ingester: check that ingester is in LEAVING state when transferring chunks and claiming tokens. Required when using memberlist client. (#1300) ... * On chunks transfer, verify that source ingester is LEAVING This is important when using gossiping ring. If source ingester is not leaving yet, than claiming its tokens will fail (merge function will not move ownership to the target ingester), and target ingester will end up with no tokens in the ring. * Register ring to prometheus. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Update check using latest version from Cortex. There is no direct usage of logging, which is consistent with the rest of this method. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Move import to group of imports with third-party packages. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Peter Štibraný	e0ec61f986	pkg/ingester: handle labels mapping to the same fast fingerprint. (#1247) ... * pkg/ingester: handle labels mapping to the same fast fingerprint. Uses slightly adapted fpMapper code from Cortex. Fixes issue #898 Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * empty commit to force new build Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Removed empty lines in imports. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Added test that pushes entries concurrently. To be run with -race. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Stream now keeps both original and mapped fingerprint Mapped fingerprint is used in streams map. Original ("rawFP") is preserved when saving chunks. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Vendoring Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Reverted previous commit (keep the test, with updated checks). Preserving raw fingerprints could lead to data loss when doing chunk deduplication while storing chunks. We don't want that. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * go mod vendor Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	6 years ago
Julien Pivotto	eb7b495565	Metrics: use Namespace everywhere when declaring metrics (#1320) ... Code linting to make things consistent. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Owen Diehl	b5c03b0802	Fix duplicate labels (update cortex) (#1280) ... * Update to latest cortex to fix duplicate labels Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Start the licycler in the ingester Required since this cortex change in 00566f61af\#diff-dcd65a4ff2de424be70a561f141e067bR180 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go mod tidy Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move cortex to a fork for ARM support Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * updates cortex to master Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * go mod tidy/vendor Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	6 years ago
Joe Elliott	0f7f200c03	Improved compression ratio histogram (#1143) ... * Updated compression ratio histogram to better cover the actual range Signed-off-by: Joe Elliott <number101010@gmail.com> * Guard against divide by 0 Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed temp test Signed-off-by: Joe Elliott <number101010@gmail.com>	6 years ago
Robert Fratto	2cb3c82cc8	pkg/ingester: prevent shutdowns from processing during joining handoff ... This commit fixes a race condition where an ingester that is shut down during the joining handoff (i.e., receiving chunks from a leaving ingester before claiming its tokens) hangs and can never shut down cleanly. A shutdown mutex is implemented which is obtained at the start of the handoff process and released after the handoff process completes. This race condition also prevented the leaving ingester from completing its shutdown, since it waits for the joining ingester to claim the tokens. This means that if a brand new ingester is shut down, it will always have finished receiving chunks from the previous leaving ingester and have finished obtaining the tokens from the ingester attempting to leave.	6 years ago
Joe Elliott	b835e2082a	Fix Loki Chunks Dashboard (#1126) ... This commit cleans up the Loki Chunks dashboard either by renaming incorrectly named series or adding needed metrics. Added new metrics: - loki_ingester_memory_chunks - loki_ingester_memory_streams - loki_ingester_chunk_utilization - loki_ingester_chunk_compression_ratio The following metrics were already being published with a different name. Fixed in the dashboard: - cortex_chunk_store_index_entries_per_chunk - cortex_ingester_flush_queue_length - loki_ingester_chunk_entries	6 years ago
Sandeep Sukhani	569748ff18	logging: removed some noise in logs from live-tailing (#1100) ... * Removed some noise in logs by detecting cases where connection is erroring due to stopped or closed tail request * fixed log formats	6 years ago
Marco Pracucci	42040f2d5b	Skip ingester tailer filtering if no filter is set	6 years ago
Hiroshi Hatake	0cfa33d90d	fluent-bit shared object go plugin (#847) ... * Import fluent-bit-go-loki plugin sources * Add fluent-bit-go package constraint * Ensure dependencies with dep ``` $ dep ensure ``` * Update .gitignore * Move fluent-bit-go-loki to fluent-bit/fluent-bit-go-loki directory * Add documentation for fluent-bit-go-loki * Fix for lint * Fix for goimports errors * Display fluent-bit-go-loki plugin version * Use more descriptive description for fluent-bit Loki plugin * Remove needless newlines * Rectify fluent-bit-go-loki makefile Remove needless parts and rename meaningless target * Use more descriptive description in README * refactor(fluent-bit): move make targets into root Makefile * Organize imports * Reorder imports with goimports $ goimports -w out_loki.go * Remove needless LICENSE file Because it is the same for Loki repository. * Rely on Loki client retry mechanism * Make descriptions more clearly * Change default unit * BatchWait: msec -> sec * BatchSize: KiB -> Byte * Use logql to parse Loki native labels format * Use logger instead of fmt.Printf * Update documentation to reflect recent changes * Use prometheus version module to print version * Dump error message with logger * Use DYN_GO_FLAGS instead of custom FLAGS for fluent-bit-plugin * Support RemoveKeys parameter * use logql.ParseMatchers() * refactor fluent-bit go plugin * add ci step for docker image * fix lint issues * fix a failing url test * fix a failing url test * add build images for drone ci * Adding labelmap file to complex record to labels Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add helm chart and fix a bug with labelmap * New push path * dep sync * Move fluent-bit folder and use jsonnet for drone * fix build * Adds more documentation and include a basic config in the container Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * remove binaries * Some readme grammar fixes * docs: really nitpicky grammar change in clients README * Add helm documentation * fixes bad config * bump chart * fixes .gitignore	6 years ago
Sandeep Sukhani	ec5bc7054d	limits: limits implementation for loki (#948) ... * limits implementation for loki Added following new limits: Length of query Number of active streams Number of streams matcher per query Ingestion rate * periodically reload ingestion rate limits to apply latest rates from overrides	6 years ago
Ward Bekker	3b965105ec	Improved Ingester out-of-order error for faster troubleshooting (#1008)	6 years ago
Sandeep Sukhani	107dfd3607	Use grpc client config from cortex for Ingester to get more control (#755) ... * Use grpc client config from cortex for Ingester for more control over the connection * moved ingester client config block at right place in production ksonnet	6 years ago
Marco Pracucci	29f08ccb1c	Fix panic in tailer due to race condition between send() and close() (#986)	7 years ago
Cyril Tovena	7f3db9d7f0	LogQL: Vector and Range Vector Aggregation. ... - adds avg,min,max,top,bottomk,stddev,stdvar,count - updates api documentation - adds tests Improve yacc & go lexer to understand duration Remove support for regexp in all queries Clean up querier and logselector	7 years ago
Sandeep Sukhani	b687ec6e5b	vendoring: update cortex to latest master (#938) ... * update cortex to latest master using v3.4.0-rc.1 for go.etcd.io/etcd v0.0.2 for github.com/prometheus/procfs v1.0.0 for github.com/prometheus/client_golang latest master for github.com/weaveworks/common with changes from https://github.com/weaveworks/common/pull/153 * fixed failing tests * use large instance for test and lint jobs in CircleCI * running only 6 test binaries in parallel * removed resource type change for CircleCI * changed GOGC to 10 for lint makefile target	7 years ago
Cyril Tovena	4d48972939	Query storage by iterating through chunks by batches. ... Also adds more tests for the store	7 years ago
Robert Fratto	e64c7e5faf	ingester: log error if closing client after transfer fails	7 years ago
Robert Fratto	e4e5e3c1f1	ingester: fix feedback from PR review	7 years ago
Robert Fratto	15cbba7242	ingester: clean up chunk transfer code	7 years ago
Robert Fratto	f8155a1109	ingester: Add test for chunk transfers	7 years ago
Robert Fratto	2eb666b0a8	ingester: fix lint issues for chunk transfers	7 years ago
Robert Fratto	d34b1f7450	ingester: support chunk transfers on ingester shutdown. ... This commit introduces chunk transfers, borrowing the mechanism from Cortex's implementation: when an ingester is shut down with claim on rollout enabled, the ingester will find a pending ingester and transfer all of its chunks to it.	7 years ago
Cyril Tovena	4bf77662d3	rounds nanoseconds boundaries to milliseconds (#771) ... * rounds nanoseconds boundaries to milliseconds * convert also store query	7 years ago
Cyril Tovena	3346ce1b40	Storage memory improvement (#713) ... * add benchmark for storage queries * improve iterator to load only on next * fix memory retained by lazy chunks * reverse backward lazy iterator	7 years ago
Sandeep Sukhani	2a8a03795a	Improvements in live tailing of logs (#541) ... * Improvements in live tailing of logs Instead of polling for new logs, a grpc stream is opened between ingester and querier to get live logs Querier reconnects to disconnected or newly added ingesters * Added more comments to code for live log tailing * Some code refactoring in live log tailing * handling delayfor in logcli, max delayfor to be 5 seconds * some changes in tail response in live tailing * Fixed issue with stopping ingesters gracefully when live tailing is being used * Added tests for tailer in querier * Live tailing made logql compatible, some code refactoring suggested in PR	7 years ago
Sandeep Sukhani	3d3c05717a	Update cortex vendor (#610) ... * Update cortex vendor Use query max look back from cortex Update config changes from cortex Fixed breaking code due to updating cortex vendor * fixed linter error	7 years ago
Cyril Tovena	ec7d453fef	fixes ingester querier not honoring filters (#594)	7 years ago
Steven Sheehy	c1603363ec	Improve readiness probe output (#556) ... Signed-off-by: Steven Sheehy <ssheehy@firescope.com>	7 years ago
Goutham Veeramachaneni	3d7119d8a7	Add a histogram for chunk encoding time (#565) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Tom Wilkie	beb0052ad0	Remove unnecessarily long wait times in tests. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Edward Welch	6446503593	Close all chunks before flushing. ... Moved the check for already closed to make sure we honor this check when `immediate` flushing is requested	7 years ago
Tom Wilkie	80bbd04778	Add an extra filter step if the user specifies a regex using old API. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	32d2f64b31	Fix ingester test. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	bfb34e4070	Wire up LogQL filters in the ingester. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	b16da1d7c3	Parse pipelines of regex matches. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Cyril Tovena	fa4f936fcb	improves memory usage of golangci-linter	7 years ago
Cyril Tovena	8d7efddde2	switch to golangci-lint linter (#539) ... * switch to golangci-lint linter * fixes for review comments	7 years ago
Goutham Veeramachaneni	017bcefb3f	Better buckets for chunk sizes (#529) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Ed	fe53924ff8	using the timestamp from the log file might lead to undesired behavior as it could be way in the past or future depending on the clock for the source system generating the logs. Given that it's compared to the servers time.Now() when looking for idle chunks it's probably best to use the server time when setting the value. (#546)	7 years ago
Steven Sheehy	85a9e82663	Fix #531 chunk_block_size not found ... Signed-off-by: Steven Sheehy <ssheehy@firescope.com>	7 years ago
Xiang Dai	aa4684ee52	support config chunk size (#464) ... * support config chunk size Signed-off-by: Xiang Dai <764524258@qq.com> * fix lint Signed-off-by: Xiang Dai <764524258@qq.com> * review feedback	7 years ago
Goutham Veeramachaneni	c1bc17e778	fix linting (#514) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Goutham Veeramachaneni	e54a1b21c3	vendor: update cortex (#512) ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago