prometheus

Commit Graph

Author	SHA1	Message	Date
johncming	252265fa81	tsdb/chunks: remove un-used code. (#6944 ) Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Bartlomiej Plotka	e6d7cc5fa4	tsdb: Added ChunkQueryable implementations to db; unified MergeSeriesSets and vertical to single struct. (#7069 ) * tsdb: Added ChunkQueryable implementations to db; unified compactor, querier and fanout block iterating. Chained to https://github.com/prometheus/prometheus/pull/7059 * NewMerge(Chunk)Querier now takies multiple primaries allowing tsdb DB code to use it. * Added single SeriesEntry / ChunkEntry for all series implementations. * Unified all vertical, and non vertical for compact and querying to single merge series / chunk sets by reusing VerticalSeriesMergeFunc for overlapping algorithm (same logic as before) * Added block (Base/Chunk/)Querier for block querying. We then use populateAndTomb(Base/Chunk/) to iterate over chunks or samples. * Refactored endpoint tests and querier tests to include subtests. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comments from Brian and Beorn. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed snapshot test and added chunk iterator support for DBReadOnly. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed race when iterating over Ats first. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed tests. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed populate block tests. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed endpoints test. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed test. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added test & fixed case of head open chunk. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed DBReadOnly tests and bug producing 1 sample chunks. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added cases for partial block overlap for multiple full chunks. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added extra tests for chunk meta after compaction. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed small vertical merge bug and added more tests for that. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Ganesh Vernekar	b8a7e80f9b	Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	7a763ff61e	Avoid empty mmap files by using .tmp files to write headers Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Brad Walker	3348930df5	Replace fileutil.ReadDir with ioutil.ReadDir (#7029 ) (#7033 ) * tsdb: Replace fileutil.ReadDir with ioutil.ReadDir (#7029) Signed-off-by: Brad Walker <brad@bradmwalker.com> * tsdb: Remove fileutil.ReadDir (#7029) Signed-off-by: Brad Walker <brad@bradmwalker.com>	5 years ago
Bartlomiej Plotka	d5c33877f9	storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. (#7005 ) * storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. ## Rationales: In many places (e.g. chunk Remote read, Thanos Receive fetching chunk from TSDB), we operate on encoded chunks not samples. This means that we unnecessary decode/encode, wasting CPU, time and memory. This PR adds chunk iterator interfaces and makes the merge code to be reused between both seriesSets I will make the use of it in following PR inside tsdb itself. For now fanout implements it and mergers. All merges now also allows passing series mergers. This opens doors for custom deduplications other than TSDB vertical ones (e.g. offline one we have in Thanos). ## Changes * Added Chunk versions of all iterating methods. It all starts in Querier/ChunkQuerier. The plan is that Storage will implement both chunked and samples. * Added Seek to chunks.Iterator interface for iterating over chunks. * NewMergeChunkQuerier was added; Both this and NewMergeQuerier are now using generigMergeQuerier to share the code. Generic code was added. * Improved tests. * Added some TODO for further simplifications in next PRs. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Moved s/Labeled/SeriesLabels as per Krasi suggestion. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Krasi's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Second iteration of Krasi comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Another round of comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Ganesh Vernekar	e50fdbc70c	Live m-mapping of chunks on disk (#6830 ) * Live m-mapping of chunks on disk Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Part 2 Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Part 3 Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Part 4 Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Attempt to fix windows bug Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	0a27df92f0	Refactor tsdb/chunks/chunks.go for future PRs (#6754 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Bartlomiej Plotka	c2e083c30f	Fixed race in Chunks method. (#6515 ) Added regression test. Fixes #6512 Before (not deterministic result due to concurrency): ``` === RUN TestChunkReader_ConcurrentRead --- FAIL: TestChunkReader_ConcurrentRead (0.01s) db_test.go:2702: unexpected error: checksum mismatch expected:597ad276, actual:597ad276 db_test.go:2702: unexpected error: checksum mismatch expected:dd0cdbc2, actual:dd0cdbc2 FAIL ``` After succuess on multiple runs. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	6 years ago
Bartlomiej Plotka	3b8ef6386c	Fixed race in Chunks method. (#6515 ) Added regression test. Fixes #6512 Before (not deterministic result due to concurrency): ``` === RUN TestChunkReader_ConcurrentRead --- FAIL: TestChunkReader_ConcurrentRead (0.01s) db_test.go:2702: unexpected error: checksum mismatch expected:597ad276, actual:597ad276 db_test.go:2702: unexpected error: checksum mismatch expected:dd0cdbc2, actual:dd0cdbc2 FAIL ``` After succuess on multiple runs. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	6 years ago
Krasimir Georgiev	549164f252	return err instead of panic for corrupted chunk (#6040 ) * Fix tsdb panic when querying corrupted chunks. check that the chunk segment has enough data to read all chunk pieces. * refactor, simplify and add tests. * simpfiy WriteChunks implementation Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>	6 years ago
Ganesh Vernekar	5ecef3542d	Cleanup after merging tsdb into prometheus Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Ganesh Vernekar	7cf09b0395	Moving tsdb into its own subdirectory Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Yao Zengzeng	56578602ab	remove redundant condition judgement whether cut new chunk (#669 ) Signed-off-by: YaoZengzeng <yaozengzeng@zju.edu.cn>	6 years ago
Yao Zengzeng	bff3ef3db4	minor enhancement: return multi error when closeAll() (#663 ) Signed-off-by: YaoZengzeng <yaozengzeng@zju.edu.cn>	6 years ago
Ganesh Vernekar	3bc1ea3d75	Reuse byte buffer in WriteChunks and writeHash (#653 ) * Re-use byte buffer in WriteChunks and writeHash Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add CHANGELOG entry Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Remove old comment Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
johncming	84d71073f1	chunks: correct the interface name Signed-off-by: johncming <johncming@yahoo.com>	6 years ago
Ganesh Vernekar	b1cd829030	Reuse Chunk Iterator (#642 ) * Reset method for chunkenc.Iterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Reset method only for XORIterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Use Reset(...) in querier.go Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Reuse deletedIterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Another way of reusing chunk iterators Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Unexport xorIterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix memSeries.iterator(...) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add some comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	31f7990d1d	Re-encode chunks that are still being appended to when snapshoti… (#641 ) * re encode all head chunks outside the time range. Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>	6 years ago
Krasi Georgiev	8eeb70fee1	remove Fsync workaround for macos. (#574 ) since golang 1.12 no special handling is required for file.Sync() @pborzenkov thanks for the pointer. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Pavel Borzenkov	4f204dcbc1	chunks: close all opened MmapFiles on NewDirReader error path (#575 ) Signed-off-by: Pavel Borzenkov <pavel.borzenkov@gmail.com>	6 years ago
zhulongcheng	e11e01f68d	Fix exported function comments (#566 ) Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>	6 years ago
zhulongcheng	4d03c70800	Verify chunk format version (#544 ) Verify chunk format version.	6 years ago
zhulongcheng	b7cac504c9	Remove unused NewReader function (#547 ) The `NewReader` function is not used anywhere.	6 years ago
Alec	0861a9b14f	[bugfix] update 'last' variable in chunks.MergeOverlappingChunks() (#539 ) * update Signed-off-by: naivewong <867245430@qq.com> * update comment Signed-off-by: naivewong <867245430@qq.com> * Update failing test. Signed-off-by: naivewong <867245430@qq.com> * Update comment Signed-off-by: naivewong <867245430@qq.com> * Add changelog entry. Signed-off-by: naivewong <867245430@qq.com> * Update CHANGELOG.md Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Ganesh Vernekar	c59ed492b2	Vertical query merging and compaction (#370 ) * Vertical series iterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Select overlapped blocks first in compactor Plan() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Added vertical compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Code cleanup and comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add benchmark for compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Perform vertical compaction only when blocks are overlapping. Actions for vertical compaction: * Sorting chunk metas * Calling chunks.MergeOverlappingChunks on the chunks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for vertical compaction * BenchmarkNormalCompaction => BenchmarkCompaction * Moved the benchmark from db_test.go to compact_test.go Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for query iterator and seek for non overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Vertical query merge only for overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Simplify logging in Compact(...) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG.md Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Calculate overlapping inside populateBlock Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * MinTime and MaxTime for BlockReader. Using this to find overlapping blocks in populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Sort blocks w.r.t. MinTime in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping in LeveledCompactor.write() instead of returning bool Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping inside LeveledCompactor.populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor createBlock to take optional []Series Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * review1 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com> * Updated CHANGELOG and minor nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor iterator and seek benchmarks for Querier. Also has as overlapping blocks. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Additional test case Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Split genSeries into genSeries and populateSeries Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Check error in benchmark Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Warn about overlapping blocks in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	bf2239079d	refactor multi errors Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	ee99718ff6	rename chunk reader vars to make it easier to follow. (#508 ) * rename chunk reader vars to make it easyer to understand. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Alec	051a7ae1a7	Missing the length of the encoding byte when calling b.Range Signed-off-by: naivewong <867245430@qq.com>	7 years ago
mknapphrt	ebf5d74325	Added storage size based retention method and new metrics (#343 ) Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed. 2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded. Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>	7 years ago
naivewong	bff5aa4d21	Missing the len of crc32 when calculating maxLen in WriteChunks (#494 ) Signed-off-by: naivewong <867245430@qq.com>	7 years ago
Simon Pasquier	d5d7a097e1	Update Makefile.common This change also uses the latest staticcheck version which comes with new verifications, hence some clean up in the code. Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Benoît Knecht	1e1b2e163d	Make interval overlap comparisons more explicit Blocks are half-open intervals [a, b), while all other intervals (chunks, head, ...) are closed intervals [a, b]. Make that distinction explicit by defining `OverlapsClosedInterval()` methods for blocks and chunks, and using them in place of the more generic `intervalOverlap()` function. This change also fixes `db.Querier()` and `db.Delete()`, which could previously return one extraneous block at the end of the specified interval. Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>	7 years ago
Simon Pasquier	ee5fe8ea9f	chunks: fix potential "index out of range" error When binary.Uvarint() fails, the returned number of bytes is less than or equal to zero. Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Mario Trangoni	c0e888e82b	fix megacheck issues: os.SEEK_SET is deprecated: Use io.SeekStart, io.SeekCurrent, and io.SeekEnd.	7 years ago
Mario Trangoni	09142e4dd1	fix unconvert issues: unnecessary conversion	7 years ago
Fabian Reinartz	1e55b7987f	Improve comments, handle allPostingsKey properly	8 years ago
Fabian Reinartz	67f0ca8f0e	Move index and chunk encoders to own packages	8 years ago
Fabian Reinartz	ac5bd71d8f	Doc fixes	8 years ago
Fabian Reinartz	d578b10d5c	chunk: make reader accept abstract ByteSlice	8 years ago
ranbochen	a27cf34a36	fix bugs on platform windows to pass all test case. (#192 ) * fix bugs on platform windows to pass all test case. * fix bugs on platform windows to pass all test case * clean up codes	8 years ago
ranbo chen	90de381cdb	fix bugs on platform windows to pass all test case	8 years ago
ranbo chen	b82b2d0c1b	fix bugs on platform windows to pass all test case.	8 years ago
Fabian Reinartz	fb9da52b11	Add more verbose error handling for closing, reduce locking This commit introduces error returns in various places and is explicit about closing persisted blocks. {Index,Chunk,Tombstone}Readers are more consistent about their Close() method. Whenever a reader is retrieved, the corresponding close method must eventually be called. We use this to track pending readers against persisted blocks. Querier's against the DB no longer hold a read lock for their entire lifecycle. This avoids long running queriers to starve new ones when we have to acquire a write lock when reloading blocks.	8 years ago
Fabian Reinartz	bbe72dccb9	Remove dependency on etcd/pkg/fileutil	8 years ago
Fabian Reinartz	5cf2662074	Refactor WAL into Head and misc improvements	8 years ago
Matt Layher	78b15c3434	Add newCRC32 function to simplify hash initialization	8 years ago
Goutham Veeramachaneni	7438ed7035	Expose Intervals type for use by TombstoneReader. TombstoneReader is exposed but Intervals is not. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	66ff7b12e9	Pool Chunk objects during compaction	8 years ago
Fabian Reinartz	2644c8665c	Don't allocate ChunkMetas, reuse postings slices	8 years ago

12 Commits (lbl-optimize)