72 Commits (c3ffdf1a9968dc505a68d541cac0d7201ccfaba1)

Author	SHA1	Message	Date
Krasi Georgiev	c3ffdf1a99	Test createBlock and check all os.RemoveAll in the tests for errors. (#549) ... Testing that createBlock creates blocks that can be opened. and checking the os.RemoveAll for errors will catch errors for un-closed files under windows. Many missing .Close() calls were added for fixing failing os.RemoveAll Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Ganesh Vernekar	c59ed492b2	Vertical query merging and compaction (#370) ... * Vertical series iterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Select overlapped blocks first in compactor Plan() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Added vertical compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Code cleanup and comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add benchmark for compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Perform vertical compaction only when blocks are overlapping. Actions for vertical compaction: * Sorting chunk metas * Calling chunks.MergeOverlappingChunks on the chunks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for vertical compaction * BenchmarkNormalCompaction => BenchmarkCompaction * Moved the benchmark from db_test.go to compact_test.go Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for query iterator and seek for non overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Vertical query merge only for overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Simplify logging in Compact(...) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG.md Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Calculate overlapping inside populateBlock Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * MinTime and MaxTime for BlockReader. Using this to find overlapping blocks in populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Sort blocks w.r.t. MinTime in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping in LeveledCompactor.write() instead of returning bool Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping inside LeveledCompactor.populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor createBlock to take optional []Series Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * review1 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com> * Updated CHANGELOG and minor nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor iterator and seek benchmarks for Querier. Also has as overlapping blocks. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Additional test case Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Split genSeries into genSeries and populateSeries Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Check error in benchmark Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Warn about overlapping blocks in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	dac2b97dfd	make createBlock more generic so it can be used in other tests. (#489) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
mknapphrt	ebf5d74325	Added storage size based retention method and new metrics (#343) ... Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed. 2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded. Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>	7 years ago
Simon Pasquier	d5d7a097e1	Update Makefile.common ... This change also uses the latest staticcheck version which comes with new verifications, hence some clean up in the code. Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Krasi Georgiev	eb6586f513	rename createPopulatedBlock to createBlock and use it instead ot createEmptyBlock (#488) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Krasi Georgiev	a90a7195d4	createPopulatedBlock returns the block dir instead of the block (#487) ... It is easy to forget to close the block returned by createPopulatedBlock which causes failures for windows so instead it returns the block dir and which can be used by OpenBlock explicitly. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Krasi Georgiev	090b6852e1	remove unused `PrefixMatcher` (#474) ... * remove unused `PrefixMatcher` Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Krasi Georgiev	bac9cbed2e	no overlapping on compaction when an existing block is not within default boundaries. (#461) ... closes https://github.com/prometheus/prometheus/issues/4643 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Krasi Georgiev	5a9ddeecef	fix lint errors (#439) ... unexported NewMemTombstones as this returns unexported memTombstones type which will not be shows in godoc. Added missing comments for exported methods. Removed unused RecordLogger,RecordReader interfaces. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Ganesh Vernekar	3a08a71d86	LabelNames() method to get all unique label names (#369) ... * LabelNames() method to get all unique label names Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	7 years ago
Krasi Georgiev	d804a27062	refactor util funcs to allow re-usage. (#419) ... * refactor util funcs to allow reusage. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Ben	a8351dc9d0	Using filepath.Join() instead of strings with slashes (#428) ... fixes: https://github.com/prometheus/tsdb/issues/426 Using `filepath.Join()` instead of strings containing forward slash path delimiters (needed for non-*nix OSes), as suggested by @krasi-georgiev	7 years ago
Chris Marchbanks	f4afc7dff2	Add benchmark for querying a persisted block (#425) ... Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	7 years ago
Bartek Płotka	047b1b1357	compact: Verify for chunks outside of compacted time range. Added unit test for populateBlocs. (#349) ... * compact: Verify for chunks outside of compacted time range. Unit test for populateBlocs. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Krasi Georgiev	d05611c027	removed some unused code and moved mockSeriesSet in querier_test (#394) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Alexey Palazhchenko	9be883670e	fix the test data file path in benchmarks. (#345) ... Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>	7 years ago
codwu	bc6ef0b94e	rename `mts` to `intvlGroups` ... Signed-off-by: codwu <wuhan9087@163.com>	7 years ago
codwu	e4444ca48c	update `addInterval` function and test. ... Signed-off-by: codwu <wuhan9087@163.com>	7 years ago
codwu	cd145c90d5	remove `put` function and use RLock in `Iter` function ... Signed-off-by: codwu <wuhan9087@163.com>	7 years ago
codwu	84a45cb79a	add rwmutex to prevent concurrent map read when delete series ... Signed-off-by: codwu <wuhan9087@163.com>	7 years ago
Fabian Reinartz	54205903f7	Move index and chunk encoders to own packages	8 years ago
Fabian Reinartz	adb6be73dd	move label read function int labels pkg	8 years ago
Fabian Reinartz	67f0ca8f0e	Move index and chunk encoders to own packages	8 years ago
Callum Styan	ca4e817372	move test utils to it's own package so we can use it in tests for other packages in tsdb repo	8 years ago
Callum Styan	a1a91ea96a	use test utils in querier_test	8 years ago
Fabian Reinartz	3ef4326114	Refactor tombstone reader types	8 years ago
Fabian Reinartz	e5ce2bef43	Add explicit error to Querier.Select ... This has been a frequent source of debugging pain since errors are potentially delayed to a much later point. They bubble up in an unrelated execution path.	8 years ago
Fabian Reinartz	665955da48	Clarify postings index semantics, handle staleness ... The postings list index may point to series that no longer exist during garbage collection. This clarifies that this is valid behavior. It would be possible, though more complex, to always keep them in sync. However, series existance means nothing in itself as the queried time range defines whether there's actual data. Thus our definition is sane overall as long as drift is kept small.	8 years ago
Fabian Reinartz	c36d574290	Replace single head lock with granular locks ... This adds various new locks to replace the single big lock on the head. All parts now must be COW as they may be held by clients after initial retrieval. Series by ID and hashes are now held in a stripe lock to reduce contention and total holding time during GC. This should reduce starvation of readers.	8 years ago
Fabian Reinartz	1ddedf2b30	Change series ID from uint32 to uint64	8 years ago
Fabian Reinartz	3901b6e70b	Remove multiple heads ... This changes the structure to a single WAL backed by a single head block. Parts of the head block can be compacted. This relieves us from any head amangement and greatly simplifies any consistency and isolation concerns by just having a single head.	8 years ago
Goutham Veeramachaneni	7438ed7035	Expose Intervals type for use by TombstoneReader. ... TombstoneReader is exposed but Intervals is not. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	2644c8665c	Don't allocate ChunkMetas, reuse postings slices	8 years ago
Fabian Reinartz	96d7f540d4	Persist series without allocating the full set ... Change index persistence for series to not be accumulated in memory before being written as one large batch. `Labels` and `ChunkMeta` objects are reused. This cuts down memory spikes during compaction of multiple blocks significantly. As part of the the Index{Reader,Writer} now have an explicit notion of symbols and series must be inserted in order.	8 years ago
Dmitry Ilyevsky	37194b7a30	Add prefix label matcher. ... Implement labels.PrefixMatcher and use interface conversion in querier to optimize label tuples search. [unit-tests]: Fix bug and populate label index for mock index. Signed-off-by: Dmitry Ilyevsky <ilyevsky@gmail.com>	8 years ago
Fabian Reinartz	bc3fee4820	Add benchmark for mergedSeriesSet	8 years ago
Fabian Reinartz	bda3ed20ac	Fix omitting of chunk on Seek()	8 years ago
Goutham Veeramachaneni	a280169427	Add regression test ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	f29fb62fba	Make TombstoneReader a Getter. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	244b73fce1	Rename for clarity and consistency. ... Misc. changes for code cleanliness. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	7ccf9d0211	Add test on blockQuerier which uses tombstones. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	d6bd64357b	Fix Delete on HeadBlock ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	22c1b5b492	Make SeriesSets use tombstones. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	7bbbd55aad	Fix bug where having one chunk can cause panic ... When we have only one chunk that is out of range, then we are returning it unpopulated (w/o calling `Chunk(ref)`). This would cause a panic downstream. Fixes: prometheus/prometheus#2629 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	8b43b0d2c1	Fix broken tests ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	6169c33fb8	Fix #59 ... Mutating the chunks can change their length. Hence referencing using previous indices can cause panics. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	6178de9acc	Removed unused feild ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	725b69caa1	Incorporate feedback on #36 ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	239f8b9eb5	Modify tests to check for mint, maxt ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago