222 Commits (61b000ee0e6a6095ef194e18f47a42ebb51c2acb)

Author	SHA1	Message	Date
Fabian Reinartz	0db4c227b7	Fix min/max time handling and concurrent crc32 usage	8 years ago
Fabian Reinartz	81222849bc	Filter WAL data in Head, misc fixes	8 years ago
Fabian Reinartz	33e9bdf403	WAL refactoring and truncation fixes and test	8 years ago
Fabian Reinartz	c36d574290	Replace single head lock with granular locks ... This adds various new locks to replace the single big lock on the head. All parts now must be COW as they may be held by clients after initial retrieval. Series by ID and hashes are now held in a stripe lock to reduce contention and total holding time during GC. This should reduce starvation of readers.	8 years ago
Fabian Reinartz	1ddedf2b30	Change series ID from uint32 to uint64	8 years ago
Goutham Veeramachaneni	1698c516ad	[WIP]: WAL implementation ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	893b6ec506	Add tests for GC and chunk truncation	8 years ago
Fabian Reinartz	4f037da462	Remove defer statement in hot path	8 years ago
Fabian Reinartz	5cf2662074	Refactor WAL into Head and misc improvements	8 years ago
Fabian Reinartz	8209e3ec23	Add various metrics	8 years ago
Fabian Reinartz	3901b6e70b	Remove multiple heads ... This changes the structure to a single WAL backed by a single head block. Parts of the head block can be compacted. This relieves us from any head amangement and greatly simplifies any consistency and isolation concerns by just having a single head.	8 years ago
Goutham Veeramachaneni	7438ed7035	Expose Intervals type for use by TombstoneReader. ... TombstoneReader is exposed but Intervals is not. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	905af27cf9	Refactor compactor	8 years ago
Fabian Reinartz	66ff7b12e9	Pool Chunk objects during compaction	8 years ago
Fabian Reinartz	2644c8665c	Don't allocate ChunkMetas, reuse postings slices	8 years ago
Fabian Reinartz	96d7f540d4	Persist series without allocating the full set ... Change index persistence for series to not be accumulated in memory before being written as one large batch. `Labels` and `ChunkMeta` objects are reused. This cuts down memory spikes during compaction of multiple blocks significantly. As part of the the Index{Reader,Writer} now have an explicit notion of symbols and series must be inserted in order.	8 years ago
Goutham Veeramachaneni	f1ae239c20	Persist the right MaxTime when snapshotting ... This is because we cut a new block from where the snapshotted block ends if we restore from backups and highTimestamp would be where we should be starting from. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	1e74c155eb	Return empty string to signal non-caching	8 years ago
Fabian Reinartz	3410559c1b	Compact head block early ... Let older head blocks be compacted once the newest once has samples at 50% of its total range. This allows the memory of the compacted blocks to be released and garbage collected before a new head block gets created. Thereby the number of head blocks is 1 or 2 instead of 2 or 3 and memory spikes are reduced.	8 years ago
Goutham Veeramachaneni	73cc5bae51	Colocate defer statements near relevant functions ... Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>	8 years ago
Goutham Veeramachaneni	b51a05044e	Fix race condition for 2 appenders having same ts ... Race: Suppose we have 100 existing series inside a HeadBlock. Now we open two appenders in two routines A1, A2 and append 30 new series and 60 new series respectively with some common series. Both try to commit at the same time and the following happens in the given order: A2 executes createSeries() A1 executes createSeries() (with its common series referencing the ids from A2) A1 persists its newlabels, samples A2 persists its newlabels, samples Now when reading it back, we read A1's samples which reference A2's id and thereby fail. Ref: prometheus/promtheus#2795 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	05e411a8eb	Improve heuristic to spread chunks across block	8 years ago
Goutham Veeramachaneni	a110a64abd	Add full Snapshot support ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	a1c8425357	Initial implementation of HeadBlock Snapshots ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	29c73f05f2	Make sure that mint and maxt are not modified. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	44e9ae38b5	Incorporate PR feedback. ... * Expose Stone as it is used in an exported method. * Move from tombstoneReader to []Stone for the same reason as above. * Make WAL reading a little cleaner. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	6febabeb28	Final delete fixes. ... * Make sure no reads happen on the block when delete is in progress. * Fix bugs in compaction. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	c211ec4f49	Fix concurrent map access. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	f29fb62fba	Make TombstoneReader a Getter. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	9bf7aa9af1	Misc. fixes incorporating feedback. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	31cf939448	Add NumTombstones to BlockMeta. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	3eb4119ab1	Make HeadBlock use WAL. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	244b73fce1	Rename for clarity and consistency. ... Misc. changes for code cleanliness. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	662d8173fe	Make Appends after Delete visible. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	d6bd64357b	Fix Delete on HeadBlock ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	45d3db4e9e	Use a *mapTombstoneReader instead of map ... We need to recalculate the sorted ref list everytime we make a Tombstones() call. This avoids that. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	39df7e2bba	Switch blocks to ULID directories, drop sequenc numbers	8 years ago
Fabian Reinartz	285bc07030	Switch append refs to string	8 years ago
Goutham Veeramachaneni	22c1b5b492	Make SeriesSets use tombstones. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	34a86af3c6	Move tombstones to their own thing. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	cea3c88f17	Add Tombstones() method to Block. ... Also add Seek() to TombstoneReader Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	4f1d857590	Implement Delete on HeadBlock ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	5579efbd5b	Initial implentation of Deletes on persistedBlock ... Very much a WIP Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	8b51b7e2be	Make WAL for HeadBlock composeable.	8 years ago
Fabian Reinartz	4862b261d0	Abstract WAL into interface	8 years ago
Fabian Reinartz	535532ca02	Export refdSample ... The type was part of a exported method signatures and should therefore be exported as well.	8 years ago
Fabian Reinartz	5534e6c53c	Make HeadBlock impl public, make interface private	8 years ago
Goutham Veeramachaneni	2fa647f50b	Fix missing postings in Merge and Intersect (#77) ... * Test for a previous implematation of Intersect Before we were moving the postings list everytime we create a new chained `intersectPostings`. That was causing some postings to be skipped. This test fails on the older version. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Advance on Seek only when valid. Issue: Before in mergedPostings and others we advance everytime we `Seek`, which causes issues with `Intersect`. Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek. Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which would advance a and b beyond 30. Fix: Now we advance only when the seeking value is greater than the current value, as the definition specifies. Also, posting 0 will not be a valid posting and will be used to signal finished or un-initialized PostingsList. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Add test for Merge+Intersect edgecase. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Add comments to trivial tests. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	8096d11e4e	Add bounds check to headBlockAppender ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	adaf4d2099	Handle duplicate & out of order values in same txn ... Add docs about not erroring out on exact dupes. Moved tests to require.* Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago