49 Commits (fb9da52b1139947a8124de486d45f1d228bd3920)

Author	SHA1	Message	Date
Fabian Reinartz	fb9da52b11	Add more verbose error handling for closing, reduce locking ... This commit introduces error returns in various places and is explicit about closing persisted blocks. {Index,Chunk,Tombstone}Readers are more consistent about their Close() method. Whenever a reader is retrieved, the corresponding close method must eventually be called. We use this to track pending readers against persisted blocks. Querier's against the DB no longer hold a read lock for their entire lifecycle. This avoids long running queriers to starve new ones when we have to acquire a write lock when reloading blocks.	8 years ago
Fabian Reinartz	bbe72dccb9	Remove dependency on etcd/pkg/fileutil	8 years ago
Fabian Reinartz	78df406dac	Allocate and cache strings for persisted blocks ... This change loads the full symbol table when we open a persisted block and allocates a string for each. This ensures that strings retrieved through the index can be used after the block was closed. Before we backed the strings by the mmap'd byte regions which would segfault in this case. Also remove an inconsistency in the disk format and move both offset tables to the end (breaking change).	8 years ago
Goutham Veeramachaneni	8919baef03	Expose NewIndexReader() and cleanups ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	b09d90c79c	Add decoding method to retrieve unsafe strings ... When decoding data from mmaped blocks, we would like to retrieve a string backed by the mmaped region. As the underlying byte slice never changes, this is safe.	8 years ago
Goutham Veeramachaneni	afaf12fe45	Compress the series chunk details in index. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	1ddedf2b30	Change series ID from uint32 to uint64	8 years ago
Matt Layher	78b15c3434	Add newCRC32 function to simplify hash initialization	8 years ago
Fabian Reinartz	2644c8665c	Don't allocate ChunkMetas, reuse postings slices	8 years ago
Fabian Reinartz	96d7f540d4	Persist series without allocating the full set ... Change index persistence for series to not be accumulated in memory before being written as one large batch. `Labels` and `ChunkMeta` objects are reused. This cuts down memory spikes during compaction of multiple blocks significantly. As part of the the Index{Reader,Writer} now have an explicit notion of symbols and series must be inserted in order.	8 years ago
Goutham Veeramachaneni	a110a64abd	Add full Snapshot support ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	9 years ago
Goutham Veeramachaneni	34a86af3c6	Move tombstones to their own thing. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	9 years ago
Goutham Veeramachaneni	4f1d857590	Implement Delete on HeadBlock ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	9 years ago
Goutham Veeramachaneni	5579efbd5b	Initial implentation of Deletes on persistedBlock ... Very much a WIP Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	9 years ago
Fabian Reinartz	2032a11d98	Add padding between fixed-sized index sections	9 years ago
Fabian Reinartz	34ba92eeeb	Move CRC back to chunks file, alignment for fixed-sized ints	9 years ago
Fabian Reinartz	a54f46d5e7	Migrate last IndexWriter pieces to decbuf	9 years ago
Fabian Reinartz	94f3fd9812	Move encoding helpers into separate file	9 years ago
Fabian Reinartz	35b62f001e	Change offset table layout, add TOC, ...	9 years ago
Fabian Reinartz	8b1f514a2d	index: validate current write stages	9 years ago
Fabian Reinartz	9b4eafcc4c	Simplify and document postings serialization	9 years ago
Fabian Reinartz	0aad526d1a	Simplify label value index ... This removes the flag from the label value index and serializes it using encbufs. TODO: move CRC32 checksum into label value index hash table for referntial integrity.	9 years ago
Fabian Reinartz	d30b181406	Switch series serialization to use encbufs	9 years ago
Fabian Reinartz	2ebaf1af4f	Add encode buffer and simplify symbol serialization	9 years ago
Fabian Reinartz	433e73f865	Change series and symbol table format	9 years ago
Fabian Reinartz	df96d97dab	Move chunk checksum	9 years ago
Julius Volz	8d1fb4fa01	Minor comment fixes and additions.	9 years ago
Fabian Reinartz	778103b450	Add liecence file and headers	9 years ago
Fabian Reinartz	7de2217011	Add fast-path for equality matching	9 years ago
Fabian Reinartz	10c7c9acbe	Adjust import names to new repository organisation	9 years ago
Goutham Veeramachaneni	71e05a22c7	Add mockIndex And Refactor Tests To Use That	9 years ago
Goutham Veeramachaneni	7b94a4e17d	Rename bytePostings To bigEndianPostings ... * To be more specific about the contents of the byte slice.	9 years ago
Goutham Veeramachaneni	efb0dfe1be	Implement Postings Iterator Over Bytes ... Closes fabxc/tsdb#18	9 years ago
Fabian Reinartz	2ef3682560	Hotfix erroneous "label index missing" error	9 years ago
Fabian Reinartz	a8e8903350	Use ChunkMeta references for clarity ... This has been a common source of hard to debug issues. Its a premature and unbenchmarked optimization and semantically, we want ChunkMetas to be references in all changed cases.	9 years ago
Fabian Reinartz	8a7addfc44	Split persistence by chunk/index instead of read/write	9 years ago
Fabian Reinartz	3e569bc964	Optimize memory allocations in writer	9 years ago
Fabian Reinartz	9a5dfadb09	Tweak IndexWriter interface, serialize directly into dir	9 years ago
Fabian Reinartz	78780cd2ba	Segment chunk file ... This adds write path support for segmented chunk data files. Files of 512MB are pre-allocated and written to. If the file size is exceeded, the next file is started. On completion, files are truncated to their final size.	9 years ago
Fabian Reinartz	c808928b90	SeriesWriter -> ChunkWriter ... This commit simplifies a SeriesWriter into a ChunkWriter and detaches it entirely from the notion of a series and the series index.	9 years ago
Fabian Reinartz	a3d042b54e	Support multiple chunk files in read path	9 years ago
Fabian Reinartz	afa084920c	Use Castagnoli polynomial for CRC32s; cache them	9 years ago
Fabian Reinartz	ac5229e1b4	Correctly write empty blocks and extend appenders by new blocks	9 years ago
Fabian Reinartz	30d8866c13	Limit compaction range, make cut and compact concurrent	9 years ago
Fabian Reinartz	d4779b374c	Properly track and write meta file	9 years ago
Fabian Reinartz	9ddbd64d00	Move stats into meta.json file, cleanup, docs	9 years ago
Fabian Reinartz	0dffd52238	Use page writer in compaction	9 years ago
Fabian Reinartz	1943f8d1bb	Fix head block stats races	9 years ago
Fabian Reinartz	dbd2b21d2e	Make persistence atomic	9 years ago
Fabian Reinartz	62b8ded0a5	Rename iterator value getters to At()	9 years ago