6 Commits (096cf0cfaf3fe62cb1bcc86945229e64dfa326ed)

Author	SHA1	Message	Date
Owen Diehl	05b8537e79	WAL Replay counter (#3974) ... * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting * introduces an entry counter into streams & wal records * sets & resets entrycount on checkpoint recovery and after wal replay finishes * pushing to streams checks counter * reject pre-applied entries test * removes commented test	4 years ago
Owen Diehl	862cfa4e1f	adds Start() method to WAL interface to delay checkpointing until after replay (#3339)	4 years ago
Owen Diehl	8a9b94a8bc	removes recover flag (#3326)	4 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295) ... * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	4 years ago
Owen Diehl	224e68a50b	WAL backpressure (#3218) ... * adds WAL replay backpressure * backpressure fn doesnt lock during passed fn for concurrency * replayController * replayController supports a RemoveFlushedChunks fn * custom wal flusher * encoding test determinism * standardizes recoveredBytesTotal to replayControllers ownership * linting * default wal memory threshold to 4gb, adds docs * fix test after v3 chunk schema change * more lenient expectCheckpoint function * replayController protects against flush race condition * adds replay_memory_ceiling to wal jsonnet * replay ceiling help msg	4 years ago
Owen Diehl	4d9865acd4	Adds WAL support (experimental) (#2981) ... * marshalable chunks * wal record types custom serialization * proto types for wal checkpoints * byteswith output unaffected by buffer * wal & record pool ifcs * wal record can hold entries from multiple series * entry pool * ingester uses noopWal * removes duplicate argument passing in ingester code. adds ingester config validation & derives chunk encoding. * segment writing * [WIP] wal recovery from segments * replay uses sync.Maps & preserves WAL fingerprints * in memory wal recovery * wal segment recovery * ingester metrics struct * wal replay locks streamsMtx in instances, adds checkpoint codec * ingester metrics * checkpointer * WAL checkpoint writer * checkpointwriter can write multiple checkpoints * reorgs checkpointing * wires up checkpointwriter to wal * ingester SeriesIter impl * wires up ingesterRecoverer to consume checkpoints * generic recovery fn * generic recovery fn * recover from both wal types * cleans up old tmp checkpoints & allows aborting in flight checkpoints * wires up wal checkpointing * more granular wal logging * fixes off by 1 wal truncation & removes double logging * adds userID to wal records correctly * wire chunk encoding tests * more granular wal metrics * checkpoint encoding test * ignores debug bins * segment replay ignores out of orders * fixes bug between WAL reading []byte validity and proto unmarshalling refs * conf validations, removes comments * flush on shutdown config * POST /ingester/shutdown * renames flush on shutdown * wal & checkpoint use same segment size * writes entries to wal regardless of tailers * makes wal checkpoing duration default to 5m * recovery metrics * encodes headchunks separately for wal purposes * merge upstream * linting * addresses pr feedback uses entry pool in stream push/tailer removes unnecessary pool interaction checkpointbytes comment fillchunk helper, record resetting in tests via pool redundant comment defers wg done in recovery s/num/count/ checkpoint wal uses a logger encodeWithTypeHeader now creates its own []byte removes pool from decodeEntries wal stop can error * prevent shared access bug with tailers and entry pool * removes stream push entry pool optimization	5 years ago