269 Commits (3ad25a6dc3d94d09f94de00fe3ef9e374007b131)

Author	SHA1	Message	Date
Marco Pracucci	db19e05d93	Add option to customise head chunks write buffer size (#8201) ... * Add option to customise head chunks write buffer size Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed tests Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Bartlomiej Plotka	4513537034	Exposed DeletionIterator and CompactMetas functions. (#8161) ... * Exposed DeletionIterator and CompactMetas functions. Required for CLI for deletions in Thanos: https://github.com/thanos-io/thanos/pull/3421 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Removed Thanos usage mentions. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Ganesh Vernekar	601a3ef0bb	Read repair empty last file in chunks_head (#8061) ... * Read repair empty file in chunks_head Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor and introduce repairLastChunkFile Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Attempt windows test fix Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Julien Pivotto	df6eed2358	Calculate head chunk size based on actual disk usage (#8139) ... Cherry-picks 8bc369bf9b Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	8bc369bf9b	Calculate head chunk size based on actual disk usage (#8139) ... Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Marco Pracucci	63be30dcee	Fixed WAL corruption on partial writes within a page (#8125) ... * Fixed WAL corruption on partial writes Signed-off-by: Marco Pracucci <marco@pracucci.com> * Renamed variable Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed test Signed-off-by: Marco Pracucci <marco@pracucci.com> * Addressed review comments Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Julien Pivotto	6c56a1faaa	Testify: move to require (#8122) ... * Testify: move to require Moving testify to require to fail tests early in case of errors. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * More moves Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Bartlomiej Plotka	3d8826a3d4	MultiError: Refactored MultiError for more concise and safe usage. (#8066) ... * MultiError: Refactored MultiError for more concise and safe usage. * Less lines * Goland IDE was marking every usage of old MultiError "potential nil" error * It was easy to forgot using Err() when error was returned, now it's safely assured on compile time. NOTE: Potentially I would rename package to merrors. (: In different PR. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed review comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fix after rebase. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
johncming	28ca0965f0	tsdb/chunkenc: fix typo of return error. (#7670) ... * tsdb/chunkenc: fix typo of return error. Signed-off-by: johncming <johncming@yahoo.com> * tsdb: fix typo of function in markdonw. Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Ganesh Vernekar	3245b3267b	Don't use returned DB to close resources on TSDB startup error (#8113) ... * Don't use returned DB to close resources on TSDB startup error Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add unit test and fix another panic Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comment Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Julien Pivotto	1282d1b39c	Refactor test assertions (#8110) ... * Refactor test assertions This pull request gets rid of assert.True where possible to use fine-grained assertions. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	4e5b1722b3	Move away from testutil, refactor imports (#8087) ... Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Arthur Silva Sens	c5a832b394	Close resources after failing to startup TSDB (#8031) ... * Close resources after failing to startup TSDB Signed-off-by: arthursens <arthursens2005@gmail.com> * Return close error instead of logging Signed-off-by: arthursens <arthursens2005@gmail.com> * Change named return's name Signed-off-by: arthursens <arthursens2005@gmail.com>	5 years ago
Brian Brazil	fdf1c29224	Fix metric description of prometheus_tsdb_symbol_table_size_bytes. (#8080) ... This is how much memory we use to load in the on-disk symbol tables, not the size of the tables themselves. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Ganesh Vernekar	ce4b3ac282	Simplify TestHeadReadWriter_Truncate (#7437) ... * Simplify TestHeadReadWriter_Truncate Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	2624d827fa	Read repair empty last file in chunks_head (#8061) ... * Read repair empty file in chunks_head Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor and introduce repairLastChunkFile Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Attempt windows test fix Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Bartlomiej Plotka	2fe1e9fa93	Create a checkpoint only at the end of Compact call (#8067) ... * Create a checkpoint only at the end of Compact call Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix Bartek's offline reviews Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Introduce TruncateInMemory and TruncateWAL Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Small enhancements and test fixing attempts Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add TestOneCheckpointPerCompactCall Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Don't truncate WAL on block compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Simplified the algo. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Better protection around calling truncateWAL, truncate WAL on Head compaction error Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	d9cd913219	Add links to TSDB blog post series in tsdb/README.md (#8078) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Julien Pivotto	59733b1a26	TSDB: use `blocks` instead of db.blocks in condition (#8068) ... Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Ganesh Vernekar	5de7d15879	Consistent naming and better names for fields in TSDB format docs (#8077) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Bartlomiej Plotka	9981b3f3ee	labels: Reduce allocated memory and latency of Hash method; Added tests. (#8025) ... * labels: Reduce allocated memory by Hash method in edge cases; Added tests. Old: /tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$ goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/pkg/labels BenchmarkLabels_Hash BenchmarkLabels_Hash/typical_labels_under_1KB BenchmarkLabels_Hash/typical_labels_under_1KB-12 5366161 259 ns/op 0 B/op 0 allocs/op BenchmarkLabels_Hash/bigger_labels_over_1KB BenchmarkLabels_Hash/bigger_labels_over_1KB-12 1700371 767 ns/op 2048 B/op 1 allocs/op BenchmarkLabels_Hash/extremely_large_label_value_10MB BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 356 3743115 ns/op 10523442 B/op 1 allocs/op PASS New: /tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$ goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/pkg/labels BenchmarkLabels_Hash BenchmarkLabels_Hash/typical_labels_under_1KB BenchmarkLabels_Hash/typical_labels_under_1KB-12 4758883 259 ns/op 0 B/op 0 allocs/op BenchmarkLabels_Hash/bigger_labels_over_1KB BenchmarkLabels_Hash/bigger_labels_over_1KB-12 3324492 357 ns/op 80 B/op 1 allocs/op BenchmarkLabels_Hash/extremely_large_label_value_10MB BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 1087 1083949 ns/op 9734 B/op 1 allocs/op PASS Process finished with exit code 0 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Kemal's comment. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Moved to v2 xxhash for improvements. New: /tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$ -test.benchtime 10s goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/pkg/labels BenchmarkLabels_Hash BenchmarkLabels_Hash/typical_labels_under_1KB BenchmarkLabels_Hash/typical_labels_under_1KB-12 53447894 221 ns/op 0 B/op 0 allocs/op BenchmarkLabels_Hash/bigger_labels_over_1KB BenchmarkLabels_Hash/bigger_labels_over_1KB-12 42341754 326 ns/op 0 B/op 0 allocs/op BenchmarkLabels_Hash/extremely_large_label_value_10MB BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 10000 1248546 ns/op 0 B/op 0 allocs/op PASS Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Removed old xxhash package. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Pined minor version. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Ganesh Vernekar	4ed3ec7b46	More info in mmap error message (#8058) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
frbimo	260cd84d8f	log on empty head compaction result (#8037) ... Signed-off-by: frbimo <fr.bimo@gmail.com>	5 years ago
Gayathri Venkatesh	73e2ce1bd6	Do not ignore reload errors in compactHead (#8051) ... * Modified unknownRefs to unknownRefs.Load() Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com> * Modified db.go Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com> * Revert "Modified unknownRefs to unknownRefs.Load()" This reverts commit 79caf595fa9b9193878dc0dd9dec17d58851ae42. Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com> * Made changes to reload error in db.go Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>	5 years ago
Goutham Veeramachaneni	1cc02930b6	Implement series API query hint to not hit chunks (#8050) ... Fixes https://github.com/prometheus/prometheus/issues/5547 Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	5 years ago
Ling Jin	9145200842	tsdb: fix unkown ref in log (#8048) ... Signed-off-by: JinLingChristopher <jinl1037@hotmail.com>	5 years ago
Arthur Silva Sens	4f45e201cc	Promtool tsdb list now prints block sizes (#7993) ... * promtool tsdb list now prints blocks' size Signed-off-by: arthursens <arthursens2005@gmail.com>	5 years ago
Jess G	90680b092c	Add a tsdb writer interface (#7950) ... * add a tsdb writer interface Signed-off-by: jessicagreben <jessicagrebens@gmail.com> * use block writer in tsdbutil block creator Signed-off-by: jessicagreben <jessicagrebens@gmail.com> * pass thru context Signed-off-by: jessicagreben <jessicagrebens@gmail.com> * use block writer in blockutil and also in tests Signed-off-by: jessicagreben <jessicagrebens@gmail.com> * fix link err Signed-off-by: jessicagreben <jessicagrebens@gmail.com> * add err handling for defer Signed-off-by: jessicagreben <jessicagrebens@gmail.com> * add blockwriter test Signed-off-by: jessicagreben <jessicagrebens@gmail.com> * mv initHead into NewBlockWriter, update tests Signed-off-by: jessicagreben <jessicagrebens@gmail.com> * use blocksize arg, make test more concise Signed-off-by: jessicagreben <jessicagrebens@gmail.com>	5 years ago
garanews	c38816828f	fix few typo (#8023) ... Signed-off-by: garanews <puntogtg@tiscali.it>	5 years ago
Brian Brazil	073e93c768	Gracefully handle unknown WAL record types. (#8004) ... As we're looking to expand what's in the WAL, having old Prometheus servers ignore the new record types rather than treating them as corruption allows for better upgrade/downgrade paths. Adjust some tests accordingly, so they're still testing what they're meant to test. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Ganesh Vernekar	50ae4e298b	Fix magic number in docs (#7998) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	f0d87b5d86	Encode and Decode method for tombstones (#7967) ... * Encode and Decode method for tombstones Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Max Neverov	7e1c27b853	Add tsdb startup duration metric (#7737) ... * Add tsdb wal replay duration metric Signed-off-by: Max Neverov <neverov.max@gmail.com>	5 years ago
Jorge Vallecillo	7aa5fb01bf	tsdb/chunks/head_chunks_test.go: Fix typo (#7953) ... tsdb/db_test.go: Fix typo Signed-off-by: Jorge Vallecillo <jorgevallecilloc@gmail.com>	5 years ago
Tobias Klauser	b5b3bc34da	tsdb/fileutil: Use FcntlFstore from golang.org/x/sys/unix (#7949) ... Direct syscalls using syscall.Syscall(SYS_*, ...) should no longer be used on darwin, see [1]. Instead, use the FcntlFstore libSystem wrapper provided by the golang.org/x/sys/unix package to implement preallocFixed. [1] https://golang.org/doc/go1.12#darwin Signed-off-by: Tobias Klauser <tklauser@distanz.ch>	5 years ago
Xiaochao Dong	a282d25099	tsdb: remove duplicate values set to reduce memory usage(map overhead) (#7915) ... Signed-off-by: Xiaochao Dong (@damnever) <dxc.wolf@gmail.com>	5 years ago
zhulongcheng	5ea6bdca1f	tsdb: sync temporary checkpoint directory before rename (#7004) ... Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	5 years ago
johncming	75ae384192	tsdb: remove redundant fields. (#7869) ... Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Ganesh Vernekar	2255b6f62f	Refactor WAL.Segments method to be part of the wal package (#6477) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
johncming	252265fa81	tsdb/chunks: remove un-used code. (#6944) ... Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Ganesh Vernekar	c806262206	Fix 'chunks.HeadReadWriter: maxt of the files are not set' error (#7856) ... * Fix chunks.HeadReadWriter: maxt of the files are not set Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Yukun Sun	cfd4e05c9e	fix: return a corruption error when iterator function find a chunk that is out of sequence (#7855) ... Signed-off-by: sunyukun <sunyukun@didiglobal.com> Co-authored-by: sunyukun <sunyukun@didiglobal.com>	5 years ago
Max Neverov	bb5c6b38e2	Fix Possible Race Condition in TSDB (#7815) ... * Replace tsdb chunk mapper size with atomic; protect mmappedChunkFiles with read path mutex on DeleteCorrupted Signed-off-by: Max Neverov <neverov.max@gmail.com> * PR fixes Signed-off-by: Max Neverov <neverov.max@gmail.com>	5 years ago
johncming	3e7b463908	add testcase for rangeForTimestamp. (#6454) ... Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Guangwen Feng	9439b2150d	Fix golint warning caused by misspell (#7842) ... Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>	5 years ago
johncming	b521612042	tsdb: simplify code. (#7792) ... Signed-off-by: johncming <johncming@yahoo.com>	6 years ago
gastonqiu	5a7d398d19	Chore: Log segment number when segment read failed (#7687) ... * Chore: Log segment number when segment read failed To manually fix the WAL files, it is good to know where the corrupt happened so we should log the segment number when the read failed. Related Issue #7506 Signed-off-by: gaston.qiu <gaston.qiu@umbocv.com>	6 years ago
johncming	d19fc71903	tsdb: use NewRangeHead instead. (#7793) ... Signed-off-by: johncming <johncming@yahoo.com>	6 years ago
Harold Dost	21a753c4e2	Make file permissions set to allow for wider umask options. (#7782) ... 0644 -> 0666 on all non vendored code. Fixes #7717 Signed-off-by: Harold Dost <harolddost@gmail.com>	6 years ago
Bartlomiej Plotka	f16cbc20d6	tsdb: Bug fix for further continued deletions after crash deletions; added more tests. (#7777) ... * tsdb: Bug fix for further continued after crash deletions; added more tests. Additionally: Added log line for block removal. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comment. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	6 years ago