prometheus

Commit Graph

Author	SHA1	Message	Date
Ganesh Vernekar	3e4a094dbb	Add stale_series_compaction_threshold config file option Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	3 months ago
Ganesh Vernekar	4f3de8da29	tsdb: Add unit tests for stale series compaction Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	3 months ago
Ganesh Vernekar	43dc23afe7	tsdb: Clear stale series from the Head during WAL replay Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	3 months ago
Ganesh Vernekar	43e69388df	tsdb: Add stale series compaction support in the DB Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	3 months ago
Ganesh Vernekar	9b444b57af	tsdb: Add StaleHead and GC for stale series in the Head block Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	3 months ago
Bartlomiej Plotka	bec70227f1	feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager constructor (#17872 ) * feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager optionally to V1 Signed-off-by: bwplotka <bwplotka@gmail.com> * Update scrape/manager.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * fixes after rebase Signed-off-by: bwplotka <bwplotka@gmail.com> * Apply suggestions from code review Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
yy	0e77dcfdb9	fix: typo in error message (#17878 ) Signed-off-by: Yohei Yamamoto <yhymmt123@gmail.com>	3 months ago
Arve Knudsen	572f247b4d	tsdb: add auto-cleanup to newTestHead and remove redundant cleanup calls (#17890 ) Add automatic cleanup to newTestHeadWithOptions so that heads created with newTestHead are automatically closed when the test ends. This simplifies test code by removing the need for manual cleanup in most cases. Changes: - Add t.Cleanup in newTestHeadWithOptions immediately after creating the head, using _ = h.Close() to handle double-close gracefully - Remove redundant t.Cleanup, defer, and explicit Close calls from tests that use newTestHead - Add cleanup for heads created with NewHead directly in restart patterns (e.g., restartHeadAndVerifySeriesCounts, startHead) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Arve Knudsen	dd85d7ca97	tsdb: fix memory leaks in buffer pools (#17879 ) * tsdb: fix memory leaks in buffer pools Clear reference fields when returning buffers to pools to avoid retaining data after the buffer is reused. Affected pools: - refSeriesPool: clear Labels - histogramsPool: clear H pointer - floatHistogramsPool: clear FH pointer - metadataPool: clear Unit and Help strings Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Clear slices Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Clear metadata buffer Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Remove tests Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Julius Hinze	e49d11c88d	tsdb: fix rare case of OOO exemplar insertion pointing to the exemplar being deleted (#17865 ) Signed-off-by: Julius Hinze <julius.hinze@grafana.com>	3 months ago
Julius Hinze	ccb7468b09	tsdb: fix grow/shrink nextIndex calculation (#17863 ) Signed-off-by: Julius Hinze <julius.hinze@grafana.com>	3 months ago
George Krajcsovits	06a59346fe	Update tsdb/chunkenc/chunk.go Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	3 months ago
bwplotka	49c3aea56d	feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage impl Signed-off-by: bwplotka <bwplotka@gmail.com>	3 months ago
György Krajcsovits	adf734db7a	update remaining tests Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
György Krajcsovits	a5ac0bff1d	update ooo_head.go but only with TODOs Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
György Krajcsovits	28dca34f4f	auto update head sample use in tests find . -name "*.go" -type f -exec sed -E -i \ 's/([^[:alpha:]]sample\{)([^,{:]+,[^,]+,[^,]+,[^,]+\})/\10, \2/g' {} + I've omitted tsdb/ooo_head.go from the commit because I'm also adding todo there. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
György Krajcsovits	f616689f09	tsdb/head.go: add start timestamp to sample type used in tests Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
György Krajcsovits	70bc06718d	feat(tsdb): new AppenderV2 and AtST interface for chunks No implementation yet. Just to test the shape of the interface. AtST is implemented for trivial cases, anything else is hard coded to return 0. Ref: https://github.com/prometheus/prometheus/issues/17791 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
Ben Kochie	72a23934ad	Refactor various tsdb sub-packages (#17847 ) Migrate various tsdb related packages from `tsdb/errors` to the standard library `errors` package. Signed-off-by: SuperQ <superq@gmail.com>	3 months ago
Bryan Boreham	f1719fa1d4	[BUGFIX] Agent: fix crash from invalid type in pool (#17802 ) We have separate pools for Appender and AppenderV2 objects, and must not put another kind of object into them. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 months ago
Julius Hinze	22463b1e9f	tsdb: add support for OOO exemplars in CircularExemplarStorage (#17469 ) * tsdb: add support for OOO exemplars in CircularExemplarStorage Doubly linked exemplar storage resize. Split exemplar buffer resize into shrink and grow functions. Skip duplicate OOO exemplars, re-initialize emptied index after deleting its last exemplar. Signed-off-by: Julius Hinze <julius.hinze@grafana.com>	4 months ago
Patryk Prus	dcda4840a7	tsdb/index: export sentinel errors for size limit failures (#17773 ) * tsdb/index: export sentinel errors for size limit failures --------- Signed-off-by: Patryk Prus <p@trykpr.us> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	4 months ago
Ben Kochie	e14795bbf4	Remove copyright date from headers (#17785 ) Remove copyright dates from various files as part of [PROM-50]. [PROM-50]: https://github.com/prometheus/proposals/blob/main/proposals/0050-remove-copyright-dates.md Signed-off-by: SuperQ <superq@gmail.com>	4 months ago
Jarkko Pöyry	e4b6d443fc	tsdb: fix handle leak on mmap failure on MS Windows (#17310 ) Signed-off-by: Jarkko Pöyry <jarkko.poyry@metaplay.io>	4 months ago
bwplotka	96ff5b8f9c	addressed comment Signed-off-by: bwplotka <bwplotka@gmail.com>	4 months ago
bwplotka	bab7614d1b	fix: ensure no race on lset Signed-off-by: bwplotka <bwplotka@gmail.com>	4 months ago
bwplotka	fc019d6628	refactor(scrape): DRY-ed getOrCreate flow Signed-off-by: bwplotka <bwplotka@gmail.com>	4 months ago
Charles Korn	69aa3ac67b	tsdb: correctly log error in `headAppenderV2.appendExemplars` Signed-off-by: Charles Korn <charles.korn@grafana.com>	4 months ago
NamanParlecha	c94101d023	TSDB: Option to configure TSDB Block Reload Interval (#16728 ) Add --storage.tsdb.block-reload-interval flag to configure TSDB block reload interval. --------- Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> Signed-off-by: NamanParlecha <namanparlecha@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	4 months ago
bwplotka	ad367b504b	refactor(tsdb/agent)[PART3]: add AppenderV2 support to agent Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	763b935b45	refactor(tsdb/agent): 1:1 copy of db.go and db_test.go for starting point Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
Bryan Boreham	41665a4a55	[CHORE] TSDB: Remove unused LabelValueFor function The last use of this was removed 4 years ago. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	5 months ago
Bryan Boreham	583bc01cc9	Refactor: TSDB: small improvement to Postings (#17661 ) * [TESTS] TSDB: Check that ListPostings are sorted `newListPostings` is a convenient place to do this; move it into the test file because it's not needed for anything else. Also simplify the existing `SliceIsSorted` check. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * [COMMENTS] TSDB: Document that Postings are ordered. The description of `Seek()` implies they are, but it's better to be explicit. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * [REFACTOR] Unexport ListPostings type It's only used within the one package, and it would be surprising if some downstream code did rely on this type, given all of its members are unexported. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> --------- Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	5 months ago
Julien Pivotto	a5671a002f	API: Add a /api/v1/features endpoint Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	5 months ago
bwplotka	e7e45090e4	refactor(appenderV2): port TSDB non-head tests Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	0b70a07572	refactor(appenderV2): add TSDB AppenderV2 implementation Signed-off-by: bwplotka <bwplotka@gmail.com> tmp Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	a41e1144dd	refactor(appenderV2): 1:1 copy of db_test.go to db_append_v2_test.go (starting point) Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	efdfb8fed6	refactor(appenderV2): 1:1 copy of head append test files for v2 (starting point) Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	129650df9d	refactor(appenderV2): 1:1 copy of head_append.go -> head_append_v2.go (starting point) Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
dongjiang	3239723098	Update golangci-lint and add modernize check (#17640 ) * add modernize check Signed-off-by: dongjiang1989 <dongjiang1989@126.com> * fix golangci lint Signed-off-by: dongjiang1989 <dongjiang1989@126.com> --------- Signed-off-by: dongjiang1989 <dongjiang1989@126.com>	5 months ago
Bartlomiej Plotka	f6ca7145ca	refactor(tsdb): use one test newTestDB constructor (#17638 ) For tests only, we had various ways of opening DB. Reduced to one instead of: * Open * newTestDB * newTestDBOpts * openTestDB This so https://github.com/prometheus/prometheus/pull/17629 is smaller and bit easier. Also for test maintainability and consistency. Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
Łukasz Mierzwa	8a1086a128	feat: Add flag that blocks lvl 1 compactions until upload is confirmed in an external JSON file (#17435 ) * Delay compactions until Thanos uploads all blocks Using Thanos sidecar with Prometheus requires us to disable TSDB compactions on Prometheus side by setting --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration to the same value. See https://thanos.io/tip/components/sidecar.md. The main problem this avoids is that Prometheus might compact given block before Thanos uploads it, creating a gap in Thanos metrics. Thanos does not upload compacted blocks because that would upload the same sample multiple times. You can tell Thanos to upload compacted blocks but that is aimed at one time migrations. This patch creates a bridge between Thanos and Prometheus by allowing Prometheus to read the shipper file Thanos creates, where it tracks which blocks were already uploaded, and using that data delays compaction of blocks until they are marked as uploaded by Thanos. Thanks to this both services can coordinate with each other (in a way) and we can stop disabling compaction on Prometheus side when Thanos uploads are enabled. The reason to have this is that disabling compactions have very dramatic performance cost. Since most time series exist for longer than a single block duration (2h by default) large chunks of block index will reference the same series, so 10 * 2h blocks will each have an index that is usually fairly big and is almost the same for all 10 blocks. Compaction de-duplicates the index so merging 10 blocks together would leave us with a single index that is around the same size as each of these 10 2h blocks would have (plus some extra for series that only exists in some blocks, but not all). Every range query that iterates over all 10 blocks would then have to read each index and so we're doing 10x more work then if we had a single compacted block. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Rename structs and functions to make this more generic Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Address review comments Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Cache UploadMeta for 1 minute Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> --------- Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	5 months ago
Björn Rabenstein	b8d19543b8	Add histogram validation in remote-read and during reducing resolution (#17561 ) ReduceResolution is currently called before validation during ingestion. This will cause a panic if there are not enough buckets in the histogram. If there are too many buckets, the spurious buckets are ignored, and therefore the error in the input histogram is masked. Furthermore, invalid negative offsets might cause problems, too. Therefore, we need to do some minimal validation in reduceResolution. Fortunately, it is easy and shouldn't slow things down. Sadly, it requires to return errors, which triggers a bunch of code changes. Even here is a bright side, we can get rud of a few panics. (Remember: Don't panic!) In different news, we haven't done a full validation of histograms read via remote-read. This is not so much a security concern (as you can throw off Prometheus easily by feeding it bogus data via remote-read) but more that remote-read sources might be makeshift and could accidentally create invalid histograms. We really don't want to panic in that case. So this commit does not only add a check of the spans and buckets as needed for resolution reduction but also a full validation during remote-read. Signed-off-by: beorn7 <beorn@grafana.com>	5 months ago
0xkato	ae00fd45ab	tsdb: guard chunk length overflow in head chunk reader (#17533 ) Signed-off-by: 0xkato <0xkkato@gmail.com>	5 months ago
Bartlomiej Plotka	f50ff0a40a	feat: rename CreatedTimestamp to StartTimestamp (#17523 ) Partially fixes https://github.com/prometheus/prometheus/issues/17416 by renaming all CT* names to ST* in the whole codebase except RW2 (this is done in separate [PR](https://github.com/prometheus/prometheus/pull/17411)) and PrometheusProto exposition proto. ``` CreatedTimestamp -> StartTimestamp CreatedTimeStamp -> StartTimestamp created_timestamp -> start_timestamp CT -> ST ct -> st ``` Signed-off-by: bwplotka <bwplotka@gmail.com>	6 months ago
Ben Ye	2e609511bb	Register missing metric prometheus_tsdb_sample_ooo_delta (#17477 ) * register missing metric prometheus_tsdb_sample_ooo_delta Signed-off-by: yeya24 <benye@amazon.com> * changelog Signed-off-by: yeya24 <benye@amazon.com> --------- Signed-off-by: yeya24 <benye@amazon.com>	6 months ago
Bryan Boreham	c1e0ab11c6	[PERF] TSDB: Speed up intersectPostings.Next Check if the next position is already a match, in which case we don't have to call `Seek`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	0e1e7441e4	[PERF] TSDB: ListPostings: check next item before binary search It is fairly common that the next item is the one we want, and cheap to check. We could also start the binary search one position on, but strangely that slows it down. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	be8307db58	[TEST] Refactor BenchmarkIntersect to reduce memory allocations Extract functions which pre-create all the memory for the benchmark itself. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	393ab9e12e	[TEST] TSDB: More realistic BenchmarkIntersect 100,000 matchers is not something that could happen while using Prometheus. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago

1 2 3 4 5 ...

1497 Commits (3e4a094dbb3e2b60cd5a4cab2c83a56213c321ce)