prometheus

Commit Graph

Author	SHA1	Message	Date
Callum Styan	ed0c3b25c7	refactor new version flag to make it easier to pick a specific format instead of having multiple flags, plus add new formats for testing Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Callum Styan	f1992591b2	Implement code paths for new proto format Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Márcio Carôso	dff1c395f6	Expose --storage.tsdb.retention.time in metric prometheus_tsdb_retention_limit_seconds (#12986 ) * Expose --storage.tsdb.retention.time in a metric Signed-off-by: Marcio Caroso <msscaroso@gmail.com> --------- Signed-off-by: Marcio Caroso <msscaroso@gmail.com>	2 years ago
Jeanette Tan	71a36d2396	Very minor refactor of the integer overflow fix Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Bryan Boreham	26fa2e8356	TSDB: Pre-size buffer to read samples from WAL When reading the WAL this method is called with buffers from a pool, on multiple goroutines. Pre-allocating sufficient size avoids slow growth and many reallocations in `append`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
George Krajcsovits	7d7b9eacff	Fix int32 overflow issues (#12978 ) On a 32 bit architecture the size of int is 32 bits. Thus converting from int64, uint64 can overflow it and flip the sign. Try for yourself in playground: package main import "fmt" func main() { x := int64(0x1F0000001) y := int64(1) z := int32(x - y) // numerically this is 0x1F0000000 fmt.Printf("%v\n", z) } Prints -268435456 as if x was smaller. Followup to #12650 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Paschalis Tsilias	42b8f2f5fc	tsdb/agent: allow ingestion of OOO samples (#12897 ) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com> Signed-off-by: Levi Harrison <git@leviharrison.dev>	2 years ago
Ganesh Vernekar	4df2f2432b	Additionally wrap WBL replay error (#12406 ) * Additionally wrap WBL replay error Although WBL replay is already wrapped with errLoadWbl, there are other errors that can happen during a WBL replay. We should not try to repair WAL in those cases. This commit additionally wraps the final error in Head.Init again with errLoadWbl so that WBL replay errors can be identified properly. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com> Signed-off-by: Levi Harrison <git@leviharrison.dev>	2 years ago
Paschalis Tsilias	afab845e65	tsdb/agent: allow ingestion of OOO samples (#12897 ) Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>	2 years ago
Ganesh Vernekar	f5913266a1	Additionally wrap WBL replay error (#12406 ) * Additionally wrap WBL replay error Although WBL replay is already wrapped with errLoadWbl, there are other errors that can happen during a WBL replay. We should not try to repair WAL in those cases. This commit additionally wraps the final error in Head.Init again with errLoadWbl so that WBL replay errors can be identified properly. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>	2 years ago
Oleg Zaytsev	fe90dcccff	Revert ListPostings change (#12955 ) Reverts change from https://github.com/prometheus/prometheus/pull/12906 The benchmarks show that it's slower when intersecting, which is a common usage for ListPostings (when intersecting matchers from Head) (old is before #12906, new is #12906): │ old │ new │ │ sec/op │ sec/op vs base │ Intersect/LongPostings1-16 20.54µ ± 1% 21.11µ ± 1% +2.76% (p=0.000 n=20) Intersect/LongPostings2-16 51.03m ± 1% 52.40m ± 2% +2.69% (p=0.000 n=20) Intersect/ManyPostings-16 194.2m ± 3% 332.1m ± 1% +71.00% (p=0.000 n=20) geomean 5.882m 7.161m +21.74% Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Oleg Zaytsev	5bd8c8c561	Clarify Postings.At() contract (#12921 ) It's implicit, but should be explicit. It is invalid to call At() after a failed call to Next() or Seek(). Following up on https://github.com/prometheus/prometheus/pull/12906 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Björn Rabenstein	ec9170b8bf	Merge pull request #12874 from krajorama/outof-order-chunks Fix duplicate sample detection at chunk size limit Signed-off-by: SuperQ <superq@gmail.com>	2 years ago
Oleg Zaytsev	1492031ef2	Optimize ListPostings Next() (#12906 ) The Next() call of ListPostings() was updating two values, while we can just update the position. This is up to 30% faster for high number of Postings. goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/tsdb/index cpu: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz │ old │ new │ │ sec/op │ sec/op vs base │ ListPostings/count=100-16 819.2n ± 0% 732.6n ± 0% -10.58% (p=0.000 n=20) ListPostings/count=1000-16 2.685µ ± 1% 2.017µ ± 0% -24.88% (p=0.000 n=20) ListPostings/count=10000-16 21.43µ ± 1% 14.81µ ± 0% -30.91% (p=0.000 n=20) ListPostings/count=100000-16 209.4µ ± 1% 143.3µ ± 0% -31.55% (p=0.000 n=20) ListPostings/count=1000000-16 2.086m ± 1% 1.436m ± 1% -31.18% (p=0.000 n=20) geomean 29.02µ 21.41µ -26.22% We're talking about microseconds here, but they just keep adding. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Arve Knudsen	de7e057d3c	tsdb: Tighten up sub-benchmark scope in BenchmarkQuerier (#12718 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Linas Medziunas	1aad4004c3	Additional test case for ValidateHistogram Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Bryan Boreham	6dcbd653e9	tsdb: register metrics after Head is initialized (#12876 ) This avoids situations where metrics are scraped before the data they are trying to look at is initialized. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Goutham Veeramachaneni	86729d4d7b	Update exp package (#12650 )	2 years ago
Dimitar Dimitrov	6f1284ac93	Fix exit condition of TestQuerierIndexQueriesRace The test was introduced in # but was changed during the code review and not reran with the faulty code since then. Closes # Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>	2 years ago
György Krajcsovits	9dbd100a5e	Refactor solution to not repeat code Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
György Krajcsovits	96d03b6f46	Fix duplicate sample detection at chunks size limit Before cutting a new XOR chunk in case the chunk goes over the size limit, check that the timestamp is in order and not equal or older than the latest sample in the old chunk. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
György Krajcsovits	56b3a015b6	Add regression test for duplicate detection at chunk size limit TestHeadDetectsDuplcateSampleAtSizeLimit tests a regression where a duplicate sample,is appended to the head, right when the head chunk is at the size limit. The test adds all samples as duplicate, thus expecting that the result has exactly half of the samples. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
George Krajcsovits	3512b2d678	storage: make histogram reset handling consistent in chainSampleIterator (#12779 ) storage: make histogram reset handling consistent in chainSampleIterator --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Alan Protasio	959c98441b	Add context argument to tsdb.PostingsForMatchers Signed-off-by: Alan Protasio <alanprot@gmail.com>	2 years ago
zenador	69edd8709b	Add warnings (and annotations) to PromQL query results (#12152 ) Return annotations (warnings and infos) from PromQL queries This generalizes the warnings we have already used before (but only for problems with remote read) as "annotations". Annotations can be warnings or infos (the latter could be false positives). We do not treat them different in the API for now and return them all as "warnings". It would be easy to distinguish them and return infos separately, should that appear useful in the future. The new annotations are then used to create a lot of warnings or infos during PromQL evaluations. Partially these are things we have wanted for a long time (e.g. inform the user that they have applied `rate` to a metric that doesn't look like a counter), but the new native histograms have created even more needs for those annotations (e.g. if a query tries to aggregate float numbers with histograms). The annotations added here are not yet complete. A prominent example would be a warning about a range too short for a rate calculation. But such a warnings is more tricky to create with good fidelity and we will tackle it later. Another TODO is to take annotations into account when evaluating recording rules. --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Arve Knudsen	156222cc50	Add context argument to LabelQuerier.LabelValues (#12665 ) Add context argument to LabelQuerier.LabelValues and LabelQuerier.SortedLabelValues. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Arve Knudsen	a964349e97	Add context argument to LabelQuerier.LabelNames (#12666 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Arve Knudsen	4451ba10b4	Add context argument to IndexReader.Postings (#12667 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Arve Knudsen	6ef9ed0bc3	Add context argument to DB.Delete (#12834 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
György Krajcsovits	b2fa4d910a	Fix more counterResetInAnyBucket edgecases Case a) empty span is at the beginning of the spans. Case b) two consequtive empty spans with positive offsets. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Fiona Liao	4419399e4e	Do WBL mmap marker replay concurrently (#12801 ) * Benchmark WBL Extended WAL benchmark test with WBL parts too - added basic cases for OOO handling - a percentage of series have a percentage of samples set as OOO ones. Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>	2 years ago
Shirley	d3a1044354	WBL loading: don't send empty buffers over chan (#12808 ) Signed-off-by: Shirley Leu <4163034+fridgepoet@users.noreply.github.com> Co-authored-by: Fiona Liao <fiona.y.liao@gmail.com>	2 years ago
Arve Knudsen	6daee89e5f	Add context argument to Querier.Select (#12660 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Fiona Liao	f211fcd92d	Remove duplicated ms.mmMaxTime check in WAL Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>	2 years ago
George Krajcsovits	b6f903b5f9	Fix handling of explicit counter reset header in histograms. (#12772 ) * Fix handling of explicit counter reset header in histograms. Explicit counter reset were being ignored. Also there was no unit test coverage. Add test case for the first sample in a chunk. Add test case for non first sample in chunk. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Dimitar Dimitrov	b40865833d	PostingsForMatchers race with creating new series (#12558 ) Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>	2 years ago
Bryan Boreham	bdc7983956	TSDB: re-use iterator when moving between series Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	0d283effa8	promql: force mmap of head chunks in BenchmarkRangeQuery Otherwise we have a highly unusual situation of over 100 chunks in the headChunks list of each series, which heavily skews performance. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Gregor Zeitlinger	f01718262a	Unit tests for native histograms (#12668 ) promql: Extend testing framework to support native histograms This includes both the internal testing framework as well as the rules unit test feature of promtool. This also adds a bunch of basic tests. Many of the code level tests can now be converted to tests within the framework, and more tests can be added easily. --------- Signed-off-by: Harold Dost <h.dost@criteo.com> Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com> Signed-off-by: Stephen Lang <stephen.lang@grafana.com> Co-authored-by: Harold Dost <h.dost@criteo.com> Co-authored-by: Stephen Lang <stephen.lang@grafana.com> Co-authored-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>	2 years ago
Michal Biesek	04d7b4dbee	lint: Fix `SA1019` Using a deprecated function `rand.Read` has been deprecated since Go 1.20 `crypto/rand.Read` is more appropriate Ref: https://tip.golang.org/doc/go1.20 Signed-off-by: Michal Biesek <michalbiesek@gmail.com>	2 years ago
Justin Lei	8ef7dfdeeb	Add a chunk size limit in bytes (#12054 ) Add a chunk size limit in bytes This creates a hard cap for XOR chunks of 1024 bytes. The limit for histogram chunk is also 1024 bytes, but it is a soft limit as a histogram has a dynamic size, and even a single one could be larger than 1024 bytes. This also avoids cutting new histogram chunks if the existing chunk has fewer than 10 histograms yet. In that way, we are accepting "jumbo chunks" in order to have at least 10 histograms in a chunk, allowing compression to kick in. Signed-off-by: Justin Lei <justin.lei@grafana.com>	2 years ago
beorn7	aa82fe198f	tsdb: Fix histogram validation So far, `ValidateHistogram` would not detect if the count did not include the count in the zero bucket. This commit fixes the problem and updates all the tests that have been undetected offenders so far. Note that this problem would only ever create false negatives, so we never falsely rejected to store a histogram because of it. On the other hand, `ValidateFloatHistogram` has been to strict with the count being at least as large as the sum of the counts in all the buckets. Float precision issues could create false positives here, see products of PromQL evaluations, it's actually quite hard to put an upper limit no the floating point imprecision. Users could produce the weirdest expressions, maxing out float precision problems. Therefore, this commit simply removes that particular check from `ValidateFloatHistogram`. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Mustafa Ateş Uzun	e5e51bebef	fix: error message typo Signed-off-by: Mustafa Ateş Uzun <mustafauzun0@gmail.com>	2 years ago
SuperQ	8d38d59fc5	Cleanup temporary chunk snapshot dirs Simlar to cleanup of WAL files on startup, cleanup temporary chunk_snapshot dirs. This prevents storage space leaks due to terminated snapshots on shutdown. Signed-off-by: SuperQ <superq@gmail.com>	2 years ago
Oleg Zaytsev	6ea6def0d3	Use zeropool when replaying agent's DB WAL (#12651 ) Same as https://github.com/prometheus/prometheus/pull/12189 but for tsdb/agent/db.go Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Oleg Zaytsev	c810e7cae3	Fix typo in Appender.AppendHistogram() arg name Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Oleg Zaytsev	61daa30bb1	Pass ref to SeriesLifecycleCallback.PostDeletion (#12626 ) When a particular SeriesLifecycleCallback tries to optimize and run closer to the Head, keeping track of the HeadSeriesRef instead of the labelsets, it's impossible to handle the PostDeletion callback properly as there's no way to know which series refs were deleted from the head. This changes the callback to provide the series refs alongside the labelsets, so the implementation can choose what to do. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Oleg Zaytsev	cd7d0b69a2	Check nil err first when committing (#12625 ) The most common case is to have a nil error when appending series, so let's check that first instead of checking the 3 error types first. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
cui fliter	f26dfc95e6	fix struct name in comment (#12624 ) Signed-off-by: cui fliter <imcusg@gmail.com>	2 years ago
Łukasz Mierzwa	3c80963e81	Use a linked list for memSeries.headChunk (#11818 ) Currently memSeries holds a single head chunk in-memory and a slice of mmapped chunks. When append() is called on memSeries it might decide that a new headChunk is needed to use for given append() call. If that happens it will first mmap existing head chunk and only after that happens it will create a new empty headChunk and continue appending our sample to it. Since appending samples uses write lock on memSeries no other read or write can happen until any append is completed. When we have an append() that must create a new head chunk the whole memSeries is blocked until mmapping of existing head chunk finishes. Mmapping itself uses a lock as it needs to be serialised, which means that the more chunks to mmap we have the longer each chunk might wait for it to be mmapped. If there's enough chunks that require mmapping some memSeries will be locked for long enough that it will start affecting queries and scrapes. Queries might timeout, since by default they have a 2 minute timeout set. Scrapes will be blocked inside append() call, which means there will be a gap between samples. This will first affect range queries or calls using rate() and such, since the time range requested in the query might have too few samples to calculate anything. To avoid this we need to remove mmapping from append path, since mmapping is blocking. But this means that when we cut a new head chunk we need to keep the old one around, so we can mmap it later. This change makes memSeries.headChunk a linked list, memSeries.headChunk still points to the 'open' head chunk that receives new samples, while older, yet to be mmapped, chunks are linked to it. Mmapping is done on a schedule by iterating all memSeries one by one. Thanks to this we control when mmapping is done, since we trigger it manually, which reduces the risk that it will have to compete for mmap locks with other chunks. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2 years ago

1 2 3 4 5 ...

838 Commits (bwplotka/prw-initial)