prometheus

Commit Graph

Author	SHA1	Message	Date
Matthieu MOREL	9c4782f1cc	golangci-lint: enable testifylint linter (#13254 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
Xiaochao Dong	28d8f1650c	tsdb: Make sure the cache for postings cardinality properly honors the label name (#12653 ) Add a string remembering which label and limit the cache corresponds to. Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>	2 years ago
Arve Knudsen	1200c89d0c	Fix tsdb.stripeSeries.gc so it handles conflicts properly (#13195 ) * Fix tsdb.stripeSeries.gc so it handles conflicts properly tsdb.stripeSeries.gc needs to prune seriesHashmap.conflicts first, otherwise seriesHashmap replaces the unique field with the first among the conflicts. Also add regression test. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * TestStripeSeries_gc: Support stringlabels, don't use internals Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Fiona Liao	5bee0cfce2	Change `ChunkReader.Chunk()` to `ChunkOrIterable()` The ChunkReader interface's Chunk() has been changed to ChunkOrIterable(). This is a precursor to OOO native histogram support - with OOO native histograms, the chunks.Meta passed to Chunk() can result in multiple chunks being returned rather than just a single chunk (e.g. if oooMergedChunk has a counter reset in the middle). To support this, ChunkOrIterable() requires either a single chunk or an iterable to be returned. If an iterable is returned, the caller has the responsibility of converting the samples from the iterable into possibly multiple chunks. The OOOHeadChunkReader now returns an iterable rather than a chunk to prepare for the native histograms case. Also as a beneficial side effect, oooMergedChunk and boundedChunk has been simplified as they only need to implement the Iterable interface now, not the full Chunk interface. --------- Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	2 years ago
Arve Knudsen	ecc37588b0	tsdb: seriesHashmap.set by making receiver a pointer (#13193 ) * Fix tsdb.seriesHashmap.set by making receiver a pointer The method tsdb.seriesHashmap.set currently doesn't set the conflicts field properly, due to the receiver being a non-pointer. Fix by turning the receiver into a pointer, and add a corresponding regression test. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
George Krajcsovits	acc114fe55	Fix panic during tsdb Commit (#13092 ) * Fix panic during tsdb Commit Fixes the following panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x19deb45] goroutine 651118930 [running]: github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0xc19100f7c0) /drone/src/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:855 +0x245 github.com/prometheus/prometheus/tsdb.dbAppender.Commit({{0x35bd6f0?, 0xc19100f7c0?}, 0xc000fa4c00?}) /drone/src/vendor/github.com/prometheus/prometheus/tsdb/db.go:1159 +0x2f We theorize that the panic happened due the the series referenced by the exemplar being removed between AppendExemplar and Commit due to being idle. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
George Krajcsovits	39a35d92bc	tsdb/head: wlog exemplars after samples (#13113 ) When samples are committed in the head, they are also written to the WAL. The order of WAL records should be sample then exemplar, but this was not the case for native histogram samples. This PR fixes that. The problem with the wrong order is that remote write reads the WAL and sends the recorded timeseries in the WAL order, which means exemplars arrived before histogram samples. If the receiving side is Prometheus TSDB and the series has not existed before then the exemplar does not currently create the series. Which means the exemplar is rejected and lost. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Linas Medziunas	1f8aea11d6	Move histogram validation code to model/histogram Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Linas Medziunas	1cd6c1cde5	ValidateHistogram: strict Count check in absence of NaNs Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Oleksandr Redko	8e5f0387a2	ci(lint): enable nolintlint and remove redundant comments (#12926 ) Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>	2 years ago
Ganesh Vernekar	4df2f2432b	Additionally wrap WBL replay error (#12406 ) * Additionally wrap WBL replay error Although WBL replay is already wrapped with errLoadWbl, there are other errors that can happen during a WBL replay. We should not try to repair WAL in those cases. This commit additionally wraps the final error in Head.Init again with errLoadWbl so that WBL replay errors can be identified properly. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com> Signed-off-by: Levi Harrison <git@leviharrison.dev>	2 years ago
Ganesh Vernekar	f5913266a1	Additionally wrap WBL replay error (#12406 ) * Additionally wrap WBL replay error Although WBL replay is already wrapped with errLoadWbl, there are other errors that can happen during a WBL replay. We should not try to repair WAL in those cases. This commit additionally wraps the final error in Head.Init again with errLoadWbl so that WBL replay errors can be identified properly. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>	2 years ago
Björn Rabenstein	ec9170b8bf	Merge pull request #12874 from krajorama/outof-order-chunks Fix duplicate sample detection at chunk size limit Signed-off-by: SuperQ <superq@gmail.com>	2 years ago
Linas Medziunas	1aad4004c3	Additional test case for ValidateHistogram Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
György Krajcsovits	56b3a015b6	Add regression test for duplicate detection at chunk size limit TestHeadDetectsDuplcateSampleAtSizeLimit tests a regression where a duplicate sample,is appended to the head, right when the head chunk is at the size limit. The test adds all samples as duplicate, thus expecting that the result has exactly half of the samples. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Arve Knudsen	156222cc50	Add context argument to LabelQuerier.LabelValues (#12665 ) Add context argument to LabelQuerier.LabelValues and LabelQuerier.SortedLabelValues. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Arve Knudsen	a964349e97	Add context argument to LabelQuerier.LabelNames (#12666 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Arve Knudsen	4451ba10b4	Add context argument to IndexReader.Postings (#12667 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Arve Knudsen	6ef9ed0bc3	Add context argument to DB.Delete (#12834 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Fiona Liao	4419399e4e	Do WBL mmap marker replay concurrently (#12801 ) * Benchmark WBL Extended WAL benchmark test with WBL parts too - added basic cases for OOO handling - a percentage of series have a percentage of samples set as OOO ones. Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>	2 years ago
Arve Knudsen	6daee89e5f	Add context argument to Querier.Select (#12660 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Gregor Zeitlinger	f01718262a	Unit tests for native histograms (#12668 ) promql: Extend testing framework to support native histograms This includes both the internal testing framework as well as the rules unit test feature of promtool. This also adds a bunch of basic tests. Many of the code level tests can now be converted to tests within the framework, and more tests can be added easily. --------- Signed-off-by: Harold Dost <h.dost@criteo.com> Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com> Signed-off-by: Stephen Lang <stephen.lang@grafana.com> Co-authored-by: Harold Dost <h.dost@criteo.com> Co-authored-by: Stephen Lang <stephen.lang@grafana.com> Co-authored-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>	2 years ago
Justin Lei	8ef7dfdeeb	Add a chunk size limit in bytes (#12054 ) Add a chunk size limit in bytes This creates a hard cap for XOR chunks of 1024 bytes. The limit for histogram chunk is also 1024 bytes, but it is a soft limit as a histogram has a dynamic size, and even a single one could be larger than 1024 bytes. This also avoids cutting new histogram chunks if the existing chunk has fewer than 10 histograms yet. In that way, we are accepting "jumbo chunks" in order to have at least 10 histograms in a chunk, allowing compression to kick in. Signed-off-by: Justin Lei <justin.lei@grafana.com>	2 years ago
beorn7	aa82fe198f	tsdb: Fix histogram validation So far, `ValidateHistogram` would not detect if the count did not include the count in the zero bucket. This commit fixes the problem and updates all the tests that have been undetected offenders so far. Note that this problem would only ever create false negatives, so we never falsely rejected to store a histogram because of it. On the other hand, `ValidateFloatHistogram` has been to strict with the count being at least as large as the sum of the counts in all the buckets. Float precision issues could create false positives here, see products of PromQL evaluations, it's actually quite hard to put an upper limit no the floating point imprecision. Users could produce the weirdest expressions, maxing out float precision problems. Therefore, this commit simply removes that particular check from `ValidateFloatHistogram`. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Łukasz Mierzwa	3c80963e81	Use a linked list for memSeries.headChunk (#11818 ) Currently memSeries holds a single head chunk in-memory and a slice of mmapped chunks. When append() is called on memSeries it might decide that a new headChunk is needed to use for given append() call. If that happens it will first mmap existing head chunk and only after that happens it will create a new empty headChunk and continue appending our sample to it. Since appending samples uses write lock on memSeries no other read or write can happen until any append is completed. When we have an append() that must create a new head chunk the whole memSeries is blocked until mmapping of existing head chunk finishes. Mmapping itself uses a lock as it needs to be serialised, which means that the more chunks to mmap we have the longer each chunk might wait for it to be mmapped. If there's enough chunks that require mmapping some memSeries will be locked for long enough that it will start affecting queries and scrapes. Queries might timeout, since by default they have a 2 minute timeout set. Scrapes will be blocked inside append() call, which means there will be a gap between samples. This will first affect range queries or calls using rate() and such, since the time range requested in the query might have too few samples to calculate anything. To avoid this we need to remove mmapping from append path, since mmapping is blocking. But this means that when we cut a new head chunk we need to keep the old one around, so we can mmap it later. This change makes memSeries.headChunk a linked list, memSeries.headChunk still points to the 'open' head chunk that receives new samples, while older, yet to be mmapped, chunks are linked to it. Mmapping is done on a schedule by iterating all memSeries one by one. Thanks to this we control when mmapping is done, since we trigger it manually, which reduces the risk that it will have to compete for mmap locks with other chunks. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2 years ago
George Krajcsovits	6cd2d1621f	Hide histogram chunk append and reset header internals (#12352 ) tsdb: Hide histogram chunk append and reset header internals Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	2 years ago
Justin Lei	32d87282ad	Add Zstandard compression option for wlog (#11666 ) Snappy remains as the default compression but there is now a flag to switch the compression algorithm. Signed-off-by: Justin Lei <justin.lei@grafana.com>	2 years ago
Marc Tudurí	4851ced266	tsdb: Support native histograms in snapshot on shutdown (#12258 ) Signed-off-by: Marc Tuduri <marctc@protonmail.com>	2 years ago
Justin Lei	4c4454e4c9	Group args to append to memSeries in chunkOpts Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Justin Lei	89af351730	Remove samplesPerChunk from memSeries (#12390 ) Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
beorn7	5b53aa1108	style: Replace `else if` cascades with `switch` Wiser coders than myself have come to the conclusion that a `switch` statement is almost always superior to a statement that includes any `else if`. The exceptions that I have found in our codebase are just these two: * The `if else` is followed by an additional statement before the next condition (separated by a `;`). * The whole thing is within a `for` loop and `break` statements are used. In this case, using `switch` would require tagging the `for` loop, which probably tips the balance. Why are `switch` statements more readable? For one, fewer curly braces. But more importantly, the conditions all have the same alignment, so the whole thing follows the natural flow of going down a list of conditions. With `else if`, in contrast, all conditions but the first are "hidden" behind `} else if `, harder to spot and (for no good reason) presented differently from the first condition. I'm sure the aforemention wise coders can list even more reasons. In any case, I like it so much that I have found myself recommending it in code reviews. I would like to make it a habit in our code base, without making it a hard requirement that we would test on the CI. But for that, there has to be a role model, so this commit eliminates all `if else` occurrences, unless it is autogenerated code or fits one of the exceptions above. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	c3c7d44d84	lint: Adjust to the lint warnings raised by current versions of golint-ci We haven't updated golint-ci in our CI yet, but this commit prepares for that. There are a lot of new warnings, and it is mostly because the "revive" linter got updated. I agree with most of the new warnings, mostly around not naming unused function parameters (although it is justified in some cases for documentation purposes – while things like mocks are a good example where not naming the parameter is clearer). I'm pretty upset about the "empty block" warning to include `for` loops. It's such a common pattern to do something in the head of the `for` loop and then have an empty block. There is still an open issue about this: https://github.com/mgechev/revive/issues/810 I have disabled "revive" altogether in files where empty blocks are used excessively, and I have made the effort to add individual `// nolint:revive` where empty blocks are used just once or twice. It's borderline noisy, though, but let's go with it for now. I should mention that none of the "empty block" warnings for `for` loop bodies were legitimate. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Justin Lei	c3e6b85631	Reverse test changes Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Justin Lei	052993414a	Add storage.tsdb.samples-per-chunk flag Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	3 years ago
beorn7	817a2396cb	Name float values as "floats", not as "values" In the past, every sample value was a float, so it was fine to call a variable holding such a float "value" or "sample". With native histograms, a sample might have a histogram value. And a histogram value is still a value. Calling a float value just "value" or "sample" or "V" is therefore misleading. Over the last few commits, I already renamed many variables, but this cleans up a few more places where the changes are more invasive. Note that we do not to attempt naming in the JSON APIs or in the protobufs. That would be quite a disruption. However, internally, we can call variables as we want, and we should go with the option of avoiding misunderstandings. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Justin Lei	73ff91d182	Test fixes Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Oleg Zaytsev	6e2905a4d4	Use zeropool.Pool to workaround SA6002 (#12189 ) * Use zeropool.Pool to workaround SA6002 I built a tiny library called https://github.com/colega/zeropool to workaround the SA6002 staticheck issue. While searching for the references of that SA6002 staticheck issues on Github first results was Prometheus itself, with quite a lot of ignores of it. This changes the usages of `sync.Pool` to `zeropool.Pool[T]` where a pointer is not available. Also added a benchmark for HeadAppender Append/Commit when series already exist, which is one of the most usual cases IMO, as I didn't find any. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Improve BenchmarkHeadAppender with more cases Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * A little copying is better than a little dependency https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Fix imports order Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Add license header Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Copyright should be on one of the first 3 lines Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Use require.Equal for testing I don't depend on testify in my lib, but here we have it available. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Avoid flaky test Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Also use zeropool for pointsPool in engine.go Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> --------- Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago
Abhijit Mukherjee	8f6d5dcd45	Fix: getting rid of EncOOOXOR chunk encoding (#12111 ) Signed-off-by: mabhi <abhijit.mukherjee@infracloud.io>	3 years ago
Rens Groothuijsen	d33eb3ab17	Automatically remove incorrect snapshot with index that is ahead of WAL (#11859 ) Signed-off-by: Rens Groothuijsen <l.groothuijsen@alumni.maastrichtuniversity.nl> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	d504c950a2	Remove unnecessary chunk fetch in Head queries `safeChunk` is only obtained from the `headChunkReader.Chunk` call where the chunk is already fetched and stored with the `safeChunk`. So, when getting the iterator for the `safeChunk`, we don't need to get the chunk again. Also removed a couple of unnecessary fields from `safeChunk` as a part of this. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Justin Lei	af1d9e01c7	Refactor tsdbutil for tests/native histograms (#11948 ) * Add float histograms to ChunkFromSamplesGeneric Signed-off-by: Justin Lei <justin.lei@grafana.com> * Add GenerateSamples functions to tsdbutil Signed-off-by: Justin Lei <justin.lei@grafana.com> PR responses Signed-off-by: Justin Lei <justin.lei@grafana.com> --------- Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
beorn7	1cfc8f65a3	histograms: Return actually useful counter reset hints This is a bit more conservative than we could be. As long as a chunk isn't the first in a block, we can be pretty sure that the previous chunk won't disappear. However, the incremental gain of returning NotCounterReset in these cases is probably very small and might not be worth the code complications. Wwith this, we now also pay attention to an explicitly set counter reset during ingestion. While the case doesn't show up in practice yet, there could be scenarios where the metric source knows there was a counter reset even if it might not be visible from the values in the histogram. It is also useful for testing. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ganesh Vernekar	38fa151a7c	tsdb: Only initialise out-of-order fields when required Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
beorn7	6dcd03dbf3	tsdb: Add integer gauge histogram support This follows what #11783 has done for float gauge histograms. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ganesh Vernekar	3c2ea91a83	tsdb: Test gauge float histograms Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	c155c0e312	tsdb: Test staleness handling of FloatHistogram Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Marc Tudurí	9474610baf	Support FloatHistogram in TSDB (#11522 ) Extends Appender.AppendHistogram function to accept the FloatHistogram. TSDB supports appending, querying, WAL replay, for this new type of histogram. Signed-off-by: Marc Tudurí <marctc@protonmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Bryan Boreham	10b27dfb84	Simplify IndexReader.Series interface Instead of passing in a `ScratchBuilder` and `Labels`, just pass the builder and the caller can extract labels from it. In many cases the caller didn't use the Labels value anyway. Now in `Labels.ScratchBuilder` we need a slightly different API: one to assign what will be the result, instead of overwriting some other `Labels`. This is safer and easier to reason about. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	4b6a4d1425	Update package tsdb tests for new labels.Labels type Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago

1 2 3 4

190 Commits (9c4782f1cc34f85f07daaaed14af2e7ad1ff56a0)