prometheus

Commit Graph

Author	SHA1	Message	Date
Justin Lei	c3e6b85631	Reverse test changes Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	3 years ago
beorn7	717a3f8e25	storage: Manually expand `genericAdd` for specific types This commit is doing what I would have expected that Go generics do for me. However, the PromQL benchmarks show a significant runtime and allocation increase with `genericAdd`, so this replaces it with hand-coded non-generic versions of it. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	817a2396cb	Name float values as "floats", not as "values" In the past, every sample value was a float, so it was fine to call a variable holding such a float "value" or "sample". With native histograms, a sample might have a histogram value. And a histogram value is still a value. Calling a float value just "value" or "sample" or "V" is therefore misleading. Over the last few commits, I already renamed many variables, but this cleans up a few more places where the changes are more invasive. Note that we do not to attempt naming in the JSON APIs or in the protobufs. That would be quite a disruption. However, internally, we can call variables as we want, and we should go with the option of avoiding misunderstandings. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	462240bc78	storage: add specialized buffers to sampleRing This utilizes the fact that most sampleRings will only contain samples of one type. In this case, the generic interface is circumvented, and a bespoke buffer for the one actually occurring sample type is used. Should a sampleRing receive a sample of a different kind later, it will transparently switch to the generic behavior. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	630bcb494b	storage: Use separate sample types for histogram vs. float Previously, we had one “polymorphous” `sample` type in the `storage` package. This commit breaks it up into `fSample`, `hSample`, and `fhSample`, each still implementing the `tsdbutil.Sample` interface. This reduces allocations in `sampleRing.Add` but inflicts the penalty of the interface wrapper, which makes things worse in total. This commit therefore just demonstrates the step taken. The next commit will tackle the interface overhead problem. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Justin Lei	f90013a5a0	Update storage/remote/codec.go Co-authored-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Justin Lei <97976793+leizor@users.noreply.github.com>	3 years ago
Justin Lei	83f43982c9	Add support for native histograms to concreteSeriesIterator Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Justin Lei	73ff91d182	Test fixes Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Xiaochao Dong (@damnever)	2b7202c4cc	Validate the metric names and labels in the remote write handler Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>	3 years ago
Bryan Boreham	b987afa7ef	labels: simplify call to get Labels from Builder It took a `Labels` where the memory could be re-used, but in practice this hardly ever benefitted. Especially after converting `relabel.Process` to `relabel.ProcessBuilder`. Comparing the parameter to `nil` was a bug; `EmptyLabels` is not `nil` so the slice was reallocated multiple times by `append`. Lastly `Builder.Labels()` now estimates that the final size will depend on labels added and deleted. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Oleg Zaytsev	beb7d3b80f	remote.Client: store urlString During remote write, we call url.String() twice: - to add the Endpoint() to the span - to actually know where whe should send the request This value does not change over time, and it's not really that lightweight to calculate. I wrote this simple benchmark: func BenchmarkURLString(b testing.B) { u, err := url.Parse("https://remote.write.com/api/v1") require.NoError(b, err) b.Run("string", func(b testing.B) { count := 0 for i := 0; i < b.N; i++ { count += len(u.String()) } }) } And the results are ~200ns/op, 80B/op, 3 allocs/op. Yes, we're going to go to the network here, which is a huge amount of resources compared to this, but still, on agents that send 500 requests per second, that is 1500 wasteful allocations per second. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago
Justin Lei	60ad864667	Remove hacky promql.Test native histogram thing Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Justin Lei	c16b6a0185	Handle native histograms in remote read Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Arve Knudsen	bc9a82f5a1	remote: Improve some comments (#12102 ) Improve some comments in storage/remote/queue_manager.go, wrt. general language and a typo. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Arve Knudsen	435b500de7	remote: Convert to RecoverableError using errors.As (#12103 ) In storage/remote, try converting to RecoverableError using errors.As, instead of through direct casting. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Justin Lei	af1d9e01c7	Refactor tsdbutil for tests/native histograms (#11948 ) * Add float histograms to ChunkFromSamplesGeneric Signed-off-by: Justin Lei <justin.lei@grafana.com> * Add GenerateSamples functions to tsdbutil Signed-off-by: Justin Lei <justin.lei@grafana.com> PR responses Signed-off-by: Justin Lei <justin.lei@grafana.com> --------- Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Fish-pro	43d77f7c41	Use http constants instead of string Signed-off-by: Fish-pro <zechun.chen@daocloud.io>	3 years ago
Charles Korn	0a1de58f7e	Mark Histogram.(Positive\|Negative)Spans as non-nullable. As far as I understand it, we'd never expect to receive a nil span, and remote.spansProtoToSpans would panic if we received a nil span. Marking the fields as non-nullable also means the generated Golang code doesn't use pointers for these fields, reducing allocations. Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 years ago
beorn7	49c5b1fae4	histograms: Fix counter reset header during merging See detailed discussion: https://github.com/prometheus/prometheus/pull/11864#issuecomment-1403963451 Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	1cfc8f65a3	histograms: Return actually useful counter reset hints This is a bit more conservative than we could be. As long as a chunk isn't the first in a block, we can be pretty sure that the previous chunk won't disappear. However, the incremental gain of returning NotCounterReset in these cases is probably very small and might not be worth the code complications. Wwith this, we now also pay attention to an explicitly set counter reset during ingestion. While the case doesn't show up in practice yet, there could be scenarios where the metric source knows there was a counter reset even if it might not be visible from the values in the histogram. It is also useful for testing. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
György Krajcsovits	2d9a9cbc08	Fix storage/remote/codec ignoreing histogram reset hint Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 years ago
Bryan Boreham	faac4c066d	package storage: fix up test not to access Labels internals Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Jesus Vazquez	136956cca4	Attempt to append ooo sample at the end first (#11615 ) This is an optimization on the existing append in OOOChunk. What we've been doing so far is find the place inside the out-of-order slice where the new sample should go in and then place it there and move any samples to the right if necessary. This is OK but requires a binary search every time the slice is bigger than 0. The optimization is opinionated and suggests that although out-of-order samples can be out-of-order amongst themselves they'll probably be in order thus we can probably optimistically append at the end and if not do the binary search. OOOChunks are capped to 30 samples by default so this is a small optimization but everything adds up, specially if you handle many active timeseries with out-of-order samples. Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Signed-off-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Marc Tudurí	721f33dbb0	histograms: Add remote-write support for Float Histograms (#11817 ) * adapt code.go and write_handler.go to support float histograms * adapt watcher.go to support float histograms * wip adapt queue_manager.go to support float histograms * address comments for metrics in queue_manager.go * set test cases for queue manager * use same counts for histograms and float histograms * refactor createHistograms tests * fix float histograms ref in watcher_test.go * address PR comments Signed-off-by: Marc Tuduri <marctc@protonmail.com>	3 years ago
beorn7	d121db7a65	federate: Fix PeekBack usage In most cases, there is no sample at `maxt`, so `PeekBack` has to be used. So far, `PeekBack` did not return a float histogram, and we disregarded even any returned normal histogram. This fixes both, and also tweaks the unit test to discover the problem (by using an earlier timestamp than "now" for the samples in the TSDB). Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Xiaochao Dong (@damnever)	2d61d012ff	Avoid copy during remote write if external labels is empty Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>	3 years ago
Fish-pro	6ed71a229e	Use errors.Is to check for a specific error Signed-off-by: Fish-pro <zechun.chen@daocloud.io>	3 years ago
Marc Tudurí	9474610baf	Support FloatHistogram in TSDB (#11522 ) Extends Appender.AppendHistogram function to accept the FloatHistogram. TSDB supports appending, querying, WAL replay, for this new type of histogram. Signed-off-by: Marc Tudurí <marctc@protonmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Sniper91	46fb802791	reset frameBytesLeft after writing (#11689 ) Signed-off-by: sniper91 <kevinzhao91@outlook.com> Signed-off-by: sniper91 <kevinzhao91@outlook.com>	3 years ago
Bryan Boreham	047585360b	Update package storage/remote tests for new labels.Labels type Use ScratchBuilder to create labels. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	abd9909595	Update package storage/remote for new labels.Labels type `QueueManager.externalLabels` becomes a slice rather than a `Labels` so we can index into it when doing the merge operation. Note we avoid calling `Labels.Len()` in `labelProtosToLabels()`. It isn't necessary - `append()` will enlarge the buffer and we're expecting to re-use it many times. Also, we now validate protobuf input before converting to Labels. This way we can detect errors first, and we don't place unnecessary requirements on the Labels structure. Re-do seriesFilter using labels.Builder (albeit N^2). Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	d6b97f631a	Update package storage for new labels.Labels type Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
sniper91	16738b00e3	Do no re-use result slice in chunkSetToSeriesSet This is required to preserve the interface property of SeriesSet that says "At returns full series. Returned series should be iterable even after Next is called." Signed-off-by: sniper91 <kevinzhao91@outlook.com>	3 years ago
Bryan Boreham	0853250695	Review feedback Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	463f5cafdd	storage: re-use iterators to save garbage Re-use previous memory if it is already of the correct type. In `NewListSeries` we hoist the conversion to an interface value out so it only allocates once. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	3c7de69059	storage: allow re-use of iterators Patterned after `Chunk.Iterator()`: pass the old iterator in so it can be re-used to avoid allocating a new object. (This commit does not do any re-use; it is just changing all the method signatures so re-use is possible in later commits.) Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	7ce09b4e39	storage: fix BenchmarkMergeSeriesSet The SeriesSets to be merged must be created each time round the loop, otherwise the benchmark is not doing any real work. Don't call ExpandSeries, because it spends most of its time allocating a memory buffer to hold the result, which we don't look at. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Fix up merge test again	3 years ago
Julius Volz	1a2c645dfa	Correctly handle error unwrapping in rules and remote write receiver errors.Unwrap() actually dangerously returns nil if the error does not have an Unwrap() method, which is the case in at least one of these places where I noticed that no error was being logged at all when it should have. Signed-off-by: Julius Volz <julius.volz@gmail.com>	3 years ago
tanghengjian	982007ecab	GetRefByhash will query a label's ref with hash value rather than lset.Hash(). Signed-off-by: tanghengjian <1040104807@qq.com>	3 years ago
Giedrius Statkevičius	d1d2566055	remote/read_handler: pool input to Marshal() (#11357 ) * remote/read_handler: pool input to Marshal() Use a sync.Pool to reuse byte slices between calls to Marshal() in the remote read handler. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * remote: add microbenchmark for remote read handler Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	3 years ago
Björn Rabenstein	bd26ce440c	storage: Avoid duplicate function call in sampleRingIterator (#11502 ) Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Signed-off-by: Jesus Vazquez	3362bf6d79	Fix merge conflicts Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Jesus Vazquez	775d90d5f8	TSDB: Rename wal package to wlog (#11352 ) The wlog.WL type can now be used to create a Write Ahead Log or a Write Behind Log. Before the prefix for wbl metrics was 'prometheus_tsdb_out_of_order_wal_' and has been replaced with 'prometheus_tsdb_out_of_order_wbl_'. Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Signed-off-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com> Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>	3 years ago
Bryan Boreham	8120af22e2	benchmarks: SetBytes takes bytes per operation Where the code was multiplying bytes by number of operations, this resulted in absurdly high throughput numbers. Also, in `BenchmarkParse()`, don't run the `expfmt` case twice. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	3330d85ba8	Replace sort.Strings and sort.Ints with faster slices.Sort (#11318 ) Use new experimental package `golang.org/x/exp/slices`. slices.Sort works on values that are directly comparable, like ints, so avoids the overhad of an interface call to `.Less()`. Left tests unchanged, because they don't need the speed and it may be a cross-check that slices.Sort gives the same answer. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Jesus Vazquez	c1b669bf9b	Add out-of-order sample support to the TSDB (#11075 ) * Introduce out-of-order TSDB support This implementation is based on this design doc: https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit?usp=sharing This commit adds support to accept out-of-order ("OOO") sample into the TSDB up to a configurable time allowance. If OOO is enabled, overlapping querying are automatically enabled. Most of the additions have been borrowed from https://github.com/grafana/mimir-prometheus/ Here is the list ist of the original commits cherry picked from mimir-prometheus into this branch: - `4b2198d7ec` - `2836e5513f` - `00b379c3a5` - `ff0dc75758` - `a632c73352` - `c6f3d4ab33` - `5e8406a1d4` - `abde1e0ba1` - `e70e769889` - `df59320886` Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Dieter Plaetinck <dieter@grafana.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * gofumpt files Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Add license header to missing files Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix OOO tests due to existing chunk disk mapper implementation Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix truncate int overflow Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Add Sync method to the WAL and update tests Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * remove useless sync Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Update minOOOTime after truncating Head * Update minOOOTime after truncating Head Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Add a unit test Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Load OutOfOrderTimeWindow only once per appender Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix OOO Head LabelValues and PostingsForMatchers Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix replay of OOO mmap chunks Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Remove unnecessary err check Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Prevent panic with ApplyConfig Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Run OOO compaction after restart if there is OOO data from WBL Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Apply Bartek's suggestions Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Refactor OOO compaction Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Address comments and TODOs - Added a comment explaining why we need the allow overlapping compaction toggle - Clarified TSDBConfig OutOfOrderTimeWindow doc - Added an owner to all the TODOs in the code Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Run go format Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix remaining review comments Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix tests Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Change wbl reference when truncating ooo in TestHeadMinOOOTimeUpdate Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> * Fix TestWBLAndMmapReplay test failure on windows Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Address most of the feedback Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Refactor the block meta for out of order Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix windows error Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix review comments Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Dieter Plaetinck <dieter@grafana.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	3 years ago
Ganesh Vernekar	d354f20c2a	Add a feature flag to control native histogram ingestion (#11253 ) * Add runtime config to control native histogram ingestion Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Make the config into a CLI flag Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Bryan Boreham	3029320ce6	storage/remote: in tests use labels.FromStrings And a few cases of `EmptyLabels()`. Replacing code which assumes the internal structure of `Labels`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	ad1c257899	storage: in tests use labels.FromStrings Replacing code which assumes the internal structure of `Labels`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago

1 2 3 4 5 ...

1352 Commits (1bfb3ed062e99bd3c74e05d9ff9a7fa4e30bbe21)