prometheus

Commit Graph

Author	SHA1	Message	Date
Arthur Silva Sens	442f24e099	chore: Simplify TestHeadAppender_AppendCTZeroSample (#14812 ) Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	2 years ago
Oleg Zaytsev	ce7d830f1f	Bring back BenchmarkLoadRealWLs (#14757 ) This was part of #14525 which was reverted. I still think that having this benchmark committed in to the repo is useful. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Callum Styan	a77f5007f9	fix bug with metadata for rw2 (#14766 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Bryan Boreham	1f38ae7bca	[TESTS] TSDB: fix up OOO tests for new Series behaviour Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	cde42f30e9	TSDB: streamline reading of overlapping head chunks `getOOOSeriesChunks` was already finding sets of overlapping chunks; we store those in a `multiMeta` struct so that `ChunkOrIterable` can reconstruct an `Iterable` easily and predictably. We no longer need a `MergeOOO` flag to indicate that this Meta should be merged with other ones; this is explicit in the `multiMeta` structure. We also no longer need `chunkMetaAndChunkDiskMapperRef`. Add `wrapOOOHeadChunk` to defeat `chunkenc.Pool` - chunks are reset during compaction, but if we wrap them (like `safeHeadChunk` was doing then this is skipped) . Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	838e49e7b8	[REFACTOR] TSDB: move chunkFromSeries from headChunkReader to head Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
riskrole	406bf775aa	chore: fix some comments Signed-off-by: riskrole <yuhang@before.tech>	2 years ago
Marco Pracucci	ef649d5968	Revert " Store `mmMaxTime` in same field as `seriesShard`" Signed-off-by: Marco Pracucci <marco@pracucci.com>	2 years ago
György Krajcsovits	183bbc39a2	Make requesting merge with OOO head explicit in chunk.Meta Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
György Krajcsovits	41c076196e	New cases in Test_ChunkQuerier_OOOQuery and Test_Querier_OOOQuery Case 1: OOO in-memory head chunk overlaps with first mmaped in-order chunk. Query: \|----------------------------------------------------------------\| InO: \|------mmap---------------\|\|---------mem----------------------\| OOO: \|-----mem-----------\| This triggers ChunkOrIterableWithCopy not including OOO head chunks bug. Similar to #14693 however testing the end of the interval doesn't trigger the problem because there the in-order head chunk will be trimmed with a tombstone, causing the code to switch to ChunkOrIterable which was fixed. See `a36d1a8a92/tsdb/querier.go (L646)` where len(p.bufIter.Intervals) will be non zero, because it includes the tombstone to trim the result to the query max time. Thus a new test is added to check the overlap at the beginning of the interval that has a separate chunk, which does not need trimming. Note: same test doesn't fail for sample querier in Test_Querier_OOOQuery as that doesn't use copy, that is copyHeadChunk is false in the if condition above. Case 2: OOO mmaped head chunk overlaps with first mmaped in-order chunk. Query: \|----------------------------------------------------------------\| InO: \|------mmap---------------\|\|---------mem----------------------\| OOO: \|-----mmap-----------\| \|--mem--\| In this case the meta contains the reference of the in-order chunk and no indication that a merge is needed with the OOO mmaped chunk. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Arve Knudsen	b0aba26ed5	tsdb: Fix ValNone typo in comment Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Bryan Boreham	9a74d53935	[BUGFIX] TSDB: Fix query overlapping in-order and ooo head (#14693 ) * tsdb: Unit test query overlapping in order and ooo head Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * TSDB: Merge overlapping head chunk The basic idea is that getOOOSeriesChunks can populate Meta.Chunk, but since it only returns one Meta per overlapping time-slot, that pointer may end up in a Meta with a head-chunk ID. So we need HeadAndOOOChunkReader.ChunkOrIterable() to call mergedChunks in that case. Previously, mergedChunks was checking that meta.Ref was a valid OOO chunk reference, but it never actually uses that reference; it just finds all chunks overlapping in time. So we can delete that code. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
shandongzhejiang	b2712ff284	chore: fix some function names Signed-off-by: shandongzhejiang <shandongzhejiang@icloud.com>	2 years ago
Arve Knudsen	3a78e76282	Upgrade golangci-lint to v1.60.1 Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
cuiweiyuan	1800af54f0	chore: fix some function names Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>	2 years ago
Bryan Boreham	512c67ec26	TSDB: Never go over maximum number of OOO chunks In `mmapCurrentOOOHeadChunk`, check if the number is at the maximum and drop the data with an error log. This is not expected to happen as the maximum is over 8 million; that's 8 years of 1 sample every second. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	9135da1e4f	TSDB: Review feedback Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Re-enable check in `createHeadWithOOOSamples` which wasn't really broken. * Move code making `Block` into a `Queryable` into test file. * Make `getSeriesChunks` return a slice (renamed `appendSeriesChunks`). * Rename `oooMergedChunks` to `mergedChunks`. * Improve comment on `ChunkOrIterableWithCopy`. * Name return values from unpackHeadChunkRef. Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	7ffd3ca280	TSDB: Cosmetic: move HeadAndOOO implementations where old code was This makes the diffs easier to follow. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	e95607b276	TSDB: Lock round access to labels, where necessary Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	26b3de0438	TSDB: Remove OOOHeadIndexReader Use headIndexReader instead. OOOCompactionHeadIndexReader needs to be expanded slightly, because it previously delegated to OOOHeadIndexReader. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	a299c7b6d6	TSDB: Remove OOOHeadChunkReader Use HeadAndOOOChunkReader instead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	e7e50a3afd	TSDB: Remove code for querying OOO-head only Just query via `HeadAndOOOQuerier`, which will skip series where no in-order chunks are in range. Now we don't need `OOORangeHead`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	0a2ff76881	TSDB tests: Fix up BenchmarkQueries Was not working even on main. Some cases still error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	f261597944	TSDB: Fix up LabelValues to work for OOO-only head Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	6529d6336c	TSDB: NewHeadAndOOOChunkReader takes headChunkReader So we can pass nil and have it read just OOO chunks. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	e04d137649	[PERF] TSDB: Query head and ooo-head together Add `HeadAndOOOQuerier` which iterates just once over series, then where necessary merges chunks from in-order and out-of-order lists. Add a ChunkQuerier for in-order and ooo together Add copy-last-chunk behaviour to HeadAndOOOChunkReader Out-of-order chunk IDs are distinguished from in-order by setting bit 23. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	da31da3ea6	Refactor: extract selectSeriesSet and selectChunkSeriesSet Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	7e24844d08	Refactor: extract headChunkReader.chunkFromSeries() For when you have a series locked already. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	a32aca0cd7	Refactoring: extract getOOOSeriesChunks Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	c75c8f8329	Refactoring: extract getSeriesChunks Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	0c852680bf	[Benchmark] TSDB: Add BenchmarkQuerierSelectWithOutOfOrder Refactor existing BenchmarkQuerierSelect to provide the set-up. Note that Head queries now run faster because they use a RangeHead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
György Krajcsovits	41656162fc	tsdb: prepare inserting native histograms into OOO head Rename a variable. Add parameters to memSeries.insert function. No effect on how float samples are handled. Related to #14546 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
machine424	82f38d3e9a	fix(tsdb/db_test.go): close the corrupted chunk after creating it to satisfy Windows FS Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2 years ago
György Krajcsovits	1ea3781699	Fix ToEncodedChunks minT for recoded chunks Discovered while working on #14546 OOO native histograms. Not triggered on main before #14546 as the code path is unused. There was a bug where the min time of a chunk was adjusted even if it was only recoded and not completely new. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Ben Ye	b7a58dcf3d	Add hidden flag to disable overlapping compaction (#14581 ) TSDB: add hidden flag to disable overlapping compaction Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com>	2 years ago
Filip Petkovski	6100e756a8	Ignore stale histograms for counter reset detection The histogram stats decoder keeps track of the last seen histogram sample in order to properly detect counter resets. We are seeing an issue where a histogram with UnknownResetHint gets treated as a counter reset when it follows a stale histogram sample. I believe that this is incorrect since stale samples should be completely ignored in PromQL. As a result, they should not be stored in the histogram stats iterator and the counter reset detection needs to be done against the last non-stale sample. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2 years ago
machine424	92873d3009	feat: allow to delay head compaction start time helping Prometheus instances to avoid simultaneous compactions and reduce stress on shared resources. This is enabled via `--enable-feature=delayed-compaction`. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2 years ago
Oleg Zaytsev	0833d2a230	Fix appendable: check whether last val was a histogram (#14613 ) * Fix appendable: check whether last val was a histogram When appending a float, we were checking whether lastValue was equal to current value, but we didn't check whether last value was a float value. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
György Krajcsovits	98ecdf3589	Fix corrupting spans via iterator sharing Iterator may share spans without copy, so we always have to make a copy before modification - copy-on-write. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
György Krajcsovits	d2f6fa7289	Fix lint error Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
György Krajcsovits	1b6d1366d8	Fix re-code histogram and chunk re-code conflict Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
György Krajcsovits	aff089a014	Reproduce recoding bug with new and missing buckets Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
machine424	9e43ad2e37	chore(remote_write): clean up as watcher.go is part of wlog now Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2 years ago
Bryan Boreham	015638c4b6	[BUGFIX] TSDB: Exclude OOO chunks mapped after compaction starts Otherwise the writer can end up with invalid chunks. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	bded853035	[Test] TSDB: TestOOOCompaction with samples added after compaction starts Test fails due to bug. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
George Krajcsovits	00ab05c3b9	Native histograms: fix spurios counter reset when merging recoded chunk to normal chunk (#14513 ) * chunkenc: allow missing empty buckets on histogram append Allow appending to chunks when the histogram to be added is missing some buckets, but the missing buckets are empty in the chunk. For example bucket at index 5 is present in the chunk, but its value is 0 and the new histogram doesn't have a bucket at index 5. This fixes an issue of merging chunks where one chunk was recoded to retroactively have some empty buckets in all the histograms and we are merging in a histogram that doesn't have the empty bucket (because it was not recoded yet). The operation alters the histogram that is being added, however this has already been the case when appending gauge histograms. Thus the test TestHistogramSeriesToChunks in storage package is changed to explicitly test what happened to the appended histogram - Compact(0) call is removed. The new expandIntSpansAndBuckets and expandFloatSpansAndBuckets functions are a merge of expandSpansForward and counterResetInAnyBucket and counterResetInAnyFloatBucket. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Max Amin	84b819a69f	feat: add Google cloud roundtripper for remote write (#14346 ) * feat: Google Auth for remote write Signed-off-by: Max Amin <maxamin@google.com> --------- Signed-off-by: Max Amin <maxamin@google.com>	2 years ago
Oleg Zaytsev	0300ad58a9	Revert the option regardless of error Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Oleg Zaytsev	d8e1b6bdfd	Store mmMaxTime in same field as seriesShard We don't use seriesShard during DB initialization, so we can use the same 8 bytes to store mmMaxTime, and save those during the rest of the lifetime of the database. This doesn't affect CPU performance. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago

1 2 3 4 5 ...

1152 Commits (deabb30b207c22eeda4cdfd15e15529b7c98c55b)