Estimate chunk size on compaction looking at data uncompressed size (#9099)

**What this PR does / why we need it**:
At `compactedIndex.ToIndexFile` we were not computing the size of the
chunk correctly. We were looking at the size of the chunk ref. Since the
size of the chunk ref is always a bunk of bytes, the chunk meta KB field
would always be 0.

4e893a0a88/pkg/storage/stores/tsdb/compactor.go (L367)

Instead, we should get the approx KBs by looking at the chunks data
uncompressed size. As we do at `store.IndexChunk`:

d405162516/pkg/storage/stores/tsdb/store.go (L189)
pull/9035/head^2
Salva Corts 2 years ago committed by GitHub
parent b82ad11f92
commit fd5d78784f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 1
      CHANGELOG.md
  2. 3
      pkg/storage/stores/tsdb/compactor.go
  3. 5
      pkg/storage/stores/tsdb/compactor_test.go

@ -12,6 +12,7 @@
* [8971](https://github.com/grafana/loki/pull/8971) **dannykopping**: Stats: fix `Cache.Chunk.BytesSent` statistic and loki_chunk_fetcher_fetched_size_bytes metric with correct chunk size.
* [8979](https://github.com/grafana/loki/pull/8979) **slim-bean**: Fix the case where a logs query with start time == end time was returning logs when none should be returned.
* [9099](https://github.com/grafana/loki/pull/9099) **salvacorts**: Fix the estimated size of chunks when writing a new TSDB file during compaction.
### All Changes

@ -360,11 +360,12 @@ func (c *compactedIndex) ToIndexFile() (index_shipper.Index, error) {
b.Del(labels.MetricName)
ls := b.Labels(nil)
approxKB := math.Round(float64(chk.Data.UncompressedSize()) / float64(1<<10))
err := c.builder.InsertChunk(ls.String(), index.ChunkMeta{
Checksum: chk.Checksum,
MinTime: int64(chk.From),
MaxTime: int64(chk.Through),
KB: uint32(chk.Size()) / (1 << 10),
KB: uint32(approxKB),
Entries: uint32(chk.Data.Entries()),
})
if err != nil {

@ -202,6 +202,7 @@ func buildChunkMetas(from, to int64) index.ChunkMetas {
MaxTime: i + 1,
Checksum: uint32(i),
Entries: 1,
KB: 1,
})
}
@ -897,6 +898,10 @@ type dummyChunkData struct {
chunk.Data
}
func (d dummyChunkData) UncompressedSize() int {
return 1 << 10 // 1KB
}
func (d dummyChunkData) Entries() int {
return 1
}

Loading…
Cancel
Save