**What this PR does / why we need it**:
This PR limits the size of the blocks created by the compactor.
The block builder keeps adding series' blooms to a block until the limit
is exceeded, meaning that blocks may grow beyond the configured maximum.
This is needed so we avoid having tiny blocks which had space for small
blooms but later a bigger blooms didn't fit.
Blocks are built lazily: the generator returns an iterator that builds
one block at a time.
**Special notes for your reviewer**:
The maximum size is currently set to 50 MBs. We will make this
configurable on a followup PR.
**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Documentation added
- [x] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e3ec)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](0d4416a4b0)
---------
Co-authored-by: Christian Haudum <christian.haudum@gmail.com>
level.Warn(logger).Log("msg","failed to get schema for block","err",err)
skippedBlocks=append(skippedBlocks,md)
// Close unneeded block
closeErrors.Add(block.Close())
continue
}
if!s.opts.Schema.Compatible(schema){
level.Warn(logger).Log("msg","block schema incompatible with options","generator_schema",fmt.Sprintf("%+v",s.opts.Schema),"block_schema",fmt.Sprintf("%+v",schema))
skippedBlocks=append(skippedBlocks,md)
// Close unneeded block
closeErrors.Add(block.Close())
continue
}
level.Debug(logger).Log("msg","adding compatible block to bloom generation inputs")
level.Debug(s.logger).Log("msg","generating bloom filters for blocks","num_blocks",len(blocksMatchingSchema),"skipped_blocks",len(skippedBlocks),"schema",fmt.Sprintf("%+v",s.opts.Schema))