apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Bryan Boreham	08ac5336d5	WAL: store pointers in record pool (#8667 ) What this PR does / why we need it: Storing a slice in `sync.Pool` wastes 24 bytes every time it is returned to the heap; see https://staticcheck.io/docs/checks#SA6002 Storing a pointer to a slice allows the Pool to work without waste. This came up in discussion about the benchmark in #8489 - with this change the memory usage comes down: ``` name old time/op new time/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 7.37ms ±13% 7.45ms ± 9% ~ (p=0.841 n=5+5) name old alloc/op new alloc/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 727kB ± 0% 679kB ± 0% -6.63% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Writer_WriteEntries/1000_lines,_1_different_label_sets-4 12.0k ± 0% 10.0k ± 0% -16.68% (p=0.000 n=5+4) ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Pablo	fda14ca156	Re-license and export `pkg/ingester` WAL code to be used in Promtail's WAL (#8315 )	3 years ago
Susana Ferreira	68a4d3271a	Fix documentation typos from #7916 (#7934 ) What this PR does / why we need it: Fix documentation typos from https://github.com/grafana/loki/pull/7916#pullrequestreview-1217192817 Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/pull/7916#pullrequestreview-1217192817 Special notes for your reviewer: Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916 ) What this PR does / why we need it: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](`5550cd65ec/docs/sources/configuration/_index.md`) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. NOTE: this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki-private/issues/83 Special notes for your reviewer: Files: * [docs/sources/configuration/index.template](`5550cd65ec/docs/sources/configuration/index.template`): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](`c32e5d0acb/docs/sources/configuration/_index.md`): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](`5550cd65ec/tools/doc-generator`) directory with documentation generation tool. Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Robert Fratto	85392a9728	Update Prometheus dependency to latest release (v2.40.4) (#7826 ) Closes #7811, which is needed for Grafana Agent to update to v2.40 and add support for native histograms. I did not add support for native histograms to Loki, sorry :)	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Trevor Whitney	ed025e0cd6	Change a few default config values and improve application of common storage config (#4543 ) * further config simplifications and better defaults * default ruler api to on * register defaults for storage configs in the common config (ie. signature version for S3) * enable ingester wal by default * disable chunk retries by default * allow common path prefix to include or omit trailing slash * allow partial storage config to be defined for overriding things like bucket names per component * fix tests * add changelog and upgrade docs * explain impact of wal change in upgrading.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * rename add WithPrefix to RegisterFlags * fix (frontend OR worker) and scheduler boot order when both are running (#4545) Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484 ) * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Owen Diehl	05b8537e79	WAL Replay counter (#3974 ) * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting * introduces an entry counter into streams & wal records * sets & resets entrycount on checkpoint recovery and after wal replay finishes * pushing to streams checks counter * reject pre-applied entries test * removes commented test	5 years ago
Owen Diehl	862cfa4e1f	adds Start() method to WAL interface to delay checkpointing until after replay (#3339 )	5 years ago
Owen Diehl	8a9b94a8bc	removes recover flag (#3326 )	5 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295 ) * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	5 years ago
Owen Diehl	224e68a50b	WAL backpressure (#3218 ) * adds WAL replay backpressure * backpressure fn doesnt lock during passed fn for concurrency * replayController * replayController supports a RemoveFlushedChunks fn * custom wal flusher * encoding test determinism * standardizes recoveredBytesTotal to replayControllers ownership * linting * default wal memory threshold to 4gb, adds docs * fix test after v3 chunk schema change * more lenient expectCheckpoint function * replayController protects against flush race condition * adds replay_memory_ceiling to wal jsonnet * replay ceiling help msg	5 years ago
Owen Diehl	4d9865acd4	Adds WAL support (experimental) (#2981 ) * marshalable chunks * wal record types custom serialization * proto types for wal checkpoints * byteswith output unaffected by buffer * wal & record pool ifcs * wal record can hold entries from multiple series * entry pool * ingester uses noopWal * removes duplicate argument passing in ingester code. adds ingester config validation & derives chunk encoding. * segment writing * [WIP] wal recovery from segments * replay uses sync.Maps & preserves WAL fingerprints * in memory wal recovery * wal segment recovery * ingester metrics struct * wal replay locks streamsMtx in instances, adds checkpoint codec * ingester metrics * checkpointer * WAL checkpoint writer * checkpointwriter can write multiple checkpoints * reorgs checkpointing * wires up checkpointwriter to wal * ingester SeriesIter impl * wires up ingesterRecoverer to consume checkpoints * generic recovery fn * generic recovery fn * recover from both wal types * cleans up old tmp checkpoints & allows aborting in flight checkpoints * wires up wal checkpointing * more granular wal logging * fixes off by 1 wal truncation & removes double logging * adds userID to wal records correctly * wire chunk encoding tests * more granular wal metrics * checkpoint encoding test * ignores debug bins * segment replay ignores out of orders * fixes bug between WAL reading []byte validity and proto unmarshalling refs * conf validations, removes comments * flush on shutdown config * POST /ingester/shutdown * renames flush on shutdown * wal & checkpoint use same segment size * writes entries to wal regardless of tailers * makes wal checkpoing duration default to 5m * recovery metrics * encodes headchunks separately for wal purposes * merge upstream * linting * addresses pr feedback uses entry pool in stream push/tailer removes unnecessary pool interaction checkpointbytes comment fillchunk helper, record resetting in tests via pool redundant comment defers wg done in recovery s/num/count/ checkpoint wal uses a logger encodeWithTypeHeader now creates its own []byte removes pool from decodeEntries wal stop can error * prevent shared access bug with tailers and entry pool * removes stream push entry pool optimization	5 years ago

14 Commits (8abe080fd6d04dc5efb074ee0764a9bfd39c82d2)