Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Ivan Kalita 3f62bb7207 chore(compactor): Allow overflowed logsobj builders (#22148) ... I'm changing the behavior of logsobj builders to allow appends to overflowed builders. Before this PR: (1) Processor tries to append a record to a builder (2) If a builder is full it doesn't append and returns an error ErrBuilderFull (3) Processor checks the error and flushes the builder (4) Processor tries to append the same record as in the step (1) to the now empty builder This approach works fine but becomes problematic once we introduce multiple TOC windows builders. The issue is that with a single builder we have atomic "try append - flush if error" behavior guarantee. With multiple builders this becomes a problem as a single kafka record might contain entries that belong to different builders. Example: - we have a record R that has a log entry L0 for TOC window W0 AND a log entry L1 for TOC window W1 - we attempt to append L0 to the first builder B0 (for TOC W0) -> success - we attempt to append L1 to the second builder B1 (for TOC W1) -> not appended, err builder full - we flush all the builders (have to flush all of them because otherwise we can lose data as Kafka offset is committed after flushing) - we attempt to append R again -> builder B0 will receive a duplicate of log L0 To simplify the processing we can allow appends to full builders ("one last time"). For the example above we'd allow the append of L1 to B1 even though B1 is already full. This way the dataobjects might become slightly bigger than the target size but this could already be the case given that the existing size estimates are estimates.		6 days ago
..
objtest.go	chore(compactor): Allow overflowed logsobj builders (#22148)	6 days ago