Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
loki/pkg/compactor/table_locker.go

53 lines
1.1 KiB

compactor: do not block compation when retention is taking too long (#9884) **What this PR does / why we need it**: Currently, we perform compaction and apply retention in the same loop. Although we have a flag for not applying retention every time we perform compaction, we still see compaction getting blocked when processing some intensive delete requests(processed while applying retention). This PR separates out the compaction and retention to run in a separate loop. I have added a table-locking feature to avoid compaction and retention from processing the same tables at a time. However, compaction and retention would treat locked tables differently, as explained below: * When compaction sees a table is locked: It would skip the table and move on to the following table. However, before skipping, it would check if the table has any uncompacted files and increment the newly added counter called `loki_compactor_skipped_compacting_locked_tables_total` to track how often we are skipping compacting tables which have uncompacted files. * When retention sees a table is locked: It would wait for the lock to be released since we can't skip any tables while processing delete requests. **Special notes for your reviewer**: * The check for tables with uncompacted files looks for count > 1 because initially, we did not support per tenant index in `boltdb-shipper`, so a table can have a single compacted multi-tenant index file. In a rare case where we have a single file which was supposed to be compacted away, it is okay to have a single uncompacted file for a while. The aim here is to not block compaction for too long in a large cell with too many uncompacted files. * Retention only works on the compacted index, so we first compact down any uncompacted files while applying retention. **Checklist** - [x] Tests updated --------- Co-authored-by: Ashwanth <iamashwanth@gmail.com>
2 years ago
package compactor
import "sync"
type lockWaiterChan chan struct{}
type tableLocker struct {
lockedTables map[string]lockWaiterChan
lockedTablesMtx sync.RWMutex
}
func newTableLocker() *tableLocker {
return &tableLocker{
lockedTables: map[string]lockWaiterChan{},
}
}
// lockTable attempts to lock a table. It returns true if the lock gets acquired for the caller.
// It also returns a channel which the caller can watch to detect unlocking of table if it was already locked by some other caller.
func (t *tableLocker) lockTable(tableName string) (bool, <-chan struct{}) {
locked := false
t.lockedTablesMtx.RLock()
c, ok := t.lockedTables[tableName]
t.lockedTablesMtx.RUnlock()
if ok {
return false, c
}
t.lockedTablesMtx.Lock()
defer t.lockedTablesMtx.Unlock()
c, ok = t.lockedTables[tableName]
if !ok {
t.lockedTables[tableName] = make(chan struct{})
c = t.lockedTables[tableName]
locked = true
}
return locked, c
}
func (t *tableLocker) unlockTable(tableName string) {
t.lockedTablesMtx.Lock()
defer t.lockedTablesMtx.Unlock()
c, ok := t.lockedTables[tableName]
if ok {
close(c)
}
delete(t.lockedTables, tableName)
}