Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
loki/pkg/util/evicting_queue.go

94 lines
1.5 KiB

Ruler: Recording Rules (#3766) * WIP: hack to get recording rules working and pushing to Cortex/Prometheus Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring Adding remote_write config for ruler Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Minor refactorings Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving manager subpackage into ruler package to avoid dependency cycles This also mirrors Cortex's package structure Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Minor refactorings Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Skipping commit if remote-write client is not defined Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating use of cortex client Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Memoizing appenders, using queue for samples & labels Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding buffer size configurability Refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metric to show current buffer size Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring for better responsibility separation & testability Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding per-tenant overrides of remote-write queue capacity Renaming "buffer size" to "queue capacity" to be more accurate Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding tests for evicting queue Minor refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding more tests and refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding queue benchmark Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Reducing redundancy in metric names Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Testing that only metric queries can be run Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Minor fixes pre-review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Guarding against unprotected nil pointer dereference in Prometheus remote.Client Adding remote-write client validation Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Setting tenant ID header on remote-write client Adding User-Agent string Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating benchmark to use complex struct rather than int to be more reflective of usage Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Registering flags Removing extraneous checks Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metric to track remote-write commit errors Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring based on review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Performance improvements based on review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Return error on invalid queue capacity Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing global queue capacity config - using limits Minor refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Reusing memory in request preparation Refactoring for testability Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving remote-write metrics into struct Refactoring Unexporting and refactoring memstore metrics to match Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Applying review suggestions Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Allowing for runtime changing of per-tenant remote-write queue capacity Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
5 years ago
package util
import (
"errors"
"sync"
)
type EvictingQueue struct {
sync.RWMutex
capacity int
entries []interface{}
onEvict func()
}
func NewEvictingQueue(capacity int, onEvict func()) (*EvictingQueue, error) {
if err := validateCapacity(capacity); err != nil {
return nil, err
}
queue := &EvictingQueue{
onEvict: onEvict,
entries: make([]interface{}, 0, capacity),
}
err := queue.SetCapacity(capacity)
if err != nil {
return nil, err
}
return queue, nil
}
func (q *EvictingQueue) Append(entry interface{}) {
q.Lock()
defer q.Unlock()
if len(q.entries) >= q.capacity {
q.evictOldest()
}
q.entries = append(q.entries, entry)
}
func (q *EvictingQueue) evictOldest() {
q.onEvict()
start := (len(q.entries) - q.Capacity()) + 1
q.entries = append(q.entries[:0], q.entries[start:]...)
}
func (q *EvictingQueue) Entries() []interface{} {
q.RLock()
defer q.RUnlock()
return q.entries
}
func (q *EvictingQueue) Length() int {
q.RLock()
defer q.RUnlock()
return len(q.entries)
}
func (q *EvictingQueue) Capacity() int {
return q.capacity
}
func (q *EvictingQueue) SetCapacity(capacity int) error {
if err := validateCapacity(capacity); err != nil {
return err
}
q.capacity = capacity
return nil
}
func (q *EvictingQueue) Clear() {
q.Lock()
defer q.Unlock()
q.entries = q.entries[:0]
}
func validateCapacity(capacity int) error {
if capacity <= 0 {
// a queue of 0 (or smaller) capacity is invalid
return errors.New("queue cannot have a zero or negative capacity")
}
return nil
}