Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
loki/pkg/storage/cache_test.go

154 lines
4.4 KiB

package storage
import (
"errors"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/grafana/loki/pkg/iter"
"github.com/grafana/loki/pkg/logproto"
)
func Test_CachedIterator(t *testing.T) {
stream := logproto.Stream{
Labels: `{foo="bar"}`,
Entries: []logproto.Entry{
{Timestamp: time.Unix(0, 1), Line: "1"},
{Timestamp: time.Unix(0, 2), Line: "2"},
{Timestamp: time.Unix(0, 3), Line: "3"},
},
}
c := newCachedIterator(iter.NewStreamIterator(stream), 3)
assert := func() {
// we should crash for call of entry without next although that's not expected.
require.Equal(t, stream.Labels, c.Labels())
require.Equal(t, stream.Entries[0], c.Entry())
require.Equal(t, true, c.Next())
require.Equal(t, stream.Entries[0], c.Entry())
require.Equal(t, true, c.Next())
require.Equal(t, stream.Entries[1], c.Entry())
require.Equal(t, true, c.Next())
require.Equal(t, stream.Entries[2], c.Entry())
require.Equal(t, false, c.Next())
require.Equal(t, nil, c.Error())
require.Equal(t, stream.Entries[2], c.Entry())
require.Equal(t, false, c.Next())
}
assert()
// Close the iterator reset it to the beginning.
require.Equal(t, nil, c.Close())
assert()
}
func Test_EmptyCachedIterator(t *testing.T) {
c := newCachedIterator(iter.NoopIterator, 0)
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Entry{}, c.Entry())
require.Equal(t, false, c.Next())
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Entry{}, c.Entry())
require.Equal(t, nil, c.Close())
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Entry{}, c.Entry())
require.Equal(t, false, c.Next())
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Entry{}, c.Entry())
}
func Test_ErrorCachedIterator(t *testing.T) {
c := newCachedIterator(&errorIter{}, 0)
require.Equal(t, false, c.Next())
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Entry{}, c.Entry())
require.Equal(t, errors.New("error"), c.Error())
require.Equal(t, errors.New("close"), c.Close())
}
Improve metric queries by computing samples at the edges. (#2293) * First pass breaking the code appart. Wondering how we're going to achieve fast mutation of labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. I realize I need hash for deduping lines. going to benchmark somes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tested some hash and decided which one to use. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting working on ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Trying to find a better hash function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More hash testing we have a winner. xxhash it is. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Settle on xxhash Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better params interfacing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add interface for queryparams for things that exist in both type of params. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add storage sample iterator implementations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests and verifying we don't get collions for the hashing method. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing ingesters tests and refactoring utility function/tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing and testing that stats are still well computed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More engine tests finished. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sharding evaluator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more engine tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix error tests in the engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish fixing all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug where extractor was not passed in correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notes about upgrade. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Renamed and fix a bug. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add memchunk tests and starting test for sampleIterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test heap sample iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * working on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing testing all new iterators. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Making sure all store functions are tested. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Benchmark and verify everything is working well. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make the linter happy. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * use xxhash v2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky test because of map. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>
6 years ago
func Test_CachedSampleIterator(t *testing.T) {
series := logproto.Series{
Labels: `{foo="bar"}`,
Samples: []logproto.Sample{
{Timestamp: time.Unix(0, 1).UnixNano(), Hash: 1, Value: 1.},
{Timestamp: time.Unix(0, 2).UnixNano(), Hash: 2, Value: 2.},
{Timestamp: time.Unix(0, 3).UnixNano(), Hash: 3, Value: 3.},
},
}
c := newCachedSampleIterator(iter.NewSeriesIterator(series), 3)
assert := func() {
// we should crash for call of entry without next although that's not expected.
require.Equal(t, series.Labels, c.Labels())
require.Equal(t, series.Samples[0], c.Sample())
require.Equal(t, true, c.Next())
require.Equal(t, series.Samples[0], c.Sample())
require.Equal(t, true, c.Next())
require.Equal(t, series.Samples[1], c.Sample())
require.Equal(t, true, c.Next())
require.Equal(t, series.Samples[2], c.Sample())
require.Equal(t, false, c.Next())
require.Equal(t, nil, c.Error())
require.Equal(t, series.Samples[2], c.Sample())
require.Equal(t, false, c.Next())
}
assert()
// Close the iterator reset it to the beginning.
require.Equal(t, nil, c.Close())
assert()
}
func Test_EmptyCachedSampleIterator(t *testing.T) {
c := newCachedSampleIterator(iter.NoopIterator, 0)
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Sample{}, c.Sample())
require.Equal(t, false, c.Next())
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Sample{}, c.Sample())
require.Equal(t, nil, c.Close())
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Sample{}, c.Sample())
require.Equal(t, false, c.Next())
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Sample{}, c.Sample())
}
func Test_ErrorCachedSampleIterator(t *testing.T) {
c := newCachedSampleIterator(&errorIter{}, 0)
require.Equal(t, false, c.Next())
require.Equal(t, "", c.Labels())
require.Equal(t, logproto.Sample{}, c.Sample())
require.Equal(t, errors.New("error"), c.Error())
require.Equal(t, errors.New("close"), c.Close())
}
type errorIter struct{}
Improve metric queries by computing samples at the edges. (#2293) * First pass breaking the code appart. Wondering how we're going to achieve fast mutation of labels. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Work in progress. I realize I need hash for deduping lines. going to benchmark somes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tested some hash and decided which one to use. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Starting working on ingester. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Trying to find a better hash function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More hash testing we have a winner. xxhash it is. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Settle on xxhash Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better params interfacing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add interface for queryparams for things that exist in both type of params. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add storage sample iterator implementations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests and verifying we don't get collions for the hashing method. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing ingesters tests and refactoring utility function/tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing and testing that stats are still well computed. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing more tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More engine tests finished. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sharding evaluator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more engine tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix error tests in the engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finish fixing all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug where extractor was not passed in correctly. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notes about upgrade. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Renamed and fix a bug. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add memchunk tests and starting test for sampleIterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test heap sample iterator. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * working on test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Finishing testing all new iterators. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Making sure all store functions are tested. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Benchmark and verify everything is working well. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make the linter happy. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * use xxhash v2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix a flaky test because of map. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>
6 years ago
func (errorIter) Next() bool { return false }
func (errorIter) Error() error { return errors.New("error") }
func (errorIter) Labels() string { return "" }
func (errorIter) Entry() logproto.Entry { return logproto.Entry{} }
func (errorIter) Sample() logproto.Sample { return logproto.Sample{} }
func (errorIter) Close() error { return errors.New("close") }