45 Commits (39b57ec4eac3cbdc718aacae32ab8ff4e989709b)

Author	SHA1	Message	Date
Ravishankar	41fafd8793	feat: Apply patterns line length limit to json message key (#14296)	1 year ago
Trevor Whitney	c001a1d93a	fix: allow any level for aggregated metrics (#14255)	1 year ago
Cyril Tovena	ae955ed30d	fix(pattern): Fixes latency metric namespace for tee to pattern (#14241)	1 year ago
Sven Grossmann	63e84b476a	fix(aggregated-metrics): correctly create logfmt string (#14124)	1 year ago
Sven Grossmann	0780456662	feat(max-allowed-line-length): add config to set `max-allowed-line-length` in pattern ingester (#14070) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	1 year ago
Paul Rogers	8fb7b488bd	chore(deps): update dskit 20240819 (#13924)	1 year ago
Trevor Whitney	d2474fa989	fix: split the error and agg metric cases for clarity (#13913)	1 year ago
Trevor Whitney	913e9f9347	feat: aggregate byte and count metrics (#13731)	1 year ago
Travis Patterson	feee240f46	chore: remove unused pattern tokenization form Loki (#13783)	1 year ago
Trevor Whitney	c6ab6b31e8	chore: remove initial metric aggregation experiment (#13729)	1 year ago
Ned Andreev	7683a79191	fix: Include whitespaces in extracted tokens (#13738) ... Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	1 year ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
George Robinson	757bde0cb5	chore: replace pkg/errors with errors (#13582)	1 year ago
benclive	e23598d710	fix: Dedup pattern tokens on output (#13534)	1 year ago
benclive	5fa9c4bd56	fix: Retain original values in logfmt pattern tokenizer (#13535)	1 year ago
Ned Andreev	ac284ca00e	feat: improve placeholder replacement of byte sizes (#13508)	1 year ago
Trevor Whitney	2c053ee00c	feat: downsample aggregated metrics (#13449)	1 year ago
Cyril Tovena	e08b4a7f88	feat: Pattern ingesters add a limiter for high eviction rate (#13464)	1 year ago
Cyril Tovena	7c86e651ac	fix: Fixes pattern pruning stability (#13429)	2 years ago
Christian Haudum	5ebd24dfb2	chore: Convert iterators in pattern module to `v2.Iterator` interface (#13407) ... This is first follow up to #13273 It changes the iterator from the pattern module to use the `v2.CloseIterator` to avoid re-implementing the iterator logic. With this change, the function `Error()` on the pattern iterator is renamed to `Err()` to conform with the `v2.Iterator` interface. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Cyril Tovena	bc01e6fd64	feat: Drain uses different tokenizer based on log format (#13384)	2 years ago
benclive	e612dd3dfe	feat: exclude in pattern tokens (#13397)	2 years ago
benclive	8c8454b9db	feat: exclude and from creating new tokens in patterns (#13395)	2 years ago
benclive	a1efafdad2	perf: Limit tokens per pattern to 128 (#13376)	2 years ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
benclive	bf1d6e36f7	perf: Gather aggregate per-line and per-tenant metrics for Drain patterns (#13368)	2 years ago
benclive	5ef18cff92	perf: Prune unused pattern branches (#13329)	2 years ago
Cyril Tovena	41fbacdce7	feat(wal): Add sizing information to writer and reader. (#13267)	2 years ago
Trevor Whitney	467eb1bb1b	feat: collect and serve pre-aggregated bytes and counts (#13020) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	2 years ago
benclive	6a0fdd0880	feat: Add new Drain tokenizer that splits on most punctuation (#13143)	2 years ago
benclive	30df31e28b	feat: Tune Patterns query drain instance (#13137)	2 years ago
benclive	f6f8babf83	feat: Add utf8 support to Pattern Lexer to support utf8 chars (#13085)	2 years ago
benclive	2923a7d958	fix: Update expected patterns when pruning (#13079)	2 years ago
benclive	797bb64173	feat: Add tokenizer interface for Drain Training (#13069)	2 years ago
benclive	d0a285926b	feat: Increase drain max depth from 8 -> 30 (#13063)	2 years ago
benclive	2171f6409f	fix: Fix panic on requesting out-of-order Pattern samples (#13010)	2 years ago
benclive	bc53b33721	feat: Add metrics for number of patterns detected & evicted (#12918)	2 years ago
benclive	7630f33ecf	test: Added more test data for pattern detection tests (#12920)	2 years ago
Paul Rogers	d16a3bf139	test: First pass at unregistering metrics so that we can run multiple tests (#12927)	2 years ago
benclive	fd2301fd62	fix: Ensure Drain patterns are valid for LogQL pattern match filter (#12815)	2 years ago
Ned Andreev	40479029d7	feat: Generic logline placeholder replacement and tokenization (#12799) ... This code allows us to preprocess generic logs and replace highly variable dynamic data (timestamps, IPs, numbers, UUIDs, hex values, bytesizes and durations) with static placeholders for easier pattern extraction and more efficient and user-friendly matching by the Drain algorithm. Additionally, there is logic that splits generic log lines into discrete tokens that can be used with Drain for better results than just naively splitting the logs on every whitespace. The tokenization here handles quote counting and emits quoted strings as a part of the same token. On the other side, it also handles likely JSON logs without any white spaces in them better, by trying to split `{"key":value}` pairs (without actually parsing the JSON). All of this is done without using regular expressions and without actually parsing the log lines in any specific format. That's why it works very efficiently in terms of CPU usage and allocations, and should handle all log formats and unformatted logs equally well.	2 years ago
benclive	7b8533e435	feat: Add step param to Patterns Query API (#12703)	2 years ago
benclive	5ae33d19ff	test: Add new tests for Drain pattern matching (#12655)	2 years ago
Cyril Tovena	491d2513e7	feat: add log patterns ingestion and query API. (#12403) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Anton Kolesnikov <anton.e.kolesnikov@gmail.com> Co-authored-by: Sven Grossmann <svennergr@gmail.com> Co-authored-by: Edward Welch <edward.welch@grafana.com>	2 years ago