51 Commits (9fb2f385529c2444be98161743aa0cd8ad420bf2)

Author	SHA1	Message	Date
Cyril Tovena	90c5dbf0d3	chore: Revert syntax module (#14831)	7 months ago
Cyril Tovena	28c9645d10	chore: Introduce logql/syntax go module (#14790)	7 months ago
Trevor Whitney	80aec25482	feat: Improve pattern ingester tracing (#14707)	7 months ago
Trevor Whitney	c1fde26730	feat: move metric aggregation to a per-tenant config (#14709)	7 months ago
Salva Corts	70508975fd	feat: Configurable list of json fields to mine patterns (#14528)	7 months ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	8 months ago
Ravishankar	41fafd8793	feat: Apply patterns line length limit to json message key (#14296)	8 months ago
Trevor Whitney	c001a1d93a	fix: allow any level for aggregated metrics (#14255)	8 months ago
Cyril Tovena	ae955ed30d	fix(pattern): Fixes latency metric namespace for tee to pattern (#14241)	8 months ago
Sven Grossmann	63e84b476a	fix(aggregated-metrics): correctly create logfmt string (#14124)	8 months ago
Sven Grossmann	0780456662	feat(max-allowed-line-length): add config to set `max-allowed-line-length` in pattern ingester (#14070) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Paul Rogers	8fb7b488bd	chore(deps): update dskit 20240819 (#13924)	9 months ago
Trevor Whitney	d2474fa989	fix: split the error and agg metric cases for clarity (#13913)	9 months ago
Trevor Whitney	913e9f9347	feat: aggregate byte and count metrics (#13731)	9 months ago
Travis Patterson	feee240f46	chore: remove unused pattern tokenization form Loki (#13783)	9 months ago
Trevor Whitney	c6ab6b31e8	chore: remove initial metric aggregation experiment (#13729)	9 months ago
Ned Andreev	7683a79191	fix: Include whitespaces in extracted tokens (#13738) ... Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	10 months ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	10 months ago
George Robinson	757bde0cb5	chore: replace pkg/errors with errors (#13582)	10 months ago
benclive	e23598d710	fix: Dedup pattern tokens on output (#13534)	10 months ago
benclive	5fa9c4bd56	fix: Retain original values in logfmt pattern tokenizer (#13535)	10 months ago
Ned Andreev	ac284ca00e	feat: improve placeholder replacement of byte sizes (#13508)	10 months ago
Trevor Whitney	2c053ee00c	feat: downsample aggregated metrics (#13449)	10 months ago
Cyril Tovena	e08b4a7f88	feat: Pattern ingesters add a limiter for high eviction rate (#13464)	11 months ago
Cyril Tovena	7c86e651ac	fix: Fixes pattern pruning stability (#13429)	11 months ago
Christian Haudum	5ebd24dfb2	chore: Convert iterators in pattern module to `v2.Iterator` interface (#13407) ... This is first follow up to #13273 It changes the iterator from the pattern module to use the `v2.CloseIterator` to avoid re-implementing the iterator logic. With this change, the function `Error()` on the pattern iterator is renamed to `Err()` to conform with the `v2.Iterator` interface. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	bc01e6fd64	feat: Drain uses different tokenizer based on log format (#13384)	11 months ago
benclive	e612dd3dfe	feat: exclude in pattern tokens (#13397)	11 months ago
benclive	8c8454b9db	feat: exclude and from creating new tokens in patterns (#13395)	11 months ago
benclive	a1efafdad2	perf: Limit tokens per pattern to 128 (#13376)	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
benclive	bf1d6e36f7	perf: Gather aggregate per-line and per-tenant metrics for Drain patterns (#13368)	11 months ago
benclive	5ef18cff92	perf: Prune unused pattern branches (#13329)	11 months ago
Cyril Tovena	41fbacdce7	feat(wal): Add sizing information to writer and reader. (#13267)	11 months ago
Trevor Whitney	467eb1bb1b	feat: collect and serve pre-aggregated bytes and counts (#13020) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	11 months ago
benclive	6a0fdd0880	feat: Add new Drain tokenizer that splits on most punctuation (#13143)	12 months ago
benclive	30df31e28b	feat: Tune Patterns query drain instance (#13137)	12 months ago
benclive	f6f8babf83	feat: Add utf8 support to Pattern Lexer to support utf8 chars (#13085)	12 months ago
benclive	2923a7d958	fix: Update expected patterns when pruning (#13079)	12 months ago
benclive	797bb64173	feat: Add tokenizer interface for Drain Training (#13069)	12 months ago
benclive	d0a285926b	feat: Increase drain max depth from 8 -> 30 (#13063)	12 months ago
benclive	2171f6409f	fix: Fix panic on requesting out-of-order Pattern samples (#13010)	12 months ago
benclive	bc53b33721	feat: Add metrics for number of patterns detected & evicted (#12918)	1 year ago
benclive	7630f33ecf	test: Added more test data for pattern detection tests (#12920)	1 year ago
Paul Rogers	d16a3bf139	test: First pass at unregistering metrics so that we can run multiple tests (#12927)	1 year ago
benclive	fd2301fd62	fix: Ensure Drain patterns are valid for LogQL pattern match filter (#12815)	1 year ago
Ned Andreev	40479029d7	feat: Generic logline placeholder replacement and tokenization (#12799) ... This code allows us to preprocess generic logs and replace highly variable dynamic data (timestamps, IPs, numbers, UUIDs, hex values, bytesizes and durations) with static placeholders for easier pattern extraction and more efficient and user-friendly matching by the Drain algorithm. Additionally, there is logic that splits generic log lines into discrete tokens that can be used with Drain for better results than just naively splitting the logs on every whitespace. The tokenization here handles quote counting and emits quoted strings as a part of the same token. On the other side, it also handles likely JSON logs without any white spaces in them better, by trying to split `{"key":value}` pairs (without actually parsing the JSON). All of this is done without using regular expressions and without actually parsing the log lines in any specific format. That's why it works very efficiently in terms of CPU usage and allocations, and should handle all log formats and unformatted logs equally well.	1 year ago
benclive	7b8533e435	feat: Add step param to Patterns Query API (#12703)	1 year ago
benclive	5ae33d19ff	test: Add new tests for Drain pattern matching (#12655)	1 year ago