apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	2406ea72bc	Allows by/without to be empty and available for max/min_over_time (#3030 ) * Allows by/without to be empty and available for max/min_over_time operations. This essentially allows to aggregate over all dimensions when using by () while without() is a noop. This also makes it possible for max and min range vector aggregation to use grouping, which is simpler than doing max by (foo) max_over_time(...) Examples: - `max_over_time(...) by ()` gives your the max over time accross of all series. - `min_over_time(...) by (namespace)` gives you the min over time per namespace. - `max_over_time(...) without (namespace)` gives you the max over time removing the namespace dimension. PS: I've also refactored a bit how we optimized grouping to make it more clear. Fixes #2884 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * nit doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Vitaliy Potapov	d05529f0e7	[Docs] Clarify docker-driver configuration options (#3049 ) Clarify docker-driver configuration about `max-file` and `max-size` default values. Fixes #3046	5 years ago
Heds Simons	b195e94f39	fluent-bit: Rename Fluent Bit plugin output name. (#2974 ) This prevents a clash with the Loki plugin now included with the Fluent Bit agent by default. The output plugin is now named `grafana-loki`. Signed-off-by: Heds Simons <heds@whaleway.net>	5 years ago
Jack Baldry	2d8cf154ba	doc: logql formatting fixes (#3039 ) * doc: Remove extraneous backtick Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Fix backtick label quoting example Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	5 years ago
Diana Payton	72b4c0181a	Update _index.md (#3032 )	5 years ago
AdamKorcz	a3ae2f08f9	Added fuzzer (#3031 )	5 years ago
Richard Hartmann	0259f85d3b	Create ADOPTERS.md (#3033 ) Signed-off-by: Richard Hartmann <richih@richih.org>	5 years ago
Reinhard Nägele	1bb4301f2e	Fix multiline docs (#3035 ) Signed-off-by: Reinhard Nägele <unguiculus@gmail.com>	5 years ago
Karsten Jeschkies	d70332c3aa	Correct `multiline` documentation. (#3029 )	5 years ago
Ilias Trichopoulos	bc29c6109e	Fix docs header inconsistency (#3027 )	5 years ago
Karsten Jeschkies	afc1e53df9	Collapse multiline logs based on a start line. (#3024 ) * Collapse multiline logs based on a start line. Summary: This is a very simple approach based on #1380 to provide multiline or block log entries in promtail. A `multiline` stage is added to pipelines. This stages matches a start line. Once a start line is matched all following lines are appended to an entry and not passed on to downstream stages. Once a new start line is matched the former block of multilines is sent. If now new line arrives withing `max_wait_time` the block is flushed to the next stage and a new block is started. * Test multiline stage process. * Format code. * Flush multiline block after `max_lines`. * Capture internal state of the stage. * Process different multiline streams in parallel. * Start documenting multiline stage. * Give an example configuration for `multiline` stage. * Make linter happy. * Pass through entries until first start line. * Update pkg/logentry/stages/multiline.go	5 years ago
Cyril Tovena	c98dfa0da1	Fixes a swallowed context err in the batch storage. (#3025 ) Fixes #2999 There's still a potential cases where, the context is done (and so the Err is not nil) and we're finished with iterating over the list of batches. I didn't introduce the error here because I think I would prefer a success in this case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	c4faa579e3	Asynchronous Promtail stages (#2996 ) * Introducing go pipelines to promtail. Based off @jeschkies idea. WIP Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More tests and code cleanup. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip breaking things. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests, adding Stop to the interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar fixes for tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Close correctly client before reading. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use defer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some comments. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes lint issues Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Attempt to fix journald without linux. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journald json test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing stop in the filetarget. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/logentry/stages/match_test.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * First set of feeback review fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use newEntry as suggested by Karsten. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Owen Diehl	05f3d6e0da	Updates chunkenc doc for V3 (#2956 )	5 years ago
Jack Baldry	794c324e64	doc: Cleanup formatting (#3017 ) * doc: Add missing backticks Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Use absolute path to positions.yaml file Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Remove .. from URL paths I'm not sure what the `..` means exactly. Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	5 years ago
Cyril Tovena	315972409b	Add more information about why a tailer would stop. (#3015 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	5c9a913caa	loki wal libsonnet (#3014 )	5 years ago
Owen Diehl	854930e661	headblock checkpointing up to v3 (#3013 )	5 years ago
Cyril Tovena	dcbfecf9e5	Fixes the ruler storage with the boltdb store. (#3008 ) This ensure we have read-only local index, and also query chunk store from ingesters. Found this bug while deploying a new cluster using the ruler and boltdb store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	18cf9d2305	Fixes promtail mixin dashboard. (#3010 ) `e14463b` removed a metric that was used previously in a dashboard could not find a better way yet to use the new metric so instead I added back the recording rules. Also fixes a missing template expansion from `899e8cc`. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	a8af1c0c84	adds the ability to read chunkFormatV3 while writing v2 (#2984 )	5 years ago
Kaviraj	410b9d9adc	fix(docker-driver): Propagate promtail's `client.Stop` properly (#2898 ) * impr(Makefile): add support to take docker-driver image via ENV * impr(promtail/client): Add cancelling mechanism for batchSend stores `context` and its `cancel` in client and uses it to make upstream calls that need `ctx`. Also properly cancel's all the upstream calls that using `ctx` when client's Stop is called. * Add `StopNow()` method to stop the client without retries. Use `StopNow()` from the docker-driver * Add test for StopNow * reverting back the client.Timeout	5 years ago
Jón Tómas Grétarsson	f6fd6ae7da	Add a critical missing field to the extended config s3 example (#2991 )	5 years ago
keyolk	7b48422b6f	Fix typo (#3002 ) Signed-off-by: keyolk <chanhun.jeong@navercorp.com>	5 years ago
Nathan Smith	876be88381	Helm Chart: Fix schema of logstash default values (#3004 ) * Fix schema of logstash default values * Bump helm chart version	5 years ago
Hui Kang	aca259f9de	Fix the docker image version in compose deployment (#3006 ) - grafana v7.3.4 - loki v2.0.0	5 years ago
Jack Baldry	46637db9cc	doc: Fix query-frontend typo (#3009 ) Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	5 years ago
Owen Diehl	8c1fe88409	Introduces per stream chunks mutex (#3000 ) * marshalable chunks * wal record types custom serialization * proto types for wal checkpoints * byteswith output unaffected by buffer * wal & record pool ifcs * wal record can hold entries from multiple series * entry pool * ingester uses noopWal * removes duplicate argument passing in ingester code. adds ingester config validation & derives chunk encoding. * segment writing * [WIP] wal recovery from segments * replay uses sync.Maps & preserves WAL fingerprints * in memory wal recovery * wal segment recovery * ingester metrics struct * wal replay locks streamsMtx in instances, adds checkpoint codec * ingester metrics * checkpointer * WAL checkpoint writer * checkpointwriter can write multiple checkpoints * reorgs checkpointing * wires up checkpointwriter to wal * ingester SeriesIter impl * wires up ingesterRecoverer to consume checkpoints * generic recovery fn * generic recovery fn * recover from both wal types * cleans up old tmp checkpoints & allows aborting in flight checkpoints * wires up wal checkpointing * more granular wal logging * fixes off by 1 wal truncation & removes double logging * adds userID to wal records correctly * wire chunk encoding tests * more granular wal metrics * checkpoint encoding test * ignores debug bins * segment replay ignores out of orders * fixes bug between WAL reading []byte validity and proto unmarshalling refs * conf validations, removes comments * flush on shutdown config * POST /ingester/shutdown * renames flush on shutdown * wal & checkpoint use same segment size * writes entries to wal regardless of tailers * makes wal checkpoing duration default to 5m * recovery metrics * encodes headchunks separately for wal purposes * merge upstream * linting * adds chunkMtx to ingester streams * all flush locking locks streams -> chunks in order to prevent deadlocks * wal integration tests * addresses pr feedback uses entry pool in stream push/tailer removes unnecessary pool interaction checkpointbytes comment fillchunk helper, record resetting in tests via pool redundant comment defers wg done in recovery s/num/count/ checkpoint wal uses a logger encodeWithTypeHeader now creates its own []byte removes pool from decodeEntries wal stop can error * cleans up merge * lint * inline functions to simplify unlocking	5 years ago
Owen Diehl	4d9865acd4	Adds WAL support (experimental) (#2981 ) * marshalable chunks * wal record types custom serialization * proto types for wal checkpoints * byteswith output unaffected by buffer * wal & record pool ifcs * wal record can hold entries from multiple series * entry pool * ingester uses noopWal * removes duplicate argument passing in ingester code. adds ingester config validation & derives chunk encoding. * segment writing * [WIP] wal recovery from segments * replay uses sync.Maps & preserves WAL fingerprints * in memory wal recovery * wal segment recovery * ingester metrics struct * wal replay locks streamsMtx in instances, adds checkpoint codec * ingester metrics * checkpointer * WAL checkpoint writer * checkpointwriter can write multiple checkpoints * reorgs checkpointing * wires up checkpointwriter to wal * ingester SeriesIter impl * wires up ingesterRecoverer to consume checkpoints * generic recovery fn * generic recovery fn * recover from both wal types * cleans up old tmp checkpoints & allows aborting in flight checkpoints * wires up wal checkpointing * more granular wal logging * fixes off by 1 wal truncation & removes double logging * adds userID to wal records correctly * wire chunk encoding tests * more granular wal metrics * checkpoint encoding test * ignores debug bins * segment replay ignores out of orders * fixes bug between WAL reading []byte validity and proto unmarshalling refs * conf validations, removes comments * flush on shutdown config * POST /ingester/shutdown * renames flush on shutdown * wal & checkpoint use same segment size * writes entries to wal regardless of tailers * makes wal checkpoing duration default to 5m * recovery metrics * encodes headchunks separately for wal purposes * merge upstream * linting * addresses pr feedback uses entry pool in stream push/tailer removes unnecessary pool interaction checkpointbytes comment fillchunk helper, record resetting in tests via pool redundant comment defers wg done in recovery s/num/count/ checkpoint wal uses a logger encodeWithTypeHeader now creates its own []byte removes pool from decodeEntries wal stop can error * prevent shared access bug with tailers and entry pool * removes stream push entry pool optimization	5 years ago
Sandeep Sukhani	ae9c4b82ec	fix table deletion in table client for boltdb-shipper (#2960 )	5 years ago
Ed Welch	9ba5753e25	Loki: handle faults when opening boltdb files (#2988 ) * we also need to catch faults not just panics when opening boltdb files * fix lint * i'll get it one of these times * fix typo	5 years ago
Cyril Tovena	ec725db9d0	Add ProcessString to Pipeline. (#2972 ) * Add ProcessString to Pipeline. Most of the time, you have a buffer that you want to test again a log pipeline, when it passed you usually copy via string(buff). In some cases like headchunk and tailer you already have an allocated immutable string, since pipeline never mutate the line passed as parameter, we can create ProcessString pipeline method that will avoid re-allocating the line. benchmp: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta BenchmarkHeadBlockIterator/Size_100000-16 20264242 16112591 -20.49% BenchmarkHeadBlockIterator/Size_50000-16 10186969 7905259 -22.40% BenchmarkHeadBlockIterator/Size_15000-16 3229052 2202770 -31.78% BenchmarkHeadBlockIterator/Size_10000-16 1916537 1392355 -27.35% BenchmarkHeadBlockSampleIterator/Size_100000-16 18364773 16106425 -12.30% BenchmarkHeadBlockSampleIterator/Size_50000-16 8988422 7730226 -14.00% BenchmarkHeadBlockSampleIterator/Size_15000-16 2788746 2306161 -17.30% BenchmarkHeadBlockSampleIterator/Size_10000-16 1773766 1488861 -16.06% benchmark old allocs new allocs delta BenchmarkHeadBlockIterator/Size_100000-16 200039 39 -99.98% BenchmarkHeadBlockIterator/Size_50000-16 100036 36 -99.96% BenchmarkHeadBlockIterator/Size_15000-16 30031 31 -99.90% BenchmarkHeadBlockIterator/Size_10000-16 20029 29 -99.86% BenchmarkHeadBlockSampleIterator/Size_100000-16 100040 40 -99.96% BenchmarkHeadBlockSampleIterator/Size_50000-16 50036 36 -99.93% BenchmarkHeadBlockSampleIterator/Size_15000-16 15031 31 -99.79% BenchmarkHeadBlockSampleIterator/Size_10000-16 10029 29 -99.71% benchmark old bytes new bytes delta BenchmarkHeadBlockIterator/Size_100000-16 27604042 21203941 -23.19% BenchmarkHeadBlockIterator/Size_50000-16 13860654 10660652 -23.09% BenchmarkHeadBlockIterator/Size_15000-16 4231360 3271363 -22.69% BenchmarkHeadBlockIterator/Size_10000-16 2633436 1993420 -24.30% BenchmarkHeadBlockSampleIterator/Size_100000-16 17799137 14598973 -17.98% BenchmarkHeadBlockSampleIterator/Size_50000-16 7433260 5833137 -21.53% BenchmarkHeadBlockSampleIterator/Size_15000-16 2258099 1778096 -21.26% BenchmarkHeadBlockSampleIterator/Size_10000-16 1393600 1073605 -22.96% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add some precision about why Sum64 is not a concern. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/chunkenc/memchunk.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Cyril Tovena	06a89ea0ee	Fixes vector grouping injection. (#2975 ) * Fixes vector grouping injection. During an improvement I introduced a bug where we would pre-aggregate some range vector aggregations, although the result is not the same. We can only inject vector grouping in range vector if the operation is associative through grouping. This is not the case for max/min/stddev/stddvar/quantile/avg_over_time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	221fd382a9	Move CI to helm3. (#2986 ) * Move CI to helm3. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Rollback to v3.2.3 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Ed Welch	0b84ef8868	Promtail: Update debian image and use a newer libsystemd (#2957 ) * use a newer libsystemd which understands compression needed in some of the newer OS distros * add comments	5 years ago
Anton Nehaev	3d27b1f468	docs: added loki4j to the list of unofficial clients (#2869 )	5 years ago
Ed Welch	6d442673b7	adding a function to safely open boltdb files and recover from any panics thrown by boltdb when opening a file. (#2983 )	5 years ago
LTek-online	56830bc78b	Changed incorrect headers of sections in the promtail config docu (#2636 ) The configuration documentation of promtails contains some errors where the headers of the sections don't match the actual configs. Notably configuration keys like 'server' and 'clients' have been mislabeled as 'server_config' and 'client_config'. This merge request is aimed to correct that so that it matches the configuration as in the written config files to avoid confusion.	5 years ago
Cyril Tovena	9807f7c43e	Avoid parsing labels when tailer is sending from a stream. (#2973 ) * Avoid parsing labels when tailer is sending from a stream. This also include some code cleanup. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * I got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Björn Rabenstein	e14463b2a3	Improve promtail alerts to retain the namespace label (#2976 ) The `PromtailFileMissing` should actually work in this simplified form. Please let me know if I'm missing something.	5 years ago
Sandeep Sukhani	14a5e22c31	add missing ingester query routes in loki reads and operational dashboard (#2961 )	5 years ago
Christian Zunker	57f8159c93	[Helm Chart Loki] Add needed k8s objects for alerting config: (#2965 ) - ConfigMap containing the rules - VolumeMount for rules - scratch mount for rule parsing Fixes #2964 Signed-off-by: Christian Zunker <christian.zunker@codecentric.cloud>	5 years ago
Christian Zunker	b5817127a5	Remove `helm init` (#2967 ) As helm v2 is EOF, I removed the commands only present in helm v2. Fixes #2966 Signed-off-by: Christian Zunker <christian.zunker@codecentric.cloud>	5 years ago
Yuichi Sawada	27dddef475	Fix logcli logql document url (#2968 )	5 years ago
Cyril Tovena	8ea5fd173f	Implement io.WriteTo by chunks. (#2962 ) This allows to totally defer the buffer creation to the user. `Marshal(w io.Writer) error` Now don't even need to create a buffer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Christian Simon	a7d4774859	Add community forum to README.md (#2969 )	5 years ago
Cyril Tovena	a62f81f1cf	Improve tailer matcher function. (#2959 ) Don't create a map on every new push of a stream or appearance of a new stream. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	4da505b413	Fixes the Stringer of the byte label operator. (#2946 ) This would cause the operator to become a numeric one once stringified. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Karsten Jeschkies	29be929a8c	Test label filter for bytes. (#2941 ) * Test label filter for bytes. Summary: - Adds a new test to `ast_test` for filtering by bytes.\| - Test another case than `B`. * Test all combinations of byte sizes.	5 years ago
Owen Diehl	1b6465d940	bumps rules-action ref to logqlv2+ version (#2951 )	5 years ago

1 2 3 4 5 ...

2011 Commits (d357ad9e58a035bcdb5e0bf0a4ebebaba4de1797) All Branches Search

2011 Commits (d357ad9e58a035bcdb5e0bf0a4ebebaba4de1797)

All Branches