apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Heds Simons	d08871d4c7	fluent-bit: Rename Fluent Bit plugin output name. (#2974 ) This prevents a clash with the Loki plugin now included with the Fluent Bit agent by default. The output plugin is now named `grafana-loki`. Signed-off-by: Heds Simons <heds@whaleway.net> (cherry picked from commit `b195e94f39`)	4 years ago
Owen Diehl	b236f2ee27	adds the ability to read chunkFormatV3 while writing v2 (#2984 ) (cherry picked from commit `a8af1c0c84`)	4 years ago
Edward Welch	a54005f24e	update changelog (cherry picked from commit `d5141c05a7`)	4 years ago
Edward Welch	14385b1d54	Merge branch 'release-2.0.1' # Conflicts: # pkg/chunkenc/memchunk_test.go	4 years ago
Ed Welch	31f3c205e6	Helm: Disable CI (#3072 ) * Removing the helm chart ci build steps as the helm charts are in a new repo now. * remove helm targets from the makefile	4 years ago
Michel Hollands	48b65834a0	Make server publicly available (#3087 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Michel Hollands	e9001d8731	Make the moduleManager field public (#3077 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Torsten Walter	6c8ad5258b	Deprecate Charts as they have been moved (#2720 ) * Deprecate Charts as they have been moved Signed-off-by: Torsten Walter <torsten.walter@syncier.com> * update docs and links to new chart location Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Edward Welch	d5141c05a7	update changelog	5 years ago
Owen Diehl	35a627df49	adds the ability to read chunkFormatV3 while writing v2 (#2984 ) (cherry picked from commit `a8af1c0c84`)	5 years ago
Cyril Tovena	fbf17904f7	Fixes iota unexpected behaviour with bytes for chunk encoding. (#2945 ) * Fixes iota unexpected behaviour with bytes for chunk encoding. This was introducing incompatibilities since iota with bytes starts at 2. Seems like a weird conversion between iota int and byte. This can currenlty corrupt chunks, and previous chunks are not decodable. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * With iota correclty this time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> (cherry picked from commit `fa7ea9cc2d`)	5 years ago
Edward Welch	8cd276d4ac	v3 of chunk encoding serializes all block data (#2934 ) (cherry picked from commit `663c872a4c`)	5 years ago
Cyril Tovena	3f99a07323	Remove regexes in the operational dashboard. (#3062 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	6d85c7c212	Add absent_over_time (#3053 ) * Add absent_over_time This adds absent_over_time which is super useful when alerting, it can be combine with any logql selector including parsers and filter to returns a timeseries with 1 when logs go missing. Fixes #2893 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds comment. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	3f0800dc44	Add function label_replace. (#3047 ) You might be wondering why do we need label_replace when we already have label_format. Well for two reasons: 1. It's nice to have something similar to Prometheus, so users building metric queries will feel right at home. 2. Interestingly, they are quite different in how they execute. label_format executes for every log line, however label_replace execute for every sample. Since you aggregate log lines into sample then label_replace is way more efficient at scale. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	4e7a123c77	Support rate for unwrap expressions. (#3048 ) This allows to bytes throughput if there were printed in a log line. Examples : ```logql sum by (device) (rate({app="network-dev"} \|= "192.16.8.1" \| json \| unwrap recv_bytes [1m])) ``` Closes #3044 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	2406ea72bc	Allows by/without to be empty and available for max/min_over_time (#3030 ) * Allows by/without to be empty and available for max/min_over_time operations. This essentially allows to aggregate over all dimensions when using by () while without() is a noop. This also makes it possible for max and min range vector aggregation to use grouping, which is simpler than doing max by (foo) max_over_time(...) Examples: - `max_over_time(...) by ()` gives your the max over time accross of all series. - `min_over_time(...) by (namespace)` gives you the min over time per namespace. - `max_over_time(...) without (namespace)` gives you the max over time removing the namespace dimension. PS: I've also refactored a bit how we optimized grouping to make it more clear. Fixes #2884 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * nit doc. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Vitaliy Potapov	d05529f0e7	[Docs] Clarify docker-driver configuration options (#3049 ) Clarify docker-driver configuration about `max-file` and `max-size` default values. Fixes #3046	5 years ago
Heds Simons	b195e94f39	fluent-bit: Rename Fluent Bit plugin output name. (#2974 ) This prevents a clash with the Loki plugin now included with the Fluent Bit agent by default. The output plugin is now named `grafana-loki`. Signed-off-by: Heds Simons <heds@whaleway.net>	5 years ago
Jack Baldry	2d8cf154ba	doc: logql formatting fixes (#3039 ) * doc: Remove extraneous backtick Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Fix backtick label quoting example Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	5 years ago
Diana Payton	72b4c0181a	Update _index.md (#3032 )	5 years ago
AdamKorcz	a3ae2f08f9	Added fuzzer (#3031 )	5 years ago
Richard Hartmann	0259f85d3b	Create ADOPTERS.md (#3033 ) Signed-off-by: Richard Hartmann <richih@richih.org>	5 years ago
Reinhard Nägele	1bb4301f2e	Fix multiline docs (#3035 ) Signed-off-by: Reinhard Nägele <unguiculus@gmail.com>	5 years ago
Karsten Jeschkies	d70332c3aa	Correct `multiline` documentation. (#3029 )	5 years ago
Ilias Trichopoulos	bc29c6109e	Fix docs header inconsistency (#3027 )	5 years ago
Karsten Jeschkies	afc1e53df9	Collapse multiline logs based on a start line. (#3024 ) * Collapse multiline logs based on a start line. Summary: This is a very simple approach based on #1380 to provide multiline or block log entries in promtail. A `multiline` stage is added to pipelines. This stages matches a start line. Once a start line is matched all following lines are appended to an entry and not passed on to downstream stages. Once a new start line is matched the former block of multilines is sent. If now new line arrives withing `max_wait_time` the block is flushed to the next stage and a new block is started. * Test multiline stage process. * Format code. * Flush multiline block after `max_lines`. * Capture internal state of the stage. * Process different multiline streams in parallel. * Start documenting multiline stage. * Give an example configuration for `multiline` stage. * Make linter happy. * Pass through entries until first start line. * Update pkg/logentry/stages/multiline.go	5 years ago
Cyril Tovena	c98dfa0da1	Fixes a swallowed context err in the batch storage. (#3025 ) Fixes #2999 There's still a potential cases where, the context is done (and so the Err is not nil) and we're finished with iterating over the list of batches. I didn't introduce the error here because I think I would prefer a success in this case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	c4faa579e3	Asynchronous Promtail stages (#2996 ) * Introducing go pipelines to promtail. Based off @jeschkies idea. WIP Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More tests and code cleanup. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip breaking things. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests, adding Stop to the interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar fixes for tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Close correctly client before reading. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use defer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some comments. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes lint issues Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Attempt to fix journald without linux. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journald json test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing stop in the filetarget. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/logentry/stages/match_test.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * First set of feeback review fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use newEntry as suggested by Karsten. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Owen Diehl	05f3d6e0da	Updates chunkenc doc for V3 (#2956 )	5 years ago
Jack Baldry	794c324e64	doc: Cleanup formatting (#3017 ) * doc: Add missing backticks Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Use absolute path to positions.yaml file Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Remove .. from URL paths I'm not sure what the `..` means exactly. Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	5 years ago
Cyril Tovena	315972409b	Add more information about why a tailer would stop. (#3015 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	5c9a913caa	loki wal libsonnet (#3014 )	5 years ago
Owen Diehl	854930e661	headblock checkpointing up to v3 (#3013 )	5 years ago
Cyril Tovena	dcbfecf9e5	Fixes the ruler storage with the boltdb store. (#3008 ) This ensure we have read-only local index, and also query chunk store from ingesters. Found this bug while deploying a new cluster using the ruler and boltdb store. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	18cf9d2305	Fixes promtail mixin dashboard. (#3010 ) `e14463b` removed a metric that was used previously in a dashboard could not find a better way yet to use the new metric so instead I added back the recording rules. Also fixes a missing template expansion from `899e8cc`. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	a8af1c0c84	adds the ability to read chunkFormatV3 while writing v2 (#2984 )	5 years ago
Kaviraj	410b9d9adc	fix(docker-driver): Propagate promtail's `client.Stop` properly (#2898 ) * impr(Makefile): add support to take docker-driver image via ENV * impr(promtail/client): Add cancelling mechanism for batchSend stores `context` and its `cancel` in client and uses it to make upstream calls that need `ctx`. Also properly cancel's all the upstream calls that using `ctx` when client's Stop is called. * Add `StopNow()` method to stop the client without retries. Use `StopNow()` from the docker-driver * Add test for StopNow * reverting back the client.Timeout	5 years ago
Jón Tómas Grétarsson	f6fd6ae7da	Add a critical missing field to the extended config s3 example (#2991 )	5 years ago
keyolk	7b48422b6f	Fix typo (#3002 ) Signed-off-by: keyolk <chanhun.jeong@navercorp.com>	5 years ago
Nathan Smith	876be88381	Helm Chart: Fix schema of logstash default values (#3004 ) * Fix schema of logstash default values * Bump helm chart version	5 years ago
Hui Kang	aca259f9de	Fix the docker image version in compose deployment (#3006 ) - grafana v7.3.4 - loki v2.0.0	5 years ago
Jack Baldry	46637db9cc	doc: Fix query-frontend typo (#3009 ) Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	5 years ago
Owen Diehl	8c1fe88409	Introduces per stream chunks mutex (#3000 ) * marshalable chunks * wal record types custom serialization * proto types for wal checkpoints * byteswith output unaffected by buffer * wal & record pool ifcs * wal record can hold entries from multiple series * entry pool * ingester uses noopWal * removes duplicate argument passing in ingester code. adds ingester config validation & derives chunk encoding. * segment writing * [WIP] wal recovery from segments * replay uses sync.Maps & preserves WAL fingerprints * in memory wal recovery * wal segment recovery * ingester metrics struct * wal replay locks streamsMtx in instances, adds checkpoint codec * ingester metrics * checkpointer * WAL checkpoint writer * checkpointwriter can write multiple checkpoints * reorgs checkpointing * wires up checkpointwriter to wal * ingester SeriesIter impl * wires up ingesterRecoverer to consume checkpoints * generic recovery fn * generic recovery fn * recover from both wal types * cleans up old tmp checkpoints & allows aborting in flight checkpoints * wires up wal checkpointing * more granular wal logging * fixes off by 1 wal truncation & removes double logging * adds userID to wal records correctly * wire chunk encoding tests * more granular wal metrics * checkpoint encoding test * ignores debug bins * segment replay ignores out of orders * fixes bug between WAL reading []byte validity and proto unmarshalling refs * conf validations, removes comments * flush on shutdown config * POST /ingester/shutdown * renames flush on shutdown * wal & checkpoint use same segment size * writes entries to wal regardless of tailers * makes wal checkpoing duration default to 5m * recovery metrics * encodes headchunks separately for wal purposes * merge upstream * linting * adds chunkMtx to ingester streams * all flush locking locks streams -> chunks in order to prevent deadlocks * wal integration tests * addresses pr feedback uses entry pool in stream push/tailer removes unnecessary pool interaction checkpointbytes comment fillchunk helper, record resetting in tests via pool redundant comment defers wg done in recovery s/num/count/ checkpoint wal uses a logger encodeWithTypeHeader now creates its own []byte removes pool from decodeEntries wal stop can error * cleans up merge * lint * inline functions to simplify unlocking	5 years ago
Owen Diehl	4d9865acd4	Adds WAL support (experimental) (#2981 ) * marshalable chunks * wal record types custom serialization * proto types for wal checkpoints * byteswith output unaffected by buffer * wal & record pool ifcs * wal record can hold entries from multiple series * entry pool * ingester uses noopWal * removes duplicate argument passing in ingester code. adds ingester config validation & derives chunk encoding. * segment writing * [WIP] wal recovery from segments * replay uses sync.Maps & preserves WAL fingerprints * in memory wal recovery * wal segment recovery * ingester metrics struct * wal replay locks streamsMtx in instances, adds checkpoint codec * ingester metrics * checkpointer * WAL checkpoint writer * checkpointwriter can write multiple checkpoints * reorgs checkpointing * wires up checkpointwriter to wal * ingester SeriesIter impl * wires up ingesterRecoverer to consume checkpoints * generic recovery fn * generic recovery fn * recover from both wal types * cleans up old tmp checkpoints & allows aborting in flight checkpoints * wires up wal checkpointing * more granular wal logging * fixes off by 1 wal truncation & removes double logging * adds userID to wal records correctly * wire chunk encoding tests * more granular wal metrics * checkpoint encoding test * ignores debug bins * segment replay ignores out of orders * fixes bug between WAL reading []byte validity and proto unmarshalling refs * conf validations, removes comments * flush on shutdown config * POST /ingester/shutdown * renames flush on shutdown * wal & checkpoint use same segment size * writes entries to wal regardless of tailers * makes wal checkpoing duration default to 5m * recovery metrics * encodes headchunks separately for wal purposes * merge upstream * linting * addresses pr feedback uses entry pool in stream push/tailer removes unnecessary pool interaction checkpointbytes comment fillchunk helper, record resetting in tests via pool redundant comment defers wg done in recovery s/num/count/ checkpoint wal uses a logger encodeWithTypeHeader now creates its own []byte removes pool from decodeEntries wal stop can error * prevent shared access bug with tailers and entry pool * removes stream push entry pool optimization	5 years ago
Sandeep Sukhani	ae9c4b82ec	fix table deletion in table client for boltdb-shipper (#2960 )	5 years ago
Ed Welch	9ba5753e25	Loki: handle faults when opening boltdb files (#2988 ) * we also need to catch faults not just panics when opening boltdb files * fix lint * i'll get it one of these times * fix typo	5 years ago
Cyril Tovena	ec725db9d0	Add ProcessString to Pipeline. (#2972 ) * Add ProcessString to Pipeline. Most of the time, you have a buffer that you want to test again a log pipeline, when it passed you usually copy via string(buff). In some cases like headchunk and tailer you already have an allocated immutable string, since pipeline never mutate the line passed as parameter, we can create ProcessString pipeline method that will avoid re-allocating the line. benchmp: ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta BenchmarkHeadBlockIterator/Size_100000-16 20264242 16112591 -20.49% BenchmarkHeadBlockIterator/Size_50000-16 10186969 7905259 -22.40% BenchmarkHeadBlockIterator/Size_15000-16 3229052 2202770 -31.78% BenchmarkHeadBlockIterator/Size_10000-16 1916537 1392355 -27.35% BenchmarkHeadBlockSampleIterator/Size_100000-16 18364773 16106425 -12.30% BenchmarkHeadBlockSampleIterator/Size_50000-16 8988422 7730226 -14.00% BenchmarkHeadBlockSampleIterator/Size_15000-16 2788746 2306161 -17.30% BenchmarkHeadBlockSampleIterator/Size_10000-16 1773766 1488861 -16.06% benchmark old allocs new allocs delta BenchmarkHeadBlockIterator/Size_100000-16 200039 39 -99.98% BenchmarkHeadBlockIterator/Size_50000-16 100036 36 -99.96% BenchmarkHeadBlockIterator/Size_15000-16 30031 31 -99.90% BenchmarkHeadBlockIterator/Size_10000-16 20029 29 -99.86% BenchmarkHeadBlockSampleIterator/Size_100000-16 100040 40 -99.96% BenchmarkHeadBlockSampleIterator/Size_50000-16 50036 36 -99.93% BenchmarkHeadBlockSampleIterator/Size_15000-16 15031 31 -99.79% BenchmarkHeadBlockSampleIterator/Size_10000-16 10029 29 -99.71% benchmark old bytes new bytes delta BenchmarkHeadBlockIterator/Size_100000-16 27604042 21203941 -23.19% BenchmarkHeadBlockIterator/Size_50000-16 13860654 10660652 -23.09% BenchmarkHeadBlockIterator/Size_15000-16 4231360 3271363 -22.69% BenchmarkHeadBlockIterator/Size_10000-16 2633436 1993420 -24.30% BenchmarkHeadBlockSampleIterator/Size_100000-16 17799137 14598973 -17.98% BenchmarkHeadBlockSampleIterator/Size_50000-16 7433260 5833137 -21.53% BenchmarkHeadBlockSampleIterator/Size_15000-16 2258099 1778096 -21.26% BenchmarkHeadBlockSampleIterator/Size_10000-16 1393600 1073605 -22.96% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add some precision about why Sum64 is not a concern. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/chunkenc/memchunk.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Cyril Tovena	06a89ea0ee	Fixes vector grouping injection. (#2975 ) * Fixes vector grouping injection. During an improvement I introduced a bug where we would pre-aggregate some range vector aggregations, although the result is not the same. We can only inject vector grouping in range vector if the operation is associative through grouping. This is not the case for max/min/stddev/stddvar/quantile/avg_over_time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	221fd382a9	Move CI to helm3. (#2986 ) * Move CI to helm3. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Rollback to v3.2.3 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago

1 2 3 4 5 ...

1977 Commits (1ffd4f641ee63046baa7e08d48b72eaa4db7fb2d) All Branches Search

1977 Commits (1ffd4f641ee63046baa7e08d48b72eaa4db7fb2d)

All Branches