2332 Commits (ac63ea85003a0eee88ad16ff97a12918e6a6cef0)

 

All Branches   

Search

Author	SHA1	Message	Date
Danny Kopping	ac63ea8500	Promtail: adding pipeline stage inspector (#4011) ... * Adding promtail pipeline stage inspector Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding library Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding docs Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Further appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing old version of go-cmp to pass check-mod step Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Making screenshot narrower and correcting link Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Copy changes Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Copy changes Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Christian Simon	2fd633cded	Restrict path segments in TenantIDs (#4020) ... Updates cortex dependency to prevent paths derived from TenantIDs to become vulnerable to path traversal attacks. CVE-2021-36156 CVE-2021-36157 Signed-off-by: Christian Simon <simon@swine.de>	4 years ago
Periklis Tsirakidis	cb337c3b71	Add LogQL AST walker (#3991) ... * Make expressions public scoped * Add walkable implementation * Use walkable in logql/optimize	4 years ago
Lize Cai	8370ed3b04	fix typo in loki docker-compose (#4016)	4 years ago
Owen Diehl	a4db1a4359	Headblock interop (#3995) ... * merge feature/unordered-replay * interoperable head chunks	4 years ago
Owen Diehl	2107148acd	fixes chunk size method in facade (#3981)	4 years ago
Jarrod Funnell	267107b75e	Parser: Allow literal control chars in logfmt decoder (#3932) ... * Allow literal control chars in decoder * Ensure decoding control chars in logfmt works	4 years ago
Sandeep Sukhani	58f9d0c085	add config and documentation about index-gateway (#3956) ... * add config and documentation about index-gateway * Update docs/sources/operations/storage/boltdb-shipper.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/operations/storage/boltdb-shipper.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Owen Diehl	b58e9cdee2	fixes a badly referenced variable name in StepEvaluator code (#3996)	4 years ago
Owen Diehl	fef99775ea	aligns rule path in docker img with bundled config. closes #3952 (#3997)	4 years ago
Owen Diehl	7289c064b7	Fixes regexReplaceAll docs (#3998) ... closes #3939	4 years ago
Danny Kopping	d6dc43b958	Adding @grafana/loki-team as default CODEOWNERS (#4007) ... In order to notify the team of PRs requiring review Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Danny Kopping	b72a470d90	Output timestamp with nanosecond precision if available (#4006) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Sandeep Sukhani	1ed19f7380	add a metric for counting number of failures in opening existing files while loading tables from active index in uploads package (#3977)	4 years ago
Karen Miller	bc566fbbab	Docs: Revise/update the overview section. (#3980) ... * Docs: Revise/update the overview section. * Update docs/sources/overview/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update docs/sources/overview/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	5d8414fcd6	Intern label keys for LogQL parser. (#3990) ... * Intern label keys for LogQL parser. They often will see the same key over and over and instead of allocating/sanitizing we can re-use the previous value. This mostly speeds up parser that were using a dynamic list of keys like json and logfmt. We could intern values too specially for metrics queries. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Parser/json/no_labels_hints-16 3894 2885 -25.91% Benchmark_Parser/json/labels_hints-16 2084 2163 +3.79% Benchmark_Parser/unpack/no_labels_hints-16 915 890 -2.72% Benchmark_Parser/unpack/labels_hints-16 848 846 -0.32% Benchmark_Parser/logfmt/no_labels_hints-16 2229 1146 -48.59% Benchmark_Parser/logfmt/labels_hints-16 1230 1150 -6.50% Benchmark_Parser/regex_greedy/no_labels_hints-16 3881 3754 -3.27% Benchmark_Parser/regex_greedy/labels_hints-16 5443 3737 -31.34% Benchmark_Parser/regex_status_digits/no_labels_hints-16 318 290 -8.89% Benchmark_Parser/regex_status_digits/labels_hints-16 335 289 -13.85% Benchmark_Parser/pattern/no_labels_hints-16 151 151 -0.20% Benchmark_Parser/pattern/labels_hints-16 129 131 +1.24% benchmark old allocs new allocs delta Benchmark_Parser/json/no_labels_hints-16 69 43 -37.68% Benchmark_Parser/json/labels_hints-16 32 36 +12.50% Benchmark_Parser/unpack/no_labels_hints-16 19 17 -10.53% Benchmark_Parser/unpack/labels_hints-16 16 17 +6.25% Benchmark_Parser/logfmt/no_labels_hints-16 46 16 -65.22% Benchmark_Parser/logfmt/labels_hints-16 20 16 -20.00% Benchmark_Parser/regex_greedy/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_greedy/labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_status_digits/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_status_digits/labels_hints-16 2 2 +0.00% Benchmark_Parser/pattern/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/pattern/labels_hints-16 1 1 +0.00% benchmark old bytes new bytes delta Benchmark_Parser/json/no_labels_hints-16 1089 624 -42.70% Benchmark_Parser/json/labels_hints-16 456 480 +5.26% Benchmark_Parser/unpack/no_labels_hints-16 569 537 -5.62% Benchmark_Parser/unpack/labels_hints-16 553 537 -2.89% Benchmark_Parser/logfmt/no_labels_hints-16 576 336 -41.67% Benchmark_Parser/logfmt/labels_hints-16 256 336 +31.25% Benchmark_Parser/regex_greedy/no_labels_hints-16 194 194 +0.00% Benchmark_Parser/regex_greedy/labels_hints-16 194 194 +0.00% Benchmark_Parser/regex_status_digits/no_labels_hints-16 52 52 +0.00% Benchmark_Parser/regex_status_digits/labels_hints-16 52 52 +0.00% Benchmark_Parser/pattern/no_labels_hints-16 35 35 +0.00% Benchmark_Parser/pattern/labels_hints-16 32 32 +0.00% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move to labels.go and cap the amount of keys that can be interned to 1024. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Michel Hollands	c44c12b930	Update Cortex version (#3992) ... * Update Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Changed code due to new Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Run go mod tidy Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Cyril Tovena	4a8f62ba00	Improve head chunk allocations when reading samples. (#3968) ... * Improve head chunk allocations when reading samples. Reading sample is the worst because we are not limited and must return all of them. Re-using memory will helps ingesters read path. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta BenchmarkHeadBlockSampleIterator/Size_20000-16 2935102 1568140 -46.57% BenchmarkHeadBlockSampleIterator/Size_10000-16 1441980 750856 -47.93% BenchmarkHeadBlockSampleIterator/Size_8000-16 1146807 602796 -47.44% BenchmarkHeadBlockSampleIterator/Size_5000-16 698638 370655 -46.95% benchmark old allocs new allocs delta BenchmarkHeadBlockSampleIterator/Size_20000-16 32 16 -50.00% BenchmarkHeadBlockSampleIterator/Size_10000-16 29 15 -48.28% BenchmarkHeadBlockSampleIterator/Size_8000-16 28 15 -46.43% BenchmarkHeadBlockSampleIterator/Size_5000-16 26 15 -42.31% benchmark old bytes new bytes delta BenchmarkHeadBlockSampleIterator/Size_20000-16 2269664 484068 -78.67% BenchmarkHeadBlockSampleIterator/Size_10000-16 1073582 984 -99.91% BenchmarkHeadBlockSampleIterator/Size_8000-16 827825 819 -99.90% BenchmarkHeadBlockSampleIterator/Size_5000-16 475557 786 -99.83% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve unorderedHeadBlock too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Vincent Miszczak	d6dc8b991c	Add ingester.autoforget-unhealthy-timeout opt-in feature (#3919) ... * Add ingester.autoforget-unhealthy-timeout opt-in feature * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	e9c0687c15	Reject labels with invalid runes when using implicit extraction parser. (#3983) ... `json` and `logfmt` parser will attempt to extract all possible keys and values as labels. But if a values currently contains the rune error (replace when there's an invalid rune) PromQL parser will fail and the request will be fully rejected. This PR removes those labels values when using those two parsers. For other parser you can just decide to not extract thoses values.( At least that's my choice for now). Unfortunatelly this comes with a small cpu overhead prices: ``` ~/go/src/github.com/grafana/loki utf8-parser* ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Parser/json/no_labels_hints-16 3647 4210 +15.44% Benchmark_Parser/json/labels_hints-16 2210 2247 +1.67% benchmark old allocs new allocs delta Benchmark_Parser/json/no_labels_hints-16 72 72 +0.00% Benchmark_Parser/json/labels_hints-16 35 35 +0.00% benchmark old bytes new bytes delta Benchmark_Parser/json/no_labels_hints-16 1121 1121 +0.00% Benchmark_Parser/json/labels_hints-16 472 472 +0.00% ``` Fixes #2859 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Owen Diehl	05b8537e79	WAL Replay counter (#3974) ... * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting * introduces an entry counter into streams & wal records * sets & resets entrycount on checkpoint recovery and after wal replay finishes * pushing to streams checks counter * reject pre-applied entries test * removes commented test	4 years ago
Michel Hollands	78b073e022	Add a chunk filterer field to the config (#3979) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Karsten Jeschkies	af452047d3	Return build info under `/loki/api/v1/status/buildinfo`. (#3972) ... * Return build info under `/version`. Closes #3221 * Update documentation. * Correct docs. * Format version handler code. * Ignore write error. * Sort imports. * Use `PrometheusVersion` struct. * Set Go version.	4 years ago
Sandeep Sukhani	1cca922e6d	fix flaky retention tests (#3976)	4 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973) ... This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Owen Diehl	23ad7f2e61	Unordered head block (#3957) ... * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting	4 years ago
raymonder jin	6d8939670e	fix typos (#3965)	4 years ago
Cyril Tovena	86000dd320	Development Docker Compose Setup (#3941) ... * Adds the first version of a docker compose dev stack. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add index-gateway and code source. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better docker ignore. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better bash comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update tools/dev/loki-boltdb-storage-s3/README.md Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * lint bash scripts. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	4 years ago
Cyril Tovena	337aed06a6	Fixes negative gauge in Promtail. (#3971) ... * Fixes negative gauge in Promtail. Fixes #3281 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karen Miller	f653a93d42	Docs: added new target (docs-next) to the docs' Makefile. (#3962) ... This target builds local docs as they should look for commits that are placed into the main branch, showing the next up, but not yet released Loki docs.	4 years ago
Robert Fratto	2e84bcfaa9	remove robert fratto from list of team members (#3963)	4 years ago
Sandeep Sukhani	0cd8f31831	log name of the file failed to open during startup by ingester (#3970)	4 years ago
Kristian Klausen	021a7fdc33	Doc: Remove removed --ingester.recover-from-wal option and fix out-of-date defaults (#3559) ... * Doc: Remove removed --ingester.recover-from-wal option Removed in: 8a9b94a8 ("removes recover flag (#3326)") * Doc: Fix out-of-date defaults 0b1dbe26 ("Promtail: Add a stream lagging metric (#2618)") d3bf21e7 ("Promtail: (and also fluent-bit) change the max batch size to 1MB (#2710)")	4 years ago
Sandeep Sukhani	6a36cf6745	add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests (#3969) ... * add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests * check only for start time	4 years ago
Sandeep Sukhani	39ac2e58ad	fix a panic in compactor when retention is not enabled (#3967)	4 years ago
Sandeep Sukhani	ae3e49973e	fix panic in compactor when retention is not enabled (#3966)	4 years ago
Cyril Tovena	bee456737e	Shards Series API. (#3856) ... * Shards Series API. This PR introduces Series Queries Sharding. It does not check the boundaries of ingesters data since I'm assuming https://github.com/grafana/loki/pull/3852 will be merge first. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	97912b6f4c	Shard ingester queries. (#3852) ... * Shard ingester queries. This is still experimental but already yield from 2x to 6x faster for short period queries. I'm still playing with it but I want to share how I do it early. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notice of the code origin. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Align shards from ingester and storage. Utimately we should have a storage that relies on fingerprint, but that's harder to change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes delete index func Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test reverting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug causing non deterministic hash. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Fixes build. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted :( Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Karen Miller	055c35e47a	Docs: add structure for a release notes section (#3910)	4 years ago
Karen Miller	7eceea082b	Docs: Make identification of experimental items consistent and more obvious. (#3911)	4 years ago
Karen Miller	2aafb0d38a	Docs: revise the LogCLI subsection (#3921) ... * Docs: revise the LogCLI subsection * Update docs/sources/getting-started/logcli.md per review suggestion Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Incorporate review suggestion. Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Julien Duchesne	acf1866706	Sign drone.yml file (#3950) ... * Sign drone.yml file This is needed to set the repository as "Protected" * Add shellcheck back	4 years ago
Cyril Tovena	bfcf939c9b	Allow no overrides config for tenants. (#3949) ... Recently came accross this issue when testing Loki, if we don't provide overrides, Loki will crash on a nil pointer. This PR fixes this issue, having no overrides is a valid use case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karsten Jeschkies	0cde54c4dc	Lint script files. (#3944) ... * Lint script files. This patch adds a Drone CI step that runs shellcheck on all Bash script files. * Install Bash in shellcheck container. * Fix a few shelcheck errors. * Fix build.sh * Fix remaining shellcheck errors.	4 years ago
Cyril Tovena	5be93e0480	Improve marker file current time metrics. (#3946) ... Currently we'll keep the last recorded time, but if the marker file doesn't exist anymore we should reset it. This would allow us to build consistent alert based on marker files that are too far behind. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
ecraven	1fb06cc64f	Makefile: Only set PROMTAIL_CGO if CGO_ENABLED is not 0. (#3935) ... * Only set PROMTAIL_CGO if CGO_ENABLED is not 0. This allows to build promtail without journald support on linux distributions that don't have systemd. Run as CGO_ENABLED=0 make promtail * Fix scope.	4 years ago
Luka Hartwig	0403e9d316	Fix typo (#3936)	4 years ago
Paul Seiffert	2cee4e31d0	List 'compactor' as valid value for target option (#3938)	4 years ago
Sandeep Sukhani	7a7eeb0d63	skip applying retention when data in table is still within retention period and there are no overlapping delete requests (#3934)	4 years ago
Sandeep Sukhani	43c16ed8a1	remove boltdb files from ingesters on startup which do not have a index bucket (#3929)	4 years ago