2320 Commits (b72a470d90aa05ebc4e435023970cdedddc19471)

 

All Branches   

Search

Author	SHA1	Message	Date
Danny Kopping	b72a470d90	Output timestamp with nanosecond precision if available (#4006) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Sandeep Sukhani	1ed19f7380	add a metric for counting number of failures in opening existing files while loading tables from active index in uploads package (#3977)	5 years ago
Karen Miller	bc566fbbab	Docs: Revise/update the overview section. (#3980) ... * Docs: Revise/update the overview section. * Update docs/sources/overview/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update docs/sources/overview/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Cyril Tovena	5d8414fcd6	Intern label keys for LogQL parser. (#3990) ... * Intern label keys for LogQL parser. They often will see the same key over and over and instead of allocating/sanitizing we can re-use the previous value. This mostly speeds up parser that were using a dynamic list of keys like json and logfmt. We could intern values too specially for metrics queries. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Parser/json/no_labels_hints-16 3894 2885 -25.91% Benchmark_Parser/json/labels_hints-16 2084 2163 +3.79% Benchmark_Parser/unpack/no_labels_hints-16 915 890 -2.72% Benchmark_Parser/unpack/labels_hints-16 848 846 -0.32% Benchmark_Parser/logfmt/no_labels_hints-16 2229 1146 -48.59% Benchmark_Parser/logfmt/labels_hints-16 1230 1150 -6.50% Benchmark_Parser/regex_greedy/no_labels_hints-16 3881 3754 -3.27% Benchmark_Parser/regex_greedy/labels_hints-16 5443 3737 -31.34% Benchmark_Parser/regex_status_digits/no_labels_hints-16 318 290 -8.89% Benchmark_Parser/regex_status_digits/labels_hints-16 335 289 -13.85% Benchmark_Parser/pattern/no_labels_hints-16 151 151 -0.20% Benchmark_Parser/pattern/labels_hints-16 129 131 +1.24% benchmark old allocs new allocs delta Benchmark_Parser/json/no_labels_hints-16 69 43 -37.68% Benchmark_Parser/json/labels_hints-16 32 36 +12.50% Benchmark_Parser/unpack/no_labels_hints-16 19 17 -10.53% Benchmark_Parser/unpack/labels_hints-16 16 17 +6.25% Benchmark_Parser/logfmt/no_labels_hints-16 46 16 -65.22% Benchmark_Parser/logfmt/labels_hints-16 20 16 -20.00% Benchmark_Parser/regex_greedy/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_greedy/labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_status_digits/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_status_digits/labels_hints-16 2 2 +0.00% Benchmark_Parser/pattern/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/pattern/labels_hints-16 1 1 +0.00% benchmark old bytes new bytes delta Benchmark_Parser/json/no_labels_hints-16 1089 624 -42.70% Benchmark_Parser/json/labels_hints-16 456 480 +5.26% Benchmark_Parser/unpack/no_labels_hints-16 569 537 -5.62% Benchmark_Parser/unpack/labels_hints-16 553 537 -2.89% Benchmark_Parser/logfmt/no_labels_hints-16 576 336 -41.67% Benchmark_Parser/logfmt/labels_hints-16 256 336 +31.25% Benchmark_Parser/regex_greedy/no_labels_hints-16 194 194 +0.00% Benchmark_Parser/regex_greedy/labels_hints-16 194 194 +0.00% Benchmark_Parser/regex_status_digits/no_labels_hints-16 52 52 +0.00% Benchmark_Parser/regex_status_digits/labels_hints-16 52 52 +0.00% Benchmark_Parser/pattern/no_labels_hints-16 35 35 +0.00% Benchmark_Parser/pattern/labels_hints-16 32 32 +0.00% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move to labels.go and cap the amount of keys that can be interned to 1024. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Michel Hollands	c44c12b930	Update Cortex version (#3992) ... * Update Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Changed code due to new Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Run go mod tidy Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Cyril Tovena	4a8f62ba00	Improve head chunk allocations when reading samples. (#3968) ... * Improve head chunk allocations when reading samples. Reading sample is the worst because we are not limited and must return all of them. Re-using memory will helps ingesters read path. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta BenchmarkHeadBlockSampleIterator/Size_20000-16 2935102 1568140 -46.57% BenchmarkHeadBlockSampleIterator/Size_10000-16 1441980 750856 -47.93% BenchmarkHeadBlockSampleIterator/Size_8000-16 1146807 602796 -47.44% BenchmarkHeadBlockSampleIterator/Size_5000-16 698638 370655 -46.95% benchmark old allocs new allocs delta BenchmarkHeadBlockSampleIterator/Size_20000-16 32 16 -50.00% BenchmarkHeadBlockSampleIterator/Size_10000-16 29 15 -48.28% BenchmarkHeadBlockSampleIterator/Size_8000-16 28 15 -46.43% BenchmarkHeadBlockSampleIterator/Size_5000-16 26 15 -42.31% benchmark old bytes new bytes delta BenchmarkHeadBlockSampleIterator/Size_20000-16 2269664 484068 -78.67% BenchmarkHeadBlockSampleIterator/Size_10000-16 1073582 984 -99.91% BenchmarkHeadBlockSampleIterator/Size_8000-16 827825 819 -99.90% BenchmarkHeadBlockSampleIterator/Size_5000-16 475557 786 -99.83% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve unorderedHeadBlock too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Vincent Miszczak	d6dc8b991c	Add ingester.autoforget-unhealthy-timeout opt-in feature (#3919) ... * Add ingester.autoforget-unhealthy-timeout opt-in feature * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Cyril Tovena	e9c0687c15	Reject labels with invalid runes when using implicit extraction parser. (#3983) ... `json` and `logfmt` parser will attempt to extract all possible keys and values as labels. But if a values currently contains the rune error (replace when there's an invalid rune) PromQL parser will fail and the request will be fully rejected. This PR removes those labels values when using those two parsers. For other parser you can just decide to not extract thoses values.( At least that's my choice for now). Unfortunatelly this comes with a small cpu overhead prices: ``` ~/go/src/github.com/grafana/loki utf8-parser* ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Parser/json/no_labels_hints-16 3647 4210 +15.44% Benchmark_Parser/json/labels_hints-16 2210 2247 +1.67% benchmark old allocs new allocs delta Benchmark_Parser/json/no_labels_hints-16 72 72 +0.00% Benchmark_Parser/json/labels_hints-16 35 35 +0.00% benchmark old bytes new bytes delta Benchmark_Parser/json/no_labels_hints-16 1121 1121 +0.00% Benchmark_Parser/json/labels_hints-16 472 472 +0.00% ``` Fixes #2859 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	05b8537e79	WAL Replay counter (#3974) ... * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting * introduces an entry counter into streams & wal records * sets & resets entrycount on checkpoint recovery and after wal replay finishes * pushing to streams checks counter * reject pre-applied entries test * removes commented test	5 years ago
Michel Hollands	78b073e022	Add a chunk filterer field to the config (#3979) ... Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Karsten Jeschkies	af452047d3	Return build info under `/loki/api/v1/status/buildinfo`. (#3972) ... * Return build info under `/version`. Closes #3221 * Update documentation. * Correct docs. * Format version handler code. * Ignore write error. * Sort imports. * Use `PrometheusVersion` struct. * Set Go version.	5 years ago
Sandeep Sukhani	1cca922e6d	fix flaky retention tests (#3976)	5 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973) ... This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	5 years ago
Owen Diehl	23ad7f2e61	Unordered head block (#3957) ... * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting	5 years ago
raymonder jin	6d8939670e	fix typos (#3965)	5 years ago
Cyril Tovena	86000dd320	Development Docker Compose Setup (#3941) ... * Adds the first version of a docker compose dev stack. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add index-gateway and code source. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Better docker ignore. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * better bash comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update tools/dev/loki-boltdb-storage-s3/README.md Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * lint bash scripts. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	5 years ago
Cyril Tovena	337aed06a6	Fixes negative gauge in Promtail. (#3971) ... * Fixes negative gauge in Promtail. Fixes #3281 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Karen Miller	f653a93d42	Docs: added new target (docs-next) to the docs' Makefile. (#3962) ... This target builds local docs as they should look for commits that are placed into the main branch, showing the next up, but not yet released Loki docs.	5 years ago
Robert Fratto	2e84bcfaa9	remove robert fratto from list of team members (#3963)	5 years ago
Sandeep Sukhani	0cd8f31831	log name of the file failed to open during startup by ingester (#3970)	5 years ago
Kristian Klausen	021a7fdc33	Doc: Remove removed --ingester.recover-from-wal option and fix out-of-date defaults (#3559) ... * Doc: Remove removed --ingester.recover-from-wal option Removed in: 8a9b94a8 ("removes recover flag (#3326)") * Doc: Fix out-of-date defaults 0b1dbe26 ("Promtail: Add a stream lagging metric (#2618)") d3bf21e7 ("Promtail: (and also fluent-bit) change the max batch size to 1MB (#2710)")	5 years ago
Sandeep Sukhani	6a36cf6745	add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests (#3969) ... * add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests * check only for start time	5 years ago
Sandeep Sukhani	39ac2e58ad	fix a panic in compactor when retention is not enabled (#3967)	5 years ago
Sandeep Sukhani	ae3e49973e	fix panic in compactor when retention is not enabled (#3966)	5 years ago
Cyril Tovena	bee456737e	Shards Series API. (#3856) ... * Shards Series API. This PR introduces Series Queries Sharding. It does not check the boundaries of ingesters data since I'm assuming https://github.com/grafana/loki/pull/3852 will be merge first. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	97912b6f4c	Shard ingester queries. (#3852) ... * Shard ingester queries. This is still experimental but already yield from 2x to 6x faster for short period queries. I'm still playing with it but I want to share how I do it early. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notice of the code origin. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Align shards from ingester and storage. Utimately we should have a storage that relies on fingerprint, but that's harder to change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes delete index func Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test reverting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug causing non deterministic hash. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Fixes build. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted :( Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Karen Miller	055c35e47a	Docs: add structure for a release notes section (#3910)	5 years ago
Karen Miller	7eceea082b	Docs: Make identification of experimental items consistent and more obvious. (#3911)	5 years ago
Karen Miller	2aafb0d38a	Docs: revise the LogCLI subsection (#3921) ... * Docs: revise the LogCLI subsection * Update docs/sources/getting-started/logcli.md per review suggestion Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Incorporate review suggestion. Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Julien Duchesne	acf1866706	Sign drone.yml file (#3950) ... * Sign drone.yml file This is needed to set the repository as "Protected" * Add shellcheck back	5 years ago
Cyril Tovena	bfcf939c9b	Allow no overrides config for tenants. (#3949) ... Recently came accross this issue when testing Loki, if we don't provide overrides, Loki will crash on a nil pointer. This PR fixes this issue, having no overrides is a valid use case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Karsten Jeschkies	0cde54c4dc	Lint script files. (#3944) ... * Lint script files. This patch adds a Drone CI step that runs shellcheck on all Bash script files. * Install Bash in shellcheck container. * Fix a few shelcheck errors. * Fix build.sh * Fix remaining shellcheck errors.	5 years ago
Cyril Tovena	5be93e0480	Improve marker file current time metrics. (#3946) ... Currently we'll keep the last recorded time, but if the marker file doesn't exist anymore we should reset it. This would allow us to build consistent alert based on marker files that are too far behind. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
ecraven	1fb06cc64f	Makefile: Only set PROMTAIL_CGO if CGO_ENABLED is not 0. (#3935) ... * Only set PROMTAIL_CGO if CGO_ENABLED is not 0. This allows to build promtail without journald support on linux distributions that don't have systemd. Run as CGO_ENABLED=0 make promtail * Fix scope.	5 years ago
Luka Hartwig	0403e9d316	Fix typo (#3936)	5 years ago
Paul Seiffert	2cee4e31d0	List 'compactor' as valid value for target option (#3938)	5 years ago
Sandeep Sukhani	7a7eeb0d63	skip applying retention when data in table is still within retention period and there are no overlapping delete requests (#3934)	5 years ago
Sandeep Sukhani	43c16ed8a1	remove boltdb files from ingesters on startup which do not have a index bucket (#3929)	5 years ago
Danny Kopping	71889dcb19	Querier/Ingester: Fixing json expression parser bug (#3928) ... * Fixing json expression parser bug Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * UNRELATED CHANGE: pulling main to fix linter issue Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Cyril Tovena	22dcfcde33	Add retention label to loki_distributor_bytes_received_total metrics (#3840) ... * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/operations/observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Move ParseRequest to an apache2 package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	5 years ago
Cyril Tovena	5887096f3c	Add Danny Kopping to the Loki Team. (#3926) ... Welcome 🚀 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Javier Palomo	e16f1fef81	refactor(production/ksonnet): Remove kausal from the root element (#3917) ... Signed-off-by: Javier Palomo <javier.palomo@grafana.com>	5 years ago
Kaviraj	5e020dbdba	Sync `main` branch docs to `next` folder (#3909) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	5 years ago
Kaviraj	f00a235942	Update steps to release versioned docs (#3897) ... * Update steps to release versioned docs. This makes versioned docs in consistent with Grafana. More details: https://github.com/grafana/website/pull/4970 https://github.com/grafana/website/pull/4970 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add step to update `latest_version` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	5 years ago
Karen Miller	3c23ca6e18	Correct “ and ” with " and same with single quote mark. (#3899)	5 years ago
Sandeep Sukhani	73c87dd319	update uid of loki-deletion dashboard (#3893)	5 years ago
Kaviraj	81e100f18f	Make `overrides` configmap names and mount path as variables. (#3888) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	5 years ago
Sandeep Sukhani	c41971b184	add index-gateway to reads and reads-resources dashboards (#3891)	5 years ago
vyzigold	67b6c1aac8	logcli: Add retries to unsuccessful log queries (#3879) ... * Add retries to unsuccessful log queries.	5 years ago
Karen Miller	c9276aa989	Docs: improve section on building from source (#3882)	5 years ago