apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Sandeep Sukhani	1ed19f7380	add a metric for counting number of failures in opening existing files while loading tables from active index in uploads package (#3977 )	5 years ago
Cyril Tovena	5d8414fcd6	Intern label keys for LogQL parser. (#3990 ) * Intern label keys for LogQL parser. They often will see the same key over and over and instead of allocating/sanitizing we can re-use the previous value. This mostly speeds up parser that were using a dynamic list of keys like json and logfmt. We could intern values too specially for metrics queries. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Parser/json/no_labels_hints-16 3894 2885 -25.91% Benchmark_Parser/json/labels_hints-16 2084 2163 +3.79% Benchmark_Parser/unpack/no_labels_hints-16 915 890 -2.72% Benchmark_Parser/unpack/labels_hints-16 848 846 -0.32% Benchmark_Parser/logfmt/no_labels_hints-16 2229 1146 -48.59% Benchmark_Parser/logfmt/labels_hints-16 1230 1150 -6.50% Benchmark_Parser/regex_greedy/no_labels_hints-16 3881 3754 -3.27% Benchmark_Parser/regex_greedy/labels_hints-16 5443 3737 -31.34% Benchmark_Parser/regex_status_digits/no_labels_hints-16 318 290 -8.89% Benchmark_Parser/regex_status_digits/labels_hints-16 335 289 -13.85% Benchmark_Parser/pattern/no_labels_hints-16 151 151 -0.20% Benchmark_Parser/pattern/labels_hints-16 129 131 +1.24% benchmark old allocs new allocs delta Benchmark_Parser/json/no_labels_hints-16 69 43 -37.68% Benchmark_Parser/json/labels_hints-16 32 36 +12.50% Benchmark_Parser/unpack/no_labels_hints-16 19 17 -10.53% Benchmark_Parser/unpack/labels_hints-16 16 17 +6.25% Benchmark_Parser/logfmt/no_labels_hints-16 46 16 -65.22% Benchmark_Parser/logfmt/labels_hints-16 20 16 -20.00% Benchmark_Parser/regex_greedy/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_greedy/labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_status_digits/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/regex_status_digits/labels_hints-16 2 2 +0.00% Benchmark_Parser/pattern/no_labels_hints-16 2 2 +0.00% Benchmark_Parser/pattern/labels_hints-16 1 1 +0.00% benchmark old bytes new bytes delta Benchmark_Parser/json/no_labels_hints-16 1089 624 -42.70% Benchmark_Parser/json/labels_hints-16 456 480 +5.26% Benchmark_Parser/unpack/no_labels_hints-16 569 537 -5.62% Benchmark_Parser/unpack/labels_hints-16 553 537 -2.89% Benchmark_Parser/logfmt/no_labels_hints-16 576 336 -41.67% Benchmark_Parser/logfmt/labels_hints-16 256 336 +31.25% Benchmark_Parser/regex_greedy/no_labels_hints-16 194 194 +0.00% Benchmark_Parser/regex_greedy/labels_hints-16 194 194 +0.00% Benchmark_Parser/regex_status_digits/no_labels_hints-16 52 52 +0.00% Benchmark_Parser/regex_status_digits/labels_hints-16 52 52 +0.00% Benchmark_Parser/pattern/no_labels_hints-16 35 35 +0.00% Benchmark_Parser/pattern/labels_hints-16 32 32 +0.00% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move to labels.go and cap the amount of keys that can be interned to 1024. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Michel Hollands	c44c12b930	Update Cortex version (#3992 ) * Update Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Changed code due to new Cortex version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Run go mod tidy Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Cyril Tovena	4a8f62ba00	Improve head chunk allocations when reading samples. (#3968 ) * Improve head chunk allocations when reading samples. Reading sample is the worst because we are not limited and must return all of them. Re-using memory will helps ingesters read path. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta BenchmarkHeadBlockSampleIterator/Size_20000-16 2935102 1568140 -46.57% BenchmarkHeadBlockSampleIterator/Size_10000-16 1441980 750856 -47.93% BenchmarkHeadBlockSampleIterator/Size_8000-16 1146807 602796 -47.44% BenchmarkHeadBlockSampleIterator/Size_5000-16 698638 370655 -46.95% benchmark old allocs new allocs delta BenchmarkHeadBlockSampleIterator/Size_20000-16 32 16 -50.00% BenchmarkHeadBlockSampleIterator/Size_10000-16 29 15 -48.28% BenchmarkHeadBlockSampleIterator/Size_8000-16 28 15 -46.43% BenchmarkHeadBlockSampleIterator/Size_5000-16 26 15 -42.31% benchmark old bytes new bytes delta BenchmarkHeadBlockSampleIterator/Size_20000-16 2269664 484068 -78.67% BenchmarkHeadBlockSampleIterator/Size_10000-16 1073582 984 -99.91% BenchmarkHeadBlockSampleIterator/Size_8000-16 827825 819 -99.90% BenchmarkHeadBlockSampleIterator/Size_5000-16 475557 786 -99.83% ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve unorderedHeadBlock too. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Vincent Miszczak	d6dc8b991c	Add ingester.autoforget-unhealthy-timeout opt-in feature (#3919 ) * Add ingester.autoforget-unhealthy-timeout opt-in feature * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Cyril Tovena	e9c0687c15	Reject labels with invalid runes when using implicit extraction parser. (#3983 ) `json` and `logfmt` parser will attempt to extract all possible keys and values as labels. But if a values currently contains the rune error (replace when there's an invalid rune) PromQL parser will fail and the request will be fully rejected. This PR removes those labels values when using those two parsers. For other parser you can just decide to not extract thoses values.( At least that's my choice for now). Unfortunatelly this comes with a small cpu overhead prices: ``` ~/go/src/github.com/grafana/loki utf8-parser* ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_Parser/json/no_labels_hints-16 3647 4210 +15.44% Benchmark_Parser/json/labels_hints-16 2210 2247 +1.67% benchmark old allocs new allocs delta Benchmark_Parser/json/no_labels_hints-16 72 72 +0.00% Benchmark_Parser/json/labels_hints-16 35 35 +0.00% benchmark old bytes new bytes delta Benchmark_Parser/json/no_labels_hints-16 1121 1121 +0.00% Benchmark_Parser/json/labels_hints-16 472 472 +0.00% ``` Fixes #2859 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	05b8537e79	WAL Replay counter (#3974 ) * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting * introduces an entry counter into streams & wal records * sets & resets entrycount on checkpoint recovery and after wal replay finishes * pushing to streams checks counter * reject pre-applied entries test * removes commented test	5 years ago
Michel Hollands	78b073e022	Add a chunk filterer field to the config (#3979 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Karsten Jeschkies	af452047d3	Return build info under `/loki/api/v1/status/buildinfo`. (#3972 ) * Return build info under `/version`. Closes #3221 * Update documentation. * Correct docs. * Format version handler code. * Ignore write error. * Sort imports. * Use `PrometheusVersion` struct. * Set Go version.	5 years ago
Sandeep Sukhani	1cca922e6d	fix flaky retention tests (#3976 )	5 years ago
Nick Pillitteri	b830d65bdd	Use the Cortex wrapper for getting tenant ID from a context (#3973 ) This change replaces all uses of the weaveworks/common method for getting tenant/org ID with the Cortex wrapper which performs some extra validation. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	5 years ago
Owen Diehl	23ad7f2e61	Unordered head block (#3957 ) * speccing out unordered head block * testware & unordered serialise * common utils for iter & sampleIter * more generic forEntries * more efficient unordedHeadChunk serialise (no internal re-casting) * roundtripping unordered head block, exit headchunk iteration early, add constant for current default chunk version * adds head block write benchmarks for ordered & unordered writes * fixes bench * removes unused initializer * gofmt * linting	5 years ago
Sandeep Sukhani	0cd8f31831	log name of the file failed to open during startup by ingester (#3970 )	5 years ago
Sandeep Sukhani	6a36cf6745	add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests (#3969 ) * add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests * check only for start time	5 years ago
Sandeep Sukhani	39ac2e58ad	fix a panic in compactor when retention is not enabled (#3967 )	5 years ago
Sandeep Sukhani	ae3e49973e	fix panic in compactor when retention is not enabled (#3966 )	5 years ago
Cyril Tovena	bee456737e	Shards Series API. (#3856 ) * Shards Series API. This PR introduces Series Queries Sharding. It does not check the boundaries of ingesters data since I'm assuming https://github.com/grafana/loki/pull/3852 will be merge first. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	97912b6f4c	Shard ingester queries. (#3852 ) * Shard ingester queries. This is still experimental but already yield from 2x to 6x faster for short period queries. I'm still playing with it but I want to share how I do it early. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add notice of the code origin. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Align shards from ingester and storage. Utimately we should have a storage that relies on fingerprint, but that's harder to change. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes delete index func Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Test reverting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a bug causing non deterministic hash. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/ingester/index/index_test.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Fixes build. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted :( Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	5 years ago
Cyril Tovena	bfcf939c9b	Allow no overrides config for tenants. (#3949 ) Recently came accross this issue when testing Loki, if we don't provide overrides, Loki will crash on a nil pointer. This PR fixes this issue, having no overrides is a valid use case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	5be93e0480	Improve marker file current time metrics. (#3946 ) Currently we'll keep the last recorded time, but if the marker file doesn't exist anymore we should reset it. This would allow us to build consistent alert based on marker files that are too far behind. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	7a7eeb0d63	skip applying retention when data in table is still within retention period and there are no overlapping delete requests (#3934 )	5 years ago
Sandeep Sukhani	43c16ed8a1	remove boltdb files from ingesters on startup which do not have a index bucket (#3929 )	5 years ago
Danny Kopping	71889dcb19	Querier/Ingester: Fixing json expression parser bug (#3928 ) * Fixing json expression parser bug Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * UNRELATED CHANGE: pulling main to fix linter issue Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Cyril Tovena	22dcfcde33	Add retention label to loki_distributor_bytes_received_total metrics (#3840 ) * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/operations/observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Move ParseRequest to an apache2 package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add retention label on the loki_distributor_bytes_received_total metrics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes bad merge. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	5 years ago
vyzigold	67b6c1aac8	logcli: Add retries to unsuccessful log queries (#3879 ) * Add retries to unsuccessful log queries.	5 years ago
Sandeep Sukhani	2c5ce2ce63	feat: index-gateway for boltdb-shipper index store (#3865 ) * feat: index-gateway for boltdb-shipper index store * jsonnet for deploying index-gateway * do not start a goroutine when query count is less than maxQueriesPerGoroutine * changes suggested from PR review	5 years ago
Cyril Tovena	bdee4a3741	Logs ingester and store queries boundaries. (#3849 ) example: ``` ts=2021-06-14T08:43:30.015895283Z caller=spanlogger.go:87 org_id=4336 traceID=0b6bb74a53efd739 level=debug msg="querying ingester" params="selector={stream=\"stdout\", pod=\"loki-canary-6 64d9d8d8f-99k22\"}, direction=BACKWARD, start=2021-06-14 08:17:03.037241855 +0000 UTC, end=2021-06-14 08:17:23.037241855 +0000 UTC, limit=1000, shards=" ts=2021-06-14T08:43:30.016305592Z caller=spanlogger.go:87 org_id=4336 traceID=0b6bb74a53efd739 level=debug msg="querying store" params="selector={stream=\"stdout\", pod=\"loki-canary-664d 9d8d8f-99k22\"}, direction=BACKWARD, start=2021-06-14 08:17:03.037241855 +0000 UTC, end=2021-06-14 08:17:23.037241855 +0000 UTC, limit=1000, shards=" ``` Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	59bb6d3fba	LogQL: Pattern Parser (#3837 ) * The beginning of a fun story. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on adding ragel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding AST parsing with Yacc and Ragel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got a pattern parser working. Reworking ast to works with bytes and not runes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Setup tests and the matches algorithm. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moar tests case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some validation for the pattern expression. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking to LogQL + performance boost. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve bound check. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes generated files from being linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Review feedback Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Docs suggestions Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	5 years ago
Sandeep Sukhani	d362df7a98	ignore download of missing boltdb files possibly removed during compaction (#3563 )	5 years ago
Cyril Tovena	7e658e8766	Fixes a flaky retention test. (#3833 ) This was due because the condition was using a string array that may be not in correct order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	cd38e987d8	Logs deletion fixes (#3827 ) * fix temp file suffix in delete requests table * log error on failure in removal of temp file only when the error is not os.IsNotExist * fix the cli flag for delete requests cancellation period	5 years ago
dbluxo	88708d4b22	added support for bearer token authentication (#3749 )	5 years ago
Danny Kopping	110ff6fb9e	Extracting queue interface (#3816 ) Exporting metrics entry type Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Danny Kopping	3642face57	Allowing for multiple defined targets (#3807 ) Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Danny Kopping	0bf7b608d4	Exposing remote writer for use in integration tests (#3797 ) Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Michel Hollands	3c47735dea	Add a QueryFrontendTripperware module (#3792 ) Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	5 years ago
Danny Kopping	babed450c7	Ruler: Recording Rules (#3766 ) * WIP: hack to get recording rules working and pushing to Cortex/Prometheus Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring Adding remote_write config for ruler Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Minor refactorings Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving manager subpackage into ruler package to avoid dependency cycles This also mirrors Cortex's package structure Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Minor refactorings Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Skipping commit if remote-write client is not defined Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating use of cortex client Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Memoizing appenders, using queue for samples & labels Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding buffer size configurability Refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metric to show current buffer size Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring for better responsibility separation & testability Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding per-tenant overrides of remote-write queue capacity Renaming "buffer size" to "queue capacity" to be more accurate Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding tests for evicting queue Minor refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding more tests and refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding queue benchmark Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Reducing redundancy in metric names Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Testing that only metric queries can be run Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Minor fixes pre-review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Guarding against unprotected nil pointer dereference in Prometheus remote.Client Adding remote-write client validation Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Setting tenant ID header on remote-write client Adding User-Agent string Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating benchmark to use complex struct rather than int to be more reflective of usage Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Registering flags Removing extraneous checks Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metric to track remote-write commit errors Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Refactoring based on review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Performance improvements based on review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Return error on invalid queue capacity Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing global queue capacity config - using limits Minor refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Reusing memory in request preparation Refactoring for testability Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving remote-write metrics into struct Refactoring Unexporting and refactoring memstore metrics to match Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Applying review suggestions Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Allowing for runtime changing of per-tenant remote-write queue capacity Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Sandeep Sukhani	c90366dbe7	just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled (#3785 ) * just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled * gofmt * Update msg Co-authored-by: Ed Welch <ed@oqqer.com> Co-authored-by: Ed Welch <ed@oqqer.com>	5 years ago
Sandeep Sukhani	61b60092af	initialize retention and deletion components only when they are enabled (#3772 )	5 years ago
Sandeep Sukhani	966bed72d3	reduce allocs in delete requests manager by reusing slice for tracing non-deleted intervals for chunks (#3769 )	5 years ago
Cyril Tovena	526dc4bbe7	Fixes parser labels hint for grouping. (#3763 ) * Fixes parser labels hint for grouping. This PR include those now. Fixes #3707 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes array allocations. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	89b32835c2	Improve mark file processing. (#3762 ) This makes boltdb batch for deleting entries smaller and so goroutines for deletion of chunks are not stuck. Also improve metrics to refresh more often. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	e5a3afedff	Add fromJson to the template stage. (#3736 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	1f6453fb81	exposes loki codec (#3758 )	5 years ago
Sandeep Sukhani	3b26c63c82	Boltdb shipper deletion fixes (#3746 ) * use return instead of break in select case * skip compaction on delete requests table	5 years ago
Roger Steneteg	0679e50d6a	correct logcli instant query timestamp param name (#3739 ) Signed-off-by: Roger Steneteg <rsteneteg@ea.com>	5 years ago
Cyril Tovena	e1a3ab841c	Improve retention mark files. (#3706 ) * Improve retention mark files. This PR is two fold: 1) Rotate mark files when they reached 100k marks, this is to ensure we max out marks file and don't create file that may be too big. 2) Instead of inserting marks using the chunk id as the key, we insert using a natural sequence. This has two benefits: - Keys are ordered, and so insertion are faster. - Allows us to use boltdb fill percent to 100%, this means boltdb won't over allocate for inserting key in between unordered data. Why ? I realize that when inserting huge amount of marks, this operation can take up to hours since boltdb has to re-allocate pages over and over. This is mainly because chunkid arrive without specific order. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve key buf sizing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/storage/stores/shipper/compactor/retention/marker.go Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> * Copy the value so that it still available for the whole boltdb tx. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	5 years ago
Sandeep Sukhani	2b7a6f201c	Logs deletion support for boltdb-shipper store (#3688 ) * add support for slicing the chunks for partial chunk deletion * add delete requests store and handler for requests * add delete requests manager and a way to check for chunk expiry based on delete requests * add chunk rewriter for partially deleted chunks * hook delete requests manager into compactor and update expiration checker * use chunk client in sweeper for deleting chunks * upload delete requests table during shutdown and some code refactoring * add tests and metrics * fixes post rebase * lint * changes suggested from PR review * avoid allocation when getting userid from chunkid * use bytes.IndexByte instead	5 years ago
Ed Welch	fa247353ed	Add a flag which allows the queriers to run completely standalone and only query stored data. (#3700 )	5 years ago
Cyril Tovena	210dc355cb	Fixes a goroutine leak in the store when doing cancellation. (#3733 ) Found out that we might have no listener and trying to send the information of cancellation. I've moved the error information down the respective log and sample iterators so that we don't need to send it via the channel. This seems to have cause some pretty big memory leak on our side. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago

1 2 3 4 5 ...

988 Commits (b72a470d90aa05ebc4e435023970cdedddc19471)