1318 Commits (e65d1b8c692f11240f46f91105f266257696334a)

Author	SHA1	Message	Date
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830) ... * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Kaviraj	1b63331e3d	Add support for `X-Query-Tags` (#4814) ... * Add support for `X-Query-Tags` 1. Now client can send any metadata about query via `X-Query-Tags` HTTP header 2. Also record `X-Query-Tags` in `metrics.go` for instant and range queries. * PR remarks * Make linter happy * Fix middleware next * Make safe parsing of the header * Add bit more test cases * Minor suggestions * Add middleware chain to querier handlers * Repopulate the HTTP headers before while Encoding	4 years ago
Salva Corts	ece671c146	Enhance logging on timeout tailing logs on canary (#4825) ... * Enhance logging on timeout tailing logs * Apply CR changes	4 years ago
Christian Haudum	7f54641166	Set correct `Content-Type` header in query response (#4828) ... * Set correct `Content-Type` header in query response Because the internal query request handlers did not set the response content type explicitly to `application/json`, and the external request handler takes the response content type from the internal response to set it for the resulting HTTP response, that response was also incorrectly returning `Content-Type: text/plain`. Setting the content type in the internal response correctly results in the expected `Content-Type` header `application/json; charset=UTF-8`. Fixes #4791 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Set correct `Content-Type` header in query response Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Set correct `Content-Type` header in query response Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Cyril Tovena	45ed6d560d	Fix up a merge conflict on log filter (#4815) ... * Fix up a merge conflict on log filter Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * use the same parameter order as the stdlib Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Hiroki Sakamoto	b1b82a6000	[Hot Fix] Merge improving case sensitive search correctly (#4804) ... * Improve case sensitive search to avoid allocations. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_LineFilter/default_true_(?i)foo-16 2400 2233 -6.96% Benchmark_LineFilter/simplified_true_(?i)foo-16 201 228 +13.13% Benchmark_LineFilter/default_false_(?i)foo-16 2443 2376 -2.74% Benchmark_LineFilter/simplified_false_(?i)foo-16 185 231 +24.96% benchmark old allocs new allocs delta Benchmark_LineFilter/default_true_(?i)foo-16 0 0 +0.00% Benchmark_LineFilter/simplified_true_(?i)foo-16 1 0 -100.00% Benchmark_LineFilter/default_false_(?i)foo-16 0 0 +0.00% Benchmark_LineFilter/simplified_false_(?i)foo-16 1 0 -100.00% benchmark old bytes new bytes delta Benchmark_LineFilter/default_true_(?i)foo-16 0 0 +0.00% Benchmark_LineFilter/simplified_true_(?i)foo-16 128 0 -100.00% Benchmark_LineFilter/default_false_(?i)foo-16 0 0 +0.00% Benchmark_LineFilter/simplified_false_(?i)foo-16 128 0 -100.00% ``` It's not much but for a billions line it makes a big difference. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Simplified version Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	dc222dc98d	Improve case insensitive search to avoid allocations. (#4394) ... * Improve case sensitive search to avoid allocations. ``` ❯ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_LineFilter/default_true_(?i)foo-16 2400 2233 -6.96% Benchmark_LineFilter/simplified_true_(?i)foo-16 201 228 +13.13% Benchmark_LineFilter/default_false_(?i)foo-16 2443 2376 -2.74% Benchmark_LineFilter/simplified_false_(?i)foo-16 185 231 +24.96% benchmark old allocs new allocs delta Benchmark_LineFilter/default_true_(?i)foo-16 0 0 +0.00% Benchmark_LineFilter/simplified_true_(?i)foo-16 1 0 -100.00% Benchmark_LineFilter/default_false_(?i)foo-16 0 0 +0.00% Benchmark_LineFilter/simplified_false_(?i)foo-16 1 0 -100.00% benchmark old bytes new bytes delta Benchmark_LineFilter/default_true_(?i)foo-16 0 0 +0.00% Benchmark_LineFilter/simplified_true_(?i)foo-16 128 0 -100.00% Benchmark_LineFilter/default_false_(?i)foo-16 0 0 +0.00% Benchmark_LineFilter/simplified_false_(?i)foo-16 128 0 -100.00% ``` It's not much but for a billions line it makes a big difference. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * typo Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Simplified version Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	88feda41a0	Improve heap iterators. (#4731) ... Mostly improve allocations but I'm planning to do more using those benchmark. ``` name old time/op new time/op delta HeapIterator-16 3.87ms ± 6% 3.41ms ± 1% -11.92% (p=0.008 n=5+5) HeapSampleIterator-16 2.09ms ± 1% 2.09ms ± 1% ~ (p=0.421 n=5+5) name old alloc/op new alloc/op delta HeapIterator-16 10.5kB ± 0% 8.2kB ± 0% -21.88% (p=0.008 n=5+5) HeapSampleIterator-16 8.40kB ± 0% 6.10kB ± 0% -27.34% (p=0.000 n=5+4) name old allocs/op new allocs/op delta HeapIterator-16 12.0 ± 0% 5.0 ± 0% -58.33% (p=0.008 n=5+5) HeapSampleIterator-16 12.0 ± 0% 5.0 ± 0% -58.33% (p=0.008 n=5+5) ``` I want to introduce a difference between the need to dedupe vs order data. The later doesn't requires poping from the heap, we could actually use a sort.slice. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Karsten Jeschkies	4c9e4f5f06	Make `*` and `+` non-greedy to double regex filter speed. (#4775) ... * Optimize regular expressions in filters. * Run original RE2 tests. * Run both tests. * Update pkg/logql/log/filter.go Co-authored-by: Bryan Boreham <bjboreham@gmail.com> * Update pkg/logql/log/filter.go Co-authored-by: Bryan Boreham <bjboreham@gmail.com> * Fix tests. * Format code. Co-authored-by: Bryan Boreham <bjboreham@gmail.com>	4 years ago
Sandeep Sukhani	8ddcc884fe	add logging in querier when live tailing request starts and ends (#4783) ... * add logging in querier when live tailing request starts and ends * Suggested changes Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Update pkg/querier/http.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	4 years ago
Kaviraj	94113ee080	Logql `stdin` support only log queries (#4606) ... * Hack stdin client into LogCLI Add flag to choose what kind of client to make request to. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Label filter working Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Demo checkpoint1 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Metrics query working. Issue was with default value of `step` (1 nanosecond). StepEvaluator try to go through every nanosecond to apply aggregate on sample data during the metric queries. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Basic tests for fileClient * Tests for logqueries direction * Use HeapIterator for Entries. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove some debug statements Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove support for metric queries. Stick with only log queries Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Small rough usage doc * Remove filesampleiterator * Fix some typos and tests * Fix breaking test cases Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks. 1. Add `--stdin` in examples of the usage 2. Add `--stdin` in all the command help output * PR remarks - Use parsed labels correctly - Fix indendation with --stdin flag * Fix issue with direction * Fix linter * MaxInt64 -> MaxInt (to support even arm32 images) * Add note on calculating `step` value on the client side	4 years ago
Karsten Jeschkies	5ece938f3c	Run all benchmarks. (#4742) ... * Run all benchmarks. * Start debugging memchunk benchmark. * Do not iterate when closed. * Pass default tenant config. * Set default limits. * Fix two more benchmarks. * Enable cron trigger again.	4 years ago
Dylan Guedes	130878f3f6	Enrich WAL folder creation error message (#4705) ... * Enrich WAL folder creation error message. * Apply suggestions from code review Use %q and try absolute path. Co-authored-by: Mateusz Gozdek <mateusz@kinvolk.io> * Include file/path package. Co-authored-by: Mateusz Gozdek <mateusz@kinvolk.io>	4 years ago
Ed Welch	88ef940ede	Loki: Set querier worker max concurrent regardless of run configuration. (#4761) ... * always set the querier worker MaxConcurrentRequests * removing a test which doesn't apply as this setting isn't being set in worker_service.go anymore	4 years ago
Cyril Tovena	44d32c5ad4	Removes all GRPC warning. (#4757) ... This works for logcli, promtail and Loki. Cherry on top, removes a lot of dependencies. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	a707ced67f	index cleanup fixes while applying retention (#4741) ... * Add metric name to the label if missing for building index entries for deletion * fix calculation of table interval from table name * add changelog * add test for series cleanup without __name__ label	4 years ago
Karsten Jeschkies	1e796b070f	Give more context to rule parser errors. (#4730) ... Summary: - Give record and group name for failing parsing.	4 years ago
Karsten Jeschkies	c19a6c5bbc	Join contain filters into one. (#4528) ... * Join contain filters into one. * Update pkg/logql/log/filter.go Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> * Test Filter method. * Test old and new filter consturctors. * Remove comment. * Define more benchmark cases. * Take care of empty contain filters. * Satisfy linter. * USe different method to filter. * Simplify all filter concat. * Format code. * Push regex filters to the end. * Remove unused test. * Add a few more tests. * Make filters private again. * Fix test failure. * Only accumulate successive contain filters. * Reverse accumulation. * Satisfy linter. * Remove old version. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	4 years ago
Sandeep Sukhani	02d88462f3	allow applying retention at different interval than compaction with a config (#4736) ... * allow applying retention at different interval than compaction with a config * add changelog entry	4 years ago
Cyril Tovena	11071d4c31	Improves memchunk benchmark to account for block size. (#4645) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	d81f757ecb	Refactor and simplify stats package. (#4712) ... * Refactor and simplify stats package. This PR refactors the stats package and make it smaller and simpler to work with. - It doesn't expose the context which can causes race when mutating it. - Uses concurrency primitive to ensure correctness - Replace trailers with in-batch statistics to propagate ingesters stats. Fixes #3366 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix tests and lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * mod-check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Sandeep Sukhani	20726e6ba1	fix flaky retention tests (#4697)	4 years ago
Owen Diehl	4d8009cacc	overrides checks for nil tenant limits on AllByUserID (#4687)	4 years ago
Ed Welch	fbebf67588	Loki: check new Read target when initializing boltdb-shipper store (#4681) ... * make sure to check for new `Read` target when setting up boltdb-shipper * also need to make sure ingester querier is added as a dep for Read target * improve the comment on why we shorten the cache validity. * fix shellcheck errors * apply smart defaults to chunk retain period to avoid query gaps. * typo	4 years ago
Ed Welch	3fef08ca1a	2.4 release notes, changelog, upgrade guide (#4658) ... * update changelog, release notes, and upgrade guide for 2.4 * tweak release notes a little * make path relative for kafka config * update upgrade guide * more docs fix the cache size not applied to results cache. * fix prefix name in ring_config	4 years ago
Trevor Whitney	bfc77118a1	Update local and docker config for simple scalable deployment (#4660) ... * first draft of local config * remove a few more defaults * interface names fix means we do not need lifecycler ip * update rings to use collectors/ prefix fix replication factor chunks-retain-period default to 0s fix a bug introduced with a Go upgrade? on printing the schema date. * simplify the config for the ruler, synthesize the local directory * simplify loki-docker-config.yaml * change common filesystem config to have chunks and rules directory * now that we are calling register flags on the common ring config we set more values and some of the tests needed to be tweaked. updated a few paths as well. * fix tests Co-authored-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Susana Ferreira	895d144303	Fix dskit/ring metric with 'cortex_' prefix (#4656)	4 years ago
Cyril Tovena	72292cbab3	Add a test to verify goroutine leak in compression pools. (#4653) ... * Add a test to verify goroutine leak in compression pools. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Account for other goroutine tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve test with eventually. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Callum Styan	678495a972	Instrument s3 client DeleteObject requests. (#4649) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Trevor Whitney	3f9d067e07	compactor depends on memberlist for memberlist ring option (#4643)	4 years ago
Ed Welch	89ee022c21	Loki: fix handling of tail requests when using target `all` or `read` (#4642) ... * better handling of tail requests when the frontend and querier are running in the same process * adding function to determine if a module is enable based on the target supplied to Loki * fix tests and incorrect registering of tail routes * renamed function and made check for active recursive	4 years ago
Susana Ferreira	2d24e2ea64	Migration to dskit/ring (#4641)	4 years ago
Trevor Whitney	07ea1ed99c	Add common ring configuration (#4617) ... * add ring section to common config Co-Authored-By: Dylan Guedes <djmgguedes@gmail.com> * fix compactor test to append loopback interface it ifnames in test if present * add loopback interface to compactor ring iface names * clean-up from PR review Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> * only appy ingester config to rings with no non-default properties Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	4 years ago
Dylan Guedes	0d66319a1f	Revert distributor defaulting to inmemory. (#4638) ... - [PR 4440](https://github.com/grafana/loki/pull/4440/files) modified the distributor setup to use inmemory by default (instead of consul). That was necessary because, by changing the default rate limiting to global, a ring was necessary, and since by default the ring was set to consul, the project couldn't run with the default configs. That's not the case anymore: right now we are reusing the same config set for the ingester (which users already have properly configured otherwise they wouldn't be able to run the project). List of changes: - Remove changelog entries - Remove upgrading guide entries - Remove added tests (unnecessary checks) - Remove value overriding	4 years ago
Sandeep Sukhani	2564a810c5	Logs deletion fixes (#4625) ... * check all tables when we have delete requests to process since chunks could span multiple tables * upload modified index when the it is modified without dropping the chunk * fix broken test * lint * Fix modified check Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * suggested changes from PR review * add changelog Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Owen Diehl	0e0d0b7897	dont try to use the scheduler ring when a downstream url is configured (#4635)	4 years ago
Christian Haudum	929c8e4923	Allow HTTP POST requests on ring pages (#4630) ... The ring pages offer the ability to "forget" instances of the ring and this is implemented by a HTTP `form` element, which does a `POST` request on itself. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Danny Kopping	5b30589acf	Documenting recording rules per-tenant WAL (#4566) ... * WIP Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Document ruler config options Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Self-review updates Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Clarifying WAL benefit in response to review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding dashboard link Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Update docs/sources/configuration/_index.md Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Ensuring ruler and ingester default WAL paths do not conflict Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Ed Welch	de4993f921	explicitly define allowed HTTP methods on HTTP endpoints (#4627)	4 years ago
Trevor Whitney	62ee399e48	default ingester final sleep to 0 unless otherwise specified (#4608) ... * default ingester final sleep to 0 unless otherwise specified * add changelog and upgrading * update config docs	4 years ago
Ed Welch	fe8bc9144f	Loki: query scheduler should send shutdown to frontends when ReplicationSet changes (#4614) ... * make sure we send a shutdown message to frontends from the scheduler when the ReplicationSet changes so that inflight queries are canceled and retried in the frontends. * use Write so we find an instance incase one is unhealthy	4 years ago
Owen Diehl	308af871cf	uses ring.Write instead of ring.WriteNoExtend for compactor ring checks (#4615)	4 years ago
Kaviraj	1e0386e9e6	Fix `ip` matcher lexer to differentiate filter from identifier (#4598) ... * Fix `ip` matcher lexer to differentiate filter from identifier Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Revert small fmt fix on expr.y Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks from Cyril Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add ip line filter parser test in mix with ip as label Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Garrett	1c7df620c4	add date time sprig template functions in logql label/line formatter (#4603) ... * add date time sprig template functions in logql label/line formatter With date time sprig template functions, we will be able to filter test jobs which create time is 1 day before as the following: ``` {job="test"} | label_format nowEpoch=`{{(unixEpoch now)}}`,createDateEpoch=`{{unixEpoch (toDate "2006-01-02" .createDate)}}` | label_format dateTimeDiff="{{sub .nowEpoch .createDateEpoch}}" | dateTimeDiff > 86400 ``` * add documents for date time sprig template functions * resolve comments * add golang datetime layout doc reference * update CHANGELOG.md accordingly to include this PR	4 years ago
Owen Diehl	84220c4b02	improves scheduler & compactor ringwatcher checks (#4607)	4 years ago
Ed Welch	0c9beadaaa	Loki: Add a ring to the compactor used to control concurrency when not running standalone (#4574) ... * add a ring to the compactor * wiring up the start/stop functionality * add some monitoring for multiple compactors running at the same time. * Add compactor to `read` target * fix tests/validation requirements * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/storage/stores/shipper/compactor/compactor_ring.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * used a shared ring_config between scheduler and compactor to avoid some duplication * fix lint * better handling of the tokens file with a way to easily tell all the rings to persist tokens * fix query scheduler replication factor * removing "scheduler" from log line which may be called by components other than a scheduler in the future. * update changelog Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Hiroki Sakamoto	1ff95fc025	Respect gRPC context error when handling errors (#4593) ... * Respect rpc status when handling http errors * Format by goimport * Update pkg/util/errors.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/util/errors.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update pkg/util/errors.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Add some tests * context, rpc cancelled and another error case * context, rpc deadline exceeded and another error case Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Hiroki Sakamoto	0e21f02d76	Respect shard number in series api (#4577) ... * Respect shard number in series api * Refactor codes * remove duplicate logic in instance * format with goimport * Improve complexity of getting all fingerprints	4 years ago
Owen Diehl	0c97afadbc	cleans up redundant setting of stream.unorderedWrites=true when replaying wal (#4595)	4 years ago
Owen Diehl	633b4cab53	ignore validity window during wal replay (#4596)	4 years ago