292 Commits (0bce8d95a2f42708d7ce8148e64cd0173067df05)

Author	SHA1	Message	Date
Cyril Tovena	1a7614ff70	Avoid splitting large range vector aggregation. (#5172) ... * Reduce split by for large range vector Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/queryrange/split_by_interval.go Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * improve fn name. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
李国忠	2c20ce6857	[optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" (#4891) ... * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * Update pkg/storage/chunk/cache/instrumented.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200" #4891 * Update pkg/querier/queryrange/roundtrip.go Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * Update roundtrip.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Owen Diehl	8c348614e3	Querier worker/inflight metrics (#5124) ... * refactors querier worker metrics to use metrics struct * metrics and logs for querier worker execution * loki_querier_worker_concurrency metric * alter log line * wires up prometheus.registerer args to prevent duplicate registration errors on tests	3 years ago
Dylan Guedes	fb457e5fc6	Copy `cortex/pkg/querier` package dependency into Loki (#5122) ... * Fork cortex querier and its dependencies. * Use forked querier instead of cortex implementation. * Fix files wrongly using cortex spanlogger. * Fix file wrongly using cortex gateway. * Remove storegateway from vendor. * Fix test. * Use seriesFunc const to fix lint. * Fix linting issues.	3 years ago
Sashank Agarwal	7ff1b45260	remove split_queries_by_interval validation (#5125)	3 years ago
Dylan Guedes	634baa8aa8	Copy `cortex/pkg/querier/stats` package dependency into Loki (#5047)	3 years ago
Dylan Guedes	0e3df1938f	Copy querier subpackages dependency into Loki (#5068) ... * Fork cortex `querier/series`. * Fork cortex `querier/chunkstore`. * Fork cortex `querier/lazyquery`. * Use Loki forked cortex code. * Copy cortex `querier/queryrange` as `querier/queryrange/queryrangebase`. * Fork cortex `pkg/querier/queryrange` to `pkg/querier/queryrange/queryrangebase`. * Tell protoc to use gogo.types instead of pb.Any. * Use `pkg/querier/queryrange/queryrangebase`. * Remove cortex `queryrange` vendoring. * Fix lint. * Use `_ = ...` syntax instead of comment to fix lint.	3 years ago
Trevor Whitney	3091ccd3eb	improve default config values (#5077) ... * improve default config values * change defaults for upstream query range package * update changelog * remove max_tranfer_retries fix as it is covered in another PR, 4792 * increase query ingesters within by 1h This provides an additional buffer on top of the max_chunk_age. Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * add comment reminder to remove query range config hack Co-authored-by: Owen Diehl <ow.diehl@gmail.com> * rollback change to align_queries_with_step and parallelise_shardable_queries * add upgrading docs * re-enable parallelise_shardable_queries by default * add parallelise_shardable_queries back to upgrading doc Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Cyril Tovena	d6727398d0	Removes split-by upstream validation (#5087) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Cyril Tovena	6ef934b24c	Fix deadlock in disconnecting querier (#5063) ... * Fix deadlock in disconnecting querier Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/worker/worker_test.go Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> * Update pkg/querier/worker/worker_test.go Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> * Uses in-memory GRPC server for tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
Cyril Tovena	a7795e57b0	Fixes a possible cancellation issue (#5075) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Susana Ferreira	191a8a6f9d	Store metrics queueTime in seconds (#5052) ... * Store metrics queueTime in seconds * Address comments * Helper function to convert seconds to nanoseconds	3 years ago
Dylan Guedes	a4f218305d	Copy `cortex/pkg/frontend` package dependency into Loki (#5044) ... * Move `cortex/pkg/frontend` to `loki/pkg/lokifrontend/frontend`. * Update modules.txt with removed Cortex frontend. * Use `pkg/lokifrontend` instead of cortex `pkg/frontend`.	3 years ago
Susana Ferreira	82f8f3c1e5	Log time in queue per request (#4949) ... * Log time in queue per request * Add querierID to log message * Add unescape request url and tenant_ids to log message * Add query enqueue time to metrics.go * Remove "querier request dequeued" log from scheduler * Address comments * Rename enqueue_time to queue_time * Store queue time in nanoseconds (int64) * Use CanonicalMIMEHeaderKey for setting the httpgrpc header	3 years ago
Kaviraj	fe1d28a545	Chore: use `spanlogger` from `dskit` (#5029) ... * Move `cortex/pkg/spanlogger` into `loki`. It uses spanlogger from `dskit/pkg/spanlogger` with small wrapper created on the `pkg/util/spanlogger` in loki repo. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove `cortex/util/spanlogger` -> `loki/util/spanlogger` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter (with goimports) and `FromContext` Run goimports with `-local` flag `FromContext` takes util logger to keep the API same as currently used inside loki Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Dylan Guedes	07b583bb83	Copy `cortex/pkg/scheduler` package dependency into Loki (#4962) ... * Fork pkg/scheduler from cortex to Loki. * Replace cortex/pkg/scheduler with loki/pkg/scheduler. * Remove unused cortex subpackages from modules.txt. * Run `make format` to fix new loki/pkg/scheduler imports. * Update new scheduler .proto generated files.	3 years ago
Dylan Guedes	92c05493e1	Copy `cortex/pkg/distributor` package dependency into Loki (#4983) ... * Fork cortex `pkg/distributor/distributor_ring.go` Copy content from `pkg/distributor/distributor_ring.go` into Loki's `pkg/distributor/distributor_ring.go`. * Fork cortex `pkg/distributor/ingester_client_pool.go`. Copy Cortex `pkg/distributor/ingester_client_pool.go` into Loki's `pkg/distributor/clientpool/ingester_client_pool.go`. We couldn't copy it directly to `pkg/distributor/ingester_client_pool.go` because we distributor imports a package that imports the ingester_client_pool, which leads to import cycle. * Stop using Cortex `pkg/distributor`. Instead, use Loki's `pkg/distributor` and `pkg/distributor/clientpool`. * Remove unused Cortex distributor pkg. * Use random ports for Loki test.	3 years ago
Dylan Guedes	d5783e86a3	Copy `cortex/pkg/querier/astmapper` package dependency into Loki (#4982) ... * Fork cortex `pkg/querier/asmapper` into Loki. * Move away from cortex `pkg/querier/astmapper`. * Fix lint issues. - Use _ in argument of method `pkg/querier/astmapper/subtree_folder.go` since it is not being used - Remove unnecessary cast from `subtree_folder.go`	3 years ago
Susana Ferreira	2fdd24038b	Move querier.split-queries-by-interval to a per-tenant configuration (#5023)	3 years ago
Susana Ferreira	0c2ed01d92	Remove querier.query-ingesters-within check when sharding is enabled (#5024)	3 years ago
Sandeep Sukhani	b865afc0d7	fix splitting of queries when step is larger than split interval (#5006)	3 years ago
Karsten Jeschkies	a5c5f16938	Provide optional matchers to `LabelValuesForMetricName`. (#4933) ... * Provide optional matchers to `LabelValuesForMetricName`. * Pass all tests. * Clarify that we are using series IDs. * Use a set for filtering series ids. * Move matcher support to series. * Write changelog line. * Revert renaming. * Validate time range. * Reset base time. * Update pkg/storage/chunk/series_store.go Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> * Pre-allocate unique strings. Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944)	3 years ago
Christian Haudum	e8d1415725	Correctly use limits middleware in series query (#4929) ... * Revert "Skip middleware for unsupported type (#4924)" This reverts commit 64999ffdf9. * Use correct middlewares in query range roundtrippers This fixes a regression introduced with a47a76cdb5 that caused 500 errors on label name queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Update query range tripper middleware tests Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Dylan Guedes	64999ffdf9	Skip middleware for unsupported type (#4924) ... * Skip computing statistics for unsupported types. * Add tests for skipping statistics middleware. * Add missing CHANGELOG entry. * Implement wrapper for labelNames request and don't skip it. * Remove infinite-loop implementation for unsupported funcs. Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	a47a76cdb5	Fixes 500 when query is outside of max_query_lookback (#4902) ... * Fixes 500 when query is outside of max_query_lookback We were using cortex middleware to apply this limit, but when the max_query_lookback was enforced it was generating a reponse that is not known to Loki. And since Loki frontend support many other type of reponses (Series,Labels,...) I had to fork the code in Loki. I added a regression tests and also enforce the limit for labels API which was for some reason omitted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	dccbe51799	stats were logged without a log level making them not correctly filterable. (#4872)	4 years ago
Cyril Tovena	7f71c5f051	Promtail pull cloudflare logs (#4813)	4 years ago
Cyril Tovena	54dc737583	Reduce tracing spans (#4842) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Marco Pracucci	2b0d020146	Refactor pkg/util/server to avoid cortexpb import in downstream projects vendoring logql (#4833) ... Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Michel Hollands	e69ceefd84	Add LabelFilterer and Store wrapper (#4818) ... * Add LabelFilterer and Store wrapper Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Address review comments Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	4 years ago
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830) ... * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Kaviraj	1b63331e3d	Add support for `X-Query-Tags` (#4814) ... * Add support for `X-Query-Tags` 1. Now client can send any metadata about query via `X-Query-Tags` HTTP header 2. Also record `X-Query-Tags` in `metrics.go` for instant and range queries. * PR remarks * Make linter happy * Fix middleware next * Make safe parsing of the header * Add bit more test cases * Minor suggestions * Add middleware chain to querier handlers * Repopulate the HTTP headers before while Encoding	4 years ago
Christian Haudum	7f54641166	Set correct `Content-Type` header in query response (#4828) ... * Set correct `Content-Type` header in query response Because the internal query request handlers did not set the response content type explicitly to `application/json`, and the external request handler takes the response content type from the internal response to set it for the resulting HTTP response, that response was also incorrectly returning `Content-Type: text/plain`. Setting the content type in the internal response correctly results in the expected `Content-Type` header `application/json; charset=UTF-8`. Fixes #4791 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Set correct `Content-Type` header in query response Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Set correct `Content-Type` header in query response Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Sandeep Sukhani	8ddcc884fe	add logging in querier when live tailing request starts and ends (#4783) ... * add logging in querier when live tailing request starts and ends * Suggested changes Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Update pkg/querier/http.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	4 years ago
Kaviraj	94113ee080	Logql `stdin` support only log queries (#4606) ... * Hack stdin client into LogCLI Add flag to choose what kind of client to make request to. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Label filter working Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Demo checkpoint1 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Metrics query working. Issue was with default value of `step` (1 nanosecond). StepEvaluator try to go through every nanosecond to apply aggregate on sample data during the metric queries. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Basic tests for fileClient * Tests for logqueries direction * Use HeapIterator for Entries. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove some debug statements Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove support for metric queries. Stick with only log queries Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Small rough usage doc * Remove filesampleiterator * Fix some typos and tests * Fix breaking test cases Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks. 1. Add `--stdin` in examples of the usage 2. Add `--stdin` in all the command help output * PR remarks - Use parsed labels correctly - Fix indendation with --stdin flag * Fix issue with direction * Fix linter * MaxInt64 -> MaxInt (to support even arm32 images) * Add note on calculating `step` value on the client side	4 years ago
Ed Welch	88ef940ede	Loki: Set querier worker max concurrent regardless of run configuration. (#4761) ... * always set the querier worker MaxConcurrentRequests * removing a test which doesn't apply as this setting isn't being set in worker_service.go anymore	4 years ago
Cyril Tovena	d81f757ecb	Refactor and simplify stats package. (#4712) ... * Refactor and simplify stats package. This PR refactors the stats package and make it smaller and simpler to work with. - It doesn't expose the context which can causes race when mutating it. - Uses concurrency primitive to ensure correctness - Replace trailers with in-batch statistics to propagate ingesters stats. Fixes #3366 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix tests and lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * mod-check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	89ee022c21	Loki: fix handling of tail requests when using target `all` or `read` (#4642) ... * better handling of tail requests when the frontend and querier are running in the same process * adding function to determine if a module is enable based on the target supplied to Loki * fix tests and incorrect registering of tail routes * renamed function and made check for active recursive	4 years ago
Susana Ferreira	2d24e2ea64	Migration to dskit/ring (#4641)	4 years ago
Ed Welch	de4993f921	explicitly define allowed HTTP methods on HTTP endpoints (#4627)	4 years ago
Cyril Tovena	165ff31e2d	Fixes the Series function to hanlde properly sharding. (#4563) ... Before they would try to match the injected label but this label is only used by the store and not returned in the chunks causing all chunks to be filtered out. Fixes #4547 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Trevor Whitney	2864a4f728	enable virtual read write targets (#4498) ... Co-Authored-By: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Mauro Stettler <mauro.stettler@gmail.com>	4 years ago
Ed Welch	2b5f3000b6	Loki: Add a ring to the query scheduler to allow discovery via the ring as an alternative to DNS (#4424) ... * Fork the frontend and scheduler so we can add support for discovery via the ring. * Add a ring to the query scheduler which is then used by the frontend worker and querier workers to find the scheduler address as an alternative to using DNS * update some of the forked code to use dskit grpcclient and grpcutil since #4312 was merged after the original fork was created. * remove query scheduler dependency from querier Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * migrate logging package and make linter happy Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add SafeReadRing for instances when scheduler is not enabled on same instance * Doc changes from code review A few doc fixes Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * go mod tidy on go-kit dep * update changelog * remove ReadRing in favor of SafeReadRing Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484) ... * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Kaviraj	2e1279d996	Fix return values of Matrix and Vector during query range in QueryShardingMiddleware (#4457) ... * Fix matrics `result` type being `null` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add test for vector type as well Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Arve Knudsen	cdde2cf050	Chore: Use dskit/grpcclient (#4312)	4 years ago
Ed Welch	d5ec714ae0	Loki: Bug: frontend waiting on results which would never come (#4411) ... * We weren't also checking if the context was canceled while waiting for query results leading to goroutines waiting forever for results that would never come... * return error from context upstream if the context is canceled	4 years ago
JordanRushing	5e8a204480	Add recovery middleware to Ingester; re-add recovery middleware to Querier when not running in standalone mode (#4349) ... * Add recovery middleware to ingester; re-add recovery middleware to querier when not running in standalone mode * Re-add gRPC recovery middleware * Break out gRPC recovery middleware from setupAuthMiddleware() into new function setupGRPCRecoveryMiddleware()	4 years ago
Danny Kopping	d5b6d05557	CI: Fixing linter deprecations (#4353) ... * Fixing CI deprecations Replacing deprecated golint with revive (https://github.com/golangci/golangci-lint/pull/1965) Removing interfacer (https://github.com/golangci/golangci-lint/pull/1755) Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Further suggestions from Marco in https://github.com/grafana/loki/issues/1002 Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Clarify instruction Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago