34 Commits (f04d0db43457353dddc71239808583d1c9cbfdbd)

Author	SHA1	Message	Date
Kaviraj Kanagaraj	ce574485cf	feat(caching): Support caching `/series` and `/labels` query results (#11539) ... **What this PR does / why we need it**: Add support for caching metadata queries (both series and labels). caching happens after splitting similar to other types of queries. This pr adds the following configs to enable them. ``` cache_series_results: true|false (default false) cache_label_results: true|false (default false) ``` And the cache backend for them can be configured using `series_results_cache` and `label_results_cache` blocks under the `query_range` section. Currently the split interval for metadata queries is fixed and defaults to 24h, this pr makes it configurable by introducing `split_metadata_queries_by_interval` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Karsten Jeschkies	5b97fcfd93	Send query plan to querier. (#11246) ... **What this PR does / why we need it**: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Kaviraj Kanagaraj	30d0030bf9	inflight-logging: Add extra metadata to inflight requests logging (#11243) ... **What this PR does / why we need it**: logging: Add extra metadata to inflight requests This adds extra metadata (similar to what we have in `metrics.go`) but for queries in in-flight (both started and retrying) Changes: Adds following data 1. Query Hash 2. Start and End time 3. Start and End delta 4. Length of the query 5. Moved the helper util to `queryutil` package because of cyclic dependencies with `logql` package. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: Find the screenshots of log entries looks like (both in `retry.go` and `roundtrip.go`)   **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Danny Kopping	34b9b9a11f	Improve observability of index queries (#11064) ... **What this PR does / why we need it**: Requests to the following APIs previously had limited or incorrect logging, which made understanding their behaviour at runtime difficult. [GET /loki/api/v1/labels](https://grafana.com/docs/loki/latest/reference/api/#list-labels-within-a-range-of-time) [GET /loki/api/v1/label/<name>/values](https://grafana.com/docs/loki/latest/reference/api/#list-label-values-within-a-range-of-time) [GET /loki/api/v1/series](https://grafana.com/docs/loki/latest/reference/api/#list-series) [GET /loki/api/v1/index/stats](https://grafana.com/docs/loki/latest/reference/api/#index-stats) [GET /loki/api/v1/index/volume](https://grafana.com/docs/loki/latest/reference/api/#volume) [GET /loki/api/v1/index/volume_range](https://grafana.com/docs/loki/latest/reference/api/#volume) All of these APIs now have querier and query-frontend logs; sharding and time-based splitting are applied to these requests, and it's valuable to see all requests.	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164) ... **What this PR does / why we need it**: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. **Which issue(s) this PR fixes**: (none) **Special notes for your reviewer**: (none) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Ed Welch	23a189990c	Loki: Tweak how we handle X-Query-Tags header (#10058) ... **What this PR does / why we need it**: Adjust the regex for X-Query-Tags to allow @ and . so that email addresses can be displayed properly. Also changed the ToLower to only apply to the key not the value **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	67c0b3ed2c	Loki: log all trace ID's but include `sampled=true` when the trace is logged (#9039) ... **What this PR does / why we need it**: Currently when we log "with context" we will only add the traceID if the particular trace is being sampled. This is nice because it makes it obvious that if you go search for a trace you'll find it. However, traceID is also useful for selecting all the log lines from a single query within Loki itself and if the trace wasn't sampled we remove this useful piece of context for querying the log lines. In this PR, instead of dropping the traceID from the logs, we instead prefix it with `NS-` Where NS is short for Not Sampled Example: ``` level=debug ts=2023-04-05T15:06:24.377067817Z caller=fetcher.go:292 org_id=145265 traceID=NS-13ed6c30228e221d chunks=1 decodeRequests=1 missing=0 ``` Questions: * should we bikeshed a little on this prefix? (I wanted something concise but if it's concise it's probably confusing to folks who've never seen it before) * Instead of a prefix, we could change the logfmt key? `ID=13ed6c30228e221d` (this is less confusing in some ways but maybe more difficult/confusing in others?) **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Periklis Tsirakidis	1bcf683513	Expose optional label matcher for label values handler (#8824)	3 years ago
Ed Welch	a4eb536fb2	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761) ... **What this PR does / why we need it**: Currently the `metrics.go` log line emitted after every query includes a metric called "subqueries". This currently tracks the number of queries created by the split_by_time operations done in Loki, but does not include any counts for subqueries created as a result of sharding. It becomes difficult to make a single subqueries counter that gives useful information to determine how much a query is split by time and sharded by a shard factor, especially now that sharding in TSDB indexes is dynamic. This PR removes and deprecates the `subqueries` stat and instead creates a `splits` and `shards` statistic which records how much a query was split_by_time and the total number of shards created as well. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Ed Welch	35510ba4eb	Loki: only log "executing query" once per query in the frontend (#8337) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Dylan Guedes	797d4bce45	Loki: Log hashed queries emitted by `metrics.go` (#8244) ... - What this PR does / why we need it: * Adds a new `query_hash` label pair to queries logged by `metrics.go` and our query engine. As the name suggests, `query_hash` is just the hashed version of the query being logged. * Especially useful when grouping results/behavior of different queries. Ex, you can easily find specific query patterns with `sum by(query_hash) (...)` .	3 years ago
Danny Kopping	a4f306399a	Add store & cache download statistics (#7982)	3 years ago
Danny Kopping	fa42c12f4c	Adding "start_delta" entry to metrics.go line (#6550) ... * Adding "start_delta" entry to metrics.go line This will help us determine if a 3h query (derived from "length") is for some random 3h period in the past, or within the last 3h Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter; for once it was actually useful Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding "end_delta" as well Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing rounding Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Danny Kopping	36e0979cf5	Add cache usage statistics (#6317) ... * Adding cache statistics Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metrics to metrics.go Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Creating new stats context for use in metric queries middleware Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Clean up unnecessary log fields Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding stats tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * CHANGELOG entry Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Documenting function Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving CHANGELOG entry to appropriate section Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Implementing a stats collector cache wrapper to simplify stats collection If we keep the stats collection in pkg/storage/chunk/cache/instrumented.go, then any implementation that wraps it will cause the stats collected to be incomplete. For example: NewBackground(cacheName, cfg.Background, Instrument(cacheName, cache, reg), reg)) - the background cache requests are not collected Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Kaviraj Kanagaraj	1389857846	`metrics.go` support for metadata queries(labels and series) (#5971) ... * Add different functions to record labels and series queries * Add tests for logging series and labels queries * Add statistics to the Series and Labels proto response types * Plug in the statistics recorder on the query handler * Remove duplicate imports and cleanup comments * Remove the need to add usage statistics for metadata queries. * Add status based on returned error from the api * Fixing lint error by fixin return values order * Use explicit logger * Remove unused recorders * Add `total_entries` representing length of total result returned * Inject statscollector middleware for metadata queries * Fix query frontend stats middleware Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests with change in extra `totalEntriesReturned` field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * `make check-generated-files` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tripperware middleware for label values endpoint Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix bug with label extraction Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog entry Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix `total_entries` value for range query Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Cyril Tovena	9b6f17f168	Add the total number of subqueries to query statistics (#5397) ... * Add subqueries count Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	b3af2dbeef	Loki: include the component in metrics.go log line (#5379) ... * add a component key to our metrics.go line Signed-off-by: Edward Welch <edward.welch@grafana.com> * new computer... who dis Signed-off-by: Edward Welch <edward.welch@grafana.com> * one more lint Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187) ... * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Susana Ferreira	191a8a6f9d	Store metrics queueTime in seconds (#5052) ... * Store metrics queueTime in seconds * Address comments * Helper function to convert seconds to nanoseconds	4 years ago
Susana Ferreira	82f8f3c1e5	Log time in queue per request (#4949) ... * Log time in queue per request * Add querierID to log message * Add unescape request url and tenant_ids to log message * Add query enqueue time to metrics.go * Remove "querier request dequeued" log from scheduler * Address comments * Rename enqueue_time to queue_time * Store queue time in nanoseconds (int64) * Use CanonicalMIMEHeaderKey for setting the httpgrpc header	4 years ago
Marco Pracucci	2b0d020146	Refactor pkg/util/server to avoid cortexpb import in downstream projects vendoring logql (#4833) ... Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Kaviraj	1b63331e3d	Add support for `X-Query-Tags` (#4814) ... * Add support for `X-Query-Tags` 1. Now client can send any metadata about query via `X-Query-Tags` HTTP header 2. Also record `X-Query-Tags` in `metrics.go` for instant and range queries. * PR remarks * Make linter happy * Fix middleware next * Make safe parsing of the header * Add bit more test cases * Minor suggestions * Add middleware chain to querier handlers * Repopulate the HTTP headers before while Encoding	4 years ago
Danny Kopping	cff8324f24	Replacing go-kit/kit/log with go-kit/log (#4484) ... * Replacing go-kit/kit/log with go-kit/log Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the f&^$ing linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding depguard Cleaning go mod Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Updating weaveworks after https://github.com/weaveworks/common/pull/226 was merged Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the mod checker Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623) ... * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	5 years ago
Cyril Tovena	a29901d33a	Add limit and line_returned in the query log. (#3423) ... * Add limit and line_returned in the query log. This can be useful to investigate how many wasted queries have been made by the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * got linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Jacob Lisi	923671a17e	chore: update cortex to latest and fix refs (#3295) ... * chore: update cortex to latest and fix refs Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * tidy mod file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	5 years ago
Cyril Tovena	5738611d64	Fixes logql.QueryType. (#2913) ... The bug was introduced with LogQL extensions. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Karsten Jeschkies	55c768bf8e	Log throughput and total bytes human readable. (#2751)	5 years ago
Cyril Tovena	c81be6603e	Fixes unit in stats request log. (#2093) ... * Fix unit conversion in stats log. 🤦 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix the tests because you know ! you know ? Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Arve Knudsen	d975b1fdd8	Fix spelling of per second (#2055) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	6 years ago
Cyril Tovena	47b79f3ea5	Update cortex to latest. (#1982) ... * Update cortex to latest. This brings fixes for traceID in the logger and swift. I've also added a bit of configuration for swift. Fixes https://github.com/grafana/loki/issues/905 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update go.sum Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	9be21a7c6b	This logs queries with latency tag when recording stats. (#1733) ... * This logs queries with latency tag when recording stats. Useful for building a dashboard of slow queries. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * mod-check. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	cdd08b041a	Expose QueryType function. (#1719) ... * Expose QueryType function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Expose query type const. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago
Cyril Tovena	2af69cb57c	Frontend & Querier query statistics instrumentation. (#1661) ... * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip stats frontend Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Updates cortex and add a record metric function. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some tests in the mix Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes unused tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Replace global by an injected interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds a test to make sure the stats middleware update response statistics. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago