apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Sandeep Sukhani	0e5da205a6	structured metadata: track usage of structured metadata in queries (#11505 ) What this PR does / why we need it: Adds support for tracking queries referencing structured metadata. It tracks the following kinds of references: * Fields in structured metadata used as label filters, `drop`/`keep` stages etc. in the query. * Fields in structured metadata used in metric queries like `uwrap` and `by` or `without` clauses in aggregations. Checklist - [x] Tests updated	1 year ago
Vladyslav Diachenko	00a3c5b026	renamed `non indexed labels` feature to `structured metadata` (#10432 ) Loki team has made a decision to rename the feature because the new name is more precise. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Trevor Whitney	a05744a385	Enable results cache for volume queries (#10403 ) This PR enables the results cache for volume queries to the `index/volume` and `index/volume_range` endpoints	2 years ago
Danny Kopping	c725de6035	Querier: add chunk refs fetch time to metrics (#10362 ) What this PR does / why we need it: Adds an entry to the `metrics.go` log line to track how long a query waits for its chunk references to be returned by the index. This is useful when trying to understand where processing time is being spent.	2 years ago
Salva Corts	3f161f5c1a	Improve observability for non-indexed labels usage (#9993 ) What this PR does / why we need it: In https://github.com/grafana/loki/pull/9700, we support encoding and decoding metadata for each entry into the chunks. In this PR we: - Update the bytes processed stats to account for the bytes from those non-indexed labels - Add new stats for bytes processed for those non-indexed labels - Add new ingestion metrics to track ingested non-indexed bytes Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Ed Welch	fa6f9c638a	Loki: add 'post_filter_lines' stat for tracking how many lines are match a queries filter expression(s) (#9983 ) What this PR does / why we need it: To better understand query behavior and labeling strategies of users, logging the number of log lines "post filtering" can be very useful, we already log the total_lines processed in a query so this will allow us to see how many lines in the label selector matched what the query was looking for. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Salva Corts	6cc581bd26	Add back cache stats for index stats requests (#9816 ) What this PR does / why we need it: In https://github.com/grafana/loki/pull/9536, we added cache stats for index stats requests. That PR had a bug that inflated the query stats due to reusing the stats context in the query engine. Therefore, we had to revert the PR at https://github.com/grafana/loki/pull/9721. This PR brings back the changes from https://github.com/grafana/loki/pull/9536 but fixes the inflated starts by no longer reusing the same context in the query engine, but rather creating a new one for the shard resolver. I tested it on a dev cluster and seems to be working fine. here's the output for the same query: Stats with the bug from #9536: ``` ... Cache.StatsResult.Requests 980 Cache.StatsResult.EntriesRequested 490 Cache.StatsResult.EntriesFound 0 Cache.StatsResult.EntriesStored 490 Cache.StatsResult.BytesSent 0 B Cache.StatsResult.BytesReceived 0 B ... Summary.BytesProcessedPerSecond 43 GB Summary.LinesProcessedPerSecond 93305142 Summary.TotalBytesProcessed 945 GB Summary.TotalLinesProcessed 2059694183 ``` Stats from _main_ ``` ... Summary.BytesProcessedPerSecond 1.6 GB Summary.LinesProcessedPerSecond 3403718 Summary.TotalBytesProcessed 95 GB Summary.TotalLinesProcessed 207971404 ``` Stats with fix in this PR ``` .. Cache.StatsResult.Requests 132 Cache.StatsResult.EntriesRequested 66 Cache.StatsResult.EntriesFound 0 Cache.StatsResult.EntriesStored 66 Cache.StatsResult.BytesSent 0 B Cache.StatsResult.BytesReceived 0 B ... Summary.BytesProcessedPerSecond 4.3 GB Summary.LinesProcessedPerSecond 9468900 Summary.TotalBytesProcessed 95 GB Summary.TotalLinesProcessed 207793816 ``` As can be seen, with the changes in this PR, the summary stats are no longer inflated. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/pull/9536 Special notes for your reviewer: I think it's ok to skip reviewing the changes from the commit cherry-picking the changes from https://github.com/grafana/loki/pull/9536 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Salva Corts	b7359c5d53	Revert "Add summary stats and metrics for stats cache (#9536 )" (#9721 ) This reverts commit `af287ac3eb`. There is a bug in this PR that inflates the stats returned for the query since we reuse the stats ctx in the query execution engine.	2 years ago
Salva Corts	af287ac3eb	Add summary stats and metrics for stats cache (#9536 ) What this PR does / why we need it: When a query finishes, we return (and log) the following stats: ```go Cache.Chunk.Requests 0 Cache.Chunk.EntriesRequested 0 Cache.Chunk.EntriesFound 0 Cache.Chunk.EntriesStored 0 Cache.Chunk.BytesSent 0 B Cache.Chunk.BytesReceived 0 B Cache.Chunk.DownloadTime 0s Cache.Index.Requests 0 Cache.Index.EntriesRequested 0 Cache.Index.EntriesFound 0 Cache.Index.EntriesStored 0 Cache.Index.BytesSent 0 B Cache.Index.BytesReceived 0 B Cache.Index.DownloadTime 0s Cache.Result.Requests 13 Cache.Result.EntriesRequested 13 Cache.Result.EntriesFound 13 Cache.Result.EntriesStored 0 Cache.Result.BytesSent 0 B Cache.Result.BytesReceived 2.5 kB Cache.Result.DownloadTime 4.600266ms ``` In addition to that, we log the following in metrics.go: ``` level=info ts=2023-05-29T09:17:10.93029945Z caller=metrics.go:152 component=frontend org_id=145265 traceID=52d59b78fe6b9221 sampled=true latency=fast query="{cluster=\"dev-us-central-0\", namespace=~\"loki.\", container=~\"distributor\|ingester \|promtail\|index-gateway\|compactor\"} \|= \"thislinewillnotexist\"" query_hash=1194136170 query_type=filter range_type=range length=3h0m0s start_delta=165h37m24.930289434s end_delta=162h37m24.930289612s step=43s duration=2.473055ms status=200 lim it=30 returned_lines=0 throughput=0B total_bytes=0B lines_per_second=0 total_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=0s splits=13 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes _fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_result_req=13 cache_result_hit=13 cache_result_download_time=4.600266ms ``` With the goal of being able to better monitor how the stats cache is performing; this PR adds stats for the index stats cache, similarly to how it's done for the results cache. Here's an example of the new stats being returned and printed: ```go ... Cache.StatsResult.Requests 180 Cache.StatsResult.EntriesRequested 129 Cache.StatsResult.EntriesFound 129 Cache.StatsResult.EntriesStored 51 Cache.StatsResult.BytesSent 0 B Cache.StatsResult.BytesReceived 75 kB ... ``` And the new stats from metrics.go ``` ... caller=metrics.go:155 ... cache_stats_results_req=129 cache_stats_results_hit=129 cache_stats_results_download_ti me=156.864429ms ... ``` Special notes for your reviewer: - Blocked by https://github.com/grafana/loki/pull/9535 - Note the new`stats.GetOrCreateContext` func. It's used inside the `query.Exec` method so we don't overwrite the stats added in the stats middleware. Checklist* - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Paul Rogers	14370bb8ce	Revert "Augment statistics.." PR 9400. (#9430 ) What this PR does / why we need it: This PR reverts PR 9400. The data collected within that PR was not sufficient. When queries are done, they are filtered before the merge iterator, resulting in an inability to collect an accurate count of duplicated data. Which issue(s) this PR fixes: Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Paul Rogers	1671751cbd	Augment statistics to note how many bytes are in duplicate lines due to replicas (#9400 ) What this PR does / why we need it: This PR is for counting the number of bytes of log lines that were marked as duplicates. This will be utilized to collect better statistics.	2 years ago
Ed Welch	a4eb536fb2	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761 ) What this PR does / why we need it: Currently the `metrics.go` log line emitted after every query includes a metric called "subqueries". This currently tracks the number of queries created by the split_by_time operations done in Loki, but does not include any counts for subqueries created as a result of sharding. It becomes difficult to make a single subqueries counter that gives useful information to determine how much a query is split by time and sharded by a shard factor, especially now that sharding in TSDB indexes is dynamic. This PR removes and deprecates the `subqueries` stat and instead creates a `splits` and `shards` statistic which records how much a query was split_by_time and the total number of shards created as well. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Danny Kopping	a4f306399a	Add store & cache download statistics (#7982 )	2 years ago
Robert Fratto	85392a9728	Update Prometheus dependency to latest release (v2.40.4) (#7826 ) Closes #7811, which is needed for Grafana Agent to update to v2.40 and add support for native histograms. I did not add support for native histograms to Loki, sorry :)	3 years ago
Danny Kopping	36e0979cf5	Add cache usage statistics (#6317 ) * Adding cache statistics Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metrics to metrics.go Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Creating new stats context for use in metric queries middleware Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Clean up unnecessary log fields Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding stats tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * CHANGELOG entry Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Documenting function Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving CHANGELOG entry to appropriate section Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Implementing a stats collector cache wrapper to simplify stats collection If we keep the stats collection in pkg/storage/chunk/cache/instrumented.go, then any implementation that wraps it will cause the stats collected to be incomplete. For example: NewBackground(cacheName, cfg.Background, Instrument(cacheName, cache, reg), reg)) - the background cache requests are not collected Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	3 years ago
Kaviraj Kanagaraj	1389857846	`metrics.go` support for metadata queries(labels and series) (#5971 ) * Add different functions to record labels and series queries * Add tests for logging series and labels queries * Add statistics to the Series and Labels proto response types * Plug in the statistics recorder on the query handler * Remove duplicate imports and cleanup comments * Remove the need to add usage statistics for metadata queries. * Add status based on returned error from the api * Fixing lint error by fixin return values order * Use explicit logger * Remove unused recorders * Add `total_entries` representing length of total result returned * Inject statscollector middleware for metadata queries * Fix query frontend stats middleware Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests with change in extra `totalEntriesReturned` field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * `make check-generated-files` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tripperware middleware for label values endpoint Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix bug with label extraction Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog entry Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix `total_entries` value for range query Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Christian Simon	6c1a62d829	Ensure proto definitions are formatted [2/2] (#6000 ) * Update build image to 0.10.4 * Add a fmt target for proto defintions This uses buf to ensure the proto defintions are more consistently formatted. * Update formatted files	3 years ago
Cyril Tovena	9b6f17f168	Add the total number of subqueries to query statistics (#5397 ) * Add subqueries count Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Susana Ferreira	191a8a6f9d	Store metrics queueTime in seconds (#5052 ) * Store metrics queueTime in seconds * Address comments * Helper function to convert seconds to nanoseconds	3 years ago
Susana Ferreira	82f8f3c1e5	Log time in queue per request (#4949 ) * Log time in queue per request * Add querierID to log message * Add unescape request url and tenant_ids to log message * Add query enqueue time to metrics.go * Remove "querier request dequeued" log from scheduler * Address comments * Rename enqueue_time to queue_time * Store queue time in nanoseconds (int64) * Use CanonicalMIMEHeaderKey for setting the httpgrpc header	3 years ago
Cyril Tovena	d81f757ecb	Refactor and simplify stats package. (#4712 ) * Refactor and simplify stats package. This PR refactors the stats package and make it smaller and simpler to work with. - It doesn't expose the context which can causes race when mutating it. - Uses concurrency primitive to ensure correctness - Replace trailers with in-batch statistics to propagate ingesters stats. Fixes #3366 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix tests and lint issues. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes moar tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * mod-check Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Ed Welch	6e67d1ea5d	Loki/Promtail: Client Refactor (#3623 ) * move docker driver * move fluent-bit output plugin * move fluentd plugin * move logstash plugin * move promtail * match stage uses simple logql label matcher and filter expressions only * remove remaining logql imports * remove helpers package and move contents to util package * move validation and runtime out of util package * fix imports * move build and cfg packages into util * move ParseRequest function to util package * create a new package for logql models and move marshal/unmarshal code to util folder * move ErrorLabel to logqlmodel * move stats package to logqlmodel * decouple loghttp from logql by moving validation to logql package * move gcplog to tools * move PackedEntryKey constant to logqlmodel * fix stats proto import and rebuild protos * fix windows promtail tests * update drone to new dockerfile locations	4 years ago
Arve Knudsen	d975b1fdd8	Fix spelling of per second (#2055 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	5 years ago
Arve Knudsen	cc98d10d74	Chore: Correct typo "per seconds" (#2050 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	5 years ago
Cyril Tovena	9be21a7c6b	This logs queries with latency tag when recording stats. (#1733 ) * This logs queries with latency tag when recording stats. Useful for building a dashboard of slow queries. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * mod-check. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Cyril Tovena	93eb4ad27c	Add statistics to query_range and instant_query API. (#1615 ) * Refactor stats package to use proto. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge statistics in the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve engine logging. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes GRPC by rolling back Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add a test to verify stats in the LogQL engine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Set circle ci build image to 0.9.1 which contains protobuf regression Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix drone CI. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Change timing data from nanoseconds to seconds like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve naming of stats Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Move stats property under data property like Prometheus. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adjust queryrange frontend to use the new stats property Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Cleanup unused structs. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation for stats API. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Format proto file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds missing godoc on logql.Result Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago

22 Commits (0e5da205a68f77919f4f653eb6bccb61fca45aa7)