From f39552c7fa4f7f173f85f953f33efe2095f77f50 Mon Sep 17 00:00:00 2001 From: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Date: Mon, 16 May 2022 11:17:11 +0100 Subject: [PATCH] Add docs for updated deletion api (#5763) * Add docs for updated deletion api Signed-off-by: Michel Hollands * Fix parameter name Signed-off-by: Michel Hollands * Clarify that only the delete requests for the current tenant are shown Signed-off-by: Michel Hollands * Update docs/sources/api/_index.md Co-authored-by: Christian Simon * Update docs/sources/operations/storage/logs-deletion.md Co-authored-by: Sandeep Sukhani * Update docs/sources/operations/storage/logs-deletion.md Co-authored-by: Sandeep Sukhani * Update docs/sources/operations/storage/logs-deletion.md Co-authored-by: Sandeep Sukhani * Specify all delete endpoints use authentication Signed-off-by: Michel Hollands * Add description of delete modes Signed-off-by: Michel Hollands * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Christian Simon Co-authored-by: Sandeep Sukhani Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> --- docs/sources/api/_index.md | 25 ++++++++++-- docs/sources/configuration/_index.md | 7 ++++ .../operations/storage/logs-deletion.md | 39 +++++++++++-------- 3 files changed, 51 insertions(+), 20 deletions(-) diff --git a/docs/sources/api/_index.md b/docs/sources/api/_index.md index 639300a562..abd5ee0376 100644 --- a/docs/sources/api/_index.md +++ b/docs/sources/api/_index.md @@ -70,6 +70,9 @@ These endpoints are exposed by the ruler: These endpoints are exposed by the compactor: - [`GET /compactor/ring`](#get-compactorring) +- [`POST /loki/api/v1/delete`](#post-lokiapiv1delete) +- [`GET /loki/api/v1/delete`](#get-lokiapiv1delete) +- [`DELETE /loki/api/v1/delete`](#delete-lokiapiv1delete) A [list of clients](../clients) can be found in the clients documentation. @@ -799,10 +802,6 @@ In microservices mode, the `/ingester/flush_shutdown` endpoint is exposed by the Displays a web page with the distributor hash ring status, including the state, healthy and last heartbeat time of each distributor. -### `GET /compactor/ring` - -Displays a web page with the compactor hash ring status, including the state, healthy and last heartbeat time of each compactor. - ## `GET /metrics` `/metrics` exposes Prometheus metrics. See @@ -1109,3 +1108,21 @@ GET /prometheus/api/v1/alerts Prometheus-compatible rules endpoint to list all active alerts. For more information, please check out the Prometheus [alerts](https://prometheus.io/docs/prometheus/latest/querying/api/#alerts) documentation. + +## Compactor + +### `GET /compactor/ring` + +Displays a web page with the compactor hash ring status, including the state, health, and last heartbeat time of each compactor. + +### `POST /loki/api/v1/delete` + +Create a new delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-log-entry-deletion). + +### `GET /loki/api/v1/delete` + +List the existing delete requests for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#list-delete-requests). + +### `DELETE /loki/api/v1/delete` + +Remove a delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-cancellation-of-a-delete-request). diff --git a/docs/sources/configuration/_index.md b/docs/sources/configuration/_index.md index 8eb1e5ce1f..85930e8682 100644 --- a/docs/sources/configuration/_index.md +++ b/docs/sources/configuration/_index.md @@ -2048,6 +2048,13 @@ compacts index shards to more performant forms. # CLI flag: -boltdb.shipper.compactor.max-compaction-parallelism [max_compaction_parallelism: | default = 1] +# Deletion mode. +# Can be one of "disabled", "whole-stream-deletion", "filter-only", or "filter-and-delete". +# When set to the default value of "whole-stream-deletion", and if +# retention_enabled is true, then the log entry deletion API endpoints are available. +# CLI flag: -boltdb.shipper.compactor.deletion-mode +[deletion_mode: | default = "whole-stream-deletion"] + # The hash ring configuration used by compactors to elect a single instance for running compactions # The CLI flags prefix for this block config is: boltdb.shipper.compactor.ring [compactor_ring: ] diff --git a/docs/sources/operations/storage/logs-deletion.md b/docs/sources/operations/storage/logs-deletion.md index fa78fa8086..710fe35a5b 100644 --- a/docs/sources/operations/storage/logs-deletion.md +++ b/docs/sources/operations/storage/logs-deletion.md @@ -4,10 +4,11 @@ weight: 60 --- # Log Entry Deletion -Log entry deletion is experimental. It is only supported for the BoltDB Shipper index store. +Log entry deletion is supported _only_ for the BoltDB Shipper index store. + +Grafana Loki supports the deletion of log entries from a specified stream. +Log entries that fall within a specified time window and match an optional line filter are those that will be deleted. -Grafana Loki supports the deletion of log entries from specified streams. -Log entries that fall within a specified time window are those that will be deleted. The Compactor component exposes REST endpoints that process delete requests. Hitting the endpoint specifies the streams and the time window. @@ -17,7 +18,12 @@ Log entry deletion relies on configuration of the custom logs retention workflow ## Configuration -Enable log entry deletion by setting `retention_enabled` to true in the Compactor's configuration. See the example in [Retention Configuration](../retention#retention-configuration). +Enable log entry deletion by setting `retention_enabled` to true and `deletion_enabled` to `whole-stream-deletion`, `filter-only`, or `filter-and-delete` in the compactor's configuration. See the example in [Retention configuration](../retention#retention-configuration). + +With `whole-stream-deletion`, all the log entries matching the query given in the delete request are removed. +With `filter-only`, log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from the on-disk chunks. +With `filter-and-delete`, log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from the on-disk chunks. + A delete request may be canceled within a configurable cancellation period. Set the `delete_request_cancel_period` in the Compactor's YAML configuration or on the command line when invoking Loki. Its default value is 24h. @@ -28,24 +34,26 @@ The Compactor exposes endpoints to allow for the deletion of log entries from sp ### Request log entry deletion ``` -POST /loki/api/admin/delete -PUT /loki/api/admin/delete +POST /loki/api/v1/delete +PUT /loki/api/v1/delete ``` Query parameters: -* `match[]=`: Repeated label matcher argument that identifies the streams from which to delete. At least one `match[]` argument must be provided. +* `query=`: query argument that identifies the streams from which to delete with optional line filters. * `start=`: A timestamp that identifies the start of the time window within which entries will be deleted. If not specified, defaults to 0, the Unix Epoch time. * `end=`: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time. A 204 response indicates success. -URL encode the `match[]` parameter. This sample form of a cURL command URL encodes `match[]={foo="bar"}`: +URL encode the `query` parameter. This sample form of a cURL command URL encodes `query={foo="bar"}`: ``` -curl -g -X POST \ - 'http://127.0.0.1:3100/loki/api/admin/delete?match[]={foo="bar"}&start=1591616227&end=1591619692' \ +curl -g -X POST \ + 'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}&start=1591616227&end=1591619692' \ -H 'x-scope-orgid: 1' + + The query parameter can also include filter operations. For example `query={foo="bar"} |= "other"` will filter out lines that contain the string "other" for the streams matching the stream selector `{foo="bar"}`. ``` ### List delete requests @@ -53,14 +61,14 @@ curl -g -X POST \ List the existing delete requests using the following API: ``` -GET /loki/api/admin/delete +GET /loki/api/v1/delete ``` Sample form of a cURL command: ``` curl -X GET \ - /loki/api/admin/delete \ + /loki/api/v1/delete \ -H 'x-scope-orgid: ' ``` @@ -73,8 +81,7 @@ Loki allows cancellation of delete requests until the requests are picked up for Cancel a delete request using this Compactor endpoint: ``` -POST /loki/api/admin/cancel_delete_request -PUT /loki/api/admin/cancel_delete_request +DELETE /loki/api/v1/delete ``` Query parameters: @@ -86,7 +93,7 @@ A 204 response indicates success. Sample form of a cURL command: ``` -curl -X POST \ - '/loki/api/admin/cancel_delete_request?request_id=' \ +curl -X DELETE \ + '/loki/api/v1/delete?request_id=' \ -H 'x-scope-orgid: ' ```