The API endpoints starting with `/loki/` are [Prometheus API-compatible](https://prometheus.io/docs/prometheus/latest/querying/api/) and the result formats can be used interchangeably.
@ -102,7 +102,11 @@ The API accepts several formats for timestamps. An integer with ten or fewer dig
The timestamps can also be written in `RFC3339` and `RFC3339Nano` format, as supported by Go's [time](https://pkg.go.dev/time) package.
## `GET /loki/api/v1/query`
## Query Loki
```
GET /loki/api/v1/query
```
`/loki/api/v1/query` allows for doing queries against a single point in time. The URL
Alternatively, if the `Content-Type` header is set to `application/json`, a
JSON post body can be sent in the following format:
- **New**: Service is new, not running yet (initial state)
- **Starting**: Service is starting; if starting succeeds, service enters **Running** state
- **Running**: Service is fully running now; when service stops running, it enters **Stopping** state
- **Stopping**: Service is shutting down
- **Terminated**: Service has stopped successfully (terminal state)
- **Failed**: Service has failed in **Starting**, **Running** or **Stopping** state (terminal state)
## List build information
```
{
"streams": [
{
"labels": "<LogQLlabelkey-valuepairs>",
"entries": [
{
"ts": "<RFC3339Nanostring>",
"line": "<logline>"
}
]
}
]
}
GET /loki/api/v1/status/buildinfo
```
Loki can be configured to [accept out-of-order writes](../configuration/#accept-out-of-order-writes).
`/loki/api/v1/status/buildinfo` exposes the build information in a JSON object. The fields are `version`, `revision`, `branch`, `buildDate`, `buildUser`, and `goVersion`.
In microservices mode, `/api/prom/push` is exposed by the distributor.
This endpoint returns the list of time series that match a certain label set.
## `GET /ready`
URL query parameters:
`/ready` returns HTTP 200 when the Loki ingester is ready to accept traffic. If
running Loki on Kubernetes, `/ready` can be used as a readiness probe.
- `match[]=<series_selector>`: Repeated log stream selector argument that selects the streams to return. At least one `match[]` argument must be provided.
In microservices mode, the `/ready` endpoint is exposed by all components.
You can URL-encode these parameters directly in the request body by using the POST method and `Content-Type: application/x-www-form-urlencoded` header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits.
## `POST /flush`
In microservices mode, these endpoints are exposed by the querier.
`/flush` triggers a flush of all in-memory chunks held by the ingesters to the
backing store. Mainly used for local testing.
In microservices mode, the `/flush` endpoint is exposed by the ingester.
## `POST /ingester/flush_shutdown`
**Deprecated**: Please use `/ingester/shutdown?flush=true` instead.
`/ingester/flush_shutdown` triggers a shutdown of the ingester and notably will _always_ flush any in memory chunks it holds.
This is helpful for scaling down WAL-enabled ingesters where we want to ensure old WAL directories are not orphaned,
but instead flushed to our chunk backend.
In microservices mode, the `/ingester/flush_shutdown` endpoint is exposed by the ingester.
## `POST /ingester/shutdown`
`/ingester/shutdown` is similar to the [`/ingester/flush_shutdown`](#post-ingesterflush_shutdown)
endpoint, but accepts three URL query parameters `flush`, `delete_ring_tokens`, and `terminate`.
**URL query parameters:**
* `flush=<bool>`:
Flag to control whether to flush any in-memory chunks the ingester holds. Defaults to `true`.
* `delete_ring_tokens=<bool>`:
Flag to control whether to delete the file that contains the ingester ring tokens of the instance if the `-ingester.token-file-path` is specified.
* `terminate=<bool>`:
Flag to control whether to terminate the Loki process after service shutdown. Defaults to `true`.
This handler, in contrast to the `/ingester/flush_shutdown` handler, terminates the Loki process by default.
This behaviour can be changed by setting the `terminate` query parameter to `false`.
In microservices mode, the `/ingester/shutdown` endpoint is exposed by the ingester.
## `GET /distributor/ring`
Displays a web page with the distributor hash ring status, including the state, healthy and last heartbeat time of each distributor.
## `GET /metrics`
`/metrics` exposes Prometheus metrics. See
[Observing Loki](../operations/observability/)
for a list of exported metrics.
In microservices mode, the `/metrics` endpoint is exposed by all components.
## `GET /config`
`/config` exposes the current configuration. The optional `mode` query parameter can be used to
modify the output. If it has the value `diff` only the differences between the default configuration
and the current are returned. A value of `defaults` returns the default configuration.
In microservices mode, the `/config` endpoint is exposed by all components.
## `GET /services`
`/services` returns a list of all running services and their current states.
Services can have the following states:
- **New**: Service is new, not running yet (initial state)
- **Starting**: Service is starting; if starting succeeds, service enters **Running** state
- **Running**: Service is fully running now; when service stops running, it enters **Stopping** state
- **Stopping**: Service is shutting down
- **Terminated**: Service has stopped successfully (terminal state)
- **Failed**: Service has failed in **Starting**, **Running** or **Stopping** state (terminal state)
## `GET /loki/api/v1/status/buildinfo`
`/loki/api/v1/status/buildinfo` exposes the build information in a JSON object. The fields are `version`, `revision`, `branch`, `buildDate`, `buildUser`, and `goVersion`.
## Series
The Series API is available under the following:
- `GET /loki/api/v1/series`
- `POST /loki/api/v1/series`
- `GET /api/prom/series`
- `POST /api/prom/series`
This endpoint returns the list of time series that match a certain label set.
URL query parameters:
- `match[]=<series_selector>`: Repeated log stream selector argument that selects the streams to return. At least one `match[]` argument must be provided.
You can URL-encode these parameters directly in the request body by using the POST method and `Content-Type: application/x-www-form-urlencoded` header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits.
In microservices mode, these endpoints are exposed by the querier.
@ -1180,18 +998,343 @@ For more information, please check out the Prometheus [alerts](https://prometheu
## Compactor
### `GET /compactor/ring`
### Compactor ring status
```
GET /compactor/ring
```
Displays a web page with the compactor hash ring status, including the state, health, and last heartbeat time of each compactor.
### `POST /loki/api/v1/delete`
### Request log deletion
```
POST /loki/api/v1/delete
PUT /loki/api/v1/delete
```
Create a new delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-log-entry-deletion).
### `GET /loki/api/v1/delete`
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
Query parameters:
* `query=<series_selector>`: query argument that identifies the streams from which to delete with optional line filters.
* `start=<rfc3339 | unix_timestamp>`: A timestamp that identifies the start of the time window within which entries will be deleted. If not specified, defaults to 0, the Unix Epoch time.
* `end=<rfc3339 | unix_timestamp>`: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time.
A 204 response indicates success.
URL encode the `query` parameter. This sample form of a cURL command URL encodes `query={foo="bar"}`:
The query parameter can also include filter operations. For example `query={foo="bar"} |= "other"` will filter out lines that contain the string "other" for the streams matching the stream selector `{foo="bar"}`.
### List log deletion requests
```
GET /loki/api/v1/delete
```
List the existing delete requests for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#list-delete-requests).
### `DELETE /loki/api/v1/delete`
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
List the existing delete requests using the following API:
```
GET /loki/api/v1/delete
```
Sample form of a cURL command:
```
curl -X GET \
<compactor_addr>/loki/api/v1/delete \
-H 'x-scope-orgid: <orgid>'
```
This endpoint returns both processed and unprocessed requests. It does not list canceled requests, as those requests will have been removed from storage.
### Request cancellation of a delete request
```
DELETE /loki/api/v1/delete
```
Remove a delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-cancellation-of-a-delete-request).
Loki allows cancellation of delete requests until the requests are picked up for processing. It is controlled by the `delete_request_cancel_period` YAML configuration or the equivalent command line option when invoking Loki.
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
Cancel a delete request using this compactor endpoint:
```
DELETE /loki/api/v1/delete
```
Query parameters:
* `request_id=<request_id>`: Identifies the delete request to cancel; IDs are found using the `delete` endpoint.
Log entry deletion is supported _only_ for the BoltDB Shipper index store.
Grafana Loki supports the deletion of log entries from a specified stream.
Log entries that fall within a specified time window and match an optional line filter are those that will be deleted.
The Compactor component exposes REST endpoints that process delete requests.
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
The compactor component exposes REST [endpoints](../../../api/#compactor) that process delete requests.
Hitting the endpoint specifies the streams and the time window.
The deletion of the log entries takes place after a configurable cancellation time period expires.
Log entry deletion relies on configuration of the custom logs retention workflow as defined in [Compactor](../retention#compactor). The Compactor looks at unprocessed requests which are past their cancellation period to decide whether a chunk is to be deleted or not.
Log entry deletion relies on configuration of the custom logs retention workflow as defined for the [compactor](../retention#compactor). The compactor looks at unprocessed requests which are past their cancellation period to decide whether a chunk is to be deleted or not.
## Configuration
Enable log entry deletion by setting `retention_enabled` to true and `deletion_mode` to `filter-only` or `filter-and-delete` in the compactor's configuration.
With `filter-only`, log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from the on-disk chunks.
With `filter-and-delete`, log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from the on-disk chunks.
With `filter-only`, log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from storage.
With `filter-and-delete`, log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from storage.
A delete request may be canceled within a configurable cancellation period. Set the `delete_request_cancel_period` in the Compactor's YAML configuration or on the command line when invoking Loki. Its default value is 24h.
A delete request may be canceled within a configurable cancellation period. Set the `delete_request_cancel_period` in the compactor's YAML configuration or on the command line when invoking Loki. Its default value is 24h.
Access to the deletion API can be enabled per tenant via the `allow_deletes` setting.
## Compactor endpoints
The Compactor exposes endpoints to allow for the deletion of log entries from specified streams.
### Request log entry deletion
```
POST /loki/api/v1/delete
PUT /loki/api/v1/delete
```
Query parameters:
* `query=<series_selector>`: query argument that identifies the streams from which to delete with optional line filters.
* `start=<rfc3339 | unix_timestamp>`: A timestamp that identifies the start of the time window within which entries will be deleted. If not specified, defaults to 0, the Unix Epoch time.
* `end=<rfc3339 | unix_timestamp>`: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time.
A 204 response indicates success.
URL encode the `query` parameter. This sample form of a cURL command URL encodes `query={foo="bar"}`:
The query parameter can also include filter operations. For example `query={foo="bar"} |= "other"` will filter out lines that contain the string "other" for the streams matching the stream selector `{foo="bar"}`.
```
### List delete requests
List the existing delete requests using the following API:
```
GET /loki/api/v1/delete
```
Sample form of a cURL command:
```
curl -X GET \
<compactor_addr>/loki/api/v1/delete \
-H 'x-scope-orgid: <orgid>'
```
This endpoint returns both processed and unprocessed requests. It does not list canceled requests, as those requests will have been removed from storage.
### Request cancellation of a delete request
Loki allows cancellation of delete requests until the requests are picked up for processing. It is controlled by the `delete_request_cancel_period` YAML configuration or the equivalent command line option when invoking Loki.
Cancel a delete request using this Compactor endpoint:
```
DELETE /loki/api/v1/delete
```
Query parameters:
* `request_id=<request_id>`: Identifies the delete request to cancel; IDs are found using the `delete` endpoint.
Karen Miller
3 years ago
committed by