The API endpoints starting with `/loki/` are [Prometheus API-compatible](https://prometheus.io/docs/prometheus/latest/querying/api/) and the result formats can be used interchangeably.
The API endpoints starting with `/loki/` are [Prometheus API-compatible](https://prometheus.io/docs/prometheus/latest/querying/api/) and the result formats can be used interchangeably.
@ -102,7 +102,11 @@ The API accepts several formats for timestamps. An integer with ten or fewer dig
The timestamps can also be written in `RFC3339` and `RFC3339Nano` format, as supported by Go's [time](https://pkg.go.dev/time) package.
The timestamps can also be written in `RFC3339` and `RFC3339Nano` format, as supported by Go's [time](https://pkg.go.dev/time) package.
## `GET /loki/api/v1/query`
## Query Loki
```
GET /loki/api/v1/query
```
`/loki/api/v1/query` allows for doing queries against a single point in time. The URL
`/loki/api/v1/query` allows for doing queries against a single point in time. The URL
Alternatively, if the `Content-Type` header is set to `application/json`, a
- **New**: Service is new, not running yet (initial state)
JSON post body can be sent in the following format:
- **Starting**: Service is starting; if starting succeeds, service enters **Running** state
- **Running**: Service is fully running now; when service stops running, it enters **Stopping** state
- **Stopping**: Service is shutting down
- **Terminated**: Service has stopped successfully (terminal state)
- **Failed**: Service has failed in **Starting**, **Running** or **Stopping** state (terminal state)
## List build information
```
```
{
GET /loki/api/v1/status/buildinfo
"streams": [
{
"labels": "<LogQLlabelkey-valuepairs>",
"entries": [
{
"ts": "<RFC3339Nanostring>",
"line": "<logline>"
}
]
}
]
}
```
```
Loki can be configured to [accept out-of-order writes](../configuration/#accept-out-of-order-writes).
`/loki/api/v1/status/buildinfo` exposes the build information in a JSON object. The fields are `version`, `revision`, `branch`, `buildDate`, `buildUser`, and `goVersion`.
In microservices mode, `/api/prom/push` is exposed by the distributor.
## List series
### Examples
The Series API is available under the following:
- `GET /loki/api/v1/series`
- `POST /loki/api/v1/series`
- `GET /api/prom/series`
- `POST /api/prom/series`
```bash
This endpoint returns the list of time series that match a certain label set.
`/ready` returns HTTP 200 when the Loki ingester is ready to accept traffic. If
- `match[]=<series_selector>`: Repeated log stream selector argument that selects the streams to return. At least one `match[]` argument must be provided.
running Loki on Kubernetes, `/ready` can be used as a readiness probe.
In microservices mode, the `/ready` endpoint is exposed by all components.
You can URL-encode these parameters directly in the request body by using the POST method and `Content-Type: application/x-www-form-urlencoded` header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits.
## `POST /flush`
In microservices mode, these endpoints are exposed by the querier.
`/flush` triggers a flush of all in-memory chunks held by the ingesters to the
### Examples
backing store. Mainly used for local testing.
In microservices mode, the `/flush` endpoint is exposed by the ingester.
## `POST /ingester/flush_shutdown`
**Deprecated**: Please use `/ingester/shutdown?flush=true` instead.
`/ingester/flush_shutdown` triggers a shutdown of the ingester and notably will _always_ flush any in memory chunks it holds.
This is helpful for scaling down WAL-enabled ingesters where we want to ensure old WAL directories are not orphaned,
but instead flushed to our chunk backend.
In microservices mode, the `/ingester/flush_shutdown` endpoint is exposed by the ingester.
## `POST /ingester/shutdown`
`/ingester/shutdown` is similar to the [`/ingester/flush_shutdown`](#post-ingesterflush_shutdown)
endpoint, but accepts three URL query parameters `flush`, `delete_ring_tokens`, and `terminate`.
**URL query parameters:**
* `flush=<bool>`:
Flag to control whether to flush any in-memory chunks the ingester holds. Defaults to `true`.
* `delete_ring_tokens=<bool>`:
Flag to control whether to delete the file that contains the ingester ring tokens of the instance if the `-ingester.token-file-path` is specified.
* `terminate=<bool>`:
Flag to control whether to terminate the Loki process after service shutdown. Defaults to `true`.
This handler, in contrast to the `/ingester/flush_shutdown` handler, terminates the Loki process by default.
This behaviour can be changed by setting the `terminate` query parameter to `false`.
In microservices mode, the `/ingester/shutdown` endpoint is exposed by the ingester.
## `GET /distributor/ring`
Displays a web page with the distributor hash ring status, including the state, healthy and last heartbeat time of each distributor.
## `GET /metrics`
`/metrics` exposes Prometheus metrics. See
[Observing Loki](../operations/observability/)
for a list of exported metrics.
In microservices mode, the `/metrics` endpoint is exposed by all components.
## `GET /config`
`/config` exposes the current configuration. The optional `mode` query parameter can be used to
modify the output. If it has the value `diff` only the differences between the default configuration
and the current are returned. A value of `defaults` returns the default configuration.
In microservices mode, the `/config` endpoint is exposed by all components.
## `GET /services`
`/services` returns a list of all running services and their current states.
Services can have the following states:
- **New**: Service is new, not running yet (initial state)
- **Starting**: Service is starting; if starting succeeds, service enters **Running** state
- **Running**: Service is fully running now; when service stops running, it enters **Stopping** state
- **Stopping**: Service is shutting down
- **Terminated**: Service has stopped successfully (terminal state)
- **Failed**: Service has failed in **Starting**, **Running** or **Stopping** state (terminal state)
## `GET /loki/api/v1/status/buildinfo`
`/loki/api/v1/status/buildinfo` exposes the build information in a JSON object. The fields are `version`, `revision`, `branch`, `buildDate`, `buildUser`, and `goVersion`.
## Series
The Series API is available under the following:
- `GET /loki/api/v1/series`
- `POST /loki/api/v1/series`
- `GET /api/prom/series`
- `POST /api/prom/series`
This endpoint returns the list of time series that match a certain label set.
URL query parameters:
- `match[]=<series_selector>`: Repeated log stream selector argument that selects the streams to return. At least one `match[]` argument must be provided.
You can URL-encode these parameters directly in the request body by using the POST method and `Content-Type: application/x-www-form-urlencoded` header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits.
In microservices mode, these endpoints are exposed by the querier.
@ -1180,18 +998,343 @@ For more information, please check out the Prometheus [alerts](https://prometheu
## Compactor
## Compactor
### `GET /compactor/ring`
### Compactor ring status
```
GET /compactor/ring
```
Displays a web page with the compactor hash ring status, including the state, health, and last heartbeat time of each compactor.
Displays a web page with the compactor hash ring status, including the state, health, and last heartbeat time of each compactor.
### `POST /loki/api/v1/delete`
### Request log deletion
```
POST /loki/api/v1/delete
PUT /loki/api/v1/delete
```
Create a new delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-log-entry-deletion).
Create a new delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-log-entry-deletion).
### `GET /loki/api/v1/delete`
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
Query parameters:
* `query=<series_selector>`: query argument that identifies the streams from which to delete with optional line filters.
* `start=<rfc3339 | unix_timestamp>`: A timestamp that identifies the start of the time window within which entries will be deleted. If not specified, defaults to 0, the Unix Epoch time.
* `end=<rfc3339 | unix_timestamp>`: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time.
A 204 response indicates success.
URL encode the `query` parameter. This sample form of a cURL command URL encodes `query={foo="bar"}`:
The query parameter can also include filter operations. For example `query={foo="bar"} |= "other"` will filter out lines that contain the string "other" for the streams matching the stream selector `{foo="bar"}`.
### List log deletion requests
```
GET /loki/api/v1/delete
```
List the existing delete requests for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#list-delete-requests).
List the existing delete requests for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#list-delete-requests).
### `DELETE /loki/api/v1/delete`
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
List the existing delete requests using the following API:
```
GET /loki/api/v1/delete
```
Sample form of a cURL command:
```
curl -X GET \
<compactor_addr>/loki/api/v1/delete \
-H 'x-scope-orgid: <orgid>'
```
This endpoint returns both processed and unprocessed requests. It does not list canceled requests, as those requests will have been removed from storage.
### Request cancellation of a delete request
```
DELETE /loki/api/v1/delete
```
Remove a delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-cancellation-of-a-delete-request).
Remove a delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-cancellation-of-a-delete-request).
Loki allows cancellation of delete requests until the requests are picked up for processing. It is controlled by the `delete_request_cancel_period` YAML configuration or the equivalent command line option when invoking Loki.
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
Cancel a delete request using this compactor endpoint:
```
DELETE /loki/api/v1/delete
```
Query parameters:
* `request_id=<request_id>`: Identifies the delete request to cancel; IDs are found using the `delete` endpoint.
Log entry deletion is supported _only_ for the BoltDB Shipper index store.
Grafana Loki supports the deletion of log entries from a specified stream.
Grafana Loki supports the deletion of log entries from a specified stream.
Log entries that fall within a specified time window and match an optional line filter are those that will be deleted.
Log entries that fall within a specified time window and match an optional line filter are those that will be deleted.
The Compactor component exposes REST endpoints that process delete requests.
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
The compactor component exposes REST [endpoints](../../../api/#compactor) that process delete requests.
Hitting the endpoint specifies the streams and the time window.
Hitting the endpoint specifies the streams and the time window.
The deletion of the log entries takes place after a configurable cancellation time period expires.
The deletion of the log entries takes place after a configurable cancellation time period expires.
Log entry deletion relies on configuration of the custom logs retention workflow as defined in [Compactor](../retention#compactor). The Compactor looks at unprocessed requests which are past their cancellation period to decide whether a chunk is to be deleted or not.
Log entry deletion relies on configuration of the custom logs retention workflow as defined for the [compactor](../retention#compactor). The compactor looks at unprocessed requests which are past their cancellation period to decide whether a chunk is to be deleted or not.
## Configuration
## Configuration
Enable log entry deletion by setting `retention_enabled` to true and `deletion_mode` to `filter-only` or `filter-and-delete` in the compactor's configuration.
Enable log entry deletion by setting `retention_enabled` to true and `deletion_mode` to `filter-only` or `filter-and-delete` in the compactor's configuration.
With `filter-only`, log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from the on-disk chunks.
With `filter-only`, log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from storage.
With `filter-and-delete`, log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from the on-disk chunks.
With `filter-and-delete`, log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from storage.
A delete request may be canceled within a configurable cancellation period. Set the `delete_request_cancel_period` in the Compactor's YAML configuration or on the command line when invoking Loki. Its default value is 24h.
A delete request may be canceled within a configurable cancellation period. Set the `delete_request_cancel_period` in the compactor's YAML configuration or on the command line when invoking Loki. Its default value is 24h.
Access to the deletion API can be enabled per tenant via the `allow_deletes` setting.
Access to the deletion API can be enabled per tenant via the `allow_deletes` setting.
## Compactor endpoints
The Compactor exposes endpoints to allow for the deletion of log entries from specified streams.
### Request log entry deletion
```
POST /loki/api/v1/delete
PUT /loki/api/v1/delete
```
Query parameters:
* `query=<series_selector>`: query argument that identifies the streams from which to delete with optional line filters.
* `start=<rfc3339 | unix_timestamp>`: A timestamp that identifies the start of the time window within which entries will be deleted. If not specified, defaults to 0, the Unix Epoch time.
* `end=<rfc3339 | unix_timestamp>`: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time.
A 204 response indicates success.
URL encode the `query` parameter. This sample form of a cURL command URL encodes `query={foo="bar"}`:
The query parameter can also include filter operations. For example `query={foo="bar"} |= "other"` will filter out lines that contain the string "other" for the streams matching the stream selector `{foo="bar"}`.
```
### List delete requests
List the existing delete requests using the following API:
```
GET /loki/api/v1/delete
```
Sample form of a cURL command:
```
curl -X GET \
<compactor_addr>/loki/api/v1/delete \
-H 'x-scope-orgid: <orgid>'
```
This endpoint returns both processed and unprocessed requests. It does not list canceled requests, as those requests will have been removed from storage.
### Request cancellation of a delete request
Loki allows cancellation of delete requests until the requests are picked up for processing. It is controlled by the `delete_request_cancel_period` YAML configuration or the equivalent command line option when invoking Loki.
Cancel a delete request using this Compactor endpoint:
```
DELETE /loki/api/v1/delete
```
Query parameters:
* `request_id=<request_id>`: Identifies the delete request to cancel; IDs are found using the `delete` endpoint.
Karen Miller
3 years ago
committed by