mirror of https://github.com/grafana/loki
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1435 lines
45 KiB
1435 lines
45 KiB
![]()
5 years ago
|
---
|
||
![]()
2 years ago
|
title: Grafana Loki HTTP API
|
||
![]()
3 years ago
|
menuTitle: "HTTP API"
|
||
![]()
2 years ago
|
description: Loki exposes REST endpoints for operating on a Loki cluster. This section details the endpoints.
|
||
|
aliases:
|
||
|
- /docs/loki/latest/api
|
||
|
- /docs/loki/latest/reference
|
||
|
weight: 100
|
||
![]()
5 years ago
|
---
|
||
|
|
||
![]()
4 years ago
|
# Grafana Loki HTTP API
|
||
![]()
6 years ago
|
|
||
![]()
4 years ago
|
Grafana Loki exposes an HTTP API for pushing, querying, and tailing log data.
|
||
![]()
4 years ago
|
Note that authenticating against the API is
|
||
![]()
6 years ago
|
out of scope for Loki.
|
||
|
|
||
![]()
5 years ago
|
## Microservices mode
|
||
![]()
6 years ago
|
|
||
|
When deploying Loki in microservices mode, the set of endpoints exposed by each
|
||
|
component is different.
|
||
|
|
||
|
These endpoints are exposed by all components:
|
||
|
|
||
![]()
3 years ago
|
- [`GET /ready`](#identify-ready-loki-instance)
|
||
|
- [`GET /metrics`](#return-exposed-prometheus-metrics)
|
||
|
- [`GET /config`](#list-current-configuration)
|
||
|
- [`GET /services`](#list-running-services)
|
||
|
- [`GET /loki/api/v1/status/buildinfo`](#list-build-information)
|
||
![]()
2 years ago
|
- [`GET /loki/api/v1/format_query`](#format-query)
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
These endpoints are exposed by the querier and the query frontend:
|
||
|
|
||
![]()
3 years ago
|
- [`GET /loki/api/v1/query`](#query-loki)
|
||
|
- [`GET /loki/api/v1/query_range`](#query-loki-over-a-range-of-time)
|
||
|
- [`GET /loki/api/v1/labels`](#list-labels-within-a-range-of-time)
|
||
|
- [`GET /loki/api/v1/label/<name>/values`](#list-label-values-within-a-range-of-time)
|
||
|
- [`GET /loki/api/v1/series`](#list-series)
|
||
![]()
3 years ago
|
- [`GET /loki/api/v1/index/stats`](#index-stats)
|
||
![]()
3 years ago
|
- [`GET /loki/api/v1/tail`](#stream-log-messages)
|
||
![]()
3 years ago
|
- **Deprecated** [`GET /api/prom/tail`](#get-apipromtail)
|
||
|
- **Deprecated** [`GET /api/prom/query`](#get-apipromquery)
|
||
|
- **Deprecated** [`GET /api/prom/label`](#get-apipromlabel)
|
||
|
- **Deprecated** [`GET /api/prom/label/<name>/values`](#get-apipromlabelnamevalues)
|
||
![]()
2 years ago
|
- **Deprecated** [`GET /api/prom/series`](#list-series)
|
||
![]()
3 years ago
|
|
||
|
These endpoints are exposed by the distributor:
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
- [`POST /loki/api/v1/push`](#push-log-entries-to-loki)
|
||
|
- [`GET /distributor/ring`](#display-distributor-consistent-hash-ring-status)
|
||
![]()
2 years ago
|
- **Deprecated** [`POST /api/prom/push`](#post-apiprompush)
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
These endpoints are exposed by the ingester:
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
- [`POST /flush`](#flush-in-memory-chunks-to-backing-store)
|
||
|
- [`POST /ingester/shutdown`](#flush-in-memory-chunks-and-shut-down)
|
||
![]()
3 years ago
|
- **Deprecated** [`POST /ingester/flush_shutdown`](#post-ingesterflush_shutdown)
|
||
![]()
6 years ago
|
|
||
|
The API endpoints starting with `/loki/` are [Prometheus API-compatible](https://prometheus.io/docs/prometheus/latest/querying/api/) and the result formats can be used interchangeably.
|
||
|
|
||
![]()
5 years ago
|
These endpoints are exposed by the ruler:
|
||
|
|
||
|
- [`GET /ruler/ring`](#ruler-ring-status)
|
||
![]()
5 years ago
|
- [`GET /loki/api/v1/rules`](#list-rule-groups)
|
||
|
- [`GET /loki/api/v1/rules/{namespace}`](#get-rule-groups-by-namespace)
|
||
|
- [`GET /loki/api/v1/rules/{namespace}/{groupName}`](#get-rule-group)
|
||
|
- [`POST /loki/api/v1/rules/{namespace}`](#set-rule-group)
|
||
|
- [`DELETE /loki/api/v1/rules/{namespace}/{groupName}`](#delete-rule-group)
|
||
|
- [`DELETE /loki/api/v1/rules/{namespace}`](#delete-namespace)
|
||
![]()
5 years ago
|
- [`GET /api/prom/rules`](#list-rule-groups)
|
||
|
- [`GET /api/prom/rules/{namespace}`](#get-rule-groups-by-namespace)
|
||
|
- [`GET /api/prom/rules/{namespace}/{groupName}`](#get-rule-group)
|
||
|
- [`POST /api/prom/rules/{namespace}`](#set-rule-group)
|
||
|
- [`DELETE /api/prom/rules/{namespace}/{groupName}`](#delete-rule-group)
|
||
|
- [`DELETE /api/prom/rules/{namespace}`](#delete-namespace)
|
||
![]()
5 years ago
|
- [`GET /prometheus/api/v1/rules`](#list-rules)
|
||
![]()
5 years ago
|
- [`GET /prometheus/api/v1/alerts`](#list-alerts)
|
||
|
|
||
![]()
3 years ago
|
These endpoints are exposed by the compactor:
|
||
![]()
2 years ago
|
|
||
![]()
3 years ago
|
- [`GET /compactor/ring`](#compactor-ring-status)
|
||
|
- [`POST /loki/api/v1/delete`](#request-log-deletion)
|
||
|
- [`GET /loki/api/v1/delete`](#list-log-deletion-requests)
|
||
|
- [`DELETE /loki/api/v1/delete`](#request-cancellation-of-a-delete-request)
|
||
![]()
3 years ago
|
|
||
![]()
2 years ago
|
A [list of clients]({{< relref "../clients" >}}) can be found in the clients documentation.
|
||
![]()
6 years ago
|
|
||
![]()
5 years ago
|
## Matrix, vector, and streams
|
||
![]()
6 years ago
|
|
||
|
Some Loki API endpoints return a result of a matrix, a vector, or a stream:
|
||
|
|
||
|
- Matrix: a table of values where each row represents a different label set
|
||
|
and the columns are each sample value for that row over the queried time.
|
||
|
Matrix types are only returned when running a query that computes some value.
|
||
|
|
||
|
- Instant Vector: denoted in the type as just `vector`, an Instant Vector
|
||
|
represents the latest value of a calculation for a given labelset. Instant
|
||
|
Vectors are only returned when doing a query against a single point in
|
||
|
time.
|
||
|
|
||
|
- Stream: a Stream is a set of all values (logs) for a given label set over the
|
||
|
queried time range. Streams are the only type that will result in log lines
|
||
|
being returned.
|
||
|
|
||
![]()
3 years ago
|
## Timestamp formats
|
||
|
|
||
|
The API accepts several formats for timestamps. An integer with ten or fewer digits is interpreted as a Unix timestamp in seconds. More than ten digits are interpreted as a Unix timestamp in nanoseconds. A floating point number is a Unix timestamp with fractions of a second.
|
||
|
|
||
|
The timestamps can also be written in `RFC3339` and `RFC3339Nano` format, as supported by Go's [time](https://pkg.go.dev/time) package.
|
||
|
|
||
![]()
3 years ago
|
## Query Loki
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/query
|
||
|
```
|
||
![]()
6 years ago
|
|
||
|
`/loki/api/v1/query` allows for doing queries against a single point in time. The URL
|
||
|
query parameters support the following values:
|
||
|
|
||
![]()
2 years ago
|
- `query`: The [LogQL]({{< relref "../logql/" >}}) query to perform
|
||
![]()
3 years ago
|
- `limit`: The max number of entries to return. It defaults to `100`. Only applies to query types which produce a stream(log lines) response.
|
||
![]()
3 years ago
|
- `time`: The evaluation time for the query as a nanosecond Unix epoch or another [supported format](#timestamp-formats). Defaults to now.
|
||
![]()
3 years ago
|
- `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward`.
|
||
![]()
6 years ago
|
|
||
![]()
5 years ago
|
In microservices mode, `/loki/api/v1/query` is exposed by the querier and the frontend.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
Response format:
|
||
![]()
6 years ago
|
|
||
|
```
|
||
|
{
|
||
|
"status": "success",
|
||
|
"data": {
|
||
|
"resultType": "vector" | "streams",
|
||
![]()
3 years ago
|
"result": [<vector value>] | [<stream value>],
|
||
![]()
5 years ago
|
"stats" : [<statistics>]
|
||
![]()
6 years ago
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
where `<vector value>` is:
|
||
![]()
6 years ago
|
|
||
|
```
|
||
|
{
|
||
|
"metric": {
|
||
|
<label key-value pairs>
|
||
|
},
|
||
|
"value": [
|
||
![]()
5 years ago
|
<number: second unix epoch>,
|
||
![]()
6 years ago
|
<string: value>
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
and `<stream value>` is:
|
||
![]()
6 years ago
|
|
||
|
```
|
||
|
{
|
||
|
"stream": {
|
||
|
<label key-value pairs>
|
||
|
},
|
||
|
"values": [
|
||
|
[
|
||
|
<string: nanosecond unix epoch>,
|
||
|
<string: log line>
|
||
|
],
|
||
|
...
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
The items in the `values` array are sorted by timestamp.
|
||
|
The most recent item is first when using `direction=backward`.
|
||
|
The oldest item is first when using `direction=forward`.
|
||
|
|
||
![]()
4 years ago
|
See [statistics](#statistics) for information about the statistics returned by Loki.
|
||
![]()
5 years ago
|
|
||
![]()
6 years ago
|
### Examples
|
||
|
|
||
![]()
3 years ago
|
This example query
|
||
![]()
2 years ago
|
|
||
![]()
6 years ago
|
```bash
|
||
![]()
3 years ago
|
curl -G -s "http://localhost:3100/loki/api/v1/query" \
|
||
|
--data-urlencode \
|
||
|
'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
|
||
|
```
|
||
|
|
||
|
gave this response:
|
||
|
|
||
|
```json
|
||
![]()
6 years ago
|
{
|
||
|
"status": "success",
|
||
|
"data": {
|
||
|
"resultType": "vector",
|
||
|
"result": [
|
||
|
{
|
||
|
"metric": {},
|
||
|
"value": [
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"1267.1266666666666"
|
||
|
]
|
||
|
},
|
||
|
{
|
||
|
"metric": {
|
||
|
"level": "warn"
|
||
|
},
|
||
|
"value": [
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"37.77166666666667"
|
||
|
]
|
||
|
},
|
||
|
{
|
||
|
"metric": {
|
||
|
"level": "info"
|
||
|
},
|
||
|
"value": [
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"37.69"
|
||
|
]
|
||
|
}
|
||
![]()
5 years ago
|
],
|
||
|
"stats": {
|
||
|
...
|
||
|
}
|
||
![]()
6 years ago
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
2 years ago
|
If your cluster has
|
||
![]()
2 years ago
|
[Grafana Loki Multi-Tenancy]({{< relref "../operations/multi-tenancy/" >}}) enabled,
|
||
![]()
3 years ago
|
set the `X-Scope-OrgID` header to identify the tenant you want to query.
|
||
|
Here is the same example query for the single tenant called `Tenant1`:
|
||
|
|
||
![]()
6 years ago
|
```bash
|
||
![]()
3 years ago
|
curl -H 'X-Scope-OrgID:Tenant1' \
|
||
|
-G -s "http://localhost:3100/loki/api/v1/query" \
|
||
|
--data-urlencode \
|
||
|
'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
|
||
|
```
|
||
|
|
||
|
To query against the three tenants `Tenant1`, `Tenant2`, and `Tenant3`,
|
||
|
specify the tenant names separated by the pipe (`|`) character:
|
||
|
|
||
|
```bash
|
||
|
curl -H 'X-Scope-OrgID:Tenant1|Tenant2|Tenant3' \
|
||
|
-G -s "http://localhost:3100/loki/api/v1/query" \
|
||
|
--data-urlencode \
|
||
|
'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
|
||
|
```
|
||
|
|
||
|
The same example query for Grafana Enterprise Logs
|
||
|
uses Basic Authentication and specifies the tenant names as a `user`.
|
||
|
The tenant names are separated by the pipe (`|`) character.
|
||
|
The password in this example is an access policy token that has been
|
||
|
defined in the `API_TOKEN` environment variable:
|
||
|
|
||
|
```bash
|
||
|
curl -u "Tenant1|Tenant2|Tenant3:$API_TOKEN" \
|
||
|
-G -s "http://localhost:3100/loki/api/v1/query" \
|
||
|
--data-urlencode \
|
||
|
'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
|
||
![]()
6 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
## Query Loki over a range of time
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/query_range
|
||
|
```
|
||
![]()
6 years ago
|
|
||
|
`/loki/api/v1/query_range` is used to do a query over a range of time and
|
||
|
accepts the following query parameters in the URL:
|
||
|
|
||
![]()
2 years ago
|
- `query`: The [LogQL]({{< relref "../logql/" >}}) query to perform
|
||
![]()
3 years ago
|
- `limit`: The max number of entries to return. It defaults to `100`. Only applies to query types which produce a stream(log lines) response.
|
||
![]()
2 years ago
|
- `start`: The start time for the query as a nanosecond Unix epoch or another [supported format](#timestamp-formats). Defaults to one hour ago. Loki returns results with timestamp greater or equal to this value.
|
||
|
- `end`: The end time for the query as a nanosecond Unix epoch or another [supported format](#timestamp-formats). Defaults to now. Loki returns results with timestamp lower than this value.
|
||
![]()
2 years ago
|
- `since`: A `duration` used to calculate `start` relative to `end`. If `end` is in the future, `start` is calculated as this duration before now. Any value specified for `start` supersedes this parameter.
|
||
|
- `step`: Query resolution step width in `duration` format or float number of seconds. `duration` refers to Prometheus duration strings of the form `[0-9]+[smhdwy]`. For example, 5m refers to a duration of 5 minutes. Defaults to a dynamic value based on `start` and `end`. Only applies to query types which produce a matrix response.
|
||
![]()
3 years ago
|
- `interval`: <span style="background-color:#f3f973;">This parameter is experimental; see the explanation under Step versus interval.</span> Only return entries at (or greater than) the specified interval, can be a `duration` format or float number of seconds. Only applies to queries which produce a stream response.
|
||
![]()
6 years ago
|
- `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward.`
|
||
|
|
||
![]()
5 years ago
|
In microservices mode, `/loki/api/v1/query_range` is exposed by the querier and the frontend.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
### Step versus interval
|
||
![]()
5 years ago
|
|
||
![]()
2 years ago
|
Use the `step` parameter when making metric queries to Loki, or queries which return a matrix response. It is evaluated in exactly the same way Prometheus evaluates `step`. First the query will be evaluated at `start` and then evaluated again at `start + step` and again at `start + step + step` until `end` is reached. The result will be a matrix of the query result evaluated at each step.
|
||
![]()
5 years ago
|
|
||
![]()
2 years ago
|
Use the `interval` parameter when making log queries to Loki, or queries which return a stream response. It is evaluated by returning a log entry at `start`, then the next entry will be returned an entry with timestampe >= `start + interval`, and again at `start + interval + interval` and so on until `end` is reached. It does not fill missing entries.
|
||
![]()
5 years ago
|
|
||
![]()
2 years ago
|
<span style="background-color:#f3f973;">Note about the experimental nature of the interval parameter:</span> This flag may be removed in the future, if so it will likely be in favor of a LogQL expression to perform similar behavior, however that is uncertain at this time. [Issue 1779](https://github.com/grafana/loki/issues/1779) was created to track the discussion, if you are using `interval` please go add your use case and thoughts to that issue.
|
||
![]()
5 years ago
|
|
||
![]()
6 years ago
|
Response:
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"status": "success",
|
||
|
"data": {
|
||
|
"resultType": "matrix" | "streams",
|
||
|
"result": [<matrix value>] | [<stream value>]
|
||
![]()
5 years ago
|
"stats" : [<statistics>]
|
||
![]()
6 years ago
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
|
Where `<matrix value>` is:
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"metric": {
|
||
|
<label key-value pairs>
|
||
|
},
|
||
|
"values": [
|
||
![]()
3 years ago
|
[
|
||
|
<number: second unix epoch>,
|
||
|
<string: value>
|
||
|
],
|
||
|
...
|
||
![]()
6 years ago
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
The items in the `values` array are sorted by timestamp, and the oldest item is first.
|
||
|
|
||
![]()
6 years ago
|
And `<stream value>` is:
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"stream": {
|
||
|
<label key-value pairs>
|
||
|
},
|
||
|
"values": [
|
||
|
[
|
||
|
<string: nanosecond unix epoch>,
|
||
|
<string: log line>
|
||
|
],
|
||
|
...
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
The items in the `values` array are sorted by timestamp.
|
||
|
The most recent item is first when using `direction=backward`.
|
||
|
The oldest item is first when using `direction=forward`.
|
||
|
|
||
![]()
4 years ago
|
See [statistics](#statistics) for information about the statistics returned by Loki.
|
||
![]()
5 years ago
|
|
||
![]()
6 years ago
|
### Examples
|
||
|
|
||
|
```bash
|
||
|
$ curl -G -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' --data-urlencode 'step=300' | jq
|
||
|
{
|
||
|
"status": "success",
|
||
|
"data": {
|
||
|
"resultType": "matrix",
|
||
|
"result": [
|
||
|
{
|
||
|
"metric": {
|
||
|
"level": "info"
|
||
|
},
|
||
|
"values": [
|
||
|
[
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"137.95"
|
||
|
],
|
||
|
[
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"467.115"
|
||
|
],
|
||
|
[
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"658.8516666666667"
|
||
|
]
|
||
|
]
|
||
|
},
|
||
|
{
|
||
|
"metric": {
|
||
|
"level": "warn"
|
||
|
},
|
||
|
"values": [
|
||
|
[
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"137.27833333333334"
|
||
|
],
|
||
|
[
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"467.69"
|
||
|
],
|
||
|
[
|
||
![]()
5 years ago
|
1588889221,
|
||
![]()
6 years ago
|
"660.6933333333334"
|
||
|
]
|
||
|
]
|
||
|
}
|
||
![]()
5 years ago
|
],
|
||
|
"stats": {
|
||
|
...
|
||
|
}
|
||
![]()
6 years ago
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
|
```bash
|
||
|
$ curl -G -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="varlogs"}' | jq
|
||
|
{
|
||
|
"status": "success",
|
||
|
"data": {
|
||
|
"resultType": "streams",
|
||
|
"result": [
|
||
|
{
|
||
|
"stream": {
|
||
|
"filename": "/var/log/myproject.log",
|
||
|
"job": "varlogs",
|
||
|
"level": "info"
|
||
|
},
|
||
|
"values": [
|
||
![]()
5 years ago
|
[
|
||
![]()
6 years ago
|
"1569266497240578000",
|
||
|
"foo"
|
||
![]()
5 years ago
|
],
|
||
|
[
|
||
![]()
6 years ago
|
"1569266492548155000",
|
||
|
"bar"
|
||
![]()
5 years ago
|
]
|
||
![]()
6 years ago
|
]
|
||
|
}
|
||
![]()
5 years ago
|
],
|
||
|
"stats": {
|
||
|
...
|
||
|
}
|
||
![]()
6 years ago
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
## List labels within a range of time
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/labels
|
||
|
```
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
`/loki/api/v1/labels` retrieves the list of known labels within a given time span.
|
||
|
Loki may use a larger time span than the one specified.
|
||
|
It accepts the following query parameters in the URL:
|
||
![]()
6 years ago
|
|
||
|
- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
|
||
![]()
5 years ago
|
- `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now.
|
||
![]()
2 years ago
|
- `since`: A `duration` used to calculate `start` relative to `end`. If `end` is in the future, `start` is calculated as this duration before now. Any value specified for `start` supersedes this parameter.
|
||
![]()
6 years ago
|
|
||
![]()
6 years ago
|
In microservices mode, `/loki/api/v1/labels` is exposed by the querier.
|
||
![]()
6 years ago
|
|
||
|
Response:
|
||
|
|
||
|
```
|
||
|
{
|
||
![]()
6 years ago
|
"status": "success",
|
||
|
"data": [
|
||
![]()
6 years ago
|
<label string>,
|
||
|
...
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
### Examples
|
||
|
|
||
|
```bash
|
||
![]()
6 years ago
|
$ curl -G -s "http://localhost:3100/loki/api/v1/labels" | jq
|
||
![]()
6 years ago
|
{
|
||
![]()
6 years ago
|
"status": "success",
|
||
|
"data": [
|
||
![]()
6 years ago
|
"foo",
|
||
|
"bar",
|
||
|
"baz"
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
## List label values within a range of time
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/label/<name>/values
|
||
|
```
|
||
![]()
6 years ago
|
|
||
|
`/loki/api/v1/label/<name>/values` retrieves the list of known values for a given
|
||
![]()
3 years ago
|
label within a given time span. Loki may use a larger time span than the one specified.
|
||
|
It accepts the following query parameters in the URL:
|
||
![]()
6 years ago
|
|
||
|
- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
|
||
![]()
5 years ago
|
- `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now.
|
||
![]()
2 years ago
|
- `since`: A `duration` used to calculate `start` relative to `end`. If `end` is in the future, `start` is calculated as this duration before now. Any value specified for `start` supersedes this parameter.
|
||
![]()
2 years ago
|
- `query`: A set of log stream selector that selects the streams to match and return label values for `<name>`. Example: `{"app": "myapp", "environment": "dev"}`
|
||
![]()
6 years ago
|
|
||
|
In microservices mode, `/loki/api/v1/label/<name>/values` is exposed by the querier.
|
||
|
|
||
|
Response:
|
||
|
|
||
|
```
|
||
|
{
|
||
![]()
6 years ago
|
"status": "success",
|
||
|
"data": [
|
||
![]()
6 years ago
|
<label value>,
|
||
|
...
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
### Examples
|
||
|
|
||
|
```bash
|
||
|
$ curl -G -s "http://localhost:3100/loki/api/v1/label/foo/values" | jq
|
||
|
{
|
||
![]()
6 years ago
|
"status": "success",
|
||
|
"data": [
|
||
![]()
6 years ago
|
"cat",
|
||
|
"dog",
|
||
|
"axolotl"
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
## Stream log messages
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/tail
|
||
|
```
|
||
![]()
6 years ago
|
|
||
|
`/loki/api/v1/tail` is a WebSocket endpoint that will stream log messages based on
|
||
|
a query. It accepts the following query parameters in the URL:
|
||
|
|
||
![]()
2 years ago
|
- `query`: The [LogQL]({{< relref "../logql/" >}}) query to perform
|
||
![]()
6 years ago
|
- `delay_for`: The number of seconds to delay retrieving logs to let slow
|
||
![]()
2 years ago
|
loggers catch up. Defaults to 0 and cannot be larger than 5.
|
||
![]()
3 years ago
|
- `limit`: The max number of entries to return. It defaults to `100`.
|
||
![]()
6 years ago
|
- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
|
||
|
|
||
|
In microservices mode, `/loki/api/v1/tail` is exposed by the querier.
|
||
|
|
||
|
Response (streamed):
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"streams": [
|
||
|
{
|
||
|
"stream": {
|
||
|
<label key-value pairs>
|
||
|
},
|
||
|
"values": [
|
||
|
[
|
||
|
<string: nanosecond unix epoch>,
|
||
|
<string: log line>
|
||
|
]
|
||
|
]
|
||
|
}
|
||
|
],
|
||
|
"dropped_entries": [
|
||
|
{
|
||
|
"labels": {
|
||
|
<label key-value pairs>
|
||
|
},
|
||
|
"timestamp": "<nanosecond unix epoch>"
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
## Push log entries to Loki
|
||
|
|
||
|
```
|
||
|
POST /loki/api/v1/push
|
||
|
```
|
||
![]()
6 years ago
|
|
||
|
`/loki/api/v1/push` is the endpoint used to send log entries to Loki. The default
|
||
![]()
5 years ago
|
behavior is for the POST body to be a snappy-compressed protobuf message:
|
||
![]()
6 years ago
|
|
||
![]()
4 years ago
|
- [Protobuf definition](https://github.com/grafana/loki/blob/main/pkg/logproto/logproto.proto)
|
||
|
- [Go client library](https://github.com/grafana/loki/blob/main/clients/pkg/promtail/client/client.go)
|
||
![]()
6 years ago
|
|
||
|
Alternatively, if the `Content-Type` header is set to `application/json`, a
|
||
|
JSON post body can be sent in the following format:
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"streams": [
|
||
|
{
|
||
![]()
6 years ago
|
"stream": {
|
||
|
"label": "value"
|
||
|
},
|
||
|
"values": [
|
||
|
[ "<unix epoch in nanoseconds>", "<log line>" ],
|
||
|
[ "<unix epoch in nanoseconds>", "<log line>" ]
|
||
![]()
6 years ago
|
]
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
4 years ago
|
You can set `Content-Encoding: gzip` request header and post gzipped JSON.
|
||
|
|
||
![]()
6 years ago
|
In microservices mode, `/loki/api/v1/push` is exposed by the distributor.
|
||
|
|
||
|
### Examples
|
||
|
|
||
![]()
2 years ago
|
```console
|
||
![]()
6 years ago
|
$ curl -v -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw \
|
||
|
'{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1570818238000000000", "fizzbuzz" ] ] }]}'
|
||
![]()
6 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
## Identify ready Loki instance
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
```
|
||
|
GET /ready
|
||
![]()
6 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
`/ready` returns HTTP 200 when the Loki instance is ready to accept traffic. If
|
||
|
running Loki on Kubernetes, `/ready` can be used as a readiness probe.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
In microservices mode, the `/ready` endpoint is exposed by all components.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
## Flush in-memory chunks to backing store
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
```
|
||
|
POST /flush
|
||
|
```
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
`/flush` triggers a flush of all in-memory chunks held by the ingesters to the
|
||
|
backing store. Mainly used for local testing.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
In microservices mode, the `/flush` endpoint is exposed by the ingester.
|
||
![]()
6 years ago
|
|
||
![]()
2 years ago
|
### Tell ingester to release all resources on next SIGTERM
|
||
|
|
||
|
```
|
||
![]()
2 years ago
|
GET, POST, DELETE /ingester/prepare_shutdown
|
||
![]()
2 years ago
|
```
|
||
|
|
||
![]()
2 years ago
|
After a `POST` to the `prepare_shutdown` endpoint returns, when the ingester process is stopped with `SIGINT` / `SIGTERM`,
|
||
|
the ingester will be unregistered from the ring and in-memory time series data will be flushed to long-term storage.
|
||
![]()
2 years ago
|
This endpoint supersedes any YAML configurations and isn't necessary if the ingester is already
|
||
|
configured to unregister from the ring or to flush on shutdown.
|
||
|
|
||
![]()
2 years ago
|
A `GET` to the `prepare_shutdown` endpoint returns the status of this configuration, either `set` or `unset`.
|
||
|
|
||
|
A `DELETE` to the `prepare_shutdown` endpoint reverts the configuration of the ingester to its previous state
|
||
|
(with respect to unregistering on shutdown and flushing of in-memory time series data to long-term storage).
|
||
|
|
||
|
This API endpoint is usually used by Kubernetes-specific scale down automations such as the
|
||
|
[rollout-operator](https://github.com/grafana/rollout-operator).
|
||
|
|
||
![]()
3 years ago
|
## Flush in-memory chunks and shut down
|
||
![]()
6 years ago
|
|
||
|
```
|
||
![]()
3 years ago
|
POST /ingester/shutdown
|
||
![]()
6 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
`/ingester/shutdown` triggers a shutdown of the ingester and notably will _always_ flush any in memory chunks it holds.
|
||
|
This is helpful for scaling down WAL-enabled ingesters where we want to ensure old WAL directories are not orphaned,
|
||
|
but instead flushed to our chunk backend.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
It accepts three URL query parameters `flush`, `delete_ring_tokens`, and `terminate`.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
**URL query parameters:**
|
||
![]()
6 years ago
|
|
||
![]()
2 years ago
|
- `flush=<bool>`:
|
||
![]()
3 years ago
|
Flag to control whether to flush any in-memory chunks the ingester holds. Defaults to `true`.
|
||
![]()
2 years ago
|
- `delete_ring_tokens=<bool>`:
|
||
![]()
2 years ago
|
Flag to control whether to delete the file that contains the ingester ring tokens of the instance if the `-ingester.token-file-path` is specified. Defaults to `false.
|
||
![]()
2 years ago
|
- `terminate=<bool>`:
|
||
![]()
3 years ago
|
Flag to control whether to terminate the Loki process after service shutdown. Defaults to `true`.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
This handler, in contrast to the deprecated `/ingester/flush_shutdown` handler, terminates the Loki process by default.
|
||
|
This behaviour can be changed by setting the `terminate` query parameter to `false`.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
In microservices mode, the `/ingester/shutdown` endpoint is exposed by the ingester.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
## Display distributor consistent hash ring status
|
||
![]()
6 years ago
|
|
||
|
```
|
||
![]()
3 years ago
|
GET /distributor/ring
|
||
![]()
6 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
Displays a web page with the distributor hash ring status, including the state, healthy and last heartbeat time of each distributor.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
## Return exposed Prometheus metrics
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
```
|
||
|
GET /metrics
|
||
|
```
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
`/metrics` returns exposed Prometheus metrics. See
|
||
![]()
2 years ago
|
[Observing Loki]({{< relref "../operations/observability/" >}})
|
||
![]()
3 years ago
|
for a list of exported metrics.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
In microservices mode, the `/metrics` endpoint is exposed by all components.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
## List current configuration
|
||
![]()
6 years ago
|
|
||
|
```
|
||
![]()
3 years ago
|
GET /config
|
||
![]()
6 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
`/config` exposes the current configuration. The optional `mode` query parameter can be used to
|
||
|
modify the output. If it has the value `diff` only the differences between the default configuration
|
||
|
and the current are returned. A value of `defaults` returns the default configuration.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
In microservices mode, the `/config` endpoint is exposed by all components.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
## List running services
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
```
|
||
|
GET /services
|
||
|
```
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
`/services` returns a list of all running services and their current states.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
Services can have the following states:
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
- **New**: Service is new, not running yet (initial state)
|
||
|
- **Starting**: Service is starting; if starting succeeds, service enters **Running** state
|
||
|
- **Running**: Service is fully running now; when service stops running, it enters **Stopping** state
|
||
|
- **Stopping**: Service is shutting down
|
||
|
- **Terminated**: Service has stopped successfully (terminal state)
|
||
|
- **Failed**: Service has failed in **Starting**, **Running** or **Stopping** state (terminal state)
|
||
|
|
||
|
## List build information
|
||
![]()
6 years ago
|
|
||
|
```
|
||
![]()
3 years ago
|
GET /loki/api/v1/status/buildinfo
|
||
![]()
6 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
`/loki/api/v1/status/buildinfo` exposes the build information in a JSON object. The fields are `version`, `revision`, `branch`, `buildDate`, `buildUser`, and `goVersion`.
|
||
![]()
6 years ago
|
|
||
![]()
2 years ago
|
## Format query
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/format_query
|
||
|
POST /loki/api/v1/format_query
|
||
|
```
|
||
|
|
||
|
Params:
|
||
![]()
2 years ago
|
|
||
![]()
2 years ago
|
- `query`: A LogQL query string. Can be passed as URL param (`?query=<query>`) in case of both `GET` and `POST`. Or as form value in case of `POST`.
|
||
|
|
||
|
The `/loki/api/v1/format_query` endpoint allows to format LogQL queries. It returns an error if the passed LogQL is invalid. It is exposed by all Loki components and helps to improve readability and the debugging experience of LogQL queries.
|
||
|
|
||
![]()
2 years ago
|
The following example formats the expression LogQL `{foo= "bar"}` into
|
||
|
|
||
|
```json
|
||
|
{
|
||
|
"status" : "success",
|
||
|
"data" : "{foo=\"bar\"}"
|
||
|
}
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
## List series
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
The Series API is available under the following:
|
||
![]()
2 years ago
|
|
||
![]()
3 years ago
|
- `GET /loki/api/v1/series`
|
||
|
- `POST /loki/api/v1/series`
|
||
|
- `GET /api/prom/series`
|
||
|
- `POST /api/prom/series`
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
This endpoint returns the list of time series that match a certain label set.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
URL query parameters:
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
- `match[]=<series_selector>`: Repeated log stream selector argument that selects the streams to return. At least one `match[]` argument must be provided.
|
||
|
- `start=<nanosecond Unix epoch>`: Start timestamp.
|
||
|
- `end=<nanosecond Unix epoch>`: End timestamp.
|
||
![]()
2 years ago
|
- `since`: A `duration` used to calculate `start` relative to `end`. If `end` is in the future, `start` is calculated as this duration before now. Any value specified for `start` supersedes this parameter.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
You can URL-encode these parameters directly in the request body by using the POST method and `Content-Type: application/x-www-form-urlencoded` header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
In microservices mode, these endpoints are exposed by the querier.
|
||
![]()
6 years ago
|
|
||
![]()
3 years ago
|
### Examples
|
||
![]()
6 years ago
|
|
||
![]()
2 years ago
|
```console
|
||
![]()
4 years ago
|
$ curl -s "http://localhost:3100/loki/api/v1/series" --data-urlencode 'match[]={container_name=~"prometheus.*", component="server"}' --data-urlencode 'match[]={app="loki"}' | jq '.'
|
||
![]()
6 years ago
|
{
|
||
|
"status": "success",
|
||
|
"data": [
|
||
|
{
|
||
|
"container_name": "loki",
|
||
|
"app": "loki",
|
||
|
"stream": "stderr",
|
||
|
"filename": "/var/log/pods/default_loki-stack-0_50835643-1df0-11ea-ba79-025000000001/loki/0.log",
|
||
|
"name": "loki",
|
||
|
"job": "default/loki",
|
||
|
"controller_revision_hash": "loki-stack-757479754d",
|
||
|
"statefulset_kubernetes_io_pod_name": "loki-stack-0",
|
||
|
"release": "loki-stack",
|
||
|
"namespace": "default",
|
||
|
"instance": "loki-stack-0"
|
||
|
},
|
||
|
{
|
||
|
"chart": "prometheus-9.3.3",
|
||
|
"container_name": "prometheus-server-configmap-reload",
|
||
|
"filename": "/var/log/pods/default_loki-stack-prometheus-server-696cc9ddff-87lmq_507b1db4-1df0-11ea-ba79-025000000001/prometheus-server-configmap-reload/0.log",
|
||
|
"instance": "loki-stack-prometheus-server-696cc9ddff-87lmq",
|
||
|
"pod_template_hash": "696cc9ddff",
|
||
|
"app": "prometheus",
|
||
|
"component": "server",
|
||
|
"heritage": "Tiller",
|
||
|
"job": "default/prometheus",
|
||
|
"namespace": "default",
|
||
|
"release": "loki-stack",
|
||
|
"stream": "stderr"
|
||
|
},
|
||
|
{
|
||
|
"app": "prometheus",
|
||
|
"component": "server",
|
||
|
"filename": "/var/log/pods/default_loki-stack-prometheus-server-696cc9ddff-87lmq_507b1db4-1df0-11ea-ba79-025000000001/prometheus-server/0.log",
|
||
|
"release": "loki-stack",
|
||
|
"namespace": "default",
|
||
|
"pod_template_hash": "696cc9ddff",
|
||
|
"stream": "stderr",
|
||
|
"chart": "prometheus-9.3.3",
|
||
|
"container_name": "prometheus-server",
|
||
|
"heritage": "Tiller",
|
||
|
"instance": "loki-stack-prometheus-server-696cc9ddff-87lmq",
|
||
|
"job": "default/prometheus"
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
```
|
||
![]()
5 years ago
|
|
||
![]()
3 years ago
|
## Index Stats
|
||
|
|
||
|
The `/loki/api/v1/index/stats` endpoint can be used to query the index for the number of `streams`, `chunks`, `entries`, and `bytes` that a query resolves to.
|
||
|
|
||
|
URL query parameters:
|
||
|
|
||
![]()
2 years ago
|
- `query`: The [LogQL]({{< relref "../logql/" >}}) matchers to check (i.e. `{job="foo", env!="dev"}`)
|
||
![]()
3 years ago
|
- `start=<nanosecond Unix epoch>`: Start timestamp.
|
||
|
- `end=<nanosecond Unix epoch>`: End timestamp.
|
||
|
|
||
|
You can URL-encode these parameters directly in the request body by using the POST method and `Content-Type: application/x-www-form-urlencoded` header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits.
|
||
|
|
||
|
Response:
|
||
![]()
2 years ago
|
|
||
![]()
3 years ago
|
```json
|
||
|
{
|
||
|
"streams": 100,
|
||
|
"chunks": 1000,
|
||
|
"entries": 5000,
|
||
![]()
2 years ago
|
"bytes": 100000
|
||
![]()
3 years ago
|
}
|
||
|
```
|
||
|
|
||
|
It is an approximation with the following caveats:
|
||
![]()
2 years ago
|
|
||
|
- It does not include data from the ingesters
|
||
|
- It is a probabilistic technique
|
||
|
- streams/chunks which span multiple period configurations may be counted twice.
|
||
![]()
3 years ago
|
|
||
|
These make it generally more helpful for larger queries.
|
||
|
It can be used for better understanding the throughput requirements and data topology for a list of matchers over a period of time.
|
||
|
|
||
![]()
5 years ago
|
## Statistics
|
||
|
|
||
|
Query endpoints such as `/api/prom/query`, `/loki/api/v1/query` and `/loki/api/v1/query_range` return a set of statistics about the query execution. Those statistics allow users to understand the amount of data processed and at which speed.
|
||
|
|
||
|
The example belows show all possible statistics returned with their respective description.
|
||
|
|
||
|
```json
|
||
|
{
|
||
|
"status": "success",
|
||
|
"data": {
|
||
|
"resultType": "streams",
|
||
|
"result": [],
|
||
|
"stats": {
|
||
![]()
2 years ago
|
"ingester": {
|
||
![]()
5 years ago
|
"compressedBytes": 0, // Total bytes of compressed chunks (blocks) processed by ingesters
|
||
|
"decompressedBytes": 0, // Total bytes decompressed and processed by ingesters
|
||
|
"decompressedLines": 0, // Total lines decompressed and processed by ingesters
|
||
|
"headChunkBytes": 0, // Total bytes read from ingesters head chunks
|
||
|
"headChunkLines": 0, // Total lines read from ingesters head chunks
|
||
|
"totalBatches": 0, // Total batches sent by ingesters
|
||
|
"totalChunksMatched": 0, // Total chunks matched by ingesters
|
||
|
"totalDuplicates": 0, // Total of duplicates found by ingesters
|
||
|
"totalLinesSent": 0, // Total lines sent by ingesters
|
||
|
"totalReached": 0 // Amount of ingesters reached.
|
||
|
},
|
||
|
"store": {
|
||
|
"compressedBytes": 0, // Total bytes of compressed chunks (blocks) processed by the store
|
||
![]()
2 years ago
|
"decompressedBytes": 0, // Total bytes decompressed and processed by the store
|
||
![]()
5 years ago
|
"decompressedLines": 0, // Total lines decompressed and processed by the store
|
||
|
"chunksDownloadTime": 0, // Total time spent downloading chunks in seconds (float)
|
||
|
"totalChunksRef": 0, // Total chunks found in the index for the current query
|
||
|
"totalChunksDownloaded": 0, // Total of chunks downloaded
|
||
|
"totalDuplicates": 0 // Total of duplicates removed from replication
|
||
|
},
|
||
|
"summary": {
|
||
![]()
5 years ago
|
"bytesProcessedPerSecond": 0, // Total of bytes processed per second
|
||
![]()
5 years ago
|
"execTime": 0, // Total execution time in seconds (float)
|
||
![]()
5 years ago
|
"linesProcessedPerSecond": 0, // Total lines processed per second
|
||
![]()
3 years ago
|
"queueTime": 0, // Total queue time in seconds (float)
|
||
![]()
2 years ago
|
"totalBytesProcessed": 0, // Total amount of bytes processed overall for this request
|
||
|
"totalLinesProcessed": 0 // Total amount of lines processed overall for this request
|
||
![]()
5 years ago
|
}
|
||
|
}
|
||
|
}
|
||
|
}
|
||
|
```
|
||
![]()
5 years ago
|
|
||
|
## Ruler
|
||
|
|
||
|
The ruler API endpoints require to configure a backend object storage to store the recording rules and alerts. The ruler API uses the concept of a "namespace" when creating rule groups. This is a stand-in for the name of the rule file in Prometheus. Rule groups must be named uniquely within a namespace.
|
||
|
|
||
|
### Ruler ring status
|
||
|
|
||
|
```
|
||
|
GET /ruler/ring
|
||
|
```
|
||
|
|
||
|
Displays a web page with the ruler hash ring status, including the state, healthy and last heartbeat time of each ruler.
|
||
|
|
||
|
### List rule groups
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/rules
|
||
|
```
|
||
|
|
||
|
List all rules configured for the authenticated tenant. This endpoint returns a YAML dictionary with all the rule groups for each namespace and `200` status code on success.
|
||
|
|
||
|
#### Example response
|
||
|
|
||
|
```yaml
|
||
|
---
|
||
|
<namespace1>:
|
||
|
- name: <string>
|
||
|
interval: <duration;optional>
|
||
|
rules:
|
||
|
- alert: <string>
|
||
|
expr: <string>
|
||
|
for: <duration>
|
||
|
annotations:
|
||
|
<annotation_name>: <string>
|
||
|
labels:
|
||
|
<label_name>: <string>
|
||
|
- name: <string>
|
||
|
interval: <duration;optional>
|
||
|
rules:
|
||
|
- alert: <string>
|
||
|
expr: <string>
|
||
|
for: <duration>
|
||
|
annotations:
|
||
|
<annotation_name>: <string>
|
||
|
labels:
|
||
|
<label_name>: <string>
|
||
|
<namespace2>:
|
||
|
- name: <string>
|
||
|
interval: <duration;optional>
|
||
|
rules:
|
||
|
- alert: <string>
|
||
|
expr: <string>
|
||
|
for: <duration>
|
||
|
annotations:
|
||
|
<annotation_name>: <string>
|
||
|
labels:
|
||
|
<label_name>: <string>
|
||
|
```
|
||
|
|
||
|
### Get rule groups by namespace
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/rules/{namespace}
|
||
|
```
|
||
|
|
||
|
Returns the rule groups defined for a given namespace.
|
||
|
|
||
|
#### Example response
|
||
|
|
||
|
```yaml
|
||
|
name: <string>
|
||
|
interval: <duration;optional>
|
||
|
rules:
|
||
|
- alert: <string>
|
||
|
expr: <string>
|
||
|
for: <duration>
|
||
|
annotations:
|
||
|
<annotation_name>: <string>
|
||
|
labels:
|
||
|
<label_name>: <string>
|
||
|
```
|
||
|
|
||
|
### Get rule group
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/rules/{namespace}/{groupName}
|
||
|
```
|
||
|
|
||
|
Returns the rule group matching the request namespace and group name.
|
||
|
|
||
|
### Set rule group
|
||
|
|
||
|
```
|
||
|
POST /loki/api/v1/rules/{namespace}
|
||
|
```
|
||
|
|
||
|
Creates or updates a rule group. This endpoint expects a request with `Content-Type: application/yaml` header and the rules **YAML** definition in the request body, and returns `202` on success.
|
||
|
|
||
|
#### Example request
|
||
|
|
||
|
Request headers:
|
||
![]()
2 years ago
|
|
||
![]()
5 years ago
|
- `Content-Type: application/yaml`
|
||
|
|
||
|
Request body:
|
||
|
|
||
|
```yaml
|
||
|
name: <string>
|
||
|
interval: <duration;optional>
|
||
|
rules:
|
||
|
- alert: <string>
|
||
|
expr: <string>
|
||
|
for: <duration>
|
||
|
annotations:
|
||
|
<annotation_name>: <string>
|
||
|
labels:
|
||
|
<label_name>: <string>
|
||
|
```
|
||
|
|
||
|
### Delete rule group
|
||
|
|
||
|
```
|
||
|
DELETE /loki/api/v1/rules/{namespace}/{groupName}
|
||
|
|
||
|
```
|
||
|
|
||
|
Deletes a rule group by namespace and group name. This endpoints returns `202` on success.
|
||
|
|
||
|
### Delete namespace
|
||
|
|
||
|
```
|
||
|
DELETE /loki/api/v1/rules/{namespace}
|
||
|
```
|
||
|
|
||
|
Deletes all the rule groups in a namespace (including the namespace itself). This endpoint returns `202` on success.
|
||
|
|
||
![]()
5 years ago
|
### List rules
|
||
![]()
5 years ago
|
|
||
![]()
5 years ago
|
```
|
||
![]()
5 years ago
|
GET /prometheus/api/v1/rules
|
||
![]()
5 years ago
|
```
|
||
|
|
||
|
Prometheus-compatible rules endpoint to list alerting and recording rules that are currently loaded.
|
||
|
|
||
|
For more information, refer to the [Prometheus rules](https://prometheus.io/docs/prometheus/latest/querying/api/#rules) documentation.
|
||
|
|
||
![]()
5 years ago
|
### List alerts
|
||
|
|
||
|
```
|
||
|
GET /prometheus/api/v1/alerts
|
||
|
```
|
||
|
|
||
|
Prometheus-compatible rules endpoint to list all active alerts.
|
||
|
|
||
![]()
4 years ago
|
For more information, please check out the Prometheus [alerts](https://prometheus.io/docs/prometheus/latest/querying/api/#alerts) documentation.
|
||
![]()
3 years ago
|
|
||
|
## Compactor
|
||
|
|
||
![]()
3 years ago
|
### Compactor ring status
|
||
|
|
||
|
```
|
||
|
GET /compactor/ring
|
||
|
```
|
||
![]()
3 years ago
|
|
||
|
Displays a web page with the compactor hash ring status, including the state, health, and last heartbeat time of each compactor.
|
||
|
|
||
![]()
3 years ago
|
### Request log deletion
|
||
|
|
||
|
```
|
||
|
POST /loki/api/v1/delete
|
||
|
PUT /loki/api/v1/delete
|
||
|
```
|
||
![]()
3 years ago
|
|
||
![]()
3 years ago
|
Create a new delete request for the authenticated tenant.
|
||
![]()
2 years ago
|
The [log entry deletion]({{< relref "../operations/storage/logs-deletion/" >}}) documentation has configuration details.
|
||
![]()
3 years ago
|
|
||
![]()
3 years ago
|
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
|
||
|
|
||
|
Query parameters:
|
||
|
|
||
![]()
2 years ago
|
- `query=<series_selector>`: query argument that identifies the streams from which to delete with optional line filters.
|
||
|
- `start=<rfc3339 | unix_seconds_timestamp>`: A timestamp that identifies the start of the time window within which entries will be deleted. This parameter is required.
|
||
|
- `end=<rfc3339 | unix_seconds_timestamp>`: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time.
|
||
|
- `max_interval=<duration>`: The maximum time period the delete request can span. If the request is larger than this value, it is split into several requests of <= `max_interval`. Valid time units are `s`, `m`, and `h`.
|
||
![]()
3 years ago
|
|
||
|
A 204 response indicates success.
|
||
|
|
||
![]()
3 years ago
|
The query parameter can also include filter operations. For example `query={foo="bar"} |= "other"` will filter out lines that contain the string "other" for the streams matching the stream selector `{foo="bar"}`.
|
||
|
|
||
|
#### Examples
|
||
|
|
||
![]()
3 years ago
|
URL encode the `query` parameter. This sample form of a cURL command URL encodes `query={foo="bar"}`:
|
||
|
|
||
|
```bash
|
||
|
curl -g -X POST \
|
||
|
'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}&start=1591616227&end=1591619692' \
|
||
![]()
3 years ago
|
-H 'X-Scope-OrgID: 1'
|
||
![]()
3 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
The same example deletion request for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant name as a user; `Tenant1` is the tenant name in this example. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable. The token must be for an access policy with `logs:delete` scope for the tenant specified in the user field:
|
||
|
|
||
|
```bash
|
||
|
curl -u "Tenant1:$API_TOKEN" \
|
||
|
-g -X POST \
|
||
|
'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}&start=1591616227&end=1591619692'
|
||
|
```
|
||
![]()
3 years ago
|
|
||
|
### List log deletion requests
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/delete
|
||
|
```
|
||
![]()
3 years ago
|
|
||
![]()
3 years ago
|
List the existing delete requests for the authenticated tenant.
|
||
![]()
2 years ago
|
The [log entry deletion]({{< relref "../operations/storage/logs-deletion/" >}}) documentation has configuration details.
|
||
![]()
3 years ago
|
|
||
![]()
3 years ago
|
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
|
||
|
|
||
|
List the existing delete requests using the following API:
|
||
|
|
||
|
```
|
||
|
GET /loki/api/v1/delete
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
This endpoint returns both processed and unprocessed deletion requests. It does not list canceled requests, as those requests will have been removed from storage.
|
||
|
|
||
|
#### Examples
|
||
|
|
||
|
Example cURL command:
|
||
![]()
3 years ago
|
|
||
|
```
|
||
|
curl -X GET \
|
||
|
<compactor_addr>/loki/api/v1/delete \
|
||
![]()
3 years ago
|
-H 'X-Scope-OrgID: <orgid>'
|
||
![]()
3 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
The same example deletion request for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant name as a user; `Tenant1` is the tenant name in this example. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable. The token must be for an access policy with `logs:delete` scope for the tenant specified in the user field.
|
||
|
|
||
|
```bash
|
||
|
curl -u "Tenant1:$API_TOKEN" \
|
||
|
-X GET \
|
||
|
<compactor_addr>/loki/api/v1/delete
|
||
|
```
|
||
![]()
3 years ago
|
|
||
|
### Request cancellation of a delete request
|
||
|
|
||
|
```
|
||
|
DELETE /loki/api/v1/delete
|
||
|
```
|
||
![]()
3 years ago
|
|
||
![]()
3 years ago
|
Remove a delete request for the authenticated tenant.
|
||
![]()
2 years ago
|
The [log entry deletion]({{< relref "../operations/storage/logs-deletion/" >}}) documentation has configuration details.
|
||
![]()
3 years ago
|
|
||
![]()
3 years ago
|
Loki allows cancellation of delete requests until the requests are picked up for processing. It is controlled by the `delete_request_cancel_period` YAML configuration or the equivalent command line option when invoking Loki. To cancel a delete request that has been picked up for processing or is partially complete, pass the `force=true` query parameter to the API.
|
||
![]()
3 years ago
|
|
||
|
Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store.
|
||
|
|
||
|
Cancel a delete request using this compactor endpoint:
|
||
|
|
||
|
```
|
||
|
DELETE /loki/api/v1/delete
|
||
|
```
|
||
|
|
||
|
Query parameters:
|
||
|
|
||
![]()
2 years ago
|
- `request_id=<request_id>`: Identifies the delete request to cancel; IDs are found using the `delete` endpoint.
|
||
|
- `force=<boolean>`: When the `force` query parameter is true, partially completed delete requests will be canceled. NOTE: some data from the request may still be deleted and the deleted request will be listed as 'processed'
|
||
![]()
3 years ago
|
|
||
|
A 204 response indicates success.
|
||
|
|
||
![]()
3 years ago
|
#### Examples
|
||
|
|
||
|
Example cURL command:
|
||
![]()
3 years ago
|
|
||
|
```
|
||
|
curl -X DELETE \
|
||
|
'<compactor_addr>/loki/api/v1/delete?request_id=<request_id>' \
|
||
![]()
3 years ago
|
-H 'X-Scope-OrgID: <tenant-id>'
|
||
![]()
3 years ago
|
```
|
||
|
|
||
![]()
3 years ago
|
The same example deletion cancellation request for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant name as a user; `Tenant1` is the tenant name in this example. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable. The token must be for an access policy with `logs:delete` scope for the tenant specified in the user field.
|
||
|
|
||
|
```bash
|
||
|
curl -u "Tenant1:$API_TOKEN" \
|
||
|
-X DELETE \
|
||
|
'<compactor_addr>/loki/api/v1/delete?request_id=<request_id>'
|
||
|
```
|
||
|
|
||
![]()
3 years ago
|
## Deprecated endpoints
|
||
|
|
||
|
### `GET /api/prom/tail`
|
||
|
|
||
|
> **DEPRECATED**: `/api/prom/tail` is deprecated. Use `/loki/api/v1/tail`
|
||
|
> instead.
|
||
|
|
||
|
`/api/prom/tail` is a WebSocket endpoint that will stream log messages based on
|
||
|
a query. It accepts the following query parameters in the URL:
|
||
|
|
||
![]()
2 years ago
|
- `query`: The [LogQL]({{< relref "../logql/" >}}) query to perform
|
||
![]()
3 years ago
|
- `delay_for`: The number of seconds to delay retrieving logs to let slow
|
||
![]()
2 years ago
|
loggers catch up. Defaults to 0 and cannot be larger than 5.
|
||
![]()
3 years ago
|
- `limit`: The max number of entries to return
|
||
|
- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
|
||
|
|
||
|
In microservices mode, `/api/prom/tail` is exposed by the querier.
|
||
|
|
||
|
Response (streamed):
|
||
|
|
||
|
```json
|
||
|
{
|
||
|
"streams": [
|
||
|
{
|
||
|
"labels": "<LogQL label key-value pairs>",
|
||
|
"entries": [
|
||
|
{
|
||
|
"ts": "<RFC3339Nano timestamp>",
|
||
|
"line": "<log line>"
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
],
|
||
|
"dropped_entries": [
|
||
|
{
|
||
|
"Timestamp": "<RFC3339Nano timestamp>",
|
||
|
"Labels": "<LogQL label key-value pairs>"
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
`dropped_entries` will be populated when the tailer could not keep up with the
|
||
|
amount of traffic in Loki. When present, it indicates that the entries received
|
||
|
in the streams is not the full amount of logs that are present in Loki. Note
|
||
|
that the keys in `dropped_entries` will be sent as uppercase `Timestamp`
|
||
|
and `Labels` instead of `labels` and `ts` like in the entries for the stream.
|
||
|
|
||
|
As the response is streamed, the object defined by the response format above
|
||
|
will be sent over the WebSocket multiple times.
|
||
|
|
||
|
### `GET /api/prom/query`
|
||
|
|
||
|
> **WARNING**: `/api/prom/query` is DEPRECATED; use `/loki/api/v1/query_range`
|
||
|
> instead.
|
||
|
|
||
|
`/api/prom/query` supports doing general queries. The URL query parameters
|
||
|
support the following values:
|
||
|
|
||
![]()
2 years ago
|
- `query`: The [LogQL]({{< relref "../logql/" >}}) query to perform
|
||
![]()
3 years ago
|
- `limit`: The max number of entries to return
|
||
|
- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
|
||
|
- `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now.
|
||
![]()
2 years ago
|
- `since`: A `duration` used to calculate `start` relative to `end`. If `end` is in the future, `start` is calculated as this duration before now. Any value specified for `start` supersedes this parameter.
|
||
![]()
3 years ago
|
- `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward.`
|
||
|
- `regexp`: a regex to filter the returned results
|
||
|
|
||
|
In microservices mode, `/api/prom/query` is exposed by the querier and the frontend.
|
||
|
|
||
|
Note that the larger the time span between `start` and `end` will cause
|
||
|
additional load on Loki and the index store, resulting in slower queries.
|
||
|
|
||
|
Response:
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"streams": [
|
||
|
{
|
||
|
"labels": "<LogQL label key-value pairs>",
|
||
|
"entries": [
|
||
|
{
|
||
|
"ts": "<RFC3339Nano string>",
|
||
|
"line": "<log line>"
|
||
|
},
|
||
|
...
|
||
|
],
|
||
|
},
|
||
|
...
|
||
|
],
|
||
|
"stats": [<statistics>]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
See [statistics](#statistics) for information about the statistics returned by Loki.
|
||
|
|
||
|
#### Examples
|
||
|
|
||
![]()
2 years ago
|
```console
|
||
![]()
3 years ago
|
$ curl -G -s "http://localhost:3100/api/prom/query" --data-urlencode 'query={foo="bar"}' | jq
|
||
|
{
|
||
|
"streams": [
|
||
|
{
|
||
|
"labels": "{filename=\"/var/log/myproject.log\", job=\"varlogs\", level=\"info\"}",
|
||
|
"entries": [
|
||
|
{
|
||
|
"ts": "2019-06-06T19:25:41.972739Z",
|
||
|
"line": "foo"
|
||
|
},
|
||
|
{
|
||
|
"ts": "2019-06-06T19:25:41.972722Z",
|
||
|
"line": "bar"
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
],
|
||
|
"stats": {
|
||
|
...
|
||
|
}
|
||
|
}
|
||
|
```
|
||
|
|
||
|
### `GET /api/prom/label/<name>/values`
|
||
|
|
||
|
> **WARNING**: `/api/prom/label/<name>/values` is DEPRECATED; use `/loki/api/v1/label/<name>/values`
|
||
|
|
||
|
`/api/prom/label/<name>/values` retrieves the list of known values for a given
|
||
|
label within a given time span. It accepts the following query parameters in
|
||
|
the URL:
|
||
|
|
||
|
- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
|
||
|
- `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now.
|
||
![]()
2 years ago
|
- `since`: A `duration` used to calculate `start` relative to `end`. If `end` is in the future, `start` is calculated as this duration before now. Any value specified for `start` supersedes this parameter.
|
||
![]()
3 years ago
|
|
||
|
In microservices mode, `/api/prom/label/<name>/values` is exposed by the querier.
|
||
|
|
||
|
Response:
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"values": [
|
||
|
<label value>,
|
||
|
...
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
#### Examples
|
||
|
|
||
![]()
2 years ago
|
```console
|
||
![]()
3 years ago
|
$ curl -G -s "http://localhost:3100/api/prom/label/foo/values" | jq
|
||
|
{
|
||
|
"values": [
|
||
|
"cat",
|
||
|
"dog",
|
||
|
"axolotl"
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
### `GET /api/prom/label`
|
||
|
|
||
![]()
2 years ago
|
> **WARNING**: `/api/prom/label` is DEPRECATED; use `/loki/api/v1/labels`
|
||
![]()
3 years ago
|
|
||
|
`/api/prom/label` retrieves the list of known labels within a given time span. It
|
||
|
accepts the following query parameters in the URL:
|
||
|
|
||
|
- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
|
||
|
- `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now.
|
||
![]()
2 years ago
|
- `since`: A `duration` used to calculate `start` relative to `end`. If `end` is in the future, `start` is calculated as this duration before now. Any value specified for `start` supersedes this parameter.
|
||
![]()
3 years ago
|
|
||
|
In microservices mode, `/api/prom/label` is exposed by the querier.
|
||
|
|
||
|
Response:
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"values": [
|
||
|
<label string>,
|
||
|
...
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
#### Examples
|
||
|
|
||
![]()
2 years ago
|
```console
|
||
![]()
3 years ago
|
$ curl -G -s "http://localhost:3100/api/prom/label" | jq
|
||
|
{
|
||
|
"values": [
|
||
|
"foo",
|
||
|
"bar",
|
||
|
"baz"
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
### `POST /api/prom/push`
|
||
|
|
||
|
> **WARNING**: `/api/prom/push` is DEPRECATED; use `/loki/api/v1/push`
|
||
|
> instead.
|
||
|
|
||
|
`/api/prom/push` is the endpoint used to send log entries to Loki. The default
|
||
|
behavior is for the POST body to be a snappy-compressed protobuf message:
|
||
|
|
||
![]()
2 years ago
|
- [Protobuf definition](https://github.com/grafana/loki/blob/main/pkg/logproto/logproto.proto)
|
||
|
- [Go client library](https://github.com/grafana/loki/blob/main/clients/pkg/promtail/client/client.go)
|
||
![]()
3 years ago
|
|
||
|
Alternatively, if the `Content-Type` header is set to `application/json`, a
|
||
|
JSON post body can be sent in the following format:
|
||
|
|
||
|
```
|
||
|
{
|
||
|
"streams": [
|
||
|
{
|
||
|
"labels": "<LogQL label key-value pairs>",
|
||
|
"entries": [
|
||
|
{
|
||
|
"ts": "<RFC3339Nano string>",
|
||
|
"line": "<log line>"
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
]
|
||
|
}
|
||
|
```
|
||
|
|
||
|
In microservices mode, `/api/prom/push` is exposed by the distributor.
|
||
|
|
||
|
#### Examples
|
||
|
|
||
![]()
2 years ago
|
```console
|
||
![]()
3 years ago
|
$ curl -H "Content-Type: application/json" -XPOST -s "https://localhost:3100/api/prom/push" --data-raw \
|
||
|
'{"streams": [{ "labels": "{foo=\"bar\"}", "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "fizzbuzz" }] }]}'
|
||
|
```
|
||
|
|
||
|
### `POST /ingester/flush_shutdown`
|
||
|
|
||
|
> **WARNING**: `/ingester/flush_shutdown` is DEPRECATED; use `/ingester/shutdown?flush=true`
|
||
|
> instead.
|
||
|
|
||
|
`/ingester/flush_shutdown` triggers a shutdown of the ingester and notably will _always_ flush any in memory chunks it holds.
|
||
|
This is helpful for scaling down WAL-enabled ingesters where we want to ensure old WAL directories are not orphaned,
|
||
|
but instead flushed to our chunk backend.
|
||
|
|
||
![]()
2 years ago
|
In microservices mode, the `/ingester/flush_shutdown` endpoint is exposed by the ingester.
|