**What this PR does / why we need it**:
Legacy storage backends (that are not single store aka "shipper") have been deprecated with https://github.com/grafana/loki/pull/10341 The docs should reflect this change.
**Special notes for your reviewer**:
The structure of the page changed _slightly_:
* Made object storage a separate section under "Chunk storage"
* Refer to this section in the "Single Store" section for supported backends
* Moved Azure example underneath the rest of the single store examples
While it is not yet perfect, I tried to change as little as possible and keep the big change for when the legacy storage backends are removed.
---------
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@ -11,85 +11,110 @@ is then compressed and stored in chunks in object stores such as S3 or GCS, or
even locally on the filesystem. A small index and highly compressed chunks
simplifies the operation and significantly lowers the cost of Loki.
Until Loki 2.0, index data was stored in a separate index.
Prior to Loki 2.0, chunks and index data were stored in separate backends:
object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data.
Loki 2.0 brings an index mechanism named 'boltdb-shipper' and is what we now call Single Store Loki.
This index type only requires one store, the object store, for both the index and chunks.
Loki 2.0 brings an index mechanism named 'boltdb-shipper' and is what we now call [Single Store](#single-store).
This type only requires one store, the object store, for both the index and chunks.
More detailed information can be found on the [operations page]({{< relref "../operations/storage/boltdb-shipper.md" >}}).
Loki 2.8 adds TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki.
Some more storage details can also be found in the [operations section]({{< relref "../operations/storage/_index.md" >}}).
## Implementations - Chunks
## Single Store
### Cassandra
Single Store refers to using object storage as the storage medium for both Loki's index as well as its data ("chunks"). There are two supported modes:
Cassandra is a popular database and one of Loki's possible chunk stores and is production safe.
### TSDB (recommended)
### GCS
Starting in Loki 2.8, the [TSDB index store]({{< relref "../operations/storage/tsdb" >}}) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper".
GCS is a hosted object store offered by Google. It is a good candidate for a managed object store, especially when you're already running on GCP, and is production safe.
### BoltDB (deprecated)
### File System
Also known as "boltdb-shipper" during development (and is still the schema `store` name). The single store configurations for Loki utilize the chunk store for both chunks and the index, requiring just one store to run Loki.
Performance is comparable to a dedicated index type while providing a much less expensive and less complicated deployment.
When using Single Store, no extra [Chunk storage](#chunk-storage) and [Index storage](#index-storage) are necessary.
### Supported storage backends
See [Object Storage](#object-storage) for supported backends.
## Chunk storage
### File system
The file system is the simplest backend for chunks, although it's also susceptible to data loss as it's unreplicated. This is common for single binary deployments though, as well as for those trying out loki or doing local development on the project. It is similar in concept to many Prometheus deployments where a single Prometheus is responsible for monitoring a fleet.
### S3
### Object storage
#### Google Cloud Storage (GCS)
GCS is a hosted object store offered by Google. It is a good candidate for a managed object store, especially when you're already running on GCP, and is production safe.
#### Amazon Simple Storage Storage (S3)
S3 is AWS's hosted object store. It is a good candidate for a managed object store, especially when you're already running on AWS, and is production safe.
### Azure Blob Storage
#### Azure Blob Storage
Blob Storage is Microsoft Azure's hosted object store. It is a good candidate for a managed object store, especially when you're already running on Azure, and is production safe.
You can authenticate Blob Storage access by using a storage account name and key or by using a Service Principal.
### IBM Cloud Object Storage (COS)
#### IBM Cloud Object Storage (COS)
[COS](https://www.ibm.com/cloud/object-storage) is IBM Cloud hosted object store. It is a good candidate for a managed object store, especially when you're already running on IBM Cloud, and is production safe.
### Notable Mentions
#### Baidu Object Storage (BOS)
You may use any substitutable services, such as those that implement the S3 API like [MinIO](https://min.io/).
[BOS](https://intl.cloud.baidu.com/product/bos.html) is the Baidu CLoud hosted object storage.
## Implementations - Index
#### Alibaba Object Storage Service (OSS)
### Single-Store
[OSS](https://www.alibabacloud.com/product/object-storage-service) is the Alibaba Cloud hosted object storage.
Single-Store refers to using object storage as the storage medium for both Loki's index as well as its data ("chunks"). There are two supported modes:
#### Other notable mentions
#### tsdb (recommended)
You may use any substitutable services, such as those that implement the S3 API like [MinIO](https://min.io/).
Starting in Loki v2.8, the [TSDB index store]({{< relref "../operations/storage/tsdb" >}}) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper".
### Cassandra (deprecated)
#### BoltDB (deprecated)
Cassandra is a popular database and one of Loki's possible chunk stores and is production safe.
Also known as "boltdb-shipper" during development (and is still the schema `store` name). The single store configurations for Loki utilize the chunk store for both chunks and the index, requiring just one store to run Loki.
> **Note:** This storage type for chunks is deprecated and may be removed in future major versions of Loki.
Performance is comparable to a dedicated index type while providing a much less expensive and less complicated deployment.
## Index storage
### Cassandra
### Cassandra (deprecated)
Cassandra can also be utilized for the index store and aside from the [boltdb-shipper]({{< relref "../operations/storage/boltdb-shipper" >}}), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.
### BigTable
> **Note:** This storage type for indexes is deprecated and may be removed in future major versions of Loki.
### BigTable (deprecated)
Bigtable is a cloud database offered by Google. It is a good candidate for a managed index store if you're already using it (due to its heavy fixed costs) or wish to run in GCP.
### DynamoDB
> **Note:** This storage type for indexes is deprecated and may be removed in future major versions of Loki.
### DynamoDB (deprecated)
DynamoDB is a cloud database offered by AWS. It is a good candidate for a managed index store, especially if you're already running in AWS.
#### Rate Limiting
> **Note:** This storage type for indexes is deprecated and may be removed in future major versions of Loki.
#### Rate limiting
DynamoDB is susceptible to rate limiting, particularly due to overconsuming what is called [provisioned capacity](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html). This can be controlled via the [provisioning](#provisioning) configs in the table manager.
### BoltDB
### BoltDB (deprecated)
BoltDB is an embedded database on disk. It is not replicated and thus cannot be used for high availability or clustered Loki deployments, but is commonly paired with a `filesystem` chunk store for proof of concept deployments, trying out Loki, and development. The [boltdb-shipper]({{< relref "../operations/storage/boltdb-shipper" >}}) aims to support clustered deployments using `boltdb` as an index.
### Azure Storage Account
An Azure storage account contains all of your Azure Storage data objects: blobs, file shares, queues, tables, and disks.
> **Note:** This storage type for indexes is deprecated and may be removed in future major versions of Loki.
## Schema Configs
## Schema Config
Loki aims to be backwards compatible and over the course of its development has had many internal changes that facilitate better and more efficient storage/querying. Loki allows incrementally upgrading to these new storage _schemas_ and can query across them transparently. This makes upgrading a breeze. For instance, this is what it looks like when migrating from the v10 -> v11 schemas starting 2020-07-01:
@ -114,9 +139,9 @@ schema_config:
For all data ingested before 2020-07-01, Loki used the v10 schema and then switched after that point to the more effective v11. This dramatically simplifies upgrading, ensuring it's simple to take advantage of new storage optimizations. These configs should be immutable for as long as you care about retention.
## Table Manager
## Table Manager (deprecated)
One of the subcomponents in Loki is the `table-manager`. It is responsible for pre-creating and expiring index tables. This helps partition the writes and reads in loki across a set of distinct indices in order to prevent unbounded growth.
One of the subcomponents in Loki is the `table-manager`. It is responsible for pre-creating and expiring index tables. This helps partition the writes and reads in Loki across a set of distinct indices in order to prevent unbounded growth.
```yaml
table_manager:
@ -308,7 +333,77 @@ This guide assumes a provisioned EKS cluster.
`bucket_name` variable.
### IBM Cloud Object Storage
### Azure deployment (Azure Blob Storage Single Store)
#### Using account name and key
```yaml
schema_config:
configs:
- from: "2020-12-11"
index:
period: 24h
prefix: index_
object_store: azure
schema: v11
store: boltdb-shipper
storage_config:
azure:
# Your Azure storage account name
account_name: <account-name>
# For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal
account_key: <account-key>
# See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers
container_name: <container-name>
use_managed_identity: <true|false>
# Providing a user assigned ID will override use_managed_identity
user_assigned_id: <user-assigned-identity-id>
request_timeout: 0
# Configure this if you are using private azure cloud like azure stack hub and will use this endpoint suffix to compose container & blob storage URL. Ex: https://account_name.endpoint_suffix/container_name/blob_name
> **Note:** Cassandra as storage backend for chunks and indexes is deprecated.
**Keeping this for posterity, but this is likely not a common config. Cassandra should work and could be faster in some situations but is likely much more expensive.**
@ -359,7 +456,7 @@ schema_config:
```
### On prem deployment (MinIO Single Store)
### On premise deployment (MinIO Single Store)
We configure MinIO by using the AWS config because MinIO implements the S3 API:
@ -386,73 +483,3 @@ schema_config:
prefix: index_
period: 24h
```
### Azure Storage Account
#### Using account name and key
```yaml
schema_config:
configs:
- from: "2020-12-11"
index:
period: 24h
prefix: index_
object_store: azure
schema: v11
store: boltdb-shipper
storage_config:
azure:
# Your Azure storage account name
account_name: <account-name>
# For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal
account_key: <account-key>
# See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers
container_name: <container-name>
use_managed_identity: <true|false>
# Providing a user assigned ID will override use_managed_identity
user_assigned_id: <user-assigned-identity-id>
request_timeout: 0
# Configure this if you are using private azure cloud like azure stack hub and will use this endpoint suffix to compose container & blob storage URL. Ex: https://account_name.endpoint_suffix/container_name/blob_name
Christian Haudum
2 years ago
committed by