@ -11,16 +11,17 @@ is then compressed and stored in chunks in object stores such as S3 or GCS, or
even locally on the filesystem. A small index and highly compressed chunks
simplifies the operation and significantly lowers the cost of Loki.
Prior to Loki 2.0, chunks and index data were stored in separate backends:
object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data.
Loki 2.8 introduced TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki.
More detailed information about TSDB can be found under the [manage section](https://grafana.com/docs/loki/latest/operations/storage/tsdb/).
Loki 2.0 brings an index mechanism named 'boltdb-shipper' and is what we now call [Single Store](#single-store).
Loki 2.0 introduced an index mechanism named 'boltdb-shipper' and is what we now call [Single Store](#single-store).
This type only requires one store, the object store, for both the index and chunks.
More detailed information can be found on the [operations page]({{< relref "../operations/storage/boltdb-shipper.md" >}}).
More detailed information about 'boltdb-shipper' can be found under the [manage section](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/).
Loki 2.8 adds TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki.
Prior to Loki 2.0, chunks and index data were stored in separate backends:
object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data. These "multistore" backends have been deprecated, as noted below.
You can find more storage information in the [manage section]({{< relref "../operations/storage/_index.md" >}}).
You can find more detailed information about all of the storage options in the [manage section](https://grafana.com/docs/loki/latest/operations/storage/).
## Single Store
@ -28,7 +29,7 @@ Single Store refers to using object storage as the storage medium for both Loki'
### TSDB (recommended)
Starting in Loki 2.8, the [TSDB index store]({{< relref "../operations/storage/tsdb" >}}) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper".
Starting in Loki 2.8, the [TSDB index store](https://grafana.com/docs/loki/latest/operations/storage/tsdb/) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper".
### BoltDB (deprecated)
@ -90,7 +91,7 @@ This storage type for chunks is deprecated and may be removed in future major ve
### Cassandra (deprecated)
Cassandra can also be utilized for the index store and aside from the [boltdb-shipper]({{< relref "../operations/storage/boltdb-shipper" >}}), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.
Cassandra can also be utilized for the index store and aside from the [boltdb-shipper](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.
{{<collapsetitle="Title of hidden content">}}
This storage type for indexes is deprecated and may be removed in future major versions of Loki.
- [TSDB]({{< relref "./tsdb" >}}) index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer.
- [Single Store (boltdb-shipper)]({{< relref "./boltdb-shipper" >}}) index store which stores boltdb index files in the object store.
- [TSDB](https://grafana.com/docs/loki/latest/operations/storage/tsdb/) index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer.
- [Single Store (boltdb-shipper)](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/) index store which stores boltdb index files in the object store.
See the [AWS deployment section]({{< relref "../../storage#aws-deployment-s3-single-store" >}}) on the storage page for a detailed setup guide.
See the [AWS deployment section](https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store) on the storage page for a detailed setup guide.
When using IBM Cloud Object Storage (COS) as object storage, IAM `Writer` role is needed.
See the [IBM Cloud Object Storage section]({{< relref "../../storage#ibm-deployment-cos-single-store" >}}) on the storage page for a detailed setup guide.
See the [IBM Cloud Object Storage section](https://grafana.com/docs/loki/latest/configure/storage/#ibm-deployment-cos-single-store) on the storage page for a detailed setup guide.
@ -26,7 +26,7 @@ Grafana Labs is excited to announce the release of Loki 2.9.0 Here's a summary o
- **logfmt stage improvements**: logfmt parser now performs non-strict parsing by default which helps scan semi-structured log lines. [PR #9626](https://github.com/grafana/loki/pull/9626)
- **Deprecations**
- Legacy index and chunk stores that are not "single store" (such as `tsdb`, `boltdb-shipper`) are deprecated. These storage backends are Cassandra (`cassandra`), DynamoDB (`aws`, `aws-dynamo`), BigTable (`bigtable`, `bigtable-hashed`), GCP (`gcp`, `gcp-columnkey`), and gRPC (`grpc`). See https://grafana.com/docs/loki/latest/storage/ for more information.
- Legacy index and chunk stores that are not "single store" (such as `tsdb`, `boltdb-shipper`) are deprecated. These storage backends are Cassandra (`cassandra`), DynamoDB (`aws`, `aws-dynamo`), BigTable (`bigtable`, `bigtable-hashed`), GCP (`gcp`, `gcp-columnkey`), and gRPC (`grpc`). See https://grafana.com/docs/loki/latest/configure/storage.md for more information.
- The `table-manager` target is deprecated, because it is not used by "single store" implementations.
- The `-boltdb.shipper.compactor.*` CLI flags are deprecated in favor of `-compactor.*`.
- The `-ingester.unordered-writes` CLI flag is deprecated and will always default to `true` in the next major release.
@ -38,7 +38,7 @@ This guide assumes Loki will be installed in one of the modes above and that a `
**To grant access to S3 via an IAM role without providing credentials:**
1. Provision an IAM role, policy and S3 bucket as described in [Storage]({{< relref "../../../../storage#aws-deployment-s3-single-store" >}}).
1. Provision an IAM role, policy and S3 bucket as described in [Storage](https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store).
- If the Terraform module was used note the annotation emitted by `terraform output -raw annotation`.
1. Add the IAM role annotation to the service account in `values.yaml`:
[TSDB]({{< relref "../../../operations/storage/tsdb" >}}) is the recommended index type for Loki and is where the current development lies.
If you are running Loki with [boltb-shipper]({{< relref "../../../operations/storage/boltdb-shipper" >}}) or any of the [legacy index types]({{< relref "../../../storage#index-storage" >}}) that have been deprecated,
If you are running Loki with [boltb-shipper]({{< relref "../../../operations/storage/boltdb-shipper" >}}) or any of the [legacy index types](https://grafana.com/docs/loki/latest/configure/storage/#index-storage) that have been deprecated,
we strongly recommend migrating to TSDB.
### Configure TSDB index for an upcoming period
To begin the migration, add a new [period_config]({{< relref "../../../configure#period_config" >}}) entry in your [schema_config]({{< relref "../../../configure#schema_config" >}}).
You can read more about schema config [here]({{< relref "../../../storage#schema-config" >}}).
You can read more about schema config [here](https://grafana.com/docs/loki/latest/configure/storage/#schema-config).
{{% admonition type="note" %}}
You must roll out the new `period_config` change to all Loki components in order for it to take effect.
`path_prefix` only applies to TSDB and BoltDB indexes. This setting has no effect on [legacy indexes]({{< relref "../../storage#index-storage" >}}).
`path_prefix` only applies to TSDB and BoltDB indexes. This setting has no effect on [legacy indexes](https://grafana.com/docs/loki/latest/configure/storage/#index-storage).
{{% /admonition %}}
`path_prefix` defaults to `index/` which is same as the default value of the removed configurations.
@ -205,7 +205,7 @@ All of these are cached to the `results_cache` which is configured in the `query
#### Write dedupe cache is deprecated
Write dedupe cache is deprecated because it not required by the newer single store indexes ([TSDB]({{<relref"../../operations/storage/tsdb">}}) and [boltdb-shipper]({{< relref "../../operations/storage/boltdb-shipper" >}})).
If you using a [legacy index type]({{< relref "../../storage#index-storage" >}}), consider migrating to TSDB (recommended).
If you using a [legacy index type](https://grafana.com/docs/loki/latest/configure/storage/#index-storage), consider migrating to TSDB (recommended).
#### Embedded cache metric changes
@ -1247,7 +1247,7 @@ schema_config:
④ Make sure this matches your existing config (e.g. maybe you were using gcs for your object_store)
⑤ 24h is required for boltdb-shipper
There are more examples on the [Storage description page]({{< relref "../../storage/_index.md#examples" >}}) including the information you need to setup the `storage` section for boltdb-shipper.
There are more examples on the [Storage description page](https://grafana.com/docs/loki/latest/configure/storage/#examples) including the information you need to setup the `storage` section for boltdb-shipper.
J Stickler
2 years ago
committed by