making clearer that TSDB is the new recommended index, and make expli… (#9246)

**What this PR does / why we need it**:

making clearer that TSDB is the new recommended index, and make explicit
that non TSDB and BoltDB shipper are deprecated and not recommended
Fixes #9105 

**Special notes for your reviewer**:

This needs to be changed in more places, but this is a start as it
concerns the main storage docs

**Checklist**
- [X] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [X] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`

---------

Co-authored-by: Travis Patterson <masslessparticle@gmail.com>
Co-authored-by: J Stickler <julie.stickler@grafana.com>
Co-authored-by: Travis Patterson <travis.patterson@grafana.com>
pull/10271/head
Ward Bekker 2 years ago committed by GitHub
parent 9097f1ff42
commit 7733be0658
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 29
      docs/sources/operations/storage/_index.md
  2. 2
      docs/sources/operations/storage/boltdb-shipper.md
  3. 10
      docs/sources/operations/storage/table-manager/_index.md
  4. 2
      docs/sources/operations/storage/tsdb.md

@ -30,23 +30,38 @@ For more information:
The following are supported for the index: The following are supported for the index:
- [Single Store (boltdb-shipper) - Recommended for 2.0 and newer]({{< relref "./boltdb-shipper" >}}) index store which stores boltdb index files in the object store - [TSDB]({{< relref "./tsdb" >}}) index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer.
- [Single Store (boltdb-shipper)]({{< relref "./boltdb-shipper" >}}) index store which stores boltdb index files in the object store.
- [Amazon DynamoDB](https://aws.amazon.com/dynamodb) - [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
- [Google Bigtable](https://cloud.google.com/bigtable) - [Google Bigtable](https://cloud.google.com/bigtable)
- [Apache Cassandra](https://cassandra.apache.org) - [Apache Cassandra](https://cassandra.apache.org)
- [BoltDB](https://github.com/boltdb/bolt) (doesn't work when clustering Loki) - [BoltDB](https://github.com/boltdb/bolt) (doesn't work when clustering Loki)
The following are supported for the chunks: The following are deprecated for the index and will be removed in a future release:
- [Amazon DynamoDB](https://aws.amazon.com/dynamodb). Support for this is deprecated and will be removed in a future release.
- [Google Bigtable](https://cloud.google.com/bigtable). Support for this is deprecated and will be removed in a future release.
- [Apache Cassandra](https://cassandra.apache.org). Support for this is deprecated and will be removed in a future release.
The following are supported and recommended for the chunks:
- [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
- [Google Bigtable](https://cloud.google.com/bigtable)
- [Apache Cassandra](https://cassandra.apache.org)
- [Amazon S3](https://aws.amazon.com/s3) - [Amazon S3](https://aws.amazon.com/s3)
- [Google Cloud Storage](https://cloud.google.com/storage/) - [Google Cloud Storage](https://cloud.google.com/storage/)
- [Filesystem]({{< relref "./filesystem" >}}) (please read more about the filesystem to understand the pros/cons before using with production data) - [Filesystem]({{< relref "./filesystem" >}}) (please read more about the filesystem to understand the pros/cons before using with production data)
- [Baidu Object Storage](https://cloud.baidu.com/product/bos.html) - [Baidu Object Storage](https://cloud.baidu.com/product/bos.html)
- [IBM Cloud Object Storage](https://www.ibm.com/cloud/object-storage) - [IBM Cloud Object Storage](https://www.ibm.com/cloud/object-storage)
The following are supported for the chunks, but not typically recommended for production use:
- [Filesystem]({{< relref "./filesystem" >}}) (please read more about the filesystem to understand the pros/cons before using with production data)
The following are deprecated for the chunks and will be removed in a future release:
- [Amazon DynamoDB](https://aws.amazon.com/dynamodb). Support for this is deprecated and will be removed in a future release.
- [Google Bigtable](https://cloud.google.com/bigtable). Support for this is deprecated and will be removed in a future release.
- [Apache Cassandra](https://cassandra.apache.org). Support for this is deprecated and will be removed in a future release.
## Cloud Storage Permissions ## Cloud Storage Permissions
### S3 ### S3
@ -64,6 +79,10 @@ See the [AWS deployment section]({{< relref "../../storage#aws-deployment-s3-sin
### DynamoDB ### DynamoDB
{{% admonition type="note" %}}
DynamoDB support is deprecated and will be removed in a future release.
{{% /admonition %}}
When using DynamoDB for the index, the following permissions are needed: When using DynamoDB for the index, the following permissions are needed:
- `dynamodb:BatchGetItem` - `dynamodb:BatchGetItem`

@ -4,6 +4,8 @@ description: Single Store (boltdb-shipper)
--- ---
# Single Store (boltdb-shipper) # Single Store (boltdb-shipper)
Note that single store (boltdb-shipper) is a legacy storage option and is not recommended for new deployments. It is recommended to use the [TSDB]({{< relref "./tsdb" >}}) index instead.
BoltDB Shipper lets you run Grafana Loki without any dependency on NoSQL stores for storing index. BoltDB Shipper lets you run Grafana Loki without any dependency on NoSQL stores for storing index.
It locally stores the index in BoltDB files instead and keeps shipping those files to a shared object store i.e the same object store which is being used for storing chunks. It locally stores the index in BoltDB files instead and keeps shipping those files to a shared object store i.e the same object store which is being used for storing chunks.
It also keeps syncing BoltDB files from shared object store to a configured local directory for getting index entries created by other services of same Loki cluster. It also keeps syncing BoltDB files from shared object store to a configured local directory for getting index entries created by other services of same Loki cluster.

@ -29,11 +29,15 @@ The Table Manager supports the following backends:
- [Apache Cassandra](https://cassandra.apache.org) - [Apache Cassandra](https://cassandra.apache.org)
- [BoltDB](https://github.com/boltdb/bolt) (primarily used for local environments) - [BoltDB](https://github.com/boltdb/bolt) (primarily used for local environments)
- **Chunk store** - **Chunk store**
- [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
- [Google Bigtable](https://cloud.google.com/bigtable)
- [Apache Cassandra](https://cassandra.apache.org)
- Filesystem (primarily used for local environments) - Filesystem (primarily used for local environments)
Loki does support the following backends for both index and chunk storage, but they are deprecated and will be removed in a future release:
- [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
- [Google Bigtable](https://cloud.google.com/bigtable)
- [Apache Cassandra](https://cassandra.apache.org)
The object storages - like Amazon S3 and Google Cloud Storage - supported by Loki The object storages - like Amazon S3 and Google Cloud Storage - supported by Loki
to store chunks, are not managed by the Table Manager, and a custom bucket policy to store chunks, are not managed by the Table Manager, and a custom bucket policy
should be set to delete old data. should be set to delete old data.

@ -6,7 +6,7 @@ weight: 1000
# TSDB # TSDB
Starting with Loki v2.8, TSDB is the Loki index. It is heavily inspired by the Prometheus's TSDB [sub-project](https://github.com/prometheus/prometheus/tree/main/tsdb). For a deeper explanation you can read Owen's [blog post](https://lokidex.com/posts/tsdb/). The short version is that this new index is more efficient, faster, and more scalable. It also resides in object storage like the [boltdb-shipper]({{< relref "./boltdb-shipper" >}}) index which preceded it. Starting with Loki v2.8, TSDB is the recommended Loki index. It is heavily inspired by the Prometheus's TSDB [sub-project](https://github.com/prometheus/prometheus/tree/main/tsdb). For a deeper explanation you can read Loki maintainer Owen's [blog post](https://lokidex.com/posts/tsdb/). The short version is that this new index is more efficient, faster, and more scalable. It also resides in object storage like the [boltdb-shipper]({{< relref "./boltdb-shipper" >}}) index which preceded it.
## Example Configuration ## Example Configuration

Loading…
Cancel
Save