apitech
/
loki
mirror of https://github.com/grafana/loki
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
6499 Commits
847 Branches
311 Tags
655 MiB
 
 
 
 
 
 
Tag: Branch: Tree: 1c5a736641
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '1c5a736641'
${ noResults }
loki/docs/sources/operations/storage/_index.md

7.2 KiB
Raw Blame History

title menuTitle description weight
Manage storage Storage Describes Loki's storage needs and supported stores. <nil>

Manage storage

You can read a high level overview of Loki storage [here](https://grafana.com/docs/loki /<LOKI_VERSION>/configure/storage/)

Grafana Loki needs to store two different types of data: chunks and indexes.

Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are compressed as "chunks" and saved in the chunks store. See chunk format for how chunks are stored internally.

The index stores each stream's label set and links them to the individual chunks.

Refer to Loki's [configuration](https://grafana.com/docs/loki /<LOKI_VERSION>/configure/) for details on how to configure the storage and the index.

For more information:

Supported Stores

The following are supported for the index:

The following are deprecated for the index and will be removed in a future release:

  • Amazon DynamoDB. Support for this is deprecated and will be removed in a future release.
  • Google Bigtable. Support for this is deprecated and will be removed in a future release.
  • Apache Cassandra. Support for this is deprecated and will be removed in a future release.

The following are supported and recommended for the chunks:

The following are supported for the chunks, but not typically recommended for production use:

  • [Filesystem]({{< relref "./filesystem" >}}) (please read more about the filesystem to understand the pros/cons before using with production data)

The following are deprecated for the chunks and will be removed in a future release:

  • Amazon DynamoDB. Support for this is deprecated and will be removed in a future release.
  • Google Bigtable. Support for this is deprecated and will be removed in a future release.
  • Apache Cassandra. Support for this is deprecated and will be removed in a future release.

Cloud Storage Permissions

S3

When using S3 as object storage, the following permissions are needed:

  • s3:ListBucket
  • s3:PutObject
  • s3:GetObject
  • s3:DeleteObject (if running the Single Store (boltdb-shipper) compactor)

Resources: arn:aws:s3:::<bucket_name>, arn:aws:s3:::<bucket_name>/*

See the [AWS deployment section](https://grafana.com/docs/loki /<LOKI_VERSION>/configure/storage/#aws-deployment-s3-single-store) on the storage page for a detailed setup guide.

DynamoDB

{{% admonition type="note" %}} DynamoDB support is deprecated and will be removed in a future release. {{% /admonition %}}

When using DynamoDB for the index, the following permissions are needed:

  • dynamodb:BatchGetItem
  • dynamodb:BatchWriteItem
  • dynamodb:DeleteItem
  • dynamodb:DescribeTable
  • dynamodb:GetItem
  • dynamodb:ListTagsOfResource
  • dynamodb:PutItem
  • dynamodb:Query
  • dynamodb:TagResource
  • dynamodb:UntagResource
  • dynamodb:UpdateItem
  • dynamodb:UpdateTable
  • dynamodb:CreateTable
  • dynamodb:DeleteTable (if table_manager.retention_period is more than 0s)

Resources: arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*

  • dynamodb:ListTables

Resources: *

AutoScaling

If you enable autoscaling from table manager, the following permissions are needed:

Application Autoscaling
  • application-autoscaling:DescribeScalableTargets
  • application-autoscaling:DescribeScalingPolicies
  • application-autoscaling:RegisterScalableTarget
  • application-autoscaling:DeregisterScalableTarget
  • application-autoscaling:PutScalingPolicy
  • application-autoscaling:DeleteScalingPolicy

Resources: *

IAM
  • iam:GetRole
  • iam:PassRole

Resources: arn:aws:iam::<aws_account_id>:role/<role_name>

IBM Cloud Object Storage

When using IBM Cloud Object Storage (COS) as object storage, IAM Writer role is needed.

See the [IBM Cloud Object Storage section](https://grafana.com/docs/loki /<LOKI_VERSION>/configure/storage/#ibm-deployment-cos-single-store) on the storage page for a detailed setup guide.

Chunk Format

  -------------------------------------------------------------------
  |                               |                                 |
  |        MagicNumber(4b)        |           version(1b)           |
  |                               |                                 |
  -------------------------------------------------------------------
  |         block-1 bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |         block-2 bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |         block-n bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |                        #blocks (uvarint)                        |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  |                      checksum(from #blocks)                     |
  -------------------------------------------------------------------
  |           metasOffset - offset to the point with #blocks        |
  -------------------------------------------------------------------