apitech
/
loki
mirror of https://github.com/grafana/loki
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
943 Commits
843 Branches
310 Tags
652 MiB
 
 
 
 
 
 
Tag: Branch: Tree: 2477f70431
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '2477f70431'
${ noResults }
loki/docs/operations/storage/README.md

3.6 KiB
Raw Blame History

Loki Storage

Loki needs to store two different types of data: chunks and indexes.

Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are GZipped as "chunks" and saved in the chunks store. See chunk format for how chunks are stored internally.

The index stores each stream's label set and links them to the individual chunks.

Refer to Loki's configuration for details on how to configure the storage and the index.

For more information:

  1. Table Manager
  2. Retention

Supported Stores

The following are supported for the index:

The following are supported for the chunks:

Cloud Storage Permissions

S3

When using S3 as object storage, the following permissions are needed:

  • s3:ListBucket
  • s3:PutObject
  • s3:GetObject

DynamoDB

When using DynamoDB for the index, the following permissions are needed:

  • dynamodb:BatchGetItem
  • dynamodb:BatchWriteItem
  • dynamodb:DeleteItem
  • dynamodb:DescribeTable
  • dynamodb:GetItem
  • dynamodb:ListTagsOfResource
  • dynamodb:PutItem
  • dynamodb:Query
  • dynamodb:TagResource
  • dynamodb:UntagResource
  • dynamodb:UpdateItem
  • dynamodb:UpdateTable

Chunk Format

  -------------------------------------------------------------------
  |                               |                                 |
  |        MagicNumber(4b)        |           version(1b)           |
  |                               |                                 |
  -------------------------------------------------------------------
  |         block-1 bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |         block-2 bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |         block-n bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |                        #blocks (uvarint)                        |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  |                      checksum(from #blocks)                     |
  -------------------------------------------------------------------
  |           metasOffset - offset to the point with #blocks        |
  -------------------------------------------------------------------