apitech
/
loki
mirror of https://github.com/grafana/loki
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
959 Commits
849 Branches
312 Tags
686 MiB
 
 
 
 
 
 
Tag: Branch: Tree: 0f7f200c03
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '0f7f200c03'
${ noResults }
loki/docs/operations/storage
History
Marco Pracucci 2477f70431
docs: clarified the relation between retention period and table period (#1129) 		6 years ago
..
README.md Documentation Rewrite (#982) 6 years ago
retention.md docs: clarified the relation between retention period and table period (#1129) 6 years ago
table-manager-active-vs-inactive-tables.png docs: improved table manager documentation (#1099) 6 years ago
table-manager-periodic-tables.png docs: improved table manager documentation (#1099) 6 years ago
table-manager-retention.png docs: improved table manager documentation (#1099) 6 years ago
table-manager.md docs: improved table manager documentation (#1099) 6 years ago

README.md

Loki Storage

Loki needs to store two different types of data: chunks and indexes.

Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are GZipped as "chunks" and saved in the chunks store. See chunk format for how chunks are stored internally.

The index stores each stream's label set and links them to the individual chunks.

Refer to Loki's configuration for details on how to configure the storage and the index.

For more information:

  1. Table Manager
  2. Retention

Supported Stores

The following are supported for the index:

The following are supported for the chunks:

Cloud Storage Permissions

S3

When using S3 as object storage, the following permissions are needed:

  • s3:ListBucket
  • s3:PutObject
  • s3:GetObject

DynamoDB

When using DynamoDB for the index, the following permissions are needed:

  • dynamodb:BatchGetItem
  • dynamodb:BatchWriteItem
  • dynamodb:DeleteItem
  • dynamodb:DescribeTable
  • dynamodb:GetItem
  • dynamodb:ListTagsOfResource
  • dynamodb:PutItem
  • dynamodb:Query
  • dynamodb:TagResource
  • dynamodb:UntagResource
  • dynamodb:UpdateItem
  • dynamodb:UpdateTable

Chunk Format

  -------------------------------------------------------------------
  |                               |                                 |
  |        MagicNumber(4b)        |           version(1b)           |
  |                               |                                 |
  -------------------------------------------------------------------
  |         block-1 bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |         block-2 bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |         block-n bytes         |          checksum (4b)          |
  -------------------------------------------------------------------
  |                        #blocks (uvarint)                        |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  -------------------------------------------------------------------
  |                      checksum(from #blocks)                     |
  -------------------------------------------------------------------
  |           metasOffset - offset to the point with #blocks        |
  -------------------------------------------------------------------