apitech
/
loki
mirror of https://github.com/grafana/loki
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

chore: Update ASCII diagram of chunk binary encoding (#15386)

Browse Source 
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
pull/15487/head
Christian Haudum 1 year ago committed by GitHub
parent 2a23cc836e
commit c8a4afe2ac
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 105 additions and 50 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 77
      docs/sources/operations/storage/_index.md
  2. 78
      pkg/chunkenc/README.md

77
docs/sources/operations/storage/_index.md
Unescape View File

@ -135,29 +135,56 @@ See the [IBM Cloud Object Storage section](https://grafana.com/docs/loki/<LOKI_V
## Chunk Format
```
-------------------------------------------------------------------
| | |
| MagicNumber(4b) | version(1b) |
| | |
-------------------------------------------------------------------
| block-1 bytes | checksum (4b) |
-------------------------------------------------------------------
| block-2 bytes | checksum (4b) |
-------------------------------------------------------------------
| block-n bytes | checksum (4b) |
-------------------------------------------------------------------
| #blocks (uvarint) |
-------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
-------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
-------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
-------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
-------------------------------------------------------------------
| checksum(from #blocks) |
-------------------------------------------------------------------
| metasOffset - offset to the point with #blocks |
-------------------------------------------------------------------
// Header
+-----------------------------------+
| Magic Number (uint32, 4 bytes) |
+-----------------------------------+
| Version (1 byte) |
+-----------------------------------+
| Encoding (1 byte) |
+-----------------------------------+
// Blocks
+--------------------+----------------------------+
| block 1 (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
| block 1 (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
| ... |
+--------------------+----------------------------+
| block N (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
// Metas
+------------------------------------------------------------------------------------------------------------------------+
| #blocks (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| ... |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| checksum (uint32, 4 bytes) |
+------------------------------------------------------------------------------------------------------------------------+
// Structured Metadata
+---------------------------------+
| #labels (uvarint) |
+---------------+-----------------+
| len (uvarint) | value (n bytes) |
+---------------+-----------------+
| ... |
+---------------+-----------------+
| checksum (uint32, 4 bytes) |
+---------------------------------+
// Footer
+-----------------------+--------------------------+
| len (uint64, 8 bytes) | offset (uint64, 8 bytes) | // offset to Structured Metadata
+-----------------------+--------------------------+
| len (uint64, 8 bytes) | offset (uint64, 8 bytes) | // offset to Metas
+-----------------------+--------------------------+
```

78
pkg/chunkenc/README.md
Unescape View File

@ -1,28 +1,56 @@
# Chunk format
# Chunk v4 format
```
| | |
| MagicNumber(4b) | version(1b) |
| | |
--------------------------------------------------
| block-1 bytes | checksum (4b) |
--------------------------------------------------
| block-2 bytes | checksum (4b) |
--------------------------------------------------
| block-n bytes | checksum (4b) |
--------------------------------------------------
| #blocks (uvarint) |
--------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) |
------------------------------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) |
------------------------------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) |
------------------------------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) |
------------------------------------------------------------------------------------------------
| checksum(from #blocks) |
-------------------------------------------------------------------
| metasOffset - offset to the point with #blocks |
--------------------------------------------------
// Header
+-----------------------------------+
| Magic Number (uint32, 4 bytes) |
+-----------------------------------+
| Version (1 byte) |
+-----------------------------------+
| Encoding (1 byte) |
+-----------------------------------+
// Blocks
+--------------------+----------------------------+
| block 1 (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
| block 1 (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
| ... |
+--------------------+----------------------------+
| block N (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
// Metas
+------------------------------------------------------------------------------------------------------------------------+
| #blocks (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| ... |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| checksum (uint32, 4 bytes) |
+------------------------------------------------------------------------------------------------------------------------+
// Structured Metadata
+---------------------------------+
| #labels (uvarint) |
+---------------+-----------------+
| len (uvarint) | value (n bytes) |
+---------------+-----------------+
| ... |
+---------------+-----------------+
| checksum (uint32, 4 bytes) |
+---------------------------------+
// Footer
+-----------------------+--------------------------+
| len (uint64, 8 bytes) | offset (uint64, 8 bytes) | // offset to Structured Metadata
+-----------------------+--------------------------+
| len (uint64, 8 bytes) | offset (uint64, 8 bytes) | // offset to Metas
+-----------------------+--------------------------+
```

Write Preview
Loading…
Cancel
Save