loki/cmd/chunks-inspect

Tools for inspecting Loki chunks

This tool can parse Loki chunks and print details from them. Useful for Loki developers.

To build the tool, simply run go build in this directory. Running resulting program with chunks file name gives you some basic chunks information:

$ ./chunks-inspect db61b4eca2a5ad68\:16f89ff4164\:16f8a0cfb41\:1538ace0 

Chunks file: db61b4eca2a5ad68:16f89ff4164:16f8a0cfb41:1538ace0
Metadata length: 485
Data length: 264737
UserID: 29
From: 2020-01-09 11:10:04.644000 UTC
Through: 2020-01-09 11:25:04.193000 UTC (14m59.549s)
Labels:
	 __name__ = logs
	 app = graphite
	 cluster = us-central1
	 color = b
	 container_name = graphite
	 filename = /var/log/pods/metrictank_graphite-1-large-multitenant-b-5f9db68b5c-jh769_ca9a10b0-0d2d-11ea-b85a-42010a80017a/graphite/0.log
	 hosted_metrics = 1
	 instance = graphite-1-large-multitenant-b-5f9db68b5c-jh769
	 job = metrictank/graphite
	 namespace = metrictank
	 org = 1
	 plan = large
	 pod_template_hash = 5f9db68b5c
	 stream = stderr
Encoding: lz4
Blocks Metadata Checksum: 3444d7a3 OK
Found 5 block(s), use -b to show block details
Minimum time (from first block): 2020-01-09 11:10:04.644490 UTC
Maximum time (from last block): 2020-01-09 11:25:04.192368 UTC
Total size of original data: 1257319 file size: 265226 ratio: 4.74

To print more details about individual blocks inside chunks, use -b parameter:

$ ./chunks-inspect -b db61b4eca2a5ad68\:16f89ff4164\:16f8a0cfb41\:1538ace0

... chunk file info, see above ...
 
Block    0: position:        6, original length: 273604 (stored:  56220, ratio: 4.87), minT: 2020-01-09 11:10:04.644490 UTC maxT: 2020-01-09 11:12:53.458289 UTC, checksum: 13e73d71 OK
Block    0: digest compressed: ae657fdbb2b8be55eebe86b31a21050de2b5e568444507e5958218710ddf02fd, original: 0dad619bf3049a1152cb3153d90c6db6c3f54edbf9977753dde3c4e1b09d07b4
Block    1: position:    56230, original length: 274703 (stored:  60861, ratio: 4.51), minT: 2020-01-09 11:12:53.461855 UTC maxT: 2020-01-09 11:16:35.420787 UTC, checksum: 55269e65 OK
Block    1: digest compressed: a7999f471f68cce0458ff9790e7e7501c5bfe14cc28661d8670b9d88aeaee96f, original: a617a9e0b6c33aeaa83833470cf6164c540a7a64258e55eec6fdff483059df6f
Block    2: position:   117095, original length: 273592 (stored:  56563, ratio: 4.84), minT: 2020-01-09 11:16:35.423228 UTC maxT: 2020-01-09 11:19:28.680048 UTC, checksum: 781dba21 OK
Block    2: digest compressed: 65b59cc61c5eeea8116ce8a8c0b0d98b4d4671e8bc91656979c93717050a18fc, original: 896cc6487365ad0590097794a202aad5c89776d1c626f2cea33c652885939ac6
Block    3: position:   173662, original length: 273745 (stored:  57486, ratio: 4.76), minT: 2020-01-09 11:19:31.062836 UTC maxT: 2020-01-09 11:23:13.562630 UTC, checksum: 2a88a52b OK
Block    3: digest compressed: 4f51a64d0397cc806a898cd6662695620083466f234d179fef5c2d02c9766191, original: 15e8a1833ccbba9aa8374029141a054127526382423d3a63f321698ff8e087b5
Block    4: position:   231152, original length: 161675 (stored:  33440, ratio: 4.83), minT: 2020-01-09 11:23:15.416284 UTC maxT: 2020-01-09 11:25:04.192368 UTC, checksum: 6d952296 OK
Block    4: digest compressed: 8dd12235f1d619c30a9afb66823a6c827613257773669fda6fbfe014ed623cd1, original: 1f7e8ef8eb937c87ad3ed3e24c321c40d43534cc43662f83ab493fb3391548b2
Total size of original data: 1257319 file size: 265226 ratio: 4.74

To also print individual log lines, use -l parameter. Full help:

$ ./chunks-inspect -h
Usage of ./chunks-inspect:
  -b	print block details
  -l	print log lines
  -s	store blocks, using input filename, and appending block index to it

Parameter -s allows you to inspect individual blocks, both in compressed format (as stored in chunk file), and original raw format.