From 54847efd17b2d98b91d93b30dcfd5eb80a0c451c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Wed, 19 Jun 2024 14:07:09 -0400 Subject: [PATCH] explain how to ignore WAL files and cleanup resulting grafs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit We don't want do backup WAL files, so we should show how to actually ignore those files. Also explain what happens every 2 hours a little more clearly. Move things around so the paragraphs flow more easily. Followup for #14297. Signed-off-by: Antoine Beaupré Co-authored-by: Bryan Boreham --- docs/storage.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/storage.md b/docs/storage.md index f472cce140..76b5f3da8f 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -59,12 +59,16 @@ A Prometheus server's data directory looks something like this: Note that a limitation of local storage is that it is not clustered or replicated. Thus, it is not arbitrarily scalable or durable in the face of drive or node outages and should be managed like any other single node -database. +database. With proper architecture, it is possible to retain years of +data in local storage. [Snapshots](querying/api.md#snapshot) are recommended for backups. Backups made without snapshots run the risk of losing data that was recorded since -the last WAL sync, which typically happens every two hours. With proper -architecture, it is possible to retain years of data in local storage. +the last TSDB block was created, which typically happens every two hours, +covering the last three hours of samples. Excluding the WAL files (the +`chunks_head/`, `wal/`, and `wbl/` directories in `storage.tsdb.path`) +on backup or restore will ensure a coherent backup, in any case, at the +cost of losing the time range covered by the WAL files. Alternatively, external storage may be used via the [remote read/write APIs](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage).