Backup and Restore Operations with Vault Raft Snapshots

Vault's integrated storage (Raft) lets you capture the cluster's consistent state as a snapshot. Snapshots can be taken with minimal downtime and are effective for disaster recovery and migrations.

This article focuses on the commands and ordering you actually need in operations, with Ops certification exam tips included along the way.

Raft Snapshot Basics and Safety

Vault's integrated storage is replicated across multiple nodes via Raft. Snapshots are created from the leader's applied state, so the resulting backup is consistent. You can issue the command against any node — it is forwarded internally to the leader.

A snapshot contains Vault's entire storage state (secrets, policies, auth backend configuration, namespaces, replication metadata, and so on). Server configuration files (vault.hcl), TLS certificates, and keys are not included. Treat snapshots themselves as sensitive material — store them with access control, encryption, and integrity verification in place.

Restoration requires the original Unseal keys (shards). Even if someone obtains the snapshot, they cannot unseal the data without the keys. Version compatibility is not strict — restoring to the same version as the source or to a slightly newer compatible version is safest.

• Snapshots can be taken online (no downtime is typically required)
• Any node accepts the command; the leader handles it (when forwarding is enabled)
• Snapshots only cover storage state; configuration files and certificates are managed separately
• Restoration requires the original Unseal keys
• Version compatibility is limited; stick to the same major version as a baseline

Basic verification commands (leader check and peer status)

export VAULT_ADDR=https://vault.example.com:8200
export VAULT_TOKEN=s.xxxxx  # a token with sudo capability

# Find the leader
vault status | egrep 'HA Mode|HA Cluster|Active Node Address'

# Check the Raft peers
vault operator raft list-peers

Backup Procedure (Taking a Snapshot)

Routine backups follow this order: 1) check peers and health, 2) save the snapshot to a secure temporary location, 3) compute a hash, 4) rotate according to the retention policy, and 5) record the operation in audit logs.

Snapshot creation carves out a consistency point on the leader. In large environments, the I/O can take a while, so verify free space and throughput on the destination volume in advance.

• Pre-check: ensure vault status and raft list-peers are stable
• Reserve free space at the destination (pre-compression size roughly matches your data usage)
• Store the hash and metadata (creation time, source node, Vault version) alongside the snapshot
• Watch latency during business hours (you may need to temporarily relax monitoring thresholds)

Example: taking and verifying a snapshot

# Take it
SNAP_DIR=/var/backups/vault
SNAP_FILE=${SNAP_DIR}/vault-raft-$(date +%Y%m%d-%H%M%S).snap
mkdir -p "$SNAP_DIR"

vault operator raft snapshot save "$SNAP_FILE"

# Verify integrity (hash)
sha256sum "$SNAP_FILE" | tee -a ${SNAP_DIR}/SHA256SUMS

# Record the metadata
{
  echo "created_at=$(date -Is)"
  vault version | xargs echo "vault_version="
  vault operator raft list-peers -format=json | jq -r '.data.leader_address' | xargs echo "leader="
} | tee -a ${SNAP_FILE}.meta

# Rotate (keep 30 generations, for example)
ls -1t ${SNAP_DIR}/vault-raft-*.snap | tail -n +31 | xargs -r rm -f

Snapshot Storage, Encryption, and Retention

Replicate snapshots to at least an offsite location or a different AZ, and send the hash and metadata along with them to detect tampering. On cloud storage, versioning and WORM (object lock) further improve recovery resilience.

Double up encryption in transit and at rest where possible. Combine SSE-KMS and GPG encryption and keep the set of people holding decryption keys small. Logging backup success/failure and hashes to your audit trail also makes compliance reviews easier.

• Upload immediately to redundant storage like S3 or GCS
• Server-side encryption (SSE-KMS) or client-side encryption (GPG)
• Use WORM, versioning, and lifecycle rules for generation management
• Run restore drills quarterly and record measured recovery times

Example: transfer to S3 (SSE-KMS) and GPG encryption

# Upload to S3 (encrypted with a KMS key)
AWS_BUCKET=s3://org-backup-vault
aws s3 cp "$SNAP_FILE" "$AWS_BUCKET" \
  --sse aws:kms --sse-kms-key-id arn:aws:kms:us-east-1:123456789012:key/xxxx
aws s3 cp "${SNAP_FILE}.meta" "$AWS_BUCKET"
aws s3 cp "${SNAP_DIR}/SHA256SUMS" "$AWS_BUCKET"

# Encrypting client-side with GPG before sending
gpg --encrypt --recipient [email protected] "$SNAP_FILE"
aws s3 cp "${SNAP_FILE}.gpg" "$AWS_BUCKET"

Restore Procedure (Single-Node Recovery and Full Cluster Rebuild)

Whether you are rebuilding a single node or recovering the entire cluster, the standard approach is to first apply the snapshot to one node, start and unseal it, and then have the other nodes join. Overwriting a node that still holds existing data will fail, so the target node's Raft data directory must be empty.

Restore on the same Vault version as the source, or on a compatible one. After restoration, expired dynamic secrets and leases are cleaned up based on their TTLs. Server configuration (vault.hcl) and TLS certificates are not in the snapshot, so prepare equivalent versions separately.

• Full recovery order: apply to one node → unseal → other nodes join
• Empty the target node's data_dir (existing Raft logs cause errors)
• vault operator raft snapshot restore needs -force to acknowledge the overwrite
• After recovery, verify with raft list-peers and functional checks (auth, secret reads)

Example: full cluster recovery (assuming systemd)

# 1) Stop every Vault node
sudo systemctl stop vault

# 2) Delete the data on node A only (the recovery target)
sudo rm -rf /opt/vault/data/*

# 3) Start node A
sudo systemctl start vault
export VAULT_ADDR=https://node-a.example.com:8200
export VAULT_TOKEN=s.xxxxx

# 4) Apply the snapshot (on A)
vault operator raft snapshot restore -force /var/backups/vault/vault-raft-20240401-000000.snap

# 5) Unseal (with the original unseal keys)
vault operator unseal
vault operator unseal
vault operator unseal

# 6) Check it works
vault status
vault operator raft list-peers

# 7) Initialize and start nodes B and C, then join from each
# Run this on node B (after exporting B's VAULT_ADDR)
vault operator raft join https://node-a.example.com:8200
vault operator unseal

# The same on node C
vault operator raft join https://node-a.example.com:8200
vault operator unseal

Comparison with Alternatives and Architecture Overview

Backup options for Vault include Raft snapshots, filesystem-level copies, and Consul snapshots (when Consul storage is in use). When you are running on Raft integrated storage, Raft snapshots are almost always the safest choice.

When designing operations, decide upfront on snapshot storage location, encryption, retention, separate handling for configuration files and certificates, and recovery drill frequency. That preparation eliminates hesitation when a real incident hits.

• On Raft, use Raft snapshots as the first choice
• Configuration files and TLS materials need separate backups
• Always validate version compatibility for migrations (rehearse in staging)

Raft snapshot capture and restore flow (conceptual diagram)

For comparison: example with the Consul backend (reference only)

# For reference, when Vault uses Consul storage (not applicable to Raft integrated storage)
# A Consul snapshot
consul snapshot save consul-$(date +%Y%m%d).snap
# Restore
consul snapshot restore consul-20240401.snap

Exam Checklist and Troubleshooting

Ops exams frequently ask about the snapshot commands, the restore order, the requirement for Unseal keys, the fact that configuration is not in the snapshot, and how version compatibility is handled. Make sure you can state the recovery order — apply to one node, unseal, then join — without hesitation.

Typical errors include restoring without clearing the data directory (existing logs make it fail), leader forwarding failures during network partitions, and version-compatibility errors. Automating the restore procedure in staging beforehand minimizes hesitation when production is on the line.

• Memorize the commands: vault operator raft snapshot save / restore (restore needs -force)
• Back up configuration and TLS separately — they are not in the snapshot
• Unsealing is mandatory after restore — Unseal key management is often the bottleneck
• Run join from the new node and point it at the leader's address
• Restore on a version equal to or newer than the source (watch for backward-incompatible changes)

Commands for troubleshooting

# Is forwarding to the leader working?
vault status

# Understand the peers and their voting status
vault operator raft list-peers -format=json | jq .

# Snapshot size and recomputed hash
ls -lh /var/backups/vault/*.snap
sha256sum /var/backups/vault/*.snap

# Autopilot state (useful for judging stability)
vault operator raft autopilot state

Check with a Practice Question

Frequently Asked Questions