Vault Integrated Storage: Practical Guide to Raft Peers (Adding and Removing Nodes)

This article focuses on adding and removing nodes from a Vault Integrated Storage (Raft) cluster. It walks through how to maintain odd-numbered topologies, preserve quorum, design TLS and addressing, and execute removals during planned downtime or outages — with a strong operations bias.

For the exam, knowing which commands apply where (leader vs. any node), how to calculate quorum, the requirements for join and remove, and the role of Autopilot translates directly into points. The discussion follows the terminology and behavior in the official documentation.

Baseline and Recommended Topology: The Foundation of Raft Cluster Operations

Vault's Integrated Storage uses Raft for both replication and leader election. Availability hinges on a majority of voting nodes being reachable. The recommended topology is an odd number — 3 or 5 nodes — and any plan to add or remove members must keep the cluster above quorum at every step.

Each node holds a local raft data directory and uses an API port (default 8200) plus an internal cluster-communication port (default 8201), both expected to run over TLS. api_addr and cluster_addr must be configured with valid FQDNs/SANs, and mutual name resolution and connectivity are prerequisites for adding a node.

• Core commands: vault operator raft list-peers / join / remove-peer / autopilot state / step-down
• Keep the count odd (3 or 5). Add or remove one node at a time, and verify quorum at every step
• Before adding a node, confirm the new node's raft data directory is empty and that TLS/SAN and port (8200/8201) connectivity all work

Minimal Vault server configuration example (for a new node)

storage "raft" {
  path    = "/opt/vault/data"           # Point at an empty directory
  node_id = "vault-3"                    # Must be unique per node
}

cluster_name = "vault-prod"
api_addr     = "https://vault-3.example.com:8200"
cluster_addr = "https://vault-3.example.com:8201"

listener "tcp" {
  address        = "0.0.0.0:8200"
  tls_cert_file  = "/etc/vault.d/certs/vault.crt"
  tls_key_file   = "/etc/vault.d/certs/vault.key"
}

tls_disable = 0

Adding a Node (Online Scale-Out)

Here is the canonical procedure for adding a new node to a running cluster. You can scale out while preserving availability. Before starting the target node, configuration, certificates, and port connectivity must all be in place.

Run join from the new node. Set VAULT_ADDR to the new node's own API, and pass the API address of the existing cluster's leader (or any reachable member) as the argument.

• Prepare the new node: configuration file, TLS certificate (SAN must include the api_addr/cluster_addr hostnames), and an empty raft data directory
• Start the new node and unseal if needed (auto-unseal is recommended)
• On the new node, run vault operator raft join https://<leader>:8200
• From the existing cluster, run vault operator raft list-peers to confirm the peer was added

Node addition flow (overview)

Before (3 nodes):

  [vault-1] (Leader) ---- [vault-2] (Follower)
          \
           \---------------- [vault-3] (Follower)

Add vault-4:

  New Node: [vault-4]
    - start w/ empty raft dir
    - api_addr/cluster_addr set
    - join -> https://vault-1:8200

After (4 nodes, recommended end state is 5 nodes):

  [vault-1] (Leader) ---- [vault-2]
          \
           \---- [vault-3] ---- [vault-4]

Example commands for adding a node

# Run on the new node (targeting its own Vault API)
export VAULT_ADDR="https://vault-4.example.com:8200"
export VAULT_CACERT="/etc/vault.d/certs/ca.crt"

# Specify the existing cluster's API address and join
vault operator raft join https://vault-1.example.com:8200

# Verify peers from any node in the existing cluster
export VAULT_ADDR="https://vault-1.example.com:8200"
vault operator raft list-peers

Removing a Node (Planned Downtime and Failure Recovery)

Always perform removals in a configuration that preserves quorum. Dropping one node from a 5-node cluster leaves 4, where the majority is 3. Plan so that you never stop two nodes at the same time in the next operation.

For planned downtime (the node is healthy), if the target is the leader, run step-down first, then remove it from cluster membership with remove-peer. When the node is permanently failed and will not return, run remove-peer with the peer_id as well.

• Get the peer_id: vault operator raft list-peers (note the Node ID)
• If the target is the leader: vault operator step-down (force a new leader election first)
• Remove: vault operator raft remove-peer -peer-id=<PEER_ID>
• Stop the OS service and handle the data directory: if it will not be reused, destroy it safely (do not let it contaminate another cluster)

Safe removal procedure (planned downtime / failure)

# 1) Check peers (from any node)
export VAULT_ADDR="https://vault-1.example.com:8200"
vault operator raft list-peers
# Identify the target PEER_ID from the Node ID column

# 2) If the target is the leader, step it down first
vault operator step-down

# 3) Remove it from membership
vault operator raft remove-peer -peer-id=<PEER_ID>

# 4) Stop the service and handle data on the target node
# systemctl stop vault
# rm -rf /opt/vault/data  # Only if not reused (be careful)

Quorum and Availability Math: Guidance for Avoiding Concurrent Work

Raft refuses writes unless a majority of voting nodes is alive. For additions, removals, and rolling upgrades, always verify quorum based on the count remaining at the next instant.

Documenting basic rules in your operations runbook — never stop two or more nodes simultaneously, freeze planned work when an incident occurs, and prioritize stabilizing the cluster first — prevents avoidable incidents.

• 3-node cluster: tolerates 1 node down (majority is 2/3)
• 5-node cluster: tolerates 2 nodes down (majority is 3/5), but planned downtime should still be one node at a time
• Before any removal, check list-peers and autopilot state to confirm the cluster is healthy

Quick health check

export VAULT_ADDR="https://vault-1.example.com:8200"
# Peer information
vault operator raft list-peers
# Autopilot health (lag and candidacy)
vault operator raft autopilot state

TLS and Addressing: Typical Causes of join Failures and How to Fix Them

Most join failures come from mismatched TLS/SAN and address settings. api_addr is the URL clients and CLIs reach, while cluster_addr is used for inter-node replication (port 8201). Confirm that the certificate's SAN includes both hostnames (or an appropriate wildcard / SubjectAltName).

Another classic cause is attempting to join with a raft directory that already contains data. The new node's raft data directory must be empty.

• Firewall / NACL: allow bidirectional traffic on 8200 (API) and 8201 (cluster)
• Mismatched cluster_name values cause membership confusion — keep them unified
• Duplicate node_id values are not allowed — assign unique IDs
• Be rigorous about distributing the CA certificate and setting VAULT_CACERT / VAULT_CAPATH

Examples for pre-checking TLS/SAN and connectivity

# Check the SAN (whether the cert extension includes the api_addr/cluster_addr hosts)
openssl x509 -in /etc/vault.d/certs/vault.crt -noout -text | grep -A1 "Subject Alternative Name"

# API / cluster connectivity
curl --cacert /etc/vault.d/certs/ca.crt https://vault-1.example.com:8200/v1/sys/health
curl --cacert /etc/vault.d/certs/ca.crt https://vault-1.example.com:8201/raft/configuration || true  # On 8201 we just confirm reachability even if cert verification fails

Automation and Rolling Maintenance: Lock In Repeatable Steps

For rolling upgrades or OS patching, work one node at a time: if the target is the leader, run step-down, then stop, start, and verify health. Run list-peers and autopilot state each time to keep the cluster from dropping below quorum.

Autopilot helps with cluster health monitoring and cleaning up unneeded servers. In operations, lean on state visibility first, and apply any configuration changes (set-config) in stages — validate before promoting to production.

• step-down is the standard primitive for a safe leader handover
• Diff list-peers and autopilot state before and after each operation
• Even with automation (IaC / scripts), keep operations strictly per-node and enforce wait conditions

Sample shell snippet for rolling operations (excerpt)

# Step down if the target host is the leader
vault operator step-down || true
# Verify health
vault operator raft list-peers
vault operator raft autopilot state
# Apply OS / binary updates here -> restart
# Re-verify health and move on to the next node

Check Your Understanding

Frequently Asked Questions