Vault Request Forwarding Implementation Notes: Designing and Operating Intra-Cluster Routing Correctly

In a Vault HA cluster, Request Forwarding is enabled by default: standby nodes internally forward client requests to the active node. This lets clients and load balancers operate without ever needing to know which node is the leader.

This article summarizes the implementation essentials based on stable features and the official documentation, framed for Ops exam preparation. We avoid version-specific behavior and focus on the HA fundamentals.

Request Forwarding Basics and Prerequisites

Request Forwarding is the mechanism by which a standby node internally forwards API requests it receives to the active node within the cluster. Unlike the legacy 307 redirect, the client does not need to know where the leader is — it just sends the request to any node and Vault handles the rest.

The prerequisites are simple: use an HA-capable storage backend (Raft integrated storage or Consul), set api_addr and cluster_addr correctly on every node, and enable TLS. Once these are in place, standbys forward requests automatically.

• An HA storage backend is required (Raft or Consul)
• api_addr is for client-facing traffic, cluster_addr is for inter-node communication
• Enable TLS and include the hostnames/IPs for both api_addr and cluster_addr in the certificate SAN
• Forwarding cannot occur when there is no leader, so use LB health checks that explicitly allow standbys when appropriate

Minimal configuration example (server.hcl excerpt: Raft + TLS)

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/server.crt"
  tls_key_file  = "/opt/vault/tls/server.key"
}

# クライアントが到達する FQDN/ポート
api_addr    = "https://vault-1.example.com:8200"
# ノード間通信用アドレス（通常 8201）
cluster_addr = "https://vault-1.example.com:8201"

storage "raft" {
  path = "/opt/vault/data"
  retry_join {
    leader_api_addr = "https://vault-1.example.com:8200"
  }
}

tls_disable = 0

Understanding the Communication Flow (Intra-Cluster Routing)

Clients connect through a load balancer or directly to any node. When a standby receives a request, it forwards it to the active node over inter-node TLS using cluster_addr, and the response returns to the client via the same standby. There is an extra hop, but the client implementation is much simpler.

Forwarding works regardless of API type (write requests are forwarded too). However, if the active node is unavailable or inter-node communication is severed, forwarding fails — so health monitoring and reliable network connectivity are essential.

• Clients can send to any node. Standbys receive → forward to active → return the response
• Inter-node connectivity and TLS via cluster_addr are mandatory
• When forwarding is not possible an error is returned (mitigate with LB retries and multi-node distribution)

End-to-end Request Forwarding flow

Verifying the flow (the leader responds regardless of which node you hit)

# 任意ノード経由でリーダー情報を取得
$ curl -sS https://vault.any.example.com:8200/v1/sys/leader -k
{"ha_enabled":true,"is_self":false,"leader_address":"https://vault-1.example.com:8200"}

# スタンバイに当たっても転送され、アクティブが処理する

Load Balancer Design (L4 Recommended, Health Checks, Timeouts)

In production we recommend an L4 (TCP) load balancer. Avoid HTTP-layer rewriting or header injection, and configure a generous idle timeout to accommodate long-running requests (such as watches). Sticky sessions are not needed.

Use /v1/sys/health for health checks. Whether to treat standbys as healthy is an operational choice, but to make the most of Request Forwarding it is easier to use standbyok=true and keep every node in the LB pool.

• Use an L4/TCP load balancer; TLS should generally terminate at the node (re-encrypt if you must terminate at the LB)
• Sticky sessions and session affinity are unnecessary (no need to pin to the leader)
• Example health check: /v1/sys/health?standbyok=true
• Set idle and backend timeouts on the longer side (e.g. ≥ 65s)

Minimal HAProxy example (L4/TCP with health checks)

frontend vault_in
  bind *:8200
  default_backend vault_nodes

backend vault_nodes
  option tcp-check
  # 健全性: スタンバイも可（要件に応じて変更）
  tcp-check connect
  tcp-check send GET\ /v1/sys/health?standbyok=true\ HTTP/1.1\r\nHost:\ vault\r\n\r\n
  tcp-check expect rstring "(200|429)"
  server vault1 10.0.0.11:8200 check
  server vault2 10.0.0.12:8200 check
  server vault3 10.0.0.13:8200 check

Configuration Validation and Troubleshooting

First, verify inter-node connectivity (cluster_addr) and leader election. Next, confirm that secret reads and writes succeed via an arbitrary node, and check the logs for forwarding errors.

Common failure modes are: missing certificate SAN entries, mismatched DNS resolution for cluster_addr, an LB idle timeout that is too short, and inter-node TCP being blocked (by security groups or firewalls).

• Start by verifying cluster_addr resolution and the TLS handshake
• Use sys/leader and sys/health to inspect cluster state
• Check the logs for lines related to forwarding/cluster failures

Frequently used verification commands

# リーダー確認（任意ノードで可）
$ curl -sS https://vault-2.example.com:8200/v1/sys/leader -k | jq .

# スタンバイ許容の健全性（LB 用）
$ curl -sS "https://vault-2.example.com:8200/v1/sys/health?standbyok=true" -k

# Raft ピア確認（統合ストレージ時）
$ vault operator raft list-peers

# ノード間ポート疎通（cluster_addr 側）
$ nc -vz vault-1.example.com 8201

Availability and Behavior During Failures (Failover and Partition)

When the active node goes down, the storage layer's leader election promotes a new active node, and Request Forwarding follows the new leader. Forwarding cannot occur before promotion completes, so it is wise to give clients a retry strategy.

During a network partition, the side where standby-to-active inter-node TLS cannot be established will see forwarding errors. Distribute the LB across multiple AZs/zones and combine multi-node connectivity with retries to mitigate the impact.

• Transient 5xx and connection failures can occur during failover — retry from the client
• Register multiple nodes as LB targets and configure short backoff for brief connection failures
• Verify that certificate SANs and DNS are consistent across zones

Quick failure-scenario validation

# 例: 旧アクティブを停止して挙動を見る（検証環境でのみ）
# active ノード
$ sudo systemctl stop vault

# 別ノードから連続でヘルスチェック
$ watch -n 1 'curl -sS https://vault.any.example.com:8200/v1/sys/health -k | jq .'
# 数秒〜十数秒で新アクティブが反映され、転送が再開することを確認

Security and Audit Considerations

With Request Forwarding, the standby-to-active forwarding path is also protected by TLS. Manage the inter-node certificate so it satisfies the SANs for both api_addr and cluster_addr, and rotate certificates one node at a time during renewal.

Audit logs are reliably recorded on the node that ultimately processes the request, so the audit-device configuration on the active node is the core of the design. Audit data on the standby side may only contain forwarding metadata, so it is safer to satisfy audit requirements on the active side.

• Require inter-node TLS (tls_disable=0)
• Allow 8200/TCP (API) and 8201/TCP (cluster_addr) through the firewall
• Always enable auditing on the active node, and enforce rotation and retention rigorously

Minimal audit and network-control examples

# 監査デバイスの有効化（アクティブで実施、永続化先は運用方針に合わせる）
$ vault audit enable file file_path=/var/log/vault_audit.log

# 例: セキュリティグループ/ファイアウォールの許可（概念）
# API:    8200/TCP クライアント -> すべての Vault ノード
# Cluster:8201/TCP 各 Vault ノード間（双方向）

Check Your Understanding

Frequently Asked Questions