Vault HA deployment

Runbook for deploying HashiCorp Vault in a three-node Raft cluster across two availability zones. Carrot Ironfoundersson designed this architecture on day three of his employment. His notes were precise, numbered, and included a hand-drawn network diagram. This runbook is the formal version of those notes.

Architecture

Three Hetzner cloud instances in the Helsinki region:

Node

Hostname

Private IP

AZ

Primary

vault-1.golemtrust.am

10.0.1.1

hel1-dc3

Secondary

vault-2.golemtrust.am

10.0.1.2

hel1-dc4

Tertiary

vault-3.golemtrust.am

10.0.1.3

hel1-dc3

The primary and tertiary nodes share a datacentre; the secondary is in a separate one. This means the cluster survives the loss of any single node or any single datacentre, but not both simultaneously. Carrot found this acceptable. He called it “sound engineering within budget constraints,” which from Carrot means he was satisfied.

All three nodes run CX21 instances with Debian 12. Vault data is stored on separate Hetzner volumes mounted at /opt/vault/data to allow resizing without reprovisioning the instance.

Prerequisites

  • Three Hetzner CX21 instances provisioned and accessible via SSH key

  • A Hetzner private network (10.0.1.0/24) with all three instances attached

  • DNS A records for vault-1, vault-2, vault-3, and vault.golemtrust.am (the load-balanced cluster address)

  • Hetzner load balancer configured to route HTTPS to port 8200 on all three instances by their private network IPs ( use_private_ip), with health checks on /v1/sys/health. The private-IP bind below assumes this; a load balancer targeting the nodes’ public IPs would not reach a listener bound to 10.0.1.1

  • TLS certificates for vault.golemtrust.am and each node hostname (Certbot with Cloudflare DNS)

  • The Vault binary downloaded from HashiCorp releases

Installation on each node

Run the following on all three nodes. SSH to each in turn:

apt update && apt upgrade -y
apt install -y unzip curl

VAULT_VERSION="1.19.0"
wget "https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip"
unzip "vault_${VAULT_VERSION}_linux_amd64.zip"
mv vault /usr/local/bin/
chmod 755 /usr/local/bin/vault

useradd --system --home /opt/vault --shell /bin/false vault
mkdir -p /opt/vault/data /opt/vault/config /opt/vault/logs
chown -R vault:vault /opt/vault

Enable the mlock capability so Vault can prevent sensitive data from being swapped to disk:

setcap cap_ipc_lock=+ep /usr/local/bin/vault

Configuration

Create /opt/vault/config/vault.hcl on each node. The node_id, retry_join, and listener addresses differ per node; the template below is for vault-1, whose private IP is 10.0.1.1:

ui            = true
cluster_name  = "golemtrust-vault"
log_level     = "warn"

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  retry_join {
    leader_api_addr       = "https://10.0.1.2:8200"
    leader_tls_servername = "vault-2.golemtrust.am"
  }
  retry_join {
    leader_api_addr       = "https://10.0.1.3:8200"
    leader_tls_servername = "vault-3.golemtrust.am"
  }
}

listener "tcp" {
  address            = "10.0.1.1:8200"
  cluster_address    = "10.0.1.1:8201"
  tls_cert_file      = "/etc/letsencrypt/live/vault-1.golemtrust.am/fullchain.pem"
  tls_key_file       = "/etc/letsencrypt/live/vault-1.golemtrust.am/privkey.pem"
  tls_min_version    = "tls13"
}

api_addr     = "https://10.0.1.1:8200"
cluster_addr = "https://10.0.1.1:8201"

seal "transit" {
  address         = "https://10.0.1.4:8200"
  tls_server_name = "vault-transit.golemtrust.am"
  token           = "<transit vault root token: retrieve from Bank of Ankh-Morpork vault>"
  mount_path      = "transit/"
  key_name        = "golemtrust-unseal"
}

On vault-2, change node_id to vault-2, the listener address and cluster_address to 10.0.1.2, api_addr and cluster_addr to 10.0.1.2, and the retry_join blocks to list 10.0.1.1 and 10.0.1.3 (with each leader_tls_servername set to that peer’s hostname). Update the TLS certificate paths accordingly. Same pattern for vault-3.

Each retry_join block dials a peer by its private IP but sets leader_tls_servername to that peer’s hostname, so the certificate presented during Raft join, issued for vault-2.golemtrust.am rather than for the IP, still validates. Without it the join handshake fails on a name mismatch. The advertised api_addr and cluster_addr are private IPs so the addresses Vault hands out for client redirects and request forwarding stay on 10.0.1.0/24 rather than the public interface. The seal block dials the transit node’s private IP (10.0.1.4) directly and sets tls_server_name so its hostname certificate still validates, the same pattern as retry_join.

The Transit seal requires a separate single-node Vault instance (vault-transit.golemtrust.am) that is manually unsealed and stores only the transit key used for auto-unseal. See the Raft configuration runbook for how the transit instance is provisioned.

Systemd unit

Create /etc/systemd/system/vault.service on each node:

[Unit]
Description=HashiCorp Vault
Documentation=https://developer.hashicorp.com/vault
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/opt/vault/config/vault.hcl

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/opt/vault/config/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
Delegate=yes

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable vault
systemctl start vault

Initialisation

Initialise the cluster from vault-1 only, run from a host on the private network since the API port binds 10.0.1.1 and is closed on the public interface. This is done once and never repeated:

export VAULT_ADDR="https://10.0.1.1:8200"
export VAULT_TLS_SERVER_NAME="vault-1.golemtrust.am"
vault operator init -key-shares=5 -key-threshold=3

This outputs five unseal keys and one root token. Copy all of them now. They will not be shown again.

Distribute the five unseal keys as follows: Adora Belle holds keys 1 and 2, Carrot holds keys 3 and 4, Ponder holds key 5. The root token goes into the Bank of Ankh-Morpork vault immediately. Under no circumstances should the root token be left active after initialisation is complete; see the Raft configuration runbook for how to revoke it after bootstrap.

With auto-unseal via Transit configured correctly, the cluster will unseal itself on subsequent starts. The manual unseal keys are for disaster recovery only.

Verification

export VAULT_ADDR="https://vault.golemtrust.am:8200"
vault status

The output should show Sealed: false, HA Mode: active, and the cluster name. Check peer status:

vault operator raft list-peers

All three nodes should appear with their node IDs and addresses. If a node is missing, check journalctl -u vault -n 50 on that node and verify its retry_join addresses are correct and reachable on port 8201.

Firewall rules

A correction here, with thanks to Anes Hadjidj for the feedback. An earlier version of this section leaned on the Hetzner Cloud Firewall to filter traffic between the Vault nodes and from the application servers. It cannot do that. A Hetzner Cloud Firewall attaches to a server’s public interfaces only. Traffic on the Cloud Networks interface (the private ens10 attachment carrying 10.0.1.0/24) passes unfiltered, inbound and outbound, so a Cloud Firewall rule on 8200 or 8201 does nothing to the cluster and application traffic that rides the private link.

The Cloud Firewall still has one useful job: keeping the public interface shut. On each Vault instance it denies inbound 8200 and 8201 on the public interface, and permits nothing there beyond SSH from the admin range.

Everything on the private network needs a host firewall. Hetzner private networks are isolated per project, so the exposure is not the open internet or other tenants; it is every server attached to that network within the same project. For a Vault node that is the threat that counts anyway, since cluster and API traffic (8200/8201) rides the private link, and lateral movement from a compromised or lower-trust host on the same network is exactly what a host firewall on the private interface contains.

nftables or ufw on ens10 is the floor. A workable ruleset permits:

  • TCP 8200 from the load balancer’s private IP only (API access)

  • TCP 8200 and 8201 between the three Vault node private IPs (cluster communication)

  • TCP 8200 from the application server range within 10.0.1.0/24

  • everything else on the interface dropped

Two adjuncts pair with it. Binding the Vault listener to the node’s own private IP rather than 0.0.0.0, as the listener block above now does, makes the listening surface explicit instead of incidental. A WireGuard or similar overlay goes further, taking the Hetzner L2 out of the trust boundary altogether.

One caution when tightening the private NIC: rules written too tightly can break the metadata service on 169.254.169.254 and DHCP on that interface, which trips people up more often than the Vault ports do. Allow those first, then narrow the rest.

The load balancer handles public TLS termination and routes only to the active leader. Its health check on /v1/sys/health keeps standby nodes out of rotation, since a standby answers there with a 429 rather than a 200, so a public client normally reaches the active node without a redirect. Vault’s built-in standby-to-leader redirect still exists, but it points at api_addr, now a private address no public client can reach; the health check routing, not the redirect, is what carries clients to the leader here.

Last updated: 20 August 2026