Vault Telemetry Configuration: Practical Guide to Prometheus / StatsD Integration

Vault can expose internal counters, latencies, and other signals as telemetry. The two common consumers are Prometheus (pull) and StatsD (push). This article covers when to use each, common configuration pitfalls, and HA considerations.

For exam prep, the frequent topics are the main telemetry stanza options, the format selection for /v1/sys/metrics, typical Prometheus scrape configurations, and the port/protocol assumptions for StatsD.

Goals and Context: The Vault Telemetry Big Picture

Vault enables observability exports via the telemetry stanza in the server config. Prometheus-compatible output is available over HTTP at /v1/sys/metrics?format=prometheus, and StatsD pushes metrics over UDP.

This article assumes OSS Vault and focuses on stable options and common operational patterns. In environments with version differences, always verify the configurable keys against the documentation for the Vault version you are running.

• Prometheus: pull model. The Prometheus server periodically scrapes Vault's HTTP endpoint.
• StatsD: push model. Vault sends metrics over UDP to a local or remote StatsD daemon.
• In HA configurations, watch for metric labels and double-counting between active and standby nodes.

Minimal connectivity check (fetching in Prometheus format)

curl -s https://vault.example.com:8200/v1/sys/metrics?format=prometheus \
  --cacert /etc/ssl/certs/ca.pem | head -50

Architecture: Pull vs. Push and the HA Flow

Prometheus uses pull while StatsD uses push, so the network direction, reachability requirements, label/tag representation, and deduplication strategies all differ. When monitoring a Vault cluster, locking down how active/standby nodes are handled along with TLS and firewall design up front reduces downstream rework.

• For Prometheus, ensure scrape reachability (firewalls/ACLs) and append format=prometheus to /v1/sys/metrics.
• For StatsD, open UDP 8125 (the common default). It is not designed to be resilient to packet loss, so factor in network quality.

Data flow (conceptual diagram)

Typical HA target example (assuming Consul service discovery)

# What it looks like when Prometheus resolves the service (excerpt)
scrape_configs:
- job_name: 'vault'
  metrics_path: /v1/sys/metrics
  params:
    format: ['prometheus']
  scheme: https
  tls_config:
    ca_file: /etc/prometheus/ca.pem
  static_configs:
  - targets: ['vault-1.example.com:8200','vault-2.example.com:8200','vault-3.example.com:8200']

Vault telemetry Configuration (Minimal Setup and Safety Tips)

On the Vault side, the telemetry stanza enables emission formats and destinations. For Prometheus-only integration, a safe practical starting point is to keep the HTTP endpoint output and set the retention window (prometheus_retention_time) along with hostname-label suppression (disable_hostname).

If you also use StatsD, add statsd_address. If your observability platform (Datadog, Graphite, etc.) is already established, this keeps migration cost low.

• disable_hostname=true helps cap label cardinality, which stabilizes dashboards.
• prometheus_retention_time controls the retention window for histograms/summaries. Setting it too large drives up memory usage.
• Because StatsD is UDP, design with no expectation of retransmission on packet loss or buffer overflows.

vault.hcl excerpt: enabling Prometheus and StatsD simultaneously

telemetry {
  # Retention for the Prometheus format (24h, say)
  prometheus_retention_time = "24h"

  # Leaving the hostname out of the labels keeps cardinality down
  disable_hostname = true

  # StatsD output (enable only if you need it)
  # The usual default port is 8125/udp
  statsd_address = "127.0.0.1:8125"
}

# Apply with systemd
# sudo systemctl reload vault

Prometheus Integration: Scrape Config and Verification

From Prometheus, scrape /v1/sys/metrics with format=prometheus appended. Avoid configurations that do not terminate TLS and perform at least CA verification. In HA environments, the common approach is to target every node or resolve them dynamically via service discovery.

To dampen double-counting and failover jitter, stabilizing the stats via recording rules is a practical approach (e.g., widen the rate window slightly).

• Set metrics_path to /v1/sys/metrics and specify format=prometheus in params.
• Do not forget TLS settings (ca_file/server_name). For self-signed certificates, distribute the CA.
• For initial verification, use curl to confirm the formatted output (status 200, metrics starting with # HELP/# TYPE).

Prometheus scrape_configs (minimal example)

scrape_configs:
- job_name: 'vault'
  scheme: https
  metrics_path: /v1/sys/metrics
  params:
    format: ['prometheus']
  tls_config:
    ca_file: /etc/prometheus/ca.pem
    # Set server_name so SNI and the certificate CN/SAN are verified
    server_name: vault.example.com
  static_configs:
  - targets: ['vault.example.com:8200']

StatsD Integration: Send Configuration and Receiver Considerations

When you set statsd_address in telemetry, Vault sends internal metrics over UDP. Receivers include Graphite, Datadog, and Telegraf. If you rely on tag extensions like DogStatsD, choose an agent that supports them.

Because UDP assumes some loss, account for network congestion and restart spikes by stabilizing alert thresholds with hysteresis or compound conditions.

• Allow 8125/udp through the firewall (unnecessary for local receivers).
• Consider sampling for high-frequency metrics (depends on receiver capabilities).
• If using DogStatsD, enable the matching address/tag settings on the receiver side.

StatsD ingestion (e.g., Graphite via Telegraf)

# Telegraf statsd input (telegraf.conf excerpt)
[[inputs.statsd]]
  service_address = ":8125"
  delete_gauges = false
  delete_counters = false
  # Adjust metric_separator / templates as needed

# On the Vault side (the vault.hcl above), set statsd_address to 127.0.0.1:8125

Operational Checks and Key Exam Points

Operationally, monitor metrics for availability, accuracy, and latency. For Prometheus, watch scrape success rate and metric-count drift; for StatsD, watch receive rate and signs of drops (receiver logs/internal metrics). This catches real problems early.

On the exam, the format selection for /v1/sys/metrics, the main telemetry keys, and the design differences between pull and push are reliable scoring opportunities.

• Use curl to verify /v1/sys/metrics?format=prometheus (TLS, response code, headers).
• Watch Prometheus' scrape_duration_seconds and scrape_samples_scraped.
• For StatsD, check receiver metrics and logs for drop or buffer warnings.

Troubleshooting tips

# 1) Does it answer in Prometheus format?
curl -vk https://vault.example.com:8200/v1/sys/metrics?format=prometheus | head -20

# 2) Port reachability (StatsD/UDP)
sudo tcpdump -ni any udp port 8125 -vv -c 5

# 3) Keep the label count down (recheck the configuration)
# After setting telemetry.disable_hostname=true, compare the dashboard series

Check Your Understanding

Frequently Asked Questions