Vault Secrets Operator: Running Kubernetes Secrets the GitOps Way

If you want least privilege, automatic rotation, and auditability for Kubernetes secrets all at once, pairing Vault with GitOps is a strong choice. Vault Secrets Operator (VSO) pulls Vault values into Kubernetes Secrets through CRDs, letting you declare intent in Git.

This article walks through VSO's CRD design, sync ordering with Argo CD / Flux, least-privilege and rotation patterns, and behavior under failure — covering both real Ops work and the exam angle.

The Big Picture: VSO and GitOps

VSO declares Vault connection, authentication, and secret retrieval as Kubernetes CRDs, ultimately producing a Kubernetes Secret. In GitOps, these CRD manifests live in Git, and Argo CD or Flux sync them and trigger the operator's reconciliation loop. The upshot: distributing secrets boils down to declaring intent — a Git change — so the reason for every change and its review history are fully auditable.

The typical flow looks like this: 1) define VaultConnection / VaultAuth / VaultStaticSecret / VaultDynamicSecret in Git, 2) Argo CD / Flux syncs them, 3) VSO authenticates to Vault and fetches / refreshes the secrets, 4) application Pods consume the Kubernetes Secret.

• Store only intent in Git (which Vault path goes to which Namespace) — never the values themselves.
• Use Argo CD / Flux sync ordering to apply VaultConnection / VaultAuth → VaultStatic / DynamicSecret → Workload in that order.
• Let VSO own TTLs and rotation for dynamic secrets, and pair that with an application reload strategy (rolling restart, file watch, etc.).

Data flow across GitOps, VSO, and Vault

A minimal CRD set (the declarations you put in Git)

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vc-default
  namespace: app
spec:
  address: https://vault.example.com:8200
  # Enterprise利用時はnamespace等を指定可。TLS/CABundleもここで指定。
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: va-k8s
  namespace: app
spec:
  vaultConnectionRef: vc-default
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: app-role
    serviceAccount: app-sa
    audiences:
      - vault
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: app-kv
  namespace: app
spec:
  vaultAuthRef: va-k8s
  mount: kv
  type: kv-v2
  path: app/config
  refreshAfter: 1m
  destination:
    create: true
    name: app-config
  # 必要なキーのみ抽出
  data:
    - secretKey: DB_USER
      remoteRef:
        key: username
    - secretKey: DB_PASS
      remoteRef:
        key: password

CRDs and Least-Privilege Design

VSO's main CRDs are: VaultConnection (defines connection info), VaultAuth (binds an auth method), VaultStaticSecret (fetches static values like KV), and VaultDynamicSecret (handles dynamic values like database or cloud short-lived credentials). Splitting them per Namespace clarifies team boundaries and separation of duties.

On the Vault side, the rule of thumb is allow only the paths you need. The role that VaultAuth points at (a Kubernetes Auth role or an AppRole) should be restricted to read / issue / renew on the target paths. On the Kubernetes side, grant the VSO ServiceAccount least-privilege rights — only create / update on Secrets in the target Namespace.

• VaultAuth lets you pick an auth method (kubernetes / approle / jwt, etc.). With Kubernetes Auth, the ServiceAccount token handles authentication.
• VaultStaticSecret declares a refresh interval via refreshAfter and friends. Dynamic secrets refresh / re-issue based on TTL.
• Use Namespace splits and labels to make it clear which Argo CD / Flux instance syncs which CRD.

Minimal Vault policy and Kubernetes Auth role (conceptual example)

# Vaultポリシー（HCL）: app-policy
path "kv/data/app/config" {
  capabilities = ["read"]
}

# Kubernetes Authロール作成（概念的なCLI例）
# 実際のパラメータ名は環境に合わせて調整してください
vault write auth/kubernetes/role/app-role \
  bound_service_account_names=app-sa \
  bound_service_account_namespaces=app \
  policies=app-policy \
  ttl=1h

GitOps Patterns (Argo CD / Flux)

Sync ordering matters. With Argo CD use the sync-wave annotation; with Flux use Kustomization's dependsOn. Line them up as VaultConnection / VaultAuth → VaultStatic / DynamicSecret → Workload so the Secret already exists by the time the app is applied.

A clean repo layout puts VSO CRDs and shared policies in base, with environment-specific bits (paths, role names, destination Secret names) in overlays. For multi-cluster setups, separate them with per-cluster Kustomizations / Argo Applications.

• Argo CD: add the argocd.argoproj.io/sync-wave metadata; sync runs in order from negative → small → large.
• Flux: declare dependencies explicitly with Kustomization.dependsOn; let the controller handle retries on failure.
• Sync the Deployment after the Secret is created — this makes startup failures much easier to avoid.

Example of sync ordering in Argo CD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-secrets
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  source:
    repoURL: https://git.example.com/org/app-config.git
    path: k8s/overlays/prod/secrets
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-workload
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  source:
    repoURL: https://git.example.com/org/app-config.git
    path: k8s/overlays/prod/workload
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: app

Security and Compliance Highlights

Store only metadata in Git — Vault paths, destination Secret names — and keep the actual values exclusively in Vault. Give the VSO ServiceAccount only create / update on Secrets, and consider whether list / get can be dropped (decide based on your audit policy).

On the network side, set a NetworkPolicy that allows egress from the VSO Pod to Vault, and configure mTLS and the right CA on Vault. For backups, treat Kubernetes Secret backups as cache only — design recovery so that values are re-fetched from Vault.

• Separation of duties: have platform / SRE review the VSO CRDs and the security team own Vault paths and policies — keep role boundaries crisp.
• Map out every path that exposes secrets, and minimize who can run kubectl get secret.
• Pair short TTLs on dynamic secrets with automatic application reconnect to shrink the blast radius window.

Minimal RBAC example (permissions VSO needs to create Secrets)

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vso-secrets-writer
  namespace: app
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vso-secrets-writer-binding
  namespace: app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: vso-secrets-writer
subjects:
  - kind: ServiceAccount
    name: vault-secrets-operator
    namespace: vso-system

Day-2 Ops: Rotation and Failure Behavior

Static secrets are re-fetched on the cadence set by refreshAfter (and friends) or when the spec changes. Dynamic secrets refresh / re-issue based on TTL and switch to re-issue once the max TTL is hit. Either way, you still need to design how the application picks up the updated Kubernetes Secret — rolling restart, SIGHUP, file watch, etc.

When Vault is unreachable or authentication fails, VSO typically surfaces the failure via status conditions and events, and previously created Kubernetes Secrets are generally not deleted right away. Reconciliation resumes once Vault recovers. Behavior depends on the engine and version settings, so validate it matches your intent before applying to production.

• You can automate application restarts on Secret updates using kubectl rollout restart or a Reconciler.
• Monitoring: correlate VSO controller metrics / events with the application's authentication-failure logs.
• Let the operator handle back-off and retries on failure; do manual re-syncs through Argo CD / Flux features.

Day-to-day operational commands

# Argo CDで強制再同期
aio argocd app sync app-secrets

# Secret更新に伴うDeploymentのローリング再起動
kubectl -n app rollout restart deploy/myapp

# 失敗イベントの確認
kubectl -n app describe vaultstaticsecret app-kv
kubectl -n vso-system logs deploy/vault-secrets-operator

Exam Prep Checklist (Ops Focus)

Exam questions often probe operational design: which resources you keep in Git, what you define inside Vault, and how you guarantee dependency ordering. Make sure you have a solid grip on each VSO CRD's role, locking down Kubernetes Auth roles and Vault policies, and controlling sync with Argo CD / Flux.

Understanding failure behavior — what happens when Vault is unreachable or auth fails, whether old Secrets persist, and how dynamic-secret TTL refresh works — pays off in both real Ops and the exam.

• Keep in Git: VaultConnection / VaultAuth / VaultStatic / DynamicSecret, and Argo / Flux definitions. Never put Vault values in Git.
• Order control: use Argo's sync-wave and Flux's dependsOn to enforce Secret → Workload order.
• Least privilege: Vault policies grant only the paths you need; K8s RBAC focuses on create / update on Secrets.
• Update propagation: remember that a Secret update does not automatically flow into Pod env vars — a restart is required.

Conceptual Kustomize layout

# ディレクトリ構成（例）
# k8s/
# ├─ base/
# │   ├─ vso/                # VSO CRDインスタンス（Connection/Auth/Static/Dynamic）
# │   └─ rbac/               # 最小RBAC
# └─ overlays/
#     ├─ dev/
#     │   ├─ kustomization.yaml
#     │   └─ patches/        # Vaultパスや宛先Secret名の調整
#     └─ prod/
#         ├─ kustomization.yaml
#         └─ patches/
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base/vso
  - ../../base/rbac
namespace: app
namePrefix: prod-
# 必要に応じてpatchesでVaultパスやロール名を変える

Check Your Understanding

Frequently Asked Questions