terraform state pull/push Deep Dive: Low-level State Operations

Terraform state is the heart of IaC operations. Normally apply/plan and the backend handle it safely, but during incident response or audits you may need direct low-level operations. This article centers on terraform state pull/push and organizes the procedures and precautions for avoiding destructive changes.

We follow HashiCorp's official baseline specifications and add notes for version-dependent behavior. From an exam-prep perspective (Pro/Advanced), we also cover when to choose each command, locking, and backend differences.

How pull/push Work and Required Background

terraform state pull fetches the current state from the backend and prints it as JSON on standard output. It is read-only and does not modify infrastructure or the backend. In contrast, terraform state push writes a local tfstate file back to the backend. It depends heavily on backend and lock constraints, and misuse can cause destructive diffs and conflicts.

State carries a serial (generation number) and a lineage (lineage ID), and only the correct successor generation is accepted. A push is typically rejected on serial mismatch, but -force can override this. Forced overwrite is a last resort: stop concurrent runs, acquire a lock, take a backup immediately before, and run a plan verification immediately after.

• pull is a safe read. It does not normally take a lock, but never run it alongside an active apply
• push is a write operation. Some backends do not support it or restrict it. Always stop other running jobs first
• State can contain secrets. Handle, store, and transfer it with strict controls
• Understand your backend's locking mechanism (S3+DynamoDB, Azure Blob Lease, GCS preconditions, etc.)

Conceptual diagram: pull/push data flow and locking

Inspecting and comparing state metadata

# Fetch the current state and check metadata
terraform state pull > current.tfstate
jq '{serial, lineage, resources: (.resources|length)}' current.tfstate

# Lightweight comparison against a recent backup (count/meta)
jq '{serial, lineage}' current.tfstate backup-20240115.tfstate

diff <(jq -S '.resources[].type+"."+.resources[].name? // empty' current.tfstate | sort) \
     <(jq -S '.resources[].type+"."+.resources[].name? // empty' backup-20240115.tfstate | sort)

Safe Use of terraform state pull

pull is useful as a starting point for investigation, auditing, and backup. It is generally safe when no plan/apply is in progress. The resulting JSON may contain sensitive data, so pay attention to the output destination and access controls.

On managed backends, organization/workspace permissions may restrict fetching. Verify authentication (terraform login, etc.) and permissions before running, and restrict long-term storage to encrypted locations.

• Confirm no Terraform jobs are running
• Decide on backup naming conventions and storage location before fetching
• Summarize the JSON with jq for review (do not widely distribute the full file)
• Use output flags like -no-color when needed (handle stdout safely)

Pull example and summarization

# Take a backup (with timestamp)
ts=$(date +%Y%m%d-%H%M%S)
terraform state pull > backup-$ts.tfstate

# Confirm metadata health
jq '{serial, lineage, resCount: (.resources|length)}' backup-$ts.tfstate

# Simple scan for sensitive content (do not over-rely on this)
egrep -in '(password|secret|token|apikey)' backup-$ts.tfstate || true

Risks of terraform state push and Safe Recovery Procedures

push replaces the state. Misuse triggers large diffs and destructive changes. Do not use it during normal operations; reserve it as a last resort such as disaster recovery or rollback after a mistake. Support availability and lock prerequisites depend on the backend.

Perform safe recovery in this order: stop concurrent runs → take a backup → validate → acquire a lock (if possible) → push → double-verify with a plan immediately after. If rejected due to serial mismatch, avoid forced overwrite without investigating the cause.

• Stop all CI/CD and manual runs; declare a lock and change freeze
• Pull the current state and compare it against the recovery-candidate tfstate
• Re-confirm how the backend handles locking and generations
• After push, run terraform plan and verify there are no unexpected destroy/add operations

Push procedure (example: S3 backend)

# 1) Save the current state
terraform state pull > before-$ts.tfstate

# 2) Confirm metadata of the recovery candidate (backup-20240115.tfstate)
jq '{serial, lineage}' backup-20240115.tfstate

# 3) If possible, perform during a window where the lock is available
#    (monitor the lock via the S3+DynamoDB table)
# 4) Try the push (initially without -force)
terraform state push backup-20240115.tfstate

# Investigate the reason if rejected for serial mismatch, etc. Only when truly
# necessary, do a forced overwrite under an operations freeze.
# terraform state push -force backup-20240115.tfstate

# 5) Verify the diff immediately afterward
terraform plan -refresh=false   # First, check only the code/state alignment
terraform plan                  # Then verify the diff including the live env

Per-Backend Considerations and Constraints

Backends differ in lock mechanisms and API constraints. Confirm whether push/pull is allowed and its prerequisites in each backend's documentation. In particular, managed backends strictly control generations and permissions server-side, and push may be unsupported or restricted.

For backend migration, prefer terraform init -migrate-state over manual transfer with push. For HTTP backends, the server implementation determines what is supported, so it is safer to avoid push in environments without a defined spec.

• S3: DynamoDB locking recommended. Encrypt with SSE-KMS, etc. Enable versioning to make recovery easier
• GCS: Object Versioning plus conditional writes prevent concurrent execution
• AzureRM: Lock via Blob lease. Configure SSE/ACL appropriately
• HTTP: Depends on the server implementation. push/lock may not be supported
• Terraform Cloud/Enterprise: Service-managed state. Some CLI subcommands may be restricted. See the official procedures

Representative backend configuration (HCL excerpt)

# S3 backend (example)
terraform {
  backend "s3" {
    bucket         = "tfstate-prod"
    key            = "network/terraform.tfstate"
    region         = "ap-northeast-1"
    dynamodb_table = "tfstate-lock"
    encrypt        = true
  }
}

# GCS backend (example)
terraform {
  backend "gcs" {
    bucket = "tfstate-prod-gcs"
    prefix = "network"
  }
}

State File Security and Auditing

State contains a wide range of resource attributes, stored in plaintext unless the provider hides them via sensitive attributes. Even if the backend encrypts at rest, the local file produced by pull is plaintext JSON. Establish governance around access control and storage.

Begin audits with the minimum required metadata and a count summary; do not distribute the full file. Restrict long-term storage to systems with key management (KMS, etc.) and audit trails. Pasting into shared channels (email/chat) is strictly forbidden.

• Always enable backend encryption (SSE, CMEK) and versioning
• Apply expiring retention and least-privilege access to pull output
• Summarize with jq to limit the spread of sensitive information
• Securely erase unused backups (shred / secure delete)

Safe audit summary sample

# Extract resource count and a list of types
jq '{serial, lineage, count: (.resources|length), types: ([.resources[].type] | unique)}' backup.tfstate

# Search for explicitly sensitive candidate attributes
jq '.. | objects | to_entries[] | select(.key|test("(?i)(password|secret|token|apikey)"))' backup.tfstate

Exam Tips: Choosing Commands and Pitfalls

Questions tend to focus on which command is appropriate and what the safe order of operations is. pull/push are low-level operations; ordinarily mv/rm/import, init -migrate-state, or apply -refresh-only achieves the goal. Remember push as a last resort for disaster recovery.

It is also important to correctly articulate the relationship between state and real resources. Rewriting state does not change real resources. Conversely, importing without HCL code does not bring resources under management. Avoid mixing these concepts.

• Choose terraform init -migrate-state for backend migration
• For drift correction, prefer apply -refresh-only (note: the legacy refresh command has deprecated forms)
• Move resources with terraform state mv; exclude them with terraform state rm
• pull is read, push is replacement. Run push only under a lock and freeze

How to distinguish common options (example)

# Bring an existing VPC under IaC management
# Wrong: replace the VPC state with state push
# Right: terraform import aws_vpc.main vpc-xxxx && plan

# Provider naming change
# Right: terraform state replace-provider 'hashicorp/aws' 'hashicorp/aws'  # example of an address fix

Check with a Question

Frequently Asked Questions