Terraform GCS Backend Complete Guide: Remote State Management on GCP

In team-based Terraform workflows, the state file must be stored remotely. On GCP, using a GCS bucket as the backend is the simplest, easiest-to-operate option.

This article explains the GCS backend from a practical perspective: the basics, security design, initialization and migration, authentication and CI/CD, operations and troubleshooting, and an exam-prep checklist.

GCS Backend Overview

Terraform's GCS backend stores the state file in Google Cloud Storage and uses locks to prevent concurrent edits by your team. GCS object versioning and strong consistency enable safe reads and writes.

The backend itself is simple: all you need is a bucket name, an optional prefix, and authentication. To improve recoverability, always enable object versioning on the bucket.

• State locking: the GCS backend supports locking and prevents conflicting updates
• Recoverability: GCS object versioning makes rollback easy
• Encryption: follows the bucket's default encryption (including CMEK). There is no option to specify a KMS key directly in the backend block
• Authentication: ADC (Application Default Credentials) is recommended; service account key files are also supported if needed
• Note: variables and references cannot be used in the backend block. Use static configuration or inject via -backend-config

GCS backend data flow (including locking)

Minimal backend configuration example (static definition)

terraform {
  backend "gcs" {
    bucket = "my-tfstate-bucket"
    prefix = "envs/prod"
    # credentials は省略可（ADC を利用）。鍵ファイルを使うなら以下：
    # credentials = "/secure/path/sa-key.json"
  }
}

Architecture and Security Design

State is one of your most sensitive assets. Align bucket design with environment and project boundaries and clearly separate access boundaries. A common pattern is to split buckets per environment (prod/stg/dev) and organize the contents with prefix and workspace like directories.

Security is enforced at the bucket level. Enable Uniform bucket-level access and Public Access Prevention. Configure object versioning and a default CMEK to combine recoverability with encryption. Apply least privilege via IAM, granting the Terraform service account roles that allow reading and writing objects plus the create/delete operations required for locks.

• Naming and hierarchy: bucket my-org-tfstate-prod with prefixes per team/app (e.g. team1/appA)
• Uniform bucket-level access: enable it to disable ACLs and unify access control under IAM
• Public Access Prevention: enforced is recommended
• Object versioning: on (essential for recovering from accidental updates)
• Encryption: configure a default CMEK (Cloud KMS) on the bucket
• IAM least privilege: storage.objects.get/list/create/delete and buckets.get

Baseline bucket configuration (gsutil / gcloud examples)

# バケット作成（リージョンはチームの実行場所に近いものを）
gsutil mb -p ${PROJECT_ID} -l ${REGION} gs://${BUCKET}

# Uniform bucket-level access を有効化
gsutil uniformbucketlevelaccess set on gs://${BUCKET}

# Public Access Prevention を有効化
gcloud storage buckets update gs://${BUCKET} --public-access-prevention

# オブジェクトバージョニングを有効化
gsutil versioning set on gs://${BUCKET}

# デフォルト CMEK を設定（事前に KMS キーを用意しておく）
# projects/PRJ/locations/LOC/keyRings/RING/cryptoKeys/KEY を指定
gcloud storage buckets update gs://${BUCKET} \
  --default-encryption-key=${KMS_KEY_RESOURCE}

# IAM 付与（例: 実行 SA にオブジェクト管理権限を付与）
gsutil iam ch serviceAccount:${TF_SA}:roles/storage.objectAdmin gs://${BUCKET}

Implementation and Initialization (Including Migration)

By Terraform syntax, the backend block cannot use variables or references. Substitute values via terraform init's -backend-config. The pragmatic pattern is to hardcode most of it and inject only secrets or per-environment differences via -backend-config.

Migration from existing local state can be performed safely with terraform init -migrate-state. On the first init, if existing state is present, you will be prompted to confirm the copy.

• Backend is static: variables, locals, and module references are not allowed
• Inject per-environment differences via -backend-config=... (CLI flag or file)
• Use terraform init -migrate-state for state migration
• Per-workspace state files are stored separately under the prefix

Partial backend definition and init command examples

# main.tf（部分的に固定）
terraform {
  backend "gcs" {
    bucket = "my-tfstate-bucket"
    prefix = "team1/appA"
    # credentials は CLI 側で渡す想定（ADC 利用なら不要）
  }
}

# CLI で環境差分を注入して初期化（ADC を使わない場合の例）
terraform init \
  -backend-config="credentials=/secure/path/sa-key.json"

# 既存ローカル state からの移行
terraform init -migrate-state

# backend.hcl を使う場合（CI 向け）
# backend.hcl
# bucket = "my-tfstate-bucket"
# prefix = "team1/appA"
# credentials = "/secure/path/sa-key.json"
terraform init -backend-config=backend.hcl

Authentication and CI/CD (ADC and Key Management)

ADC (Application Default Credentials) is recommended. Locally, use gcloud auth application-default login. On GCE/Cloud Run/Cloud Build, grant least-privilege roles to the runtime service account, and you can omit credentials from the backend block.

Avoid embedding key files in CI/CD; use Workload Identity Federation or run directly as a service account. If you absolutely must use a key file, store it in Secret Manager or similar and have a rotation plan in place.

• Prefer ADC: omit credentials in the backend and resolve authentication via the environment
• Grant least privilege to service accounts (objects.* and buckets.get)
• In CI, aim for keyless operation using Workload Identity
• If you use a key, inject it via -backend-config=credentials=... and never commit it to the repo

Representative authentication examples for local and CI

# ローカル（ADC）
# ブラウザで認証し ADC を設定
gcloud auth application-default login
# credentials を省略して init
terraform init

# サービスアカウント鍵ファイルを使う場合（非推奨だが例示）
export GOOGLE_APPLICATION_CREDENTIALS=/secure/path/sa-key.json
terraform init -backend-config="credentials=${GOOGLE_APPLICATION_CREDENTIALS}"

# Cloud Build（サービスアカウントに権限付与済み / ADC 利用）
# cloudbuild.yaml のステップ内でそのまま実行可能
- name: hashicorp/terraform:light
  entrypoint: bash
  args:
    - -c
    - |
      terraform init -backend-config=backend.hcl
      terraform plan -input=false

Operations and Troubleshooting

Locks are normally released automatically, but they can linger after abnormal process termination. The error message includes the lock ID; once you have verified it is safe to clear, run force-unlock.

If you accidentally update state, you can recover via GCS object versioning by fetching a previous version. Rather than overwriting directly, copy to a different name first, then consider terraform state push/pull (push is usually disabled, so handle recovery carefully).

• 403/404 errors: verify the SA has roles/storage.objectAdmin (or equivalent least privilege) and buckets.get
• Lock contention: avoid parallel runs; in CI, consider serializing jobs
• Recovery procedure: copy an older generation to a different key name first, then diff before applying
• Region design: choose a low-latency single or multi-region option

Unlocking and version recovery examples

# ロック解除（エラーメッセージにある ID を使用）
terraform force-unlock 12345678-90ab-cdef-1234-567890abcdef

# 既存 state の取得
terraform state pull > current.tfstate

# オブジェクトの全バージョンを確認（-a オプション）
gsutil ls -a gs://${BUCKET}/${PREFIX}/default.tfstate

# 特定世代(#NUM)を別名へコピーして比較
gsutil cp gs://${BUCKET}/${PREFIX}/default.tfstate#NUM ./recovered.tfstate

# 差分を目視確認後、慎重に復旧手順を選択（例: 手動で必要エントリを移植）

Exam Prep Checklist (Associate / Pro)

Questions concentrate on the basics: backend immutability, locking, recovery, and authentication. For design-choice questions, the key is whether you pick versioning, least privilege, and keyless operation (ADC / Workload Identity).

Common topics include the correct use of migration and reinitialization flags (-migrate-state vs -reconfigure), the fact that variables cannot be used in the backend block, and the relationship between workspaces and prefix.

• The backend block is static; variables and references are not allowed
• The GCS backend supports locking; DynamoDB is an S3 backend topic
• Recovery is via GCS versioning, which must be enabled on the bucket
• Prefer ADC; key files are an exception. Workload Identity is preferred in CI
• Migrate with terraform init -migrate-state; swap configuration with -reconfigure
• Per-workspace state is stored separately (combination of prefix and workspace)

Reinitialization and backend config swap (frequently tested)

# backend 設定を変更した場合（例: prefix 変更）は再初期化を明示
terraform init -reconfigure -backend-config=backend.hcl

# 既存ローカル state を GCS に移行
terraform init -migrate-state

Check Your Understanding

Frequently Asked Questions