Terraform S3 + DynamoDB Backend Design: Getting Locking and Encryption Right

For Terraform remote state, availability and consistency come first. On AWS, the standard pattern is to store state in S3 and acquire locks via DynamoDB.

This article focuses on S3-side encryption design and DynamoDB-side lock design, balancing exam-relevant talking points with safe operational practices in the real world.

Background and Requirements

Terraform state may contain sensitive values, and accidental deletion or contention can be catastrophic. Durability, encryption, prevention of concurrent runs, and auditability are therefore non-negotiable.

S3 stores the state. DynamoDB is the single source of truth for locking. S3 versioning enables rollbacks, and DynamoDB's mutual exclusion prevents concurrent applies. The network path is protected by TLS to the AWS API.

From an exam perspective, the high-yield topics are S3 encryption method selection, KMS key permissions, bucket versioning, the DynamoDB lock table key schema, and handling force-unlock.

• S3 side: versioning is mandatory; choose SSE-S3 or SSE-KMS based on organizational requirements
• DynamoDB side: single partition key LockID, PAY_PER_REQUEST recommended, SSE enabled
• IAM: least privilege. For S3, allow Get/Put/List on the target prefix; for DynamoDB, GetItem/PutItem/DeleteItem; for KMS, Encrypt/Decrypt/GenerateDataKey/DescribeKey
• Operations: audit with CloudTrail data events and access logs, restore versions when issues occur, and treat force-unlock as a last resort

S3 Backend Design: Key Hierarchy, Versioning, and Encryption

Design your keys first and foremost to avoid workspace collisions. Using workspace_key_prefix and separating keys by env/workspace name/module is both readable and safe. Since 2020, S3 guarantees strong object consistency, but you still need locking.

Versioning is mandatory; it makes recovery from accidental deletes or overwrites realistic. Combine it with lifecycle rules to set retention for older versions and control costs. MFA Delete carries significant operational overhead, so it is commonly substituted with auditing plus versioning.

Think of encryption in two layers. Terraform's backend s3 setting encrypt=true requires SSE-S3 with S3-managed keys. If organizational policy mandates customer-managed KMS keys, specify the CMK in the bucket's default encryption and enforce SSE-KMS via the bucket policy and KMS key policy. The backend has no option to directly specify a particular KMS key.

• key: use a unique path such as env/${terraform.workspace}/appname/terraform.tfstate
• versioning: enable it and optimize cost with lifecycle rules
• encryption: SSE-S3 for simple operations, SSE-KMS for strict key management and auditing
• auditing: record GetObject/PutObject via CloudTrail data events

DynamoDB Lock Design: Table Schema and Operations

Before write operations like apply, Terraform reaches out to DynamoDB to acquire a lock. The table has a simple schema with only a LockID partition key, and the lock is acquired via a conditional PutItem on LockID. On failure, Terraform assumes an existing lock and either waits or fails.

PAY_PER_REQUEST is the recommended throughput mode; it handles the intermittent load from multiple teams and CI runners naturally. SSE is recommended, though sensitive values generally do not end up in the table. In the rare case of a failure you can use terraform force-unlock, but only after confirming there are truly no concurrent runs.

• The required partition key name is LockID (string)
• Permissions of roughly ddb:GetItem/PutItem/DeleteItem/DescribeTable are enough
• Place the table in the same region as the S3 bucket to simplify latency and permission management
• Treat force-unlock as a last resort and confirm no leftover state from a failed apply

Lock and write flow during a Terraform state update

IAM and Access Control: Locking and Encryption with Least Privilege

Keeping IAM permissions for state to a strict minimum is the safest approach. For S3, grant GetObject and PutObject on the specified prefix of the target bucket, plus ListBucket (limited by prefix), and typically do not grant DeleteObject. When using SSE-KMS, grant KMS Encrypt/Decrypt/GenerateDataKey/DescribeKey scoped to the key.

DynamoDB needs GetItem/PutItem/DeleteItem and DescribeTable for acquiring and releasing locks. Table-scan permissions are not required.

Backend configuration values do not support variable references, so write them statically or externalize them via -backend-config. The safe procedure is to first create the bucket, table, and so on with local state, then migrate with terraform init -migrate-state.

• Avoid granting DeleteObject on S3 wherever possible; guard against accidental deletion with versioning
• Register the IAM role in the KMS key policy as well; for S3 replication, a separate KMS Grant is required
• The backend's encrypt=true means SSE-S3; enforce SSE-KMS via the bucket's default encryption

An example of an S3 bucket, KMS, DynamoDB lock table, and least-privilege policy (HCL, abridged)

# 先に local backend で作成し、その後 remote backend に移行すること。

terraform {
  required_version = ">= 1.3"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.0"
    }
  }
  backend "s3" {
    bucket         = "my-tf-state-bucket"
    key            = "env/prod/app1/terraform.tfstate"
    region         = "ap-northeast-1"
    dynamodb_table = "terraform-lock"
    encrypt        = true  # SSE-S3 を要求。SSE-KMS はバケット側で設定する
    # 変数参照不可。変更時は terraform init -migrate-state
  }
}

provider "aws" {
  region = "ap-northeast-1"
}

resource "aws_kms_key" "tf_state" {
  description         = "Terraform state at rest encryption"
  enable_key_rotation = true
}

resource "aws_s3_bucket" "tf_state" {
  bucket = "my-tf-state-bucket"
}

resource "aws_s3_bucket_versioning" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.tf_state.arn
    }
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  rule {
    id     = "expire-old-versions"
    status = "Enabled"
    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }
}

resource "aws_dynamodb_table" "tf_lock" {
  name         = "terraform-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
  server_side_encryption {
    enabled     = true
  }
}

# ステート操作ロール用 IAM ポリシー例（必要最小限）
data "aws_iam_policy_document" "tf_state_access" {
  statement {
    sid     = "S3StateReadWrite"
    actions = ["s3:GetObject", "s3:PutObject"]
    resources = [
      "${aws_s3_bucket.tf_state.arn}/env/prod/app1/*"
    ]
  }
  statement {
    sid     = "S3List"
    actions = ["s3:ListBucket"]
    resources = [aws_s3_bucket.tf_state.arn]
    condition {
      test     = "StringLike"
      variable = "s3:prefix"
      values   = ["env/prod/app1/*"]
    }
  }
  statement {
    sid     = "DynamoDBLock"
    actions = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:DeleteItem", "dynamodb:DescribeTable"]
    resources = [aws_dynamodb_table.tf_lock.arn]
  }
  statement {
    sid     = "KMSForS3State"
    actions = ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey", "kms:DescribeKey"]
    resources = [aws_kms_key.tf_state.arn]
  }
}

resource "aws_iam_policy" "tf_state_access" {
  name   = "tf-state-access"
  policy = data.aws_iam_policy_document.tf_state_access.json
}

Multi-Account and DR: Handling Replication and Keys

As a rule, avoid manipulating the same state from multiple accounts. The safer pattern is to consolidate into a single account and cross account boundaries via AssumeRole. Keep the backend in one place and have working roles access it.

S3 cross-region replication is useful for disaster recovery, but state requires a single authoritative copy. Disallow writes on the replica and treat it strictly as a read-only recovery option. When using SSE-KMS, set up a KMS Grant for the replication role and configure the KMS key on the destination side.

Avoid multi-regionalizing the lock table with DynamoDB global tables; that breaks the singularity of the lock. As a rule, place the lock in a single location in the same region as the authoritative S3 copy.

• Place the authoritative backend in a single account and single region
• When using S3 replication, limit it to read-only recovery use
• Provision KMS keys per region and consider multi-region keys when needed
• Do not replicate the lock table; prepare an operational procedure for force-unlock

Operations Checklist and Troubleshooting

In day-to-day operations, regularly check S3 versioning and lifecycle, KMS key expiration and rotation, and CloudTrail data events. Follow your naming conventions for workspaces and keys to prevent collisions.

If a stale lock prevents apply from proceeding, first confirm via your monitoring stack or CI that no running session exists, then inspect the lock item in DynamoDB. Use terraform force-unlock only when truly necessary. When state is corrupted, roll back to the most recent S3 version and verify health with terraform state pull.

Permission errors most often stem from either the KMS policy or the S3 bucket policy. When using a CMK via S3 default encryption, always confirm that the IAM role in question is also permitted by the KMS key policy.

• S3: enable versioning and retain older versions for about 90 days via lifecycle
• KMS: enable_key_rotation combined with CloudTrail to audit key usage
• DynamoDB: PAY_PER_REQUEST and monitor throttling via CloudWatch
• Troubleshooting steps: confirm no running session → inspect lock item in DDB → use force-unlock if needed → restore the previous S3 version

Check Your Understanding

Frequently Asked Questions