Cloud Scheduler + Cloud Functions/Run: Scheduled Batch Automation Tutorial

Cloud Scheduler is GCP's managed cron service, capable of triggering Cloud Functions / Cloud Run / Pub/Sub / arbitrary HTTP endpoints on a schedule. This article walks through the full set of implementation patterns: authentication, retries, parallel execution, and Workflows integration.

Pattern 1: Scheduler → Cloud Functions

# Function (Python)
import functions_framework

@functions_framework.http
def daily_report(request):
    # Work: aggregate in BigQuery, then send email
    from google.cloud import bigquery
    client = bigquery.Client()
    rows = client.query("SELECT COUNT(*) FROM analytics.events WHERE DATE(timestamp)=CURRENT_DATE()").result()
    # Send the email with SendGrid...
    return "OK"

# Deploy
gcloud functions deploy daily-report \
  --gen2 --runtime=python311 --region=asia-northeast1 \
  --trigger-http --no-allow-unauthenticated \
  [email protected]

# Scheduler configuration (OIDC authentication)
gcloud scheduler jobs create http daily-report-job \
  --schedule="0 9 * * *" \
  --time-zone="Asia/Tokyo" \
  --uri="https://asia-northeast1-PROJECT.cloudfunctions.net/daily-report" \
  --http-method=POST \
  --oidc-service-account-email=scheduler-sa@PROJECT.iam.gserviceaccount.com \
  --oidc-token-audience="https://asia-northeast1-PROJECT.cloudfunctions.net/daily-report"

Pattern 2: Scheduler → Pub/Sub → Multiple Subscribers

# Pub/Sub Topic
gcloud pubsub topics create daily-trigger

# Scheduler → Pub/Sub
gcloud scheduler jobs create pubsub daily-trigger-job \
  --schedule="0 9 * * *" \
  --time-zone="Asia/Tokyo" \
  --topic=daily-trigger \
  --message-body="run"

# Several subscribers (fan-out)
gcloud pubsub subscriptions create report-sub --topic=daily-trigger \
  --push-endpoint=https://my-report-service.run.app
gcloud pubsub subscriptions create cleanup-sub --topic=daily-trigger \
  --push-endpoint=https://my-cleanup-service.run.app

Pattern 3: Scheduler → Cloud Run Job (Parallel Batch)

# Cloud Run Job (Python)
import os, sys
def main():
    task_index = int(os.environ.get("CLOUD_RUN_TASK_INDEX", 0))
    task_count = int(os.environ.get("CLOUD_RUN_TASK_COUNT", 1))
    # Handle the slice of work for this task_index
    print(f"Processing task {task_index}/{task_count}")

if __name__ == "__main__":
    main()

# Create the job (parallelism 10)
gcloud run jobs create daily-batch-job \
  --image=asia-northeast1-docker.pkg.dev/PROJECT/repo/batch:v1 \
  --region=asia-northeast1 \
  --tasks=10 \
  --max-retries=3 \
  --task-timeout=3600

# Run the job from Scheduler
gcloud scheduler jobs create http run-daily-batch \
  --schedule="0 2 * * *" \
  --time-zone="Asia/Tokyo" \
  --uri="https://asia-northeast1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT/jobs/daily-batch-job:run" \
  --http-method=POST \
  --oauth-service-account-email=scheduler-sa@PROJECT.iam.gserviceaccount.com

Pattern 4: Scheduler → Workflows (Multi-step)

# Workflow YAML
main:
  steps:
    - extract:
        call: googleapis.cloudfunctions.v2.functions.call
        args: { name: projects/.../functions/extract }
        result: extract_result
    - transform:
        call: googleapis.cloudfunctions.v2.functions.call
        args: { name: projects/.../functions/transform }
        result: transform_result
    - load:
        call: googleapis.cloudfunctions.v2.functions.call
        args: { name: projects/.../functions/load }

# Deploy the workflow
gcloud workflows deploy etl-workflow --source=etl.yaml --location=asia-northeast1

# Scheduler → Workflow
gcloud scheduler jobs create http etl-trigger \
  --schedule="0 3 * * *" \
  --uri="https://workflowexecutions.googleapis.com/v1/projects/PROJECT/locations/asia-northeast1/workflows/etl-workflow/executions" \
  --http-method=POST \
  --oauth-service-account-email=scheduler-sa@PROJECT.iam.gserviceaccount.com

Cron Format Cheat Sheet

Retry and Error Handling

• Scheduler auto-retry: up to 5 attempts with exponential backoff (5s to 1h)
• Cloud Functions: messages retained for 7 days
• Dead Letter Topic: forward failed messages to a separate topic
• Cloud Monitoring alerts: notify PagerDuty on consecutive failures
• Idempotent design: ensure the same job is safe to run multiple times

Typical Use Cases

• Daily DB backups (Firestore Export / Cloud SQL Backup)
• Scheduled report generation and Slack/Email delivery
• Cache invalidation
• SaaS data sync (Salesforce → BigQuery)
• Shutting down idle resources (e.g. stopping VMs overnight)
• Scheduled ML training (weekly continuous training)

※ Google Cloud is a trademark of Google LLC. For the latest details, see the official Cloud Scheduler documentation.