Dataflow / Apache Beam Guide: Unified Streaming and Batch Pipelines on GCP

Dataflow is Google Cloud's serverless data-processing service that runs pipelines written with the Apache Beam SDK as a managed workload. Its signature feature is the Unified model: the same code handles streaming and batch. It's the de facto standard for Pub/Sub to BigQuery ETL.

Apache Beam Unified Model

• PCollection: Distributed dataset (shared by Streaming and Batch)
• PTransform: Transformations (Map, Filter, GroupBy, etc.)
• Pipeline: A chain of PTransforms
• Runner: Execution engine (Dataflow / Spark / Flink / Direct)

Window / Trigger / Watermark

• Watermark: Beam estimates the boundary 'no data older than this will arrive'
• Trigger: AfterWatermark / AfterProcessingTime / AfterCount
• Late Data: How to handle data arriving after the Watermark (drop or re-aggregate)

Dataflow vs Dataflow Prime

• Dataflow: Standard tier; Worker type is set manually
• Dataflow Prime: Vertical Autoscaling (auto-tuned CPU/memory), Right-fitting (per-stage optimization), and built-in Streaming Engine

Key Features

• Autoscaling: Automatic worker-count adjustment (especially effective for bursty streaming workloads)
• Streaming Engine: Offloads state management to Dataflow, keeping workers lightweight
• Dataflow Shuffle: Managed shuffle (especially effective for large batch jobs)
• FlexRS: Uses Spot VMs for 40% cost savings (batch only)
• Templates: Pre-built templates plus Flex Templates for customization
• Notebooks integration: Develop directly from Vertex AI Workbench

Typical Pipeline: Pub/Sub to BigQuery

# Python Apache Beam
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions, StandardOptions

options = PipelineOptions(streaming=True)

with beam.Pipeline(options=options) as p:
  (p
   | "Read from Pub/Sub" >> beam.io.ReadFromPubSub(topic='projects/PROJECT/topics/events')
   | "Parse JSON" >> beam.Map(lambda msg: json.loads(msg))
   | "Window" >> beam.WindowInto(beam.window.FixedWindows(60))
   | "Group by user" >> beam.GroupByKey()
   | "Write to BQ" >> beam.io.WriteToBigQuery(
       'PROJECT:dataset.events',
       schema='user_id:STRING,count:INTEGER,window_start:TIMESTAMP',
       write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)
  )

Pricing

Comparison with Other Tools

Typical Use Cases

• Real-time aggregation of IoT telemetry
• Web logs to BigQuery streaming ETL
• CDC (Datastream to Dataflow to BQ)
• Fraud detection (Pub/Sub to Dataflow ML to alerts)
• Audio/video pipelines (GCS to Dataflow to ML)

Google Cloud and Apache Beam are trademarks of their respective owners. For the latest information, see the official Dataflow docs.