Dataflow & Apache Beam Guide (2026)

Dataflow is Google Cloud's serverless data-processing service that runs pipelines written with the Apache Beam SDK as a managed workload. Its signature feature is the Unified model: the same code handles streaming and batch. It's the de facto standard for Pub/Sub to BigQuery ETL.

Apache Beam Unified Model

PCollection: Distributed dataset (shared by Streaming and Batch)
PTransform: Transformations (Map, Filter, GroupBy, etc.)
Pipeline: A chain of PTransforms
Runner: Execution engine (Dataflow / Spark / Flink / Direct)

Window / Trigger / Watermark

Window Type	Use Case
Fixed (Tumbling)	Fixed intervals (e.g., aggregate every 1 minute)
Sliding	Overlapping (5-minute windows every 1 minute)
Session	Closed on inactivity (e.g., user behavior)
Global	No window (batch-oriented)

Watermark: Beam estimates the boundary 'no data older than this will arrive'
Trigger: AfterWatermark / AfterProcessingTime / AfterCount
Late Data: How to handle data arriving after the Watermark (drop or re-aggregate)

Dataflow vs Dataflow Prime

Dataflow: Standard tier; Worker type is set manually
Dataflow Prime: Vertical Autoscaling (auto-tuned CPU/memory), Right-fitting (per-stage optimization), and built-in Streaming Engine

Key Features

Autoscaling: Automatic worker-count adjustment (especially effective for bursty streaming workloads)
Streaming Engine: Offloads state management to Dataflow, keeping workers lightweight
Dataflow Shuffle: Managed shuffle (especially effective for large batch jobs)
FlexRS: Uses Spot VMs for 40% cost savings (batch only)
Templates: Pre-built templates plus Flex Templates for customization
Notebooks integration: Develop directly from Vertex AI Workbench

Typical Pipeline: Pub/Sub to BigQuery

# Python Apache Beam
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions, StandardOptions

options = PipelineOptions(streaming=True)

with beam.Pipeline(options=options) as p:
  (p
   | "Read from Pub/Sub" >> beam.io.ReadFromPubSub(topic='projects/PROJECT/topics/events')
   | "Parse JSON" >> beam.Map(lambda msg: json.loads(msg))
   | "Window" >> beam.WindowInto(beam.window.FixedWindows(60))
   | "Group by user" >> beam.GroupByKey()
   | "Write to BQ" >> beam.io.WriteToBigQuery(
       'PROJECT:dataset.events',
       schema='user_id:STRING,count:INTEGER,window_start:TIMESTAMP',
       write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)
  )

Pricing

Item	Price (us-central1)
vCPU	$0.069/h (Batch) / $0.075/h (Streaming)
Memory	$0.0046/GB/h
Shuffle	$0.011/GB (Batch) / $0.018/GB (Streaming Engine)
FlexRS (Spot)	40% discount

Comparison with Other Tools

Item	Dataflow	Dataproc (Spark)	AWS Kinesis Data Analytics
Model	Beam (Unified)	Spark / Hadoop	Flink
Serverless	◎	Has Serverless mode	○
OSS compatibility	○ (Beam OSS)	◎ (Spark/Hadoop)	○ (Flink)
Pricing	Worker time	Cluster time	KPU time

Typical Use Cases

Real-time aggregation of IoT telemetry
Web logs to BigQuery streaming ETL
CDC (Datastream to Dataflow to BQ)
Fraud detection (Pub/Sub to Dataflow ML to alerts)
Audio/video pipelines (GCS to Dataflow to ML)

How are Dataflow and Apache Beam related?

Apache Beam is an open-source unified programming model, and Dataflow is Google's managed version of its Runner (execution engine). The same Beam code can also run on Spark or Flink.

What is the difference between Streaming and Batch?

Streaming pulls data continuously from Pub/Sub, while Batch reads once from GCS or BigQuery. The same Beam pipeline handles both (the Unified model).

What are Window, Trigger, and Watermark?

Window groups data over a time range, Trigger decides when to emit results, and Watermark defines the boundary for late data. These are the three core concepts of streaming processing.

What is Dataflow Prime?

An enhanced version of Dataflow with Vertical Autoscaling (automatic resource tuning), Right-fitting, and improved shuffle. Especially effective for complex pipelines.

How is Dataflow priced?

You pay for Worker time (vCPU + memory + storage) plus Shuffle data and Streaming Engine data. Streaming runs around the clock and gets expensive, so the Autoscaler is essential for keeping costs down.

How does it compare to Apache Spark?

Spark is a processing engine, while Beam is a programming model plus a Runner abstraction. Dataflow's strength is serverless management; Spark on Dataproc's strength is OSS compatibility.

What are Dataflow Templates?

Pre-built templates from Google (Pub/Sub to BQ, GCS to BQ, and more). You can run them from the UI without writing code, and Flex Templates let you customize them.

How do I debug and monitor Dataflow jobs?

Use the Dataflow UI graph view, Worker Metrics, Cloud Profiler integration, and Dataflow Insights for automatic diagnostics.

Check what you learned with practice questions

Practice with certification-focused question sets

Visit the GCP exam prep page

Author

NicheeLab Editorial Team

NicheeLab editorial team focused on data engineering and cloud certification learning. Content is structured around practical study needs and official exam domains.

Dataflow / Apache Beam Guide: Unified Streaming and Batch Pipelines on GCP