Databricks Photon Engine Explained: Speedups, Supported Workloads, and Limits

Photon is Databricks' native vectorized engine — a from-scratch reimplementation of Apache Spark's execution layer in C++. It maintains compatibility with the traditional JVM-based Spark engine while applying CPU-level optimizations (SIMD, columnar batch processing) to deliver significant speedups, especially for SQL queries, DataFrame operations, and Delta Lake I/O.

This article covers which workloads Photon supports and does not support, how automatic JVM fallback behaves, how to enable Photon, and a cost comparison across Classic, Photon, and Serverless.

Photon Architecture: Why It's Fast

Traditional Spark runs as Java bytecode on the JVM. Photon replaces that execution layer with C++ and adds the following optimizations:

• Columnar batch processing: data is laid out in memory by column rather than by row, maximizing CPU cache efficiency
• SIMD instructions: process multiple data points per instruction (Single Instruction, Multiple Data)
• No runtime code generation: instead of JVM Whole-Stage Code Generation, pre-compiled C++ code is used
• Optimized memory management: bypass JVM garbage collection and control memory allocation directly

Thanks to these optimizations, Photon can deliver roughly 2-8x speedups over the traditional Spark engine, especially for CPU-bound workloads (heavy JOINs, aggregations, sorts, and filtering).

Supported Workloads

Photon delivers speedups on the workloads below. Most Spark SQL and DataFrame API operations are covered by Photon.

Unsupported Workloads

The workloads below do not run on Photon and automatically fall back to the JVM-based Spark engine. Even on Photon-enabled clusters, this processing runs as classic Spark.

Fallback Behavior: Automatic Switching for Unsupported Ops

Running an unsupported operation on a Photon-enabled cluster does not raise an error. Only that operation falls back to the JVM-based Spark engine, while the remaining Photon-supported operations continue to run on Photon.

# 例：Photon対応クラスタでUDFを含むクエリを実行
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

# このUDFはJVMフォールバックで実行される
@udf(returnType=StringType())
def custom_transform(value):
    return value.upper() + "_PROCESSED"

# filterとgroupByはPhotonで実行、UDF部分のみJVMで実行
result = (
    spark.read.format("delta").load("/mnt/silver/orders")
    .filter("amount > 100")                    # Photon
    .withColumn("label", custom_transform("region"))  # JVM fallback
    .groupBy("label").sum("amount")            # Photon
)
result.display()

When you inspect the query plan in the Spark UI, operators that ran on Photon carry a 'Photon' prefix (PhotonScan, PhotonHashAggregate, etc.). Operators without that prefix have fallen back to the JVM.

How to Enable Photon

You can enable Photon in any of the following ways. Setting it at cluster creation time is the most common approach.

• At cluster creation, choose a Photon-enabled Databricks Runtime (e.g., 15.4 LTS Photon)
• Enforce photon_enabled: true via a Cluster Policy
• Specify runtime_engine: PHOTON in the cluster config via the REST API or CLI

// Cluster Policy でPhotonを強制する例
{
  "runtime_engine": {
    "type": "fixed",
    "value": "PHOTON"
  },
  "spark_version": {
    "type": "regex",
    "pattern": "15\\.4\\.x-photon-scala.*",
    "defaultValue": "15.4.x-photon-scala2.12"
  }
}

On Serverless SQL Warehouse, Photon is enabled by default — you get Photon's speedups with no additional configuration.

Classic vs Photon vs Serverless: Pricing Comparison

Deciding whether to adopt Photon requires looking at both the DBU rate and the runtime. Here's an overview of the cost profile of each compute option (actual rates vary by cloud provider and region).

The most reliable way to evaluate Photon's ROI is to run the same workload on Classic and Photon and compare runtime and DBU consumption. For ETL jobs dominated by JOINs, aggregations, and sorts, the shorter runtime typically outweighs Photon's higher unit price.

Best Practices for Maximum Performance

• Rewrite custom UDFs using Spark built-in functions (UDFs fall back to the JVM)
• Migrate from the RDD API to the DataFrame API (RDDs are not supported by Photon)
• Store data in Delta Lake format so the optimized Photon Parquet reader can be used
• Combine with data skipping (Z-ORDER / Liquid Clustering) to reduce I/O
• Compact small files (OPTIMIZE) to improve scan efficiency

Key Points for the Exam

• Photon is a C++ vectorized engine (not JVM-based)
• RDDs and custom UDFs are not supported by Photon
• Unsupported operations fall back to the JVM rather than raising an error
• The difference between Photon and Serverless (Serverless = Photon + managed compute + instant start)
• DBU rates rank Classic < Photon < Serverless, but total cost depends on runtime

Check Your Understanding

Frequently Asked Questions