Databricks Spark Developer Complete Guide: DataFrame API & Spark SQL Prep

The Databricks Certified Associate Developer for Apache Spark (commonly called the Spark Developer exam) measures your ability to implement DataFrame API and Spark SQL workloads in PySpark or Scala. Delta Lake and Unity Catalog are out of scope — what's tested is deep Apache Spark API knowledge. Since code questions account for 30-35% of the exam, reading the API reference and hands-on practice are the keys to passing.

Exam Overview

Because the exam is English-only, every question, answer choice, and code snippet appears in English. Learning the technical terms in English directly will boost your reading speed.

The 7 Exam Domains and Their Weights

Domain 1: Spark Architecture (17%)

Tests whether you understand Spark internals. Conceptual multiple-choice questions dominate this domain — not code.

• Driver / Executor model: The Driver holds the SparkSession and plans jobs, while Executors run tasks in parallel. Know how to handle Executor OOM (increasing partition count, tuning memory settings).
• Lazy Evaluation: Transformations (select, filter, groupBy, etc.) are not executed until an Action is called. Know how to inspect logical and physical plans with explain().
• Narrow vs Wide Transformation: Distinguish Narrow (map, filter — completes within a partition, no shuffle) from Wide (groupBy, join, repartition — triggers a shuffle).
• Catalyst Optimizer: The 4 stages: logical plan → optimized logical plan → physical plan → code generation. Frequently tested optimization rules include Predicate Pushdown (applying filters early), Column Pruning (dropping unused columns), and Constant Folding (pre-computing constant expressions).
• Adaptive Query Execution (AQE): Dynamically switches join strategies, coalesces partitions, and splits skewed data using runtime statistics. Enabled by default on Databricks.

Domain 2: DataFrame API (17%)

One of the highest-weighted domains. Tests whether you precisely understand the arguments, return values, and behavior of the main PySpark DataFrame API methods.

Frequently Tested APIs and Question Patterns

• select / withColumn: df.select("col1", "col2") selects columns; df.withColumn("new_col", expr) adds or transforms a column. The difference: select returns only the listed columns, while withColumn returns all original columns plus the new one.
• filter / where: df.filter(col("age") > 30) and df.where("age > 30") are equivalent. To combine conditions, use & (AND) and | (OR), wrapping each condition in parentheses.
• groupBy / agg: df.groupBy("dept").agg(count("*"), avg("salary")) applies multiple aggregations at once. Know the difference vs. agg-less groupBy().count().
• join: df1.join(df2, "key", "inner")'s third argument specifies the join type. The 7 types are inner / left / right / outer / cross / semi / anti. Semi join returns no columns from the right table; anti join returns only non-matching rows.
• Window functions: Window.partitionBy("dept").orderBy("salary") defines a window, then apply row_number(), rank(), dense_rank(), lag(), lead(). row_number assigns sequential numbers even on ties; rank assigns the same number to ties and leaves gaps; dense_rank assigns the same number to ties with no gaps.

Domain 3: Spark SQL (13%)

• spark.sql("SELECT ...") returns a DataFrame. Use createOrReplaceTempView("view_name") to expose a DataFrame to SQL.
• Built-in SQL functions: COALESCE, NULLIF, CASE WHEN, CAST, and date functions (date_add, datediff, date_format).
• Subqueries: the difference between correlated subqueries (EXISTS) and non-correlated subqueries (IN), and the effect on the execution plan.

Domain 4: Data Sources (13%)

• spark.read.format("csv").option("header", "true").option("inferSchema", "true") reads CSV. Know how to read and write JSON, Parquet, ORC, and Avro, and the characteristics of each format.
• Schema definition: StructType and StructField let you specify a schema explicitly to avoid the overhead of inferSchema (which scans the entire dataset).
• Partitioned writes: df.write.partitionBy("year", "month") partitions data into a directory structure. Watch out for the small-files problem caused by over-partitioning (e.g., partitioning by high-cardinality columns).
• Write modes: append (add to existing data), overwrite (replace existing data), error/errorifexists (default — errors if the table exists), and ignore (does nothing if the table exists). Four modes total.

Domain 5: Higher-order Functions (13%)

Tests lambda operations on arrays (ArrayType) and maps (MapType). Added in Spark 3.0+, this is a common blind spot for learners compared to the other domains.

• transform: applies a lambda to each array element. transform(array_col, x -> x * 2) doubles every element.
• filter: keeps only the array elements matching a predicate. filter(array_col, x -> x > 0) keeps only positive values.
• exists: returns a Boolean indicating whether any element in the array satisfies the predicate. exists(array_col, x -> x == "target")
• aggregate: reduces array elements with a fold operation. You provide an initial value and a lambda to compute things like the sum of an array or a concatenated string.

Domain 6: Structured Streaming (13%)

• spark.readStream.format("rate") or format("socket") to create test streams.
• Output modes: append (emits only new rows; for non-aggregating streams), complete (emits the full result every batch; for aggregating streams), and update (emits only changed rows). Three modes total.
• Watermark: withWatermark("timestamp", "10 minutes") sets the tolerance for late data. Data arriving past the watermark is dropped.
• Trigger settings: trigger(processingTime="5 seconds") (sets the micro-batch interval) and trigger(availableNow=True) (processes all available data and then stops).

Domain 7: Testing & Performance (14%)

• Broadcast variables: broadcast_var = sc.broadcast(lookup_dict) distributes read-only data to every Executor. Used for large dictionaries and lookup tables.
• Accumulators: acc = sc.accumulator(0) creates a distributed counter. If you update an accumulator inside a Transformation, the value can be incremented multiple times when an Action re-runs the Transformation.
• repartition vs coalesce: repartition does a full shuffle and can either increase or decrease the partition count. coalesce only decreases partitions and avoids a shuffle. Use coalesce to control the file count before writing.
• Caching strategy: df.cache() (memory only) vs. df.persist(StorageLevel.MEMORY_AND_DISK). Caching pays off when the same DataFrame is referenced multiple times.

PySpark vs. Scala: How to Pick a Language

Check Your Understanding with a Sample Question

Frequently Asked Questions