Medallion Architecture Explained: Bronze/Silver/Gold Design Pattern

Medallion Architecture is the recommended data design pattern for the Databricks Lakehouse Platform. It organizes data into three layers — Bronze (raw), Silver (cleansed), and Gold (business-optimized) — progressively improving data quality, reliability, and usability. It is the most frequently tested architecture topic on the Data Engineer Associate exam.

The Big Picture

Raw data arriving from data sources (applications, IoT, external APIs, files) is first stored as-is in the Bronze layer, then progressively cleansed, joined, and aggregated to be promoted to Silver and Gold. Every layer is managed as a Delta Table.

Bronze Layer (Raw Data Storage)

The role of the Bronze layer is to "store raw data arriving from data sources without transformation". Data quality is not enforced; missing values, duplicates, and type mismatches are all allowed. The goal is to keep a complete record of "what was actually received".

• Store data in its original form (no transformation)
• Add metadata (ingest timestamp, source file name, etc.)
• Schema depends on the source (inferred or explicitly defined)
• Ingest tools: Auto Loader, COPY INTO, Kafka Connector
• Data quality: low (NULLs, duplicates, and type mismatches all allowed)

# Ingest into Bronze layer (Auto Loader)
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", "/checkpoint/bronze/schema")
  .load("/data/landing/events/")
  .withColumn("_ingested_at", current_timestamp())
  .withColumn("_source_file", input_file_name())
  .writeStream
  .option("checkpointLocation", "/checkpoint/bronze/events")
  .trigger(availableNow=True)
  .toTable("bronze.raw_events")
)

Silver Layer (Cleansing and Joins)

The Silver layer takes the raw data from Bronze and refines it through "general-purpose cleansing that does not depend on any business domain". This is where you handle NULLs, deduplicate, convert types, and join tables (e.g., events + a user master).

• Deduplication
• Type conversion (STRING -> TIMESTAMP, DECIMAL conversion, etc.)
• NULL handling (filter or default value)
• Table joins (Fact + Dimension)
• Data quality checks (DLT Expectations / CHECK constraints)
• Tools: MERGE INTO, Delta Live Tables, Structured Streaming + foreachBatch

# Transform into Silver layer (MERGE)
MERGE INTO silver.customers t
USING (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY updated_at DESC) AS rn
  FROM bronze.raw_customers
  WHERE _ingested_at > current_timestamp() - INTERVAL 1 HOUR
  QUALIFY rn = 1
) s
ON t.customer_id = s.customer_id
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Gold Layer (Business Optimization)

The Gold layer takes Silver data and "optimizes it for a specific business use case". Summary tables for BI dashboards, feature tables for ML, and wide tables for reporting all live here.

• Aggregate tables (daily/monthly sales summaries, etc.)
• Wide tables (denormalized, pre-joined tables)
• ML feature tables (for Feature Store)
• KPI tables (directly powering dashboards)

-- Gold layer aggregate table
CREATE OR REPLACE TABLE gold.daily_sales_summary AS
SELECT
  date_trunc('day', order_date) AS sale_date,
  region,
  product_category,
  COUNT(*) AS order_count,
  SUM(amount) AS total_amount,
  AVG(amount) AS avg_amount
FROM silver.orders o
JOIN silver.customers c ON o.customer_id = c.customer_id
GROUP BY 1, 2, 3;

Layer-by-Layer Comparison

Relationship with Delta Live Tables (DLT)

Delta Live Tables (DLT) is a Databricks feature for implementing Medallion Architecture declaratively. You define the tables for each layer with the @dlt.table decorator, and DLT automatically manages dependencies between layers. Data quality checks (Expectations) are built in, so quality assurance in the Silver layer becomes very concise.

import dlt
from pyspark.sql.functions import *

@dlt.table(comment="Bronze: raw events")
def bronze_events():
    return (spark.readStream
      .format("cloudFiles")
      .option("cloudFiles.format", "json")
      .load("/data/landing/")
    )

@dlt.table(comment="Silver: cleaned events")
@dlt.expect_or_drop("valid_amount", "amount > 0")
def silver_events():
    return dlt.read_stream("bronze_events").filter("event_type IS NOT NULL")

What the Exam Asks About

• The role and differences between Bronze / Silver / Gold layers
• Expected data quality at each layer
• Why Auto Loader is the recommended ingest tool for the Bronze layer
• Primary transformations in the Silver layer (deduplication, type conversion, joins)
• That the Gold layer is the one optimized for a specific use case
• The relationship between DLT and Medallion

Check Your Understanding

Frequently Asked Questions