Azure Data Lake Storage Gen2 (ADLS Gen2) Complete Design: HNS, Medallion Architecture, Partition Strategy

Azure Data Lake Storage Gen2 (ADLS Gen2) is a large-scale analytics storage service that adds a Hierarchical Namespace (HNS) to Azure Blob Storage. It supports both Blob-compatible APIs and HDFS-compatible APIs (ABFS Driver), allowing direct access from analytics engines such as Spark, Hadoop, Databricks, Synapse, and Fabric. HNS provides POSIX-like behavior (directory/file-level permissions and atomic rename), dramatically improving performance and usability for big data workloads. Enable the Hierarchical Namespace option when creating the Storage Account. Typical use cases include data lakes, data warehouse landing zones, ML datasets, IoT data, and log archives. Microsoft Fabric's OneLake is also built on ADLS Gen2 internally.

Key HNS benefits: 1) Atomic rename — directory renames are instant (Blob requires Copy → Delete for every blob, which is slow and expensive), dramatically speeding up Spark/Hadoop output directory renames at job completion. 2) Atomic delete — directory-level deletion is instant. 3) POSIX ACLs — fine-grained directory/file-level permission control (Blob only supports container-level). 4) Improved path-based query performance (list and filter operations). 5) HDFS-compatible APIs (ABFS Driver) enable natural Spark/Hadoop access. 6) Finer-grained Lifecycle Management (flexible rules with path conditions). HNS inherits the same tiers (Hot/Cool/Cold/Archive), replication, and encryption as Blob Storage. Pricing is nearly identical to Blob (HNS Transaction unit price is slightly higher). With virtually no downsides, ADLS Gen2 is the de facto standard choice for new data lakes.

Medallion Architecture is a data pipeline design pattern based on quality tiers in a data lake. Bronze (Raw Layer): ingest data from source systems as-is (CSV, JSON, Parquet, Avro) without schema normalization — immutable and append-only with lineage tracking. Silver (Cleansed Layer): cleanse, deduplicate, unify schemas, JOIN, and convert types from Bronze data, organized by business entity (e.g., customer master, order transactions) — typically in Delta Lake or Parquet format. Gold (Curated Layer): aggregate Silver data and compute business metrics for direct use in dashboards, reports, and ML training — star schema design is common. Standard directory structure: container/bronze/source/yyyy/mm/dd/, container/silver/entity/yyyy/mm/dd/, container/gold/business/yyyy/mm/. This is the standard pattern in Databricks, Fabric Lakehouse, and Synapse, delivering data quality assurance, traceability, and reprocessing ease.

ADLS Gen2 partition strategy significantly impacts analytics performance and cost. Common patterns: 1) Date partitioning (yyyy=2026/mm=05/dd=24/) — ideal for time-series data; Predicate Pushdown (Spark reads only required partitions) dramatically boosts performance. 2) Hash partitioning (region=us/customer_hash=001/) — balances parallel processing and avoids hot partitions. 3) Composite partitioning (yyyy/mm/dd/region/category/) — for multidimensional analysis, but beware of over-partitioning (splitting 1 PB across 1 million partitions degrades list performance). Best practice: aim for 1 GB-1 TB per partition and avoid excessive granularity. With Delta Lake, Z-Order and Liquid Clustering provide secondary ordering on top of partitions. Write partitions with Spark's partitionBy() method, and use Lifecycle Management to automatically delete or archive old partitions.

The ABFS (Azure Blob File System) Driver is the Hadoop-compatible driver for accessing ADLS Gen2. URI format: abfss://[email protected]/path/file. abfss is the TLS-encrypted version (recommended); abfs is unencrypted (not recommended). Supported natively by Spark, Hadoop, Databricks, Synapse, and Fabric. Authentication methods: 1) Storage Account Key (simple but legacy), 2) SAS Token (time-limited access), 3) Service Principal (recommended, via Workload Identity Federation), 4) Managed Identity (most recommended for service-to-service within Azure). Spark example: spark.conf.set('fs.azure.account.auth.type.<storage>.dfs.core.windows.net', 'OAuth') with Managed Identity configuration. With Databricks Unity Catalog, the ABFS URI is registered as an External Location in the catalog for unified management, eliminating the need to write URIs directly in Spark code.

ADLS Gen2 has four access control layers: Azure RBAC, POSIX ACL, Storage Account Key, and SAS Token. Azure RBAC: subscription/resource group/storage account/container level — assign built-in roles like Storage Blob Data Reader/Contributor to Microsoft Entra ID principals (Users, Groups, Service Principals, Managed Identities) — coarse-grained. POSIX ACL: Read/Write/Execute permissions at the directory/file level — the traditional POSIX Owner/Group/Others model plus extensions like Named User, Named Group, and Default ACL — fine-grained. Both are evaluated with AND (only access permitted by both passes). The recommended pattern is: Azure RBAC for coarse-grained (container-level) permissions plus POSIX ACL for fine-grained control (e.g., Engineering Team Read/Write on Bronze, Analytics Team Read-only on Silver). In Databricks Unity Catalog integrated environments, Unity Catalog acts as an abstraction layer, eliminating the need to directly manage POSIX ACLs.

ADLS Gen2 supports the same Lifecycle Management policies as Blob Storage. Typical rules: 1) Bronze data — Hot → Cool at 30 days, Cool → Cold at 90 days, Cold → Archive at 365 days, delete at 2,555 days (7 years). 2) Silver data — keep on Hot and delete old date partitions (no backup needed since regeneratable). 3) Gold data — keep on Hot, backup required. 4) Log data — Hot for 1 week → Cool at 30 days → Cold at 365 days → Archive at 5 years → delete. 5) Intermediate data (Spark Shuffle, Intermediate) — delete after 7 days. In production, the cost savings are dramatic (millions to tens of millions of yen per year) — an essential feature. Using the Cold tier (GA in 2022 — faster retrieval than Archive at slightly higher cost) further improves cost efficiency. Last Accessed Tracking + Auto Tier features enable even smarter automatic optimization.

DP-700 (Fabric Data Engineer Associate) is the flagship certification for this area, with deep coverage of OneLake (built on ADLS Gen2). DP-203 (Azure Data Engineer Associate, retired in 2024-03) had ADLS Gen2 at its core. AZ-104 (Administrator) covers Storage broadly in Domain 2; AZ-305 (Solutions Architect Expert) covers data lake design from an architect's perspective; DP-300 (DBA) covers data integration; AI-103 (GA in 2026-06) covers AI/ML dataset storage. Databricks certifications (Data Engineer Associate/Professional) also frequently include ADLS Gen2 as foundational content. Understanding ADLS Gen2 is an essential skill for data engineers.