Fabric Pipeline vs Azure Data Factory: Complete Comparison of Activities, Triggers, Integration Runtime & Selection Criteria

Both are GUI-based ETL/ELT orchestration services with functional overlap. Fabric Pipeline: runs inside a Microsoft Fabric Workspace, tightly integrated with Fabric Lakehouse/Warehouse/KQL Database, billed via Fabric Capacity (CU), slightly fewer Activity types but strong native Fabric service integration, ideal for new Fabric projects. Azure Data Factory (ADF): standalone Azure resource, 90+ connectors including ADLS Gen2/Synapse/Databricks/SQL DB/on-prem, billed per Activity Run + Integration Runtime hours, long-standing enterprise ETL track record, ideal for maintaining existing projects. Decision: new Fabric-based project → Fabric Pipeline; ongoing maintenance of existing ADF projects → ADF; Fabric + non-Fabric resource integration → use both. Microsoft is accelerating investment in Fabric Pipeline, which is expected to become the mainstream long term.

Dataflow Gen2 is a Power Query-based ETL tool provided by the Data Engineering/Data Factory Workload in Microsoft Fabric. It is the evolution of the legacy Dataflow Gen1 (inside Power BI) and is created as an independent Item in a Fabric Workspace. Key features: 1) data ingestion from 300+ connectors (SaaS, Database, File) via Power Query Online; 2) visual ETL authoring (M Language-based but GUI-centric); 3) Output Destination targeting Lakehouse/Warehouse/KQL Database/SQL DB; 4) callable from Pipelines (embedded as an Activity); 5) auto-schema inference. Typical use cases: 1) ingesting SaaS data (Salesforce, SAP, Dynamics 365); 2) ETL for Power BI users familiar with Power Query; 3) low-code ETL for SQL analysts and business users. Pipeline (Activity-based orchestration) and Dataflow Gen2 (data transformation specialty) are complementary; combining both is the standard pattern.

Main Activities in Fabric Pipeline/ADF: 1) Copy Activity (Source → Sink data copy, 90+ connectors, the most versatile); 2) Notebook Activity (runs Fabric Notebook/Synapse Notebook); 3) Stored Procedure Activity (executes Warehouse stored procedures); 4) Dataflow Activity (runs Dataflow Gen2); 5) Lookup Activity (retrieves values from a data source to feed downstream activities); 6) ForEach Activity (loop control); 7) If Condition/Switch Activity (branching); 8) Wait Activity; 9) Execute Pipeline Activity (calls another Pipeline for modular composition); 10) Web Activity (REST API calls); 11) Set Variable/Append Variable Activity (Pipeline variable operations); 12) Validation Activity (condition checks). A Pipeline is built by combining Activities to assemble complex ETL workflows via GUI, with flexible branching control through dependencies (Success/Failure/Completion/Skipped).

Pipeline execution Triggers: 1) Schedule Trigger (CRON expression or fixed interval — hourly/daily/weekly); 2) Tumbling Window Trigger (non-overlapping time windows, ideal for backfilling historical data); 3) Event-based Trigger (Storage Account Blob create/delete via Event Grid, reactive ETL); 4) Manual Trigger (manual run from the Pipeline UI); 5) Pipeline Run from Pipeline (via Execute Pipeline Activity). Typical patterns: 1) nightly batch ETL → Schedule; 2) real-time Blob upload → Event-based; 3) historical backfill → Tumbling Window; 4) ad-hoc runs → Manual. In production, Schedule Trigger is the most common, Event-based provides real-time responsiveness, and Tumbling Window guarantees data consistency.

The Integration Runtime (IR) in ADF/Synapse Pipeline is the compute environment that executes data integration workloads. Three types: 1) Azure IR (Microsoft-managed, accesses data via the public network from Azure PaaS — easiest, most scalable, standard); 2) Self-hosted IR (IR Agent installed on-prem or inside a VNet to access on-prem DBs, VNet-scoped SQL DBs, or Private Endpoints — essential for network-isolated environments); 3) Azure-SSIS IR (lift-and-shift execution of existing SSIS Packages, for SQL Server SSIS migration). Self-hosted IR runs as a single instance (up to 4-node HA), and adding nodes increases parallel processing throughput. The production standard is to combine Azure IR + Self-hosted IR (Self-hosted for on-prem portions, Azure IR for Azure portions). In Fabric Pipeline the Integration Runtime concept is simplified — workloads run inside Fabric Capacity.

Mapping Data Flow is the visual data transformation feature in ADF/Synapse Pipeline, executed internally as a Spark Job. Design ETL logic in a Source → Transformation (Filter, Aggregate, Join, Pivot, Sink) → Sink GUI with no code required. Common transformations: 1) Filter (row filtering); 2) Derived Column (new column calculations); 3) Aggregate (Group By + aggregation); 4) Join (combine multiple sources); 5) Lookup (reference data lookups); 6) Pivot/Unpivot (column-direction reshape); 7) Surrogate Key (sequential key assignment); 8) Window (window functions); 9) Conditional Split (conditional branching); 10) Sink (output destination). Because Mapping Data Flow runs as a Spark Job, it scales to large data volumes (TB-class). Fabric Pipeline does not provide Mapping Data Flow; the Fabric design philosophy is to implement equivalent functionality via Dataflow Gen2 + Notebook.

ADF cost elements: 1) Activity Run charges (around $0.001 per Copy Activity/Lookup, etc.); 2) Data Movement (DIU hours × DIU count, scales with transfer volume); 3) Pipeline Activity execution time; 4) Self-hosted IR Node VM costs (a 4-node HA setup runs into hundreds of thousands of yen per month in production). Fabric Pipeline cost: Fabric Capacity (CU) only — Pipeline/Notebook/Dataflow all share CU. Optimization tactics: 1) change Copy Activity DIU from Auto to a fixed value (4 DIU standard) to make Data Movement cost predictable; 2) stop unnecessary Triggers (review overly frequent Schedules); 3) configure Self-hosted IR auto-scale; 4) tune Pipeline parallelism (concurrent Activity limits); 5) leverage Pause/Resume in Fabric Pipeline; 6) send Activity Logs to Azure Monitor for continuous optimization. In production, monthly Pipeline execution history reviews are essential to detect abnormal increases in Activity run counts early.

DP-700 (Fabric Data Engineer Associate) is the flagship certification where Fabric Pipeline is tested in depth (Pipeline Activities, Triggers, Dataflow Gen2, Notebook integration). DP-203 (Azure Data Engineer Associate, retired 2024-03) had ADF as a core topic. AZ-305 (Solutions Architect Expert) covers ETL tool selection from an architect's perspective; DP-300 (DBA) covers Pipeline + SQL DB integration; AI-103 (GA 2026-06) covers Pipeline + Azure AI Foundry data preparation pipelines. For data engineers, understanding Pipeline/Data Factory is an essential skill, and Fabric Pipeline is strongly recommended as the future mainstream tool.