Azure Data Factory

ADF provides pre-built connectors to over 100 data sources and sinks including Azure services, AWS S3, Google BigQuery, Salesforce, SAP, Oracle, MongoDB, REST APIs, and file formats like Parquet, Avro, and JSON. Each connector handles authentication, pagination, schema detection, and data type mapping automatically. Connectors are fully managed by Microsoft, receiving regular updates for API changes and new features without requiring user intervention. Linked Services store connection configurations securely using Azure Key Vault integration, and parameterized datasets enable reusable connector definitions across multiple pipelines.

Mapping Data Flows provide a visual, code-free interface for designing complex data transformations that execute on auto-scaled Apache Spark clusters managed by ADF. Users can perform joins, aggregations, pivots, window functions, derived columns, and conditional splits through a drag-and-drop canvas with real-time data preview. The underlying Spark code is generated automatically, eliminating the need for Spark expertise. Debug mode allows interactive testing with sample data, and the data flow graph provides execution metrics including row counts, timing, and partition distribution. Transformations support schema drift handling for semi-structured data and can process datasets ranging from megabytes to terabytes.

ADF offers three Integration Runtime types: Azure IR for cloud-to-cloud data movement with auto-resolve region selection, Self-hosted IR for secure access to on-premises and private network data sources without opening firewall ports, and Azure-SSIS IR for running existing SSIS packages in a managed cloud environment. The Azure IR supports Managed Virtual Network with private endpoints for network-isolated data movement. Self-hosted IR supports high-availability clusters with multiple nodes and can be shared across data factories. Each runtime type is optimized for its connectivity scenario, and multiple runtimes can coexist within a single data factory to handle diverse network topologies.

Beyond basic schedule triggers, ADF supports tumbling window triggers for backfill scenarios with dependency chaining, storage event triggers that fire when blobs are created or deleted, and custom event triggers that respond to Azure Event Grid topics. This enables event-driven architectures where pipelines execute automatically in response to data arrival, system events, or business process signals. Tumbling window triggers maintain their own execution state, supporting retry of failed windows and catch-up execution for gaps. Triggers can be parameterized to pass runtime context (file names, timestamps, event metadata) into pipeline parameters, enabling dynamic pipeline behavior based on the triggering event.

ADF natively integrates with Azure DevOps Git and GitHub for version-controlled pipeline development, enabling branching, pull requests, and code review workflows for data pipelines. Teams can promote pipelines across dev, staging, and production environments using automated ARM template deployment via Azure DevOps release pipelines or GitHub Actions. The publish branch stores generated ARM templates, and parameterized linked services allow environment-specific configurations (connection strings, credentials, resource references) to be injected at deployment time. This enables enterprise-grade DevOps practices for data integration, including automated testing of pipeline configurations and rollback capabilities through version history.