Our Snippets Of Knowledge Weekly Blog Series
Originally based on the Azure Logic Apps runtime, Data Factory has become the mature orchestration tool to schedule, monitor and manage the execution of integration workloads across the Microsoft cloud platform, and beyond. It also offers, low-code data flow transformation capabilities as an abstraction over Apache Spark and complied DAG’s execution using Apache Airflow.
With over 120 connectors to data sources and third-party tools, plus a suite of Integration Runtime engines to reach into private networks, Data Factory offers a comprehensive bootstrap service for data ingestion and transformation workloads.
Inside the Resource, the scalable nature of the pipeline ForEach activity allows for the parallel execution of inner activities up to a maximum of 50 iterations, or many more if nested with child calls in a metadata driven dependency hierarchy. Iterations can also be handle in sequence, but parallel execution is the default behaviour.
The offering has since become the product code base for other integration pipeline capabilities in Azure Synapse Analytics and Microsoft Fabric.
See MS Learn for more information on this Resource here.
We hope you found this knowledge snippet helpful.
Check out all our posts in this series here.
Comentários