Understanding Azure Data Factory (eBook)

Sudhir Rawat is a senior software engineer at Microsoft Corporation. He has 15 years of experience in turning data to insights. He is involved in various activities, including development, consulting, troubleshooting, and speaking. He works extensively on the data platform. He has delivered sessions on platforms at Microsoft TechEd India, Microsoft Azure Conference, Great India Developer Summit, SQL Server Annual Summit, Reboot (MVP), and many more. His certifications include MCITP, MCTS, MCT on SQL Server Business Intelligence, MCPS on Implementing Microsoft Azure Infrastructure Solutions, and MS on Designing and Implementing Big Data Analytics Solutions. Abhishek Narain works as a technical program manager on the Azure Data Governance team at Microsoft. Previously he has worked as a consultant at Microsoft and Infragistics and he has worked on various Azure services and Windows app development projects. He is a public speaker and regularly speaks at various events, including Node Day, Droidcon, Microsoft TechEd, PyCon, the Great India Developer Summit and many others. Before joining Microsoft, he was awarded the Microsoft MVP designation.  

1.       Introduction to Azure Data Factory 

·         Overview 

·         Integration runtime (CIR, SHIR, SSIS IR)

·         Linked Services and datasets

·         Activities

·         Pipelines  

2.       Data Movement

·         Copy Activity

·         Scenario: 1. Hybrid and 2. Cloud

·         Performance

                                                               i.      Hybrid

                                                             ii.      Cloud

3.       Data Transformation  

·         All activities

·         Define and build solution using Various activities 

·         Scenario 

4.       Managing Flow 

·         Basically defining Control flow 

·         All control flow activity

·         Use cases/ Scenario: Multi-table load using single pipeline, Lookup and data copy.

5.       Security 

·         ADF Metadata

·         Data Movement (in transit/ rest)

·         Credential Management

·         Ports and Firewalls for hybrid scenarios

6.       Monitoring

·   &^ Activity (data engineer)

·         Integration runtime (DevOps)  

·         UI, SDK, PSH

·         Azure Monitor and OMS

7.       Executing SSIS Packages 

·         Demo/ Scenario: Setup

8.       Operationalizing Pipelines

·         Parameters & System variables 

·         Setting up Triggers

·         Scenario: end-to-end (operationalized) 

9.   Summary 

·         1. Hybrid pipelines (ETL), 2. Modern DW (UX)

·         3. ISV – SDK/ customizable (.NET/ PSH/ Python)