Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using DataFactory, DataBricks, and Delta Lake
CDC pipeline guide using Azure DataFactory with Azure DataBricks Delta Lake’s change data feed
In this post, we will look at creating an Azure data factory with a pipeline that loads Office 365 event data incrementally based on change data capture (CDC) information in the source of Change Data Feed( CDF) of a Delta lake table to an AWS S3 bucket.
What we’ll cover:
- Create an ADF…