Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using DataFactory, DataBricks, and Delta Lake

In this post, we will look at creating an Azure data factory with a pipeline that loads Office 365 event data incrementally based on change data capture (CDC) information in the source of Change Data Feed( CDF) of a Delta lake table to an AWS S3 bucket.

What we’ll cover:

  • Create an ADF…

Building a custom authentication flow using Amazon Cognito and MetaMask

Toy building block
Photo by @helloimnik on Unsplash.

When you build a blockchain DApp, you can use AWS services with custom logic, such as monitoring and troubleshooting your contract event logs using AWS CloudWatch. With Amazon Cognito, you can deliver temporary, limited-privilege credentials to your application to access AWS resources.

In this article, we will introduce a cryptographically…

Connecting AKS Applications to Azure PostgreSQL with AAD Pod Identities

Photo by luis gomes from Pexels

In this post, We will look at how to run a Java Spring Boot application on Azure Kubernetes Service (AKS) and connects to Azure PostgreSQL using Azure AD Pod identity. Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage applications based on microservices.

Using BigQuery, Airflow, and Amazon Personalize to Build a machine learning workflow

Amazon announced the availability of Amazon Managed Workflows for Apache Airflow (MWAA), a fully managed service that makes it easy to run Apache Airflow on AWS and to build data processing workflows in the cloud.

Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of…

Hands-on Tutorials

Multi-cloud bulk transfer of files using AWS Data Wrangler, Amazon S3 Inventory, Amazon S3 Batch Operations, Athena, Fargate, and AzCopy

Photo by @jsweissphoto on Unsplash

It may be a requirement of your business to move a good amount of data periodically from one public cloud to another. More specifically, you may face mandates requiring a multi-cloud solution. …

Building a Serverless Workflow using AWS Cloud Development Kit

Photo by @jeremybishop on unsplash

The AWS CDK is a software development framework used to define cloud infrastructure as code and provision it through CloudFormation. The CDK integrates fully with AWS services and allows developers to use a high-level construct to define cloud infrastructure in code.

In this article, we will build a CDK version…

Serverless Machine Learning on AWS Lambda with Amazon EFS

Photo by @exxteban on unsplash

Amazon announced the general availability of AWS Lambda support for Amazon Elastic File System. Amazon EFS is a fully managed, elastic, shared file system and designed to be consumed by other AWS services.

With the release of Amazon EFS for Lambda, we can now easily share data across function invocations…

Step-by-Step Guide to Building a Quick Q&A Chatbot

Photo by manfredsteger on pixabay

Amazon announced the general availability of Amazon Kendra a few weeks ago, Kendra is a highly accurate and easy to use enterprise search service powered by machine learning.

In this post I will build a question and answer chatbot solution using React with Amplify, WebSocket API in AWS API Gateway

Step-by-Step Guide to Building Efficient and Scalable Document Similarity Search Engine

Photo by NeONBRAND on Unsplash

Amazon Elasticsearch Service recently added support for k-nearest neighbor search. It enables you to run high scale and low latency k-NN search across thousands of dimensions with the same ease as running any regular Elasticsearch query.

k-NN similarity search is powered by Open Distro for Elasticsearch, an Apache 2.0-licensed …

Yi Ai

AWS Community Builder | AWS AZURE GCP Certified Engineer | A Cloud Technology Enthusiast | AWS Certified Security/Machine Learning/Database Analytics Specialty

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store