A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric, Azure Data Factory and Azure Synapse Analytics.
An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Azure Data Lake. Other processing takes place on Azure Data Factory, Azure Synapse and Tableau.
End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)
A python application that leverages Microsoft Azure to build a fully functional trading system.
Building a next-generation hybrid data pipeline architecture that combines the power of Microsoft Fabric, Azure Cloud, and Power BI. This pipeline is engineered to tackle the challenges of real-time data ingestion, multi-layered processing, and analytics, delivering business-critical insights.
Formula 1 race data engineering project which utilises azure services and databricks to ingest and analyse the data.
Data engineering project using Azure Data Factory and Apache Airflow
A demand forecasting pipeline deployed on Azure and AWS
Python script that cleans the JSON document used by an Azure Data Factory or Azure Synapse Copy Activity to specify the source-sink mapping.
A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.
This project demonstrates the development of a complete end-to-end data pipeline in the cloud using the Microsoft Azure ecosystem to ingest, transform, store, and visualize COVID-19 related data.
In this project, I've created an end-to-end ETL pipeline and subsequently developed a machine learning model to predict the price of Amazon products based on several product-related features.
Unofficial samples that utilise Pigment's API to integrate with data movement and serverless solutions.
ETL pipeline tailored for Olympics data
Treinamento Azure Data Factory + Azure Databricks + Azure Analysis Services + Power BI
Repository for the Visual TOM to Azure Data Factory connector
A repository to continue my education on Azure Data Services.
This branch focuses on building Data Engineering Interview Question and Answer
This project looks at datasets to identify the effect of COVID-19 in Europe in 2020.
End-to-end data engineering pipeline using Azure Blob, Data Factory, dbt, Snowflake, and Streamlit for interactive business analytics. (WIP)
Add a description, image, and links to the azure-data-factory topic page so that developers can more easily learn about it.
To associate your repository with the azure-data-factory topic, visit your repo's landing page and select "manage topics."