Data Engineer, AI, Education
In this project, I built a medallion architecture data pipeline on Azure, integrating Azure Data Factory, Databricks, and dbt to process and transform data across bronze, silver, and gold layers. It was a practical, real-world application of data engineering concepts, focusing on scalability and data quality. For a full breakdown of the process, challenges, and …
Project description in details at Medium.com: Description This project showcases the development of an end-to-end real-time streaming data pipeline using Microsoft Azure services. The objective was to ingest weather data from a public API, process it in real-time, and deliver actionable insights through visualizations and alerts, all while prioritizing cost efficiency and security. By leveraging a …
Project description in details at Medium.com: https://medium.com/@jushijun/microsoft-fabric-end-to-end-data-engineering-bing-data-analytics-4220b25bf1cd 1. Environment Setup Create an Azure resource group Add Bing News API for data retrieval Create a Microsoft Fabric workspace Create a Lakehouse for data storage 2. Data Ingestion Retrieve news feed data from the Bing News API Store raw JSON data in the Lakehouse 3. Data Transformation …
Project description in details at Medium.com: 2. Data Ingestion (Part 1): Extract data into Azure Data Lake Storage. 3. Data Transformation (Part 2): Clean and enrich data using Azure Databricks. 4. Data Pipeline (Part 2): Orchestrate workflows with Azure Data Factory. 5. Data Loading (Part 3): Load transformed data into Azure Synapse Analytics. 6. Data …