Data Engineer, AI, Education
This project is a real-time data pipeline that pulls data from an external API, processes it with Apache Kafka and Spark, and stores it in Apache Cassandra. Apache Airflow handles the scheduling, and Docker ties it all together for a clean, reproducible setup. Details: What I Learned:I got practical experience with real-time data tools, from …
This project showcases a streamlined data pipeline built to handle healthcare data using modern data engineering tools. I used Apache Airflow (via Astronomer) to orchestrate workflows, Google BigQuery as a scalable data warehouse, and dbt for transformations and quality checks. Starting with synthetic data generation, the pipeline uploads data to Google Cloud Storage, creates external …
Detailed walkthrough at Medium.com: https://medium.com/@jushijun/building-a-weather-data-pipeline-with-apache-airflow-aws-and-amazon-rds-fca4ab31540c In this project, I developed a fully automated weather data pipeline to streamline the ingestion, transformation, and storage of weather data. Using Apache Airflow for orchestration, I integrated the OpenWeatherMap API to extract real-time data, processed it with Python, and stored the results in a PostgreSQL database hosted on Amazon RDS. …