Detailed walkthrough at Medium.com: https://medium.com/@jushijun/building-a-weather-data-pipeline-with-apache-airflow-aws-and-amazon-rds-fca4ab31540c
In this project, I developed a fully automated weather data pipeline to streamline the ingestion, transformation, and storage of weather data. Using Apache Airflow for orchestration, I integrated the OpenWeatherMap API to extract real-time data, processed it with Python, and stored the results in a PostgreSQL database hosted on Amazon RDS. AWS EC2 and S3 were leveraged for compute and storage, ensuring a scalable and efficient solution. This project highlights my skills in data engineering, cloud infrastructure, and automation.
