Data Scientist, Engineer, AI, Educator
This project implements an automated MLOps pipeline for deploying a machine learning model, focusing on operational efficiency and security. It integrates Jenkins for CI/CD, Docker for containerization, and AWS ECS for scalable deployment, ensuring a repeatable and reliable workflow. Project Overview The pipeline automates the deployment of a Flask-based ML application, emphasizing Continuous Integration and …
This repo shows how to use AWS EMR and Apache Spark to process big data in the cloud. It includes scripts and notes from my Medium guide on setting up and running Spark jobs. Key Features Tech Stack Link: Read the full article on Medium Code: https://github.com/shj37/AWS-EMR-Scalable-Spark-Processing
I created a step-by-step data warehouse for banking customer data using AWS and dbt. This guide walks through schema design, cloud setup, and data transformation—perfect for anyone curious about practical data engineering. Check out the full article here: https://medium.com/@jushijun/building-a-banking-customer-data-warehouse-an-end-to-end-guide-using-aws-and-dbt-c058ebe7af35. Github: https://github.com/shj37/dbt-redshift-aws-banking-data-warehouse
This project combines Retrieval-Augmented Generation (RAG) with AWS for scalability, GitHub CI/CD for automation, and Docker for deployment reliability. It’s a practical dive into AI and DevOps that’s transforming education. Check out the full story on Medium: Read the article from developers’ perspectives, Read the article from educators’ perspectives. Github: https://github.com/shj37/medical_aws_CICD
Detailed walkthrough at Medium.com: https://medium.com/@jushijun/building-a-weather-data-pipeline-with-apache-airflow-aws-and-amazon-rds-fca4ab31540c In this project, I developed a fully automated weather data pipeline to streamline the ingestion, transformation, and storage of weather data. Using Apache Airflow for orchestration, I integrated the OpenWeatherMap API to extract real-time data, processed it with Python, and stored the results in a PostgreSQL database hosted on Amazon RDS. …
Detailed walkthrough at Medium.com: https://medium.com/@jushijun/building-an-incremental-data-pipeline-with-dbt-snowflake-and-amazon-s3-e8bee58e69d7 In this project, I developed a scalable data pipeline integrating Amazon S3, Snowflake, and dbt to efficiently manage and transform order data. The pipeline generates synthetic order data, stores it in S3, stages it in Snowflake, and uses dbt to perform incremental loads based on CDC timestamps, ensuring optimal performance by …