Data Engineer, AI, Education
This project is a real-time data pipeline that keeps tabs on YouTube video stats—like likes and views—using Kafka and ksqlDB. It pulls data from the YouTube Data API, processes it on the fly, and sends me a Telegram message when something noteworthy changes. How It Works Tech Stack Why It MattersIt’s a hands-on showcase of …
This project showcases a streamlined data pipeline built to handle healthcare data using modern data engineering tools. I used Apache Airflow (via Astronomer) to orchestrate workflows, Google BigQuery as a scalable data warehouse, and dbt for transformations and quality checks. Starting with synthetic data generation, the pipeline uploads data to Google Cloud Storage, creates external …
This project was about building a straightforward, scalable data pipeline for healthcare data using modern tools and cloud tech. For a full breakdown of the process, challenges, and insights gained, check out my article on Medium: https://medium.com/@jushijun/building-an-end-to-end-data-pipeline-for-healthcare-data-with-bigquery-dbt-and-github-ci-cd-8e772b01e318 Github: https://github.com/shj37/dbt-redshift-aws-banking-data-warehouse GoalCreate a pipeline to ingest, transform, and deploy healthcare data for actionable insights. Tools Used How …