Data Engineer, AI, Education
This project is a real-time data pipeline that keeps tabs on YouTube video stats—like likes and views—using Kafka and ksqlDB. It pulls data from the YouTube Data API, processes it on the fly, and sends me a Telegram message when something noteworthy changes. How It Works Tech Stack Why It MattersIt’s a hands-on showcase of …
This project is a real-time data pipeline that pulls data from an external API, processes it with Apache Kafka and Spark, and stores it in Apache Cassandra. Apache Airflow handles the scheduling, and Docker ties it all together for a clean, reproducible setup. Details: What I Learned:I got practical experience with real-time data tools, from …
This project showcases a streamlined data pipeline built to handle healthcare data using modern data engineering tools. I used Apache Airflow (via Astronomer) to orchestrate workflows, Google BigQuery as a scalable data warehouse, and dbt for transformations and quality checks. Starting with synthetic data generation, the pipeline uploads data to Google Cloud Storage, creates external …
This project combines Retrieval-Augmented Generation (RAG) with AWS for scalability, GitHub CI/CD for automation, and Docker for deployment reliability. It’s a practical dive into AI and DevOps that’s transforming education. Check out the full story on Medium: Read the article. Github: https://github.com/shj37/medical_aws_CICD