This project is a real-time anomaly detection system for stock trading data, built with Redpanda and Quix. It processes streaming trade data and flags unusual patterns using both rule-based and machine learning techniques.
Purpose
The goal is to detect anomalies—like sudden price jumps or high-volume trades—as they happen, using a lightweight, practical setup.
Technologies Used
- Redpanda: Streams the data (Kafka-compatible).
- Quix: Manages the pipeline.
- Python: Powers the processing and ML (Pandas, NumPy, Scikit-learn).
- Docker Compose: Runs the environment.
How It Works
- Data: Streams simulated stock trades from a CSV.
- Anomaly Detection: Uses volume thresholds, price change rules, and an Isolation Forest model.
Key Acheivements
Successfully built a system that processes live data and executes trades based on a custom strategy, demonstrating real-time data engineering in action.
Link: Read the full article on Medium
References
