Project description in details: https://medium.com/@jushijun/microsoft-fabric-end-to-end-data-engineering-bing-data-analytics-4220b25bf1cd
1. Environment Setup
- Create an Azure resource group
- Add Bing News API for data retrieval
- Create a Microsoft Fabric workspace
- Create a Lakehouse for data storage
2. Data Ingestion
- Retrieve news feed data from the Bing News API
- Store raw JSON data in the Lakehouse
3. Data Transformation (Incremental Load)
- Clean and preprocess news articles
- Implement incremental loading for efficient updates
4. Sentiment Analysis (Incremental Load)
- Apply SynapseML for sentiment analysis
- Store sentiment labels for reporting
- Implement incremental updates for analysis
5. Data Reporting
- Build interactive dashboards using Power BI
- Design visualizations for daily sentiment distribution
6. Building Pipelines
- Automate data workflows using Data Pipeline
- Orchestrate data ingestion, transformation, and analysis
- Monitor and optimize pipeline performance
7. Setting Up Alerts (Data Activator)
- Define event-driven alerts based on positive sentiment ratio
- Configure Data Activator for real-time notifications
- Integrate with Teams for alerts
