🏭 Manufacturing Data Platform

End-to-End Production Data Platform built on the M5 Forecasting Accuracy dataset — covering ingestion, validation, transformation, ODS, streaming, analytics, ML forecasting, APIs, and dashboards.

---

📐 Architecture

Raw CSV Files (M5)
       │
       ▼
  data/raw/                   ← Raw data store
       │
       ▼ (Phase 2) explore.py
  Profiling & EDA
       │
       ▼ (Phase 3) melt + join + star schema
  data/processed/ + data/mart/   ← Parquet files
       │
  ┌────┴────────────────────────────┐
  │                                 │
  ▼ (Phase 6) Airflow DAGs          ▼ (Phase 7) Kafka
  Orchestrated Pipeline         Event Streaming
  ingest→validate→transform→load   OrderCreated / PriceUpdated / InventoryUpdated
  │                                 │
  └────────────┬────────────────────┘
               ▼
         PostgreSQL ODS               ← Phase 8
         (fact_sales, dims, views)
               │
        ┌──────┴──────────┐
        ▼                 ▼
    dbt Models          FastAPI         ← Phases 9 & 12
    raw→stg→int→mart    REST Endpoints
        │                 │
        ▼                 ▼
    ML Forecasting    Streamlit         ← Phases 11 & 13
    XGBoost/LightGBM  Dashboard
               │
        ┌──────┴──────────┐
        ▼                 ▼
    CI/CD             Terraform         ← Phases 14 & 15
    GitHub Actions    GCP IaC

---

🗂 Project Structure

Data_Platform/
├── data/
│   ├── raw/               ← M5 CSV files
│   ├── processed/         ← Parquet (melted, joined)
│   └── mart/              ← Star schema Parquet
├── src/
│   ├── data_understanding/ ← Phase 2: explore.py
│   ├── transformation/     ← Phase 3: melt, join, star schema
│   ├── validation/         ← Phase 5: validators, cleaning pipeline
│   ├── eda/               ← Phase 4: eda_plots.py
│   ├── kafka/             ← Phase 7: producers + consumers
│   ├── ods/               ← Phase 8: schema.sql, loaders
│   ├── analytics/         ← Phase 10: business_queries.sql
│   ├── api/               ← Phase 12: FastAPI
│   └── dashboard/         ← Phase 13: Streamlit
├── dags/                  ← Phase 6: Airflow DAGs
├── dbt/manufacturing_dbt/ ← Phase 9: dbt models
├── ml/                    ← Phase 11: ML training
├── terraform/             ← Phase 15: GCP IaC
├── docker/                ← Dockerfiles
├── .github/workflows/     ← Phase 14: CI/CD
├── tests/                 ← Unit + integration tests
├── docs/                  ← Documentation + ER diagram
├── docker-compose.yml
└── requirements.txt

---

🚀 Quick Start

1. Create Python Virtual Environment

# Windows
scripts\setup_venv.bat

# Linux / Mac
bash scripts/setup_venv.sh

# Activate
venv\Scripts\activate        # Windows
source venv/bin/activate     # Linux

2. Environment Configuration

cp .env.example .env
# Edit .env with your credentials

3. Start Docker Services

docker-compose up -d

Services available:

Service	URL
Airflow UI	http://localhost:8081 (admin/admin)
FastAPI Docs	http://localhost:8000/docs
Streamlit Dashboard	http://localhost:8501
Kafka UI	http://localhost:8080
PostgreSQL	localhost:5433

4. Run Data Pipeline

# Step 1: Data understanding
python src/data_understanding/explore.py

# Step 2: Full transformation pipeline
python src/transformation/build_master.py

# Step 3: Validate & clean
python src/validation/cleaning_pipeline.py

# Step 4: Load ODS
python src/ods/load_dimensions.py
python src/ods/load_facts.py

5. dbt Transformations

cd dbt/manufacturing_dbt
dbt run
dbt test
dbt docs generate && dbt docs serve --port 8082

6. Train ML Model

python ml/feature_engineering.py
python ml/train_model.py
python ml/hyperparameter_tuning.py --trials 50

7. Start API

uvicorn src.api.main:app --reload
# API docs: http://localhost:8000/docs

8. Start Dashboard

streamlit run src/dashboard/app.py

---

📊 Dataset — M5 Forecasting

File	Rows	Cols	Description
calendar.csv	1,969	14	Days 2011–2016 with events & SNAP
sales_train_validation.csv	30,490	1,919	Wide-format daily sales
sell_prices.csv	~6.8M	4	Weekly item prices per store

Scope: 30,490 products × 10 stores (CA, TX, WI) × 1,913 days ≈ 58M sales records

---

🔌 API Endpoints

Method	Endpoint	Description
GET	/sales	Paginated sales with filters
GET	/sales/summary	Aggregate sales stats
GET	/products	Product catalogue
GET	/products/{item_id}	Product detail + KPIs
GET	/stores	All stores
GET	/stores/{store_id}	Store detail + monthly trend
GET	/forecast/{item_id}/{store_id}	Demand forecast (28-day default)
GET	/analytics/top-products	Top N by revenue
GET	/analytics/store-performance	Store ranking
GET	/analytics/category-revenue	Category breakdown
GET	/analytics/seasonal-trends	Seasonal patterns
GET	/analytics/holiday-impact	Event day sales lift
GET	/analytics/weekly-trends	Week-over-week

---

🤖 ML Forecasting

Models trained: XGBoost, LightGBM, Random Forest

Features:

• Lag sales (1d, 7d, 14d, 28d)
• Rolling stats (7d/28d mean, std, max)
• Cyclical calendar (month sin/cos, day-of-week sin/cos)
• Price features (current, change%, 4-week avg)
• Event / SNAP flags
• Item × Store × Category encodings

Evaluation metrics: MAE, RMSE, MAPE

---

🏗 Infrastructure (GCP via Terraform)

cd terraform
terraform init
terraform plan -var="project_id=YOUR_PROJECT" -var="db_password=SECRET"
terraform apply

Resources provisioned:

• Cloud SQL PostgreSQL 16 (VPC-private)
• GCS data lake with lifecycle rules
• Artifact Registry for Docker images
• Cloud Run for API + Dashboard
• Secret Manager for credentials
• VPC with private networking

---

🧪 Tests

# Unit tests
pytest tests/unit/ -v

# API tests (requires PostgreSQL)
pytest tests/api/ -v

# All with coverage
pytest tests/ --cov=src --cov-report=html

---

📋 Tech Stack

Layer	Technology
Orchestration	Apache Airflow 2.9
Streaming	Apache Kafka (Confluent)
Data Store	PostgreSQL 16
Transformations	dbt-postgres 1.8
ML	XGBoost, LightGBM, Optuna
API	FastAPI + SQLAlchemy
Dashboard	Streamlit + Plotly
CI/CD	GitHub Actions
IaC	Terraform (GCP)
Containers	Docker Compose
Data Format	Parquet (PyArrow)