A comprehensive repository for emotion classification using various deep learning architectures, including EfficientNetV2, Vision Mamba (Vim), ResNet18-SE, and IR50. This project includes a complete pipeline from preprocessing and training to a functional video processing demo with Explainable AI (XAI) capabilities. Note: The demo outputs below were generated using the ResNet18-SE architecture.
| GradCAM | Integrated Gradients |
|---|---|
![]() |
![]() |
The primary goal of this project was to research and evaluate various methodologies, research papers, and deep learning models to improve the accuracy of emotion prediction. Starting from scratch with almost no prior knowledge of Python and no knowledge of PyTorch, our group of four built this project from the ground up. Key objectives included:
- Training at least one model entirely from scratch.
- Experimenting with a wide range of different architectures and approaches.
- Empirically validating the effectiveness of these methods based on initial motivations and hypotheses.
Project Duration: December 18, 2025 – February 22, 2026 (Exluding Prior Research Described in Perliminary Report)
Here is an overview of the main directories in this repository:
demo: Contains scripts to demonstrate the emotion classifier in different contexts:Images: Processes a folder of images and outputs a CSV of predicted emotion probabilities.Video: Processes a video or GIF, outputting a new file overlaid with bounding boxes, predictions, and Exaplainable AI (XAI) heatmaps (GradCAM or Integrated Gradients).WebcamDemo: Real-time emotion classification on a live webcam feed, with optional live XAI headmaps.
models: Contains contributor-specific experiments and distinct architectures (Aleks, Dmytro, Daniel, Tiago). The specific architectures tested include:- ResNet18 (Including SE, CBAM, and LReLU variants)
- EfficientNetV2-S
- Vision Mamba (Vim)
- Compact Convolutional Transformer (CCT)
- ConvNeXt (V2 Atto, Custom EmoNeXt)
- VGGNet & GoogLeNet
- Custom CNNs (CNN5, CERNbased)
preprocessing: Pipeline scripts (filters, deduplication, resizing) used to clean and normalize the datasets before training.reports: Project documentation, analysis figures, and the preliminary/final reports.resources: Static assets, including test videos, sample GIFs, and outputs.transfer_learning: Scripts focused on fine-tuning pre-trained models (specifically IR50).XAI_testing: Experimental code evaluating various Explainable AI methods (some approaches here were exploratory and did not end up in the final pipeline).
- EfficientNetV2-S: 88.1% accuracy on CK+
Pre-Trained Weights: You can find and download the pre-trained weights for the models and their variations here: Google Drive Link
The video demo demonstrates real-time emotion classification on video files or GIFs. It detects faces, classifies their prevailing emotion, and optionally overlays visual explanations (heatmaps) to show which parts of the face contributed most to the decision.
The image demo expects the path to a folder containing images as an input. It loads all images in the given directory (and all subdirectories), feeds them through the model in batches and converts the resulting logits to probabilities using the Softmax function. These probabilites are then saved to a csv file together with the corresponding filepath.
Install the provided conda environment using
cd environments
conda env create -f demo_env.ymlTo run the demo, use the process_video.py script located in the demo/Video directory:
python demo/Video/process_video.py <input_video_path> [options]Examples:
# Basic usage
python demo/Video/process_video.py inputs/my_video.mp4
# Initial run with GradCAM XAI (default)
python demo/Video/process_video.py inputs/my_video.mp4 --xai gradcam
# Use Integrated Gradients for XAI
python demo/Video/process_video.py inputs/my_video.mp4 --xai ig
# Use Haar Cascade/RetinaFace detector
python demo/Video/process_video.py inputs/my_video.mp4 --face-detector haarcascade
# Save to specific output
python demo/Video/process_video.py inputs/my_video.mp4 --output results/output.mp4The process_video.py script handles the entire inference pipeline:
- Input Reading: Supports both standard video formats (
.mp4,.avi, etc.) and.giffiles via a customGifReader. - Face Detection: capable of using either RetinaFace (default, higher accuracy) or Haar Cascades (faster, lower accuracy).
- Preprocessing: Crops faces, resizes them to 64x64, and normalizes them to match training conditions.
- Inference: Runs the loaded model (e.g., ResNet18SE) to predict one of 6 emotions: anger, disgust, fear, happiness, sadness, surprise.
- XAI Overlay: Generates heatmaps using GradCAM or Integrated Gradients to visualize model focus.
- Output: Annotates the video with bounding boxes, emotion labels, confidence scores, and emojis. Saves as video or GIF depending on the output extension.
To run the demo, use the process_images.py script located in the demo/Image directory:
python demo/Image/process_images.py <input directory> [options]Examples:
python demo/Image/process_images.py ./inputs/images
python demo/Image/process_images.py ./inputs/images -of ./image_results.csvThe preprocessing pipeline ensures high-quality input data by filtering and normalizing raw images before training.
The preprocessing is managed by a pipeline system (see preprocessing/pipeline.py and preprocessing/dataset_runner.py) that applies a sequence of steps to every image in a dataset folder.
The standard preprocessing pipeline includes the following steps:
FaceExistenceFilter: Ensures a face is actually present in the image with high confidence.FaceRotationFilter: Corrects or filters images based on yaw rotation (looking left/right).FaceOrientationFilter: Corrects in-plane rotation (roll correction).LightingFilter: Filters out images with poor lighting conditions (too dark/bright).RemoveBlurredFaces: Detects and removes blurry images.CroppingFace: Tight crops around the detected face.ResizingTo64: Resizes the cropped face to a standard 64x64 resolution.ToRGB: Converts images to 3-channel RGB.
To run the preprocessing, you configure the steps array in preprocessing/pipeline/run_preprocessing.py. For example, for EmoSet:
# In run_preprocessing.py
steps = [
# ... your filter and step instances here ...
FaceExistenceFilter(),
CroppingFace(),
ResizingTo64(),
ToRGB()
]The following table details the specific preprocessing steps applied to each dataset used in this project:
| Dataset | Preprocessing Steps | Classes Removed | Classes Kept |
|---|---|---|---|
| EmoSet (118k labeled) | FaceExistenceFilter, CroppingFace, ResizingTo64, ToRGB | - | - |
| ExpW (normal, not cleaned) | CroppingFace, FaceExistenceFilter (with high accuracy), ResizingTo64, ToRGB | disgust, sadness | anger, happiness (among others) |
| jaffe | ResizingTo64, ToRGB | - | - |
| KDEF | CroppingFace, ResizingTo64, ToRGB (whole dataset used for Test only) | - | - |
| NHFI | ResizingTo64, ToRGB | - | - |
| NONAME | CroppingFace, ResizingTo64, ToRGB | - | - |
| WSEFEP | ResizingTo64, ToRGB | - | - |
| AffectNet | ResizingTo64, ToRGB | - | - |
| CKplus | ResizingTo64, ToRGB (whole dataset used for Test only) | - | - |
| FERPlus | ResizingTo64, ToRGB | - | - |
| RAF-DB | ResizingTo64, ToRGB | - | - |
| MMAFEDB | ResizingTo64, ToRGB | - | - |
Number of images per emotion after the second stage of preprocessing:
A Duplication Pre-processing step (refer to preprocessing/image_duplicates.py) is used to clean datasets. It employs Difference Hashing (dHash) to identify and remove near-duplicate images, ensuring that the model doesn't overfit to repeated data.
The transfer_learning directory contains scripts for fine-tuning pre-trained models, specifically focusing on IR50 (ResNet50-IR).
- Objective: To leverage powerful pre-trained face recognition features (originally trained on CelebA) for emotion classification. The pre-trained weights were sourced from Qualitative Content Selection (QCS).
- Evolution:
- Original: Fine-tuning the standard IR50 backbone.
- Layer Freezing: Experimenting with freezing different blocks of the backbone (
train_ir50_layer_freezing.py) to prevent overfitting and preserve low-level features. - Light Head (Final Version): The final iteration (
train_ir50_light_head.py) replaces the massive 12.8M parameter fully-connected head with a lightweight, attention-based head (~795K params).
- Key Script:
transfer_learning/train_ir50_light_head.py - Techniques:
- Light Head Architecture: Uses Spatial Attention (to focus on key facial regions like eyes/mouth) followed by Global Average Pooling, significantly reducing parameters while maintaining spatial awareness.
- Class Balanced Focal Loss (CBFL): Handles class imbalance in emotion datasets.
- Differential Learning Rates: Fine-tunes the backbone gently (low LR) while training the new head aggressively (high LR).
- Mixup: Data augmentation technique aimed at improving generalization.



