CNN-based facial emotion recognition for mental health applications

Facial emotions serve as important diagnostic markers in psychiatric assessment, functioning as objective indicators of a patient's psychological status. Unfortunately, therapists often have trouble remembering patients' emotions during sessions, and conventional methods like questionnaires and clinician-rated scales are subject to subjectivity, limiting diagnosis. On the other hand, many existing FER approaches operate in real-time, providing live classification rather than a persistent record for post-session review. Demographic bias also remains a major challenge in FER systems due to unrepresentative data, causing a representation gap for underrepresented groups. To overcome this, this project proposes a FER System designed to capture and log patient facial expressions via a CNN framework. System execution began by evaluating three prominent deep learning architectures: ResNet50V2, MobileNetV2, and EfficientNet-B0, utilizing the FER2013 dataset. The project then compared three training approaches: FER2013-trained, local dataset-trained, and FER2013-pretrained with local dataset fine-tuning. This localized Malaysian FER dataset was specifically introduced to counteract the heavy Western bias inherent in standard benchmarks like FER2013. Finally, an offline, desktop-based application, MyFERS, was developed to perform interval-based facial emotion analysis, supporting post-session clinical reflection. By generating an Emotional Trajectory Report, this offline approach prevents the cognitive overload that live, real-time FER systems typically impose on therapists. ResNet50V2 was selected as the core architecture, achieving a 69.61% accuracy rate. Furthermore, refining the model pre-trained on FER2013 with local data proved the superior training protocol, yielding an accuracy of 57.14%. This local finetuning successfully mitigated a severe domain shift, restoring system accuracy after the base FER2013 model experienced a performance drop to 44.49% when applied to local demographics. From system testing with 18 individuals, MyFERS scored 57% accuracy, 61% precision, 57% recall and 56% F1-score. Despite constraints across dataset parameters, training phases, and evaluation boundaries, this project establishes a foundational benchmark for domain-specific FER architectures.