Advanced Facial Expression Recognition using Unified Multimodal LSTM Framework with Position Attention and Hybrid Grey Wolf–Ant Lion Optimization

Abstract

Facial Expression Recognition (FER) is a cornerstone capability in human-computer interaction, affective computing, and intelligent systems. Despite significant progress, two persistent challenges remain unresolved: (i) effective integration of multimodal data streams encompassing audio, video, and text, and (ii) robust feature selection and classification under high-dimensional conditions. This paper presents a unified investigation of two complementary FER frameworks. The first framework proposes Long Short-Term Memory with a Position Attention Mechanism (LSTM-PAM) for multimodal FER, leveraging the Surrey Audio-Visual Expressed Emotion (SAVEE) and CMU Multimodal Opinion Sentiment Intensity (CMU-MOSI) datasets. Preprocessing employs Wiener filtering for audio, Contrast-Limited Adaptive Histogram Equalization (CLAHE) for video, and lemmatization for text. Feature extraction utilizes Mel-Frequency Cepstral Coefficients (MFCC), Gray-Level Co-occurrence Matrix (GLCM), and Word2Vector representations, fused into a unified feature vector and classified with LSTM-PAM. The second framework introduces a hybrid Grey Wolf Optimization and Ant Lion Optimization algorithm (GWALO) combined with a Recurrent Neural Network (RNN) for single-modality FER on the JAFFE dataset, employing Viola-Jones face detection, Gabor filtering, and Affine-Scale-Invariant Feature Transform (ASIFT). Comprehensive experiments demonstrate that LSTM-PAM achieves 98.25% and 95.78% accuracy on SAVEE and CMU-MOSI datasets, while GWALO-RNN achieves 99.7% accuracy on JAFFE, outperforming all baseline and state-of-the-art methods. This synthesis provides researchers a holistic perspective on advancing FER through complementary deep learning and hybrid optimization strategies.