CROSS-MODAL DEEP LEARNING FOR MALWARE DETECTION AND ZERO-DAY THREAT IDENTIFICATION USING BEHAVIORAL AND MEMORY FEATURES

Abstract

Zero-day malware poses a significant and continuously evolving cybersecurity threat because it exploits previously unknown vulnerabilities and employs sophisticated evasion techniques that can bypass conventional signature-based antivirus and rule-driven detection mechanisms. The inability of traditional approaches to identify previously unseen malware variants highlights the need for intelligent detection frameworks capable of learning complex behavioral and memory-level characteristics without relying exclusively on known signatures. To address this challenge, this study proposes a cross-modal hybrid deep learning framework that integrates dynamic behavioral analysis with volatile memory forensics for comprehensive malware detection, classification, and zero-day threat identification. The proposed framework consists of two complementary deep learning branches designed to capture heterogeneous characteristics of malicious software. The first branch employs a Bidirectional Long Short-Term Memory (BiLSTM) network enhanced with a self-attention mechanism to model temporal dependencies and identify discriminative patterns within API-call and system-call sequences collected during sandboxed malware execution. This branch enables the framework to capture behavioral characteristics associated with malicious activities, execution patterns, and system interactions. The second branch integrates a one-dimensional Convolutional Neural Network (1D-CNN) with a stacked autoencoder to learn representative patterns from volatile memory-resident features. While the CNN extracts localized and discriminative feature representations, the stacked autoencoder provides compact representations and reconstruction-based anomaly information that can assist in identifying samples exhibiting characteristics that deviate from previously observed malware families.

To effectively combine these heterogeneous representations, a cross-modal attention mechanism is introduced to learn the relative importance of behavioral and memory-based features and generate a unified representation for malware-family classification. In parallel, the reconstruction error generated by the autoencoder is utilized as an additional anomaly indicator for identifying potentially unseen malware samples and zero-day threats. The proposed framework was evaluated using the CIC-MalMem-2022 dataset in conjunction with a curated dynamic-behavior corpus. To provide a realistic evaluation of zero-day detection capability, a held-out malware-family protocol was adopted, ensuring that selected malware families were excluded from the training process and subsequently used to evaluate the model's ability to generalize to previously unseen threats. Experimental results demonstrate that the proposed framework achieves an accuracy of 99.3%, an F1-score of 0.986, and a zero-day detection AUC of 0.981. Furthermore, comparative experiments show that the proposed approach consistently outperforms conventional machine learning and deep learning baselines, including Random Forest, XGBoost, LightGBM, CNN-BiLSTM, and MalHyStack. The results indicate that combining dynamic behavioral evidence with memory-forensic information provides complementary information that improves the robustness and generalization of malware detection. The incorporation of cross-modal attention further enables effective fusion of heterogeneous security features, while reconstruction-based anomaly detection provides an additional capability for identifying previously unseen malware. Overall, the proposed framework demonstrates the potential of multimodal deep learning and forensic feature fusion as a proactive approach for detecting emerging and zero-day malware threats in modern cybersecurity environments.