Robust Anomaly Detection In Large-Scale Streaming Data Systems Using Deep Learning

Abstract

Streaming data generated from sensors, Internet of Things (IoT) devices, and digital platforms demands continuous monitoring, yet identifying anomalous patterns in high-velocity environments remains challenging due to evolving data distributions and scale. Existing approaches often struggle with adaptability and robustness. This research aims to design a robust deep learning (DL)model for anomaly detection in large-scale streaming systems. A total of 6543 data points were gathered from the open source Streaming Anomaly Dataset, followed by Z-score normalizationand feature extraction using Principal Component Analysis (PCA). The proposed Billiards Optimizer-driven Adaptive Convolutional Neural Network (BO-ACNN) model employs the Billiards Optimizer to fine-tune hyperparameters and enhance convergence efficiency, while the Adaptive CNN dynamically adjusts convolutional filters to capture evolving temporal–spatial features. This combination enables precise detection of irregular patterns in streaming environments. The model effectively identifies anomalies in large-scale data streams by learning complex patterns and adapting to distribution shifts. Experimental evaluation shows better performance in detection rate (98.60%), accuracy (98.90%), false positive rate (1.79%), recall (97.26%), precision (96.45%), F1-score (97.85%), ROC-AUC (98.45%), and latency (20ms) compared to conventional approaches, which were implemented in Python. The approach ensures scalable, adaptive, and reliable anomaly detection, making it suitable for intelligent monitoring applications.