Scalable Machine Learning Framework for Real-Time Data Stream Analytics and Anomaly Detection

Abstract

The ever-growing number of sensors, networked financial transactions, and Internet of Things (IoT) devices has created high-volume, high-velocity data streams with an ever-increasing number of events per second that batch-based machine learning pipelines cannot keep up with. We introduce a scalable machine learning pipeline that extracts features from data streams, runs an ensemble of lightweight streaming detectors, and automatically tracks concept drift—the target of real-time data stream analytics—all in a single pipeline by treating concept drift monitoring as an event. The framework was tested on a synthetically created high-throughput data stream with 60,000 rows of data, eight features, and 2% noise (with two abrupt concept drift events). Three basic stream anomaly detectors (Half-Space Trees, One-Class Support Vector Machine, Lightweight Online Detector of Anomalies (LODA)) were compared using the same evaluation protocol, with the use of the ADWIN and Page–Hinkley drift detectors. The detection performance for the LODA-based configuration is the best (F1-score = 0.927, precision = 0.927, recall = 0.928). The highest raw throughput is achieved by One-Class SVM (67,201 events per second), though its detection accuracy is lower than the others. Per-record inference latency across all configurations was less than a millisecond, ensuring the framework's ability to support near real-time use. The results in this work show that such lightweight ensemble streaming detectors, along with adaptive monitoring for drift, can achieve good detection accuracy without the penalty of high latency or reduced throughput in scalable anomaly detection of streaming data. The study also reflects on the shortcomings of the current assessment and presents directions for future studies, including adversarial robustness, distributed deployment, and assessment across heterogeneous real-world streams.