A Microservices-Based Large-Scale Intelligence Framework for Adaptive Anomaly Detection in Distributed Data Streams

Abstract

In the era of modern cloud-native systems, where many services run in the cloud and are distributed in the form of microservices, the conventional monolithic anomaly detection framework is no longer sufficient to process data streams that are generated at a high rate and volume. This study introduces the Microservices-Based Large-Scale Intelligence Framework (MSILF), an adaptive modular framework for real-time anomaly detection in heterogeneous distributed data streams. The MSILF combines a multi-layer ensemble consisting of a MultiLayer perceptron (MLP) classifier, a Local Outlier Factor (LOF) detector, and a Variational Autoencoder (VAE) through a weighted fusion module and is continuously updated by a reinforcement learning (RL) agent. The framework was dockerized and orchestrated using Kubernetes, resulting in horizontal scalability and fault tolerance. An F1 score of 91.9% and end-to-end detection latency of 38 ms were achieved through experimental evaluation on a synthetic microservice telemetry benchmark, outperforming five comparative baselines, including recent deep-learning-based approaches. The results confirmed the effectiveness of using ensemble intelligence with adaptive threshold management in production-scale microservices.