Distributional Shift Detection Algorithms for Proactive Model Maintenance in Production

Abstract

ML models used in production settings are vulnerable to performance deterioration due to distributional shifts, in which case the statistical characteristics of incoming data differ from those of training data samples. Early detection of distributional shifts in ML models is important for ensuring their reliability and safety. This paper introduces an algorithm for early detection of distributional shifts based on latent projection and the calculation of divergence measures of projections from production data to the training data centroid in a low-dimensional latent space obtained by means of an autoencoder embedding. Once a shift exceeds a predetermined threshold, maintenance actions are automatically taken to mitigate the issue. Tests have been performed using benchmarks consisting of artificial distributional shifts on popular image datasets Fashion-MNIST and SVHN, where the proposed method was found to perform better than existing techniques using KL-divergence or autoencoder reconstruction measures, exhibiting higher detection accuracy and lower false positive rates at moderate latencies.