Explainable Hybrid Machine Learning for Healthcare Disease Prediction Using Latent Feature and Anomaly Learning

Abstract

Anomaly detection in healthcare systems is one way through which disease prediction can be made accurate and the reliability of the data utilized in making clinical decisions ensured. The use of traditional machine learning algorithms may not be able to detect anomalies, nonlinear and complex patterns in the healthcare data. This study aims to design an anomaly aware healthcare prediction model using the integration of an autoencoder for latent feature extraction and isolation forest, K-means clustering and gradient-boosted classification for disease prediction. This model was tested using two healthcare datasets that contained 2,000 diabetes data records and 50,000 cancer data records. Various models were designed and analyzed to find out how useful the inclusion of anomaly detection, latent feature extraction, and classification is in disease prediction. Of all the tested models, the Auto Encoder (AE) + Isolation Forest + LightGBM combination provided the best performance on the diabetes dataset with an accuracy of 90.75% and ROC-AUC value of 0.9593, while Auto Encoder (AE) + K-Means + LightGBM combination provided the best performance on the cancer dataset with an accuracy of 97.51% and ROC-AUC value of 0.9969. From the above results, the paper concludes that using Autoencoders to extract latent space features along with LightGBM works well for anomaly-aware disease prediction.