AMACO-QP: Enhanced Adaptive Multi-Colony ACO-Based Query Processing with Healthcare-Aware Cost Model for Scalable Big Data Analytics

Abstract

The rapid proliferation of electronic health records (EHRs) across large-scale distributed healthcare environments has created an urgent demand for intelligent, scalable, and privacy-aware query processing frameworks. Traditional cost-based query optimizers do not consider clinical priority, evidence credibility, and regulatory considerations like HIPAA, considering all healthcare queries as equals, and do not give the query optimal query performance and clinically unreliable results. This paper presents AMACO-QP (Adaptive Multi-Colony Ant Colony Optimization-based Query Processing), a new healthcare-aware query optimization system that incorporates a multi-objective cost model, the first of its kind, with simulated federated multi-colony ACO execution on Apache Hadoop (HDFS), Apache Spark, and Trino 437 in a controlled single-node setting, which models a hospital edge deployment. The framework consists of dynamic clinical priority scoring (ICU > Emergency > General Ward), exchange of pheromones with respect to privacy preservation of different types of colonies within the hospital, and evidence-based result credibility scoring based on ICD-10 and SNOMED-CT clinical guidelines. AMACO-QP had been deployed and tested on the HealthcareData.csv dataset in the Hive-partitioned Parquet table (healthcare_gold) of a Dockerized Trino 437 cluster, with clinically significant results on query response time, privacy compliance, and result credibility observed in all tested workload conditions. When compared to five robust baselines, including Spark Catalyst using Adaptive Query Execution, Trino CBO, Apache Calcite Volcano Planner, Genetic Algorithms Query Optimizer, and Deep Reinforcement Learning Query Optimizer, AMACO-QP exhibits better query response time, throughput, and clinical prioritization compliance. The proposed framework will provide a scalable, HIPAA-compliant, and clinically meaningful base for optimization of big data queries in federated healthcare environments.