Uncertainty-Calibrated Thermal-Aware Scheduling For Deadline-Constrained Edge AI Inference

Abstract

The phenomenon of giving inference at the edge a deadline is gaining in popularity as more real-time applications move computing from the centralized cloud to the distributed edge. Latency and temperature are hard to predict, though, because of workload bursts, congested queues, network changes, and temperature fluctuations. Deterministic schedulers do not take into account this uncertainty, and might therefore choose a schedule that fails to meet the service deadlines or thermal limits. The present study is proposed to predict end-to-end latency and peak node temperature by using LightGBM model to design and develop UCTAS: Uncertainty-Calibrated Thermal-Aware Scheduler. Split conformal prediction yields calibrated uncertainty intervals where the upper bound serves for screening the feasibility on the deadline and the temperature. A multi-objective allocation function is then used to solve the problem with balancing latency, energy, deadline risk, thermal risk and uncertainty. UCTAS is tested by simulation with five heterogeneous abstract edge nodes, 100,000 inference requests and six base schedulers. The results indicate that the intervals are covered reliably with better deadline satisfaction, fewer thermal violations, lower tail latency, and reliable performance in scheduling the edges.