ST-DMFRL: Spatio-Temporal Diffusion Mean-Field Reinforcement Learning for Adaptive Load Balancing In Hyperscale Cloud Data Centers

Abstract

Cloud data centers require highly efficient load balancing mechanisms to manage dynamic and bursty workloads while maintaining strict service level agreements. Although deep reinforcement learning has shown promise in this domain, existing approaches suffer from three critical bottlenecks: Gaussian policy collapse in multi-modal action spaces, poor scalability of multi-agent reinforcement learning due to quadratic communication overhead, and delayed reward attribution in proactive virtual machine migration. To address these limitations, this paper proposes Spatio-Temporal Diffusion Mean-Field Reinforcement Learning (ST-DMFRL), a novel framework for adaptive load balancing. ST-DMFRL introduces a spatio-temporal graph transformer that jointly captures topological bottlenecks and temporal workload bursts from a dynamic data center graph. To resolve policy collapse, we replace the standard Gaussian policy with a conditional diffusion policy network that iteratively denoises Gaussian noise into traffic allocation vectors, enabling the agent to sample from distinct and equally optimal routing modes rather than averaging them. Furthermore, a mean-field game formulation is integrated so that each local agent interacts with the aggregate statistical load distribution of the cluster rather than with individual peers, reducing multi-agent complexity from quadratic to linear. Finally, a delay-aware reward shaping mechanism using temporal causal tracing solves the credit assignment problem by attributing long-term queue drainage benefits to immediate migration decisions. Extensive experiments on large-scale production cluster traces demonstrate that ST-DMFRL reduces average response latency by approximately 51%, decreases SLA violations by 85%, improves throughput by 45%, and scales seamlessly to clusters of 10,000 nodes where competing multi-agent methods fail to converge. By bridging generative diffusion models, graph transformers, and mean-field game theory, the proposed framework establishes a new state of the art for hyperscale cloud resource management.