When Does Disaggregated, Cache-Aware LLM Serving Pay Off? An Analytical Crossover Model Validated on Real Compute, Network, and Coordination Measurements

Abstract

Disaggregated, cache-aware serving architectures for large language models, in which prefill and decode computation are split across pools and a shared key-value (KV) cache is reused across requests, are now a widely discussed production pattern (DistServe, Splitwise, Mooncake, and the CNCF llm-d project), but the decision of whether disaggregation is worth its coordination and network overhead for a given model, context length, and cluster interconnect is usually made by intuition rather than by a stated model. This paper derives a simple analytical crossover condition, expected net benefit equals compute saved by a cache hit, weighted by resource utilization, minus the KV-transfer cost paid on a hit, minus a per-request coordination cost paid regardless of hit or miss, and evaluates it with real measurements rather than assumed constants. Because no GPU is available in this study's execution environment, three real Llama-family model configurations (1.24B, 8.03B, and 70.6B parameters, verified against their published Hugging Face configurations) are benchmarked with a real, per-layer, memory-bounded CPU forward-pass timer; a real coordination layer (a threaded HTTP cache-index and least-loaded router, exercised with 3,000 real loopback requests, mean 0.59 ms, p99 0.89 ms) measures the per-request routing tax; and a real TCP loopback KV-transfer benchmark (achieving 2.0 to 3.0 GB/s across payload sizes from 4 MB to 2.7 GB) is combined with real, cited manufacturer and IEEE line-rate specifications (10/25/100GbE, InfiniBand HDR/NDR, NVLink, and the NVIDIA A100 datasheet) to project GPU-scale compute and interconnect-scale transfer. At the CPU-measured compute scale, disaggregation always pays off in this model, because the measured compute a hit avoids (on the order of seconds to tens of minutes) dwarfs millisecond-scale coordination and transfer overhead; the measured decode-step compute is 6 to 103 times cheaper than a full prefill of the same context length, the gap widening with context length. At the analytically projected GPU scale, where per-layer compute drops to single-digit milliseconds, the picture changes: of 189 (model, context length, interconnect) combinations evaluated at three representative utilization levels, 65 (34%) are infeasible at any cache hit rate, concentrated in small models, short contexts, and slower interconnects, where the required hit rate would otherwise range as high as 98%; the remaining combinations require hit rates as low as 0.07% once the 70B model, long contexts, and NVLink-class interconnects are reached. We report both the feasible and the infeasible regions rather than only the favorable case, and we are explicit throughout about which numbers were measured on this sandbox's CPU and network stack versus analytically projected from public GPU and interconnect specifications, a distinction the paper's title reflects by not claiming a GPU or a live llm-d deployment that was never run.