Queueing-Aware Latency Prediction for Scalable Large Language Model Applications

Abstract

Large language model applications increasingly operate as shared, multi-tenant services in which response quality depends not only on model accuracy but on the timing of computation, memory movement, scheduling, and network exchange. Latency prediction for such systems cannot be reduced to isolated accelerator benchmarks, because the observed response time emerges from dynamic queues that form across gateways, batching schedulers, key-value caches, inference workers, autoscalers, and downstream tools. This paper develops a queueing-aware perspective on latency prediction for scalable large language model applications. It synthesizes classical queueing theory with contemporary inference serving architectures, discussing how batching, continuous scheduling, paged attention, disaggregated execution, and autoscaling create structural trade-offs among throughput, tail latency, cost, and fairness. The paper argues that prediction systems must be treated as socio-technical infrastructure rather than purely statistical components. Accurate forecasts require observability across layers, governance over service-level objectives, robust treatment of workload drift, and explicit attention to sustainability and equitable access. It also examines how queueing-aware prediction can improve admission control, routing, capacity planning, and user-facing communication while introducing risks of feedback loops, manipulation, and uneven service quality. The conclusion outlines research directions for hybrid mechanistic and machine-learning models, policy-aware service management, and resilient deployment in cloud-edge environments.