ALPS: An Efficient Request Scheduler for LLM Inference

Speaker: Dr Chen Chen, Nottingham Trent University
Series: Computer Laboratory Systems Research Group Seminar

Abstract:

The deployment of Large Language Models (LLMs) on resource-constrained edge server clusters introduces a critical systems challenge: achieving high throughput and low latency under strict GPU memory and bandwidth limits. Current schedulers are ill-suited for this environment, as they treat requests independently, incurring two major inefficiencies. They waste compute and memory on redundant prefill computations for identical, lengthy system prompts across applications, and their phase-agnostic policies create suboptimal batches that poorly overlap compute-intensive prefills with memory-bound decodes, leading to GPU under-utilization.

This paper presents ALPS, an application-aware, length-predictive scheduler for request serving for edge LLM inference. ALPS transforms the queueing layer into an intelligent orchestrator with two core techniques: (i) semantic-aware request grouping, which hashes requests by system prompt to co-locate them, reducing redundant prefill computations; and (ii) phase-aware scheduling, which uses a lightweight output token bucket predictor to estimate decode/prefill ratios and strategically reorders requests to maximize prefill-decode overlap in continuous hybrid batching. Augmented with SLO-aware preemption, ALPS is implemented as a practical scheduling layer atop vLLM. Extensive evaluations on an edge GPU testbed with diverse models and production workloads demonstrate that ALPS improves token throughput by up to 1.6× and reduces end-to-end latency by up to 2.4× compared to state-of-the-art schedulers.

Bio:

Chen has been a Lecturer in Computer Science at Nottingham Trent University since 2025. Between 2022 and 2025, he was a postdoctoral researcher with the System Research Group advised by Prof. Richard Mortier, Department of Computer Science and Technology, University of Cambridge.

His research interests are broadly in areas of AI Systems, Distributed Systems, Serverless Computing, Cloud/Edge Computing, Network Resource Orchestration, and IoTs. His latest work focuses on request scheduling in LLM inference, aiming to optimize the overall performance of the system, including latency, energy consumption and system throughput. Chen has extensive experience in collaborating with industries such as Siemens, Infineon, EMS-UK, National Physical Lab and many others. Chen’s research is mostly funded by UKRI and EU Horizon.

Online meeting link:

Microsoft Teams