Phase-Splitting and Smarter Scheduling of LLM Inference in Data Centers

Neil L. Rideout

10/15/20265 min read

Phase-Splitting and Smarter Scheduling of LLM Inference in Data Centers

The explosive growth of generative large language models has transformed data centers into critical infrastructure for AI services. Chatbots, coding assistants, enterprise knowledge systems, and agentic workflows all depend on fast, reliable inference. Yet the underlying hardware economics remain punishing. High-end GPUs such as the H100 and successor platforms are expensive to acquire, power-hungry to operate, and frequently underutilized once real traffic arrives.

A fundamental mismatch drives much of this inefficiency. LLM inference is not a uniform workload. It consists of two phases with sharply different resource profiles: a compute-intensive prefill (or prompt computation) phase and a memory-bandwidth-bound decode (token generation) phase. Traditional serving systems that keep both phases on the same GPU or the same homogeneous cluster force expensive silicon to spend long stretches of time waiting on memory or running at low utilization. Phase-splitting—also called prefill-decode disaggregation—combined with smarter, phase-aware scheduling offers a practical path to higher throughput, lower cost, and better power efficiency.

The Two Phases of LLM Inference

When a request arrives, the model first processes the entire input prompt in parallel. This prefill stage performs large matrix multiplications across all tokens simultaneously. It is heavily compute-bound and benefits from the highest available FLOPS, large batch sizes when possible, and the newest accelerators. Once the first output token is produced, the system switches to autoregressive generation. Each subsequent token depends on the previous one and on the growing key-value (KV) cache. Decode is typically memory-bandwidth limited; the arithmetic intensity is low, and the dominant cost is repeatedly reading the KV cache and model weights.

Because these phases have different bottlenecks, co-locating them creates interference. A new prefill request can stall ongoing decode tokens, inflating time-between-tokens (TBT) or inter-token latency. Continuous batching and advanced schedulers mitigate some of the pain, but they cannot erase the fundamental mismatch. Characterization studies on production traces (coding assistants and conversational services) consistently show that the decode phase underutilizes the compute capability of flagship GPUs while still consuming high power and memory capacity.

Phase-Splitting: The Core Idea

Phase-splitting physically or logically separates the two stages onto different machine pools or resource partitions. Prefill machines focus on rapid prompt processing and production of the KV cache. Decode machines specialize in sustained token generation. The KV cache (or a compact representation of request state) is transferred across the high-bandwidth interconnects that already exist in modern GPU clusters—NVLink, InfiniBand, or Ethernet fabrics with RDMA.

The original Splitwise work from Microsoft Research and collaborators demonstrated the power of this approach. By provisioning separate prompt and token pools (plus a flexible mixed pool that can absorb bursts), the system can match hardware to phase needs. Prefill can run on the latest high-FLOPS GPUs; decode can run on slightly older or lower-power devices that still deliver excellent memory bandwidth. State transfer overhead stays low—typically under 1% of end-to-end latency when optimized libraries and fast backplanes are used.

Results are striking. Relative to strong homogeneous baselines that already employ continuous batching, phase-split clusters have achieved up to 1.4× higher throughput at 20% lower cost, or 2.35× higher throughput under the same cost and power budgets. Power savings of roughly 25% at equivalent throughput have also been reported. These gains come from better hardware utilization, the ability to use heterogeneous accelerators, and reduced interference between phases.

Evolving Beyond Static Splits

Static prefill and decode pools are only the starting point. Real workloads are bursty and heavy-tailed. Prefill demand can spike while decode capacity sits idle, or vice versa. Recent systems therefore add elasticity and intelligence.

Dynamic rate matching adjusts the ratio of prefill to decode capacity in response to live traffic and service-level objectives (SLOs) such as time-to-first-token (TTFT) and TBT. Mixed or swing pools allow machines to temporarily switch roles. Prefill-deflection techniques let lightly loaded decode nodes perform chunked prefill for queued requests when doing so improves tail latency without violating ongoing decode SLOs. Because the prefill happens in place, the expensive inter-node KV transfer is avoided for those requests.

Scheduling itself becomes multi-level and phase-aware. A cluster-level scheduler routes requests (or phases of requests) using metrics such as pending tokens rather than simple request counts, often employing variants of join-the-shortest-queue. Machine-level or GPU-level schedulers maintain separate queues for prefill and decode and apply phase-specific policies—shortest-prompt-first for TTFT, aging or opportunity-cost policies to prevent head-of-line blocking, and token-window pacing to stabilize inter-token latency. Some designs further partition streaming multiprocessors (SMs) within a single GPU so that prefill and decode can co-exist with reduced interference while still sharing the same memory hierarchy.

Heterogeneous hardware is a natural fit. High-end GPUs handle prefill; previous-generation or specialized memory-optimized devices handle decode. Quantization strategies can also be phase-aware—more aggressive weight-only quantization on the decode side, dual-side approaches on prefill—further stretching scarce accelerator capacity.

System and Operational Implications for Data Centers

For data-center operators the implications are concrete. Capacity planning shifts from “how many H100s do we need?” to “what mix of prefill and decode capacity, and what interconnect bandwidth, do we need under expected traffic mixes?” Power delivery and cooling can be provisioned more precisely because the two phases have different power profiles: short, high-power prefill bursts versus longer, lower, more stable decode draws. Peak-power management techniques that are phase-aware can further reduce capital expense for power infrastructure.

Networking becomes a first-class design constraint. The fabric must support low-latency, high-bandwidth KV-cache transfers without becoming the new bottleneck. Topology-aware placement of prefill and decode pools, careful routing of state, and potentially hierarchical caching of popular prefixes all matter. Observability must track phase-specific metrics—TTFT, TBT, queue depths per pool, transfer volumes, and energy per token—rather than coarse GPU utilization alone.

Software stacks are evolving rapidly. Frameworks such as vLLM have added support for disaggregated serving; research prototypes and production systems continue to refine continuous batching, prefix caching, and intelligent routing on top of the split. The combination of phase disaggregation with prefix-cache-aware routing is especially powerful for multi-turn and agentic workloads that share long contexts.

Challenges and Open Questions

Phase-splitting is not free. Transfer overhead, although small when well engineered, still exists. Under extreme prefill-heavy or decode-heavy skew, fixed ratios waste capacity; elastic rebalancing must be fast and stable. Tail-latency guarantees become harder when requests cross machine boundaries. Debugging distributed request state requires better tooling. Security and multi-tenancy considerations grow when KV caches move between hosts.

Workload diversity also matters. Benefits are largest for larger models and for traffic with long prompts relative to output length. Short, interactive queries or highly cacheable multi-turn sessions change the optimal balance. Carbon-aware or renewable-aware scheduling adds another dimension: the ability to shift phases or entire requests across geographically distributed sites must respect both SLOs and the distinct power profiles of prefill versus decode.

Looking Ahead

As models continue to grow and inference demand outpaces training, the pressure on data-center efficiency will only intensify. Phase-splitting turns an architectural observation—two phases, two resource profiles—into a practical systems lever. Combined with smarter, multi-level, SLO-aware scheduling, dynamic pool management, heterogeneous hardware, and phase-specific optimizations, it delivers measurable gains in throughput, cost, and power.

The next wave of systems will likely push further: tighter co-design of accelerators with phase semantics, finer-grained SM or chiplet partitioning, predictive models of request structure that guide scheduling decisions, and deeper integration with data-center power and cooling controls. For operators building or expanding AI infrastructure, treating prefill and decode as first-class, independently schedulable resources is no longer a research curiosity. It is becoming a core requirement for sustainable, high-performance LLM serving.

In short, the path to efficient large-scale inference runs through deliberate separation of phases and intelligent orchestration of the resulting specialized resources. Data centers that master this separation will deliver more tokens per watt, more requests per dollar, and better user experience under real-world load.

Contact

Head Office

Green Life Enterprises LLC
8 Biltmore Est
Suite 310,
Phoenix, AZ 85016

greenlifedatacenters@gmail.com

+1-813-220-0001

Canadian Office

Green Life Enterprises LLC
3142 Nicholson Ave
Suite 10
New Waterford, Nova Scotia B1H 1N8