Inference stopped being an afterthought bolted onto a trained model. Serving engines now decide the wire format, the latency profile, and the failure modes your clients see. This piece settles what changed at the API level, why team charters are splitting along serving boundaries, and which concrete steps working teams can take before the next rollout.

The Latency Budget Rebellion

Training teams spent years optimizing throughput on large batches. Inference punishes that habit. A request arrives, a user waits, and the clock runs on decode time, not aggregate samples per second. Teams that carried batch-first assumptions into production found their dashboards green while users refreshed a spinner.

The pressure shows up in budgets. Per-token cost now sits in the same review as model quality, and a service-level objective for p95 decode latency gets argued over in pull requests rather than quarterly planning decks. An engineer proposing a larger context window has to defend the memory that window consumes against every concurrent request.

PagedAttention, introduced in 2023 by Woosuk Kwon and colleagues alongside the vLLM serving engine, forced a redesign of how teams think about the key-value cache. The method stores the cache in fixed-size blocks mapped to non-contiguous physical memory, borrowing from virtual memory and OS paging. Once that idea landed, cache layout stopped being an implementation detail and became part of the deployment conversation.

The arithmetic behind that conversation is unforgiving. A KV cache grows linearly with sequence length, with the number of layers, and with the size of the attention head dimension, and it is stored per concurrent request. Doubling the context window does not double the memory footprint of a single request; it doubles it for every request in flight at once. On a card with a fixed pool of high-bandwidth memory, the model weights take their share first, and whatever remains becomes the ceiling on concurrency. That ceiling is why a configuration that benchmarks beautifully on a single stream can collapse under a realistic burst.

There is a second-order effect that catches people late: prefill and decode compete for the same silicon. Prefill is compute-bound and chews through matrix multiplies in large chunks; decode is memory-bandwidth-bound and moves a comparatively small amount of data per step. A scheduler that admits a wave of long prompts will stall the decode steps of everything already in flight, so the p95 you promised on a quiet morning evaporates the moment a batch job hits the same endpoint. Separating the two phases onto different pools, or at least rate-limiting bulk prefill against interactive traffic, is the kind of decision that belongs in the architecture review rather than in a postmortem.

Serving Engines Become the Contract

vLLM, TensorRT-LLM, and Text Generation Inference each impose their own behavior on the wire. Choose one and you inherit its streaming semantics, its batching strategy, and its quirks around request cancellation. The engine is less a library than a contract with the rest of your stack.

Block sizes leak outward. A KV-cache block size tuned for one engine's memory allocator shapes how many tokens a partial request can hold, which in turn shapes how a client should chunk and retry. Client SDKs written before the engine swap start making assumptions that no longer hold.

Continuous batching is the other place where engines diverge. Instead of waiting for a batch to fill, the scheduler admits new requests at token boundaries, so a short completion can finish and free its slot while a long one keeps decoding. That improves utilization, and it also means the latency a given request observes depends on what else arrived in the same window. Two identical prompts sent a second apart can return at noticeably different speeds. Clients that measure a single request and extrapolate will be wrong about the tail.

Roles split along those boundaries. One group owns the model and its evaluation harness; another owns the serving tier, its memory profile, and its rollout. Handoffs between them are where most of the interesting bugs live, and where documentation tends to be thinnest.

What Changes at the Wire

Streaming deltas replace the single response payload that older clients expected. A completion arrives token by token, and any consumer that assumed a complete JSON body has to be rewritten. Partial frames, keepalive chunks, and end-of-stream markers each need explicit handling.

Token accounting shifts to per-request meters. Instead of a monthly invoice, teams watch prompt tokens and completion tokens accumulate per call, with the completion side usually dominating cost on long outputs. Retries get expensive fast, which is why idempotency keys and retry budgets deserve the same scrutiny they get elsewhere. A related piece on retry budgets and outage risk covers the general pattern.

Timeouts need to be tuned to decode, not prefill. A long prompt can take a while to process before the first token appears, and a client that times out during prefill will retry a request that was about to succeed. Splitting the two phases in your telemetry is the cheapest fix available.

Structured output deserves its own line in the contract. Constrained decoding, whether grammar-based or schema-driven, changes the token distribution the engine emits and can slow decode by a margin that varies with how tight the constraint is. A JSON schema that forbids most of the vocabulary forces the sampler to mask logits on nearly every step. If your client depends on strict schemas, benchmark with the schema in place rather than against a free-form baseline.

Sampling parameters are the quietest part of the contract and the one that breaks comparisons most often. Temperature, top-p, and any repetition penalty interact with the engine's own defaults, and two engines given the same nominal settings will not produce the same distribution. When a team migrates from one serving stack to another, an evaluation harness that only checks final answer correctness will miss the shift; a harness that also records logprob deltas on a fixed prompt set will catch it. That record doubles as a regression baseline for the next upgrade, which is the only way to tell a genuine model improvement from a change in sampler defaults.

Tooling Churn Reshapes Hiring

Job posts now ask for serving-engine fluency alongside model experience. Candidates get asked how they would size a KV cache for a given concurrency target, or what happens when block eviction kicks in under memory pressure. The questions are closer to systems interviews than to ML theory.

On-call rotations include cache eviction paging. A page that fires because the serving tier ran out of blocks at 3 a.m. is a different animal from a model accuracy regression, and the runbook for it looks more like a database incident than a training job. Teams that never wrote such a runbook are writing one now.

The broader hiring pattern echoes what this site has noted about tooling careers trading depth for breadth. Fewer pure-model roles, more systems roles that happen to sit next to a model.

Playbook Rewrites in Practice

Benchmark suites move into CI pipelines. A pull request that changes a prompt template or a sampling parameter now triggers a latency and cost benchmark, and the results gate the merge. Teams that skipped this step learned the hard way when a small change doubled completion length.

Rollouts get gated on p95 decode latency rather than average throughput. Cost dashboards pair with quality metrics so a cheaper configuration that degrades output becomes visible before it ships. Postmortems increasingly cite block-size regressions, where a configuration change quietly reduced the number of concurrent requests the engine could sustain.

Load testing has to look different too. A fixed set of identical prompts measures the engine under the friendliest possible conditions, because every request finishes at the same time and the scheduler never has to make a hard choice. Replay real traffic shapes instead: a mix of prompt lengths, a realistic spread of output lengths, and arrivals that cluster. That replay is what surfaces the eviction behavior and the queueing you will actually meet on a busy afternoon.

Capacity planning deserves the same realism. Peak concurrency on an interactive product often lands an order of magnitude above the daily mean, and it arrives in bursts measured in seconds rather than hours. Sizing the serving tier for the mean leaves you short exactly when it matters, while sizing for the absolute peak leaves expensive memory idle most of the day. The usual compromise is a smaller steady-state pool with a documented scale-out path and a queue that sheds load predictably, plus a hard cap on per-request context so one caller cannot monopolize the block pool.

The trade-off is real. Smaller teams argue that this instrumentation overhead is too heavy for a product still finding fit, and they have a point. A two-person team shipping a prototype does not need a full benchmark harness. The cost arrives later, when the prototype becomes the product and nobody remembers why the serving tier behaves the way it does.

Actions for Working Teams

Instrument prefill and decode separately in your telemetry this week, even if it means adding a timestamp pair around the first token.

Pin serving-engine versions in your lockfiles, including the CUDA and driver versions underneath them, so a silent upgrade cannot change wire behavior overnight.

Add cache-eviction tests to staging that deliberately exhaust the KV-cache blocks and confirm the service degrades predictably instead of dropping requests.

Rehearse rollback on token-accounting drift, where the meter and the actual token count diverge after a model or tokenizer change.

Document the wire contract, including streaming frame shapes and retry semantics, alongside the model card so client teams read one artifact instead of guessing.