AGENTIC LLM SERVING · HYBRID MODELS

FlashState

Flexible hybrid state management for agentic LLM serving. Resume every agent turn exactly where the last one ended, with no redundant prefill.

Paper (coming soon) Code (coming soon)

Results

2.87×

Prefill throughput

vs. vLLM, GLM-5.3-Flash, 48 sessions

−97%

Uncached tokens

Median per request vs. vLLM

−87%

p90 TTFT

Time to first token vs. vLLM

The problem

Coding agents send request after request, and each one extends the previous context. Prefix caching avoids recomputing that shared history. Hybrid models, which mix attention with recurrent layers such as Mamba2, Gated DeltaNet or Kimi Delta Attention, make this harder. To resume, the engine needs the attention KV and a recurrent checkpoint at the exact resume position.

Existing engines tie checkpoints to KV-cache block boundaries and evict them without regard to their KV. As a result, a follow-up request falls back to an earlier block and recomputes thousands of tokens it has already seen.

vLLM saves checkpoints only at block boundaries (4,352 tokens for GLM-5.3-Flash), so it recomputes up to the end of the previous turn. FlashState checkpoints at the exact turn boundary.

How it works

Arbitrary State Checkpointing

Decouples recurrent checkpoint positions from KV-cache block boundaries, so a checkpoint can sit at any token, including the end of a turn.

Cost-aware state management

Treats checkpoint admission and eviction as a memory-constrained optimization that minimizes redundant prefill, solved online with a GreedyDual-Size variant.

Built into vLLM

Supports safe state reuse across hybrid architectures: DSA + KDA, Qwen sparse attention + Gated DeltaNet, and GQA + Mamba2.

Across hybrid architectures

Model	Architecture	Prefill TPM	Uncached tokens (p50)	p90 TTFT
GLM-5.3-Flash	DSA + Kimi Delta Attention	2.87×	−97%	−87%
Nemotron 3 Super	GQA + Mamba2	2.27×	−97%	−77%
Qwen3.8-Flash-Next	Sparse attn + Gated DeltaNet	1.54×	−66%	−79%

Coding-agent traces, 48 concurrent sessions, two NVIDIA B200 GPUs per engine, same cache budget as the vLLM baseline.

Citation

@misc{flashstate,
  title = {FlashState: Unlocking Flexible Hybrid State Management
           for Agentic LLM Serving},
  note  = {Under review}
}