Scaling 10M-Token Context in Production: Disaggregated KV-Cache Fabrics, Custom Triton Prefill Kernels, and Ultra Ethernet RDMA
Key Takeaways
- 10M-token contexts overwhelm single GPUs, requiring hundreds of GBs for KV-cache alone.
- Scaling 10M-token context necessitates disaggregated prefill and decode execution tiers.
- Even INT4 quantization for 10M tokens exceeds 80GB H100 memory for KV-cache.
- Asynchronous custom Triton chunked-prefill kernels are essential for efficient context ingestion.
- Ultra Ethernet RDMA provides the lossless, multi-path networking required for disaggregated inference.
Scaling large language model context windows to 10 million tokens in production breaks the classic compute-versus-memory dynamics of monolithic inference clusters, shifting the bottleneck from FLOP throughput to distributed memory fabric capacity and interconnect topology. Making an uninterrupted 10M-token context window computationally and economically viable requires abandoning unified inference nodes in favor of a triad architecture: disaggregated prefill and decode execution tiers, asynchronous custom Triton chunked-prefill kernels, and lossless, multi-path Ultra Ethernet Consortium (UEC) RDMA networking.
1. The Physics of the 10M-Token Frontier: Footprint and Latency Divergence
The standard linear increase in context length triggers a quadratic expansion in attention operations and a linear, highly punitive expansion in memory capacity requirements. When scaling to a 10,000,000-token operational horizon, traditional collocated serving—where the same GPU handles both the prompt ingestion (prefill) phase and the autoregressive token generation (decode) phase—fails completely due to device-level out-of-memory (OOM) events and extreme resource starvation.
The physical memory footprint for retaining the Key-Value (KV) cache of a single context stream is strictly governed by the model’s structural parameters:
Where:
• L = Number of Transformer Layers
• NKV = Number of Key-Value Heads (reflecting Grouped-Query Attention)
• Dhead = Dimension of each attention head
• S = Sequence length in tokens (10,000,000)
• Pbytes = Bytes per parameter (2 for FP16/BF16, 1 for FP8, 0.5 for INT4)
Consider an enterprise-scale foundation model featuring 80 layers, a head dimension of 128, and Grouped-Query Attention (GQA) configured with 8 KV heads (comparable to modern 70B parameter open-weights architectures). Evaluating the memory footprint across distinct precision formats for a 10M context reveals the infrastructural scale required:
- Unquantized FP16 (2 bytes/element): $2 times 80 times 8 times 128 times 10^7 times 2 = 327.68 text{ GB}$ per single concurrent request stream.
- Quantized FP8 (E4M3 / E5M2, 1 byte/element): $2 times 80 times 8 times 128 times 10^7 times 1 = 163.84 text{ GB}$ per single concurrent request stream.
- Compressed INT4 (0.5 bytes/element): $2 times 80 times 8 times 128 times 10^7 times 0.5 = 81.92 text{ GB}$ per single concurrent request stream.
Even with state-of-the-art sub-byte quantization (INT4), a single 10M-token dialogue or document history exhausts the entirety of an 80GB NVIDIA H100 Tensor Core GPU’s High Bandwidth Memory (HBM3), leaving zero capacity for the model weights ($~140text{ GB}$ at FP16) or the activation memory required for intermediate tensor operations.
Beyond pure memory capacity lies the operational divergence between Time to First Token (TTFT) and Inter-Token Latency (ITL). In monolithic architectures, processing a 10M-token prefill can seize GPU Streaming Multiprocessors (SMs) for tens of seconds. Autoregressive decoding requests collocated on the same hardware stall completely, introducing multi-second jitter spikes into token streaming rates. Decoupling the phases into physically specialized compute topologies is an absolute requirement.
2. Disaggregated Serving: The Prefill-Decode Decoupling Paradigm
To resolve the interference between prefill computation and token decoding, modern production infrastructure utilizes Disaggregated Serving. Pioneered conceptually by research systems such as Splitwise and DistServe, and brought to hyperscale via production frameworks like Mooncake, this approach partitions the inference cluster into two distinct hardware pools: Prefill Nodes and Decode Nodes.

Prefill Engine Cluster (Compute-Bound)
Optimized for dense matrix multiplications ($text{GEMM}$) executed across parallel token sequences. Prefill instances maximize Tensor Core utilization, operating under broad Pipeline and Tensor Parallelism topologies ($TP=8, PP=4$) backed by SXM5/NVLink substrates. They calculate the initial context KV cache states rapidly, then immediately stream those tensors out of local HBM into the disaggregated fabric.
Decode Engine Cluster (Memory Bandwidth-Bound)
Optimized for memory bandwidth and latency-critical memory lookups ($text{GEMV}$). Operating primarily under high continuous batching ($DP ge 16$), these nodes do not retain static prompt contexts locally. Instead, they dynamically query and bind pre-computed KV tensors through non-blocking direct-memory-access primitives as new tokens are sequentially sampled.
The linchpin of this architecture is the Tiered KV Cache Storage Fabric. At a sequence length of 10M tokens, retaining millions of inactive context vectors in Tier-1 GPU memory ($HBM3e$) is economically non-viable. Instead, a multi-tier memory hierarchy must be managed by an orchestrator operating at the virtual memory block level, reminiscent of operating system page swaps:
- Tier-1 (GPU HBM3e): Retains only the active context windows and sliding-window KV blocks directly accessed by current generation kernels. Throughput: $3.35 text{ TB/s}$ to $4.8 text{ TB/s}$.
- Tier-2 (Host Dynamic RAM / CXL 3.1): Retains recent conversation histories and speculative decoding trees via host-pinned memory mapped over PCIe Gen5 ($128 text{ GB/s}$) or Compute Express Link ($CXL 3.1$) switching fabrics, delivering near-zero-overhead transfers.
- Tier-3 (Distributed Non-Volatile Fabric): Remote NVMe-oF pools backed by high-density enterprise SSD arrays. KV cache segments matching cold historical context chunks are offloaded here, retrieved via RDMA only when backward attention masks demand historical verification.
Architecture Flow: End-to-End 10M Token Pipeline
[User Prompt: 10M Tokens]
│
▼
[Global API / Ingress Gateway] ──▶ Dispatches chunked ingestion to Prefill Worker Pool
│
├──▶ Node A (Prefill Engine): Computes Chunk 0..2M via Triton Parallel Kernel
├──▶ Node B (Prefill Engine): Computes Chunk 2..4M via Triton Parallel Kernel
└──▶ Nodes C-E: Compute remaining sequence blocks concurrently via RingAttention
│
▼ [Asynchronous RDMA Stream via Ultra Ethernet Consortium (UEC) Transport]
[Disaggregated Distributed KV-Cache Store (Shared NVRAM / Host RAM Fabric)]
│
▼ [On-Demand Paged Block Fetching via Non-blocking Memory Handles]
[Decode Engine Pool] ──▶ Generates auto-regressive tokens at steady-state ITL (< 25ms/tok)
3. Custom Triton Chunked-Prefill Kernels
Standard implementation kernels such as vanilla FlashAttention-2 or FlashAttention-3 are designed for sequences that reside fully within continuous device allocation buffers. When sequence length $S$ reaches $10^7$, two critical failure modes arise:
- SRAM Capacity Overflow: Intermediate accumulators for the softmax normalizer exceed the dynamic shared memory capacity ($228text{ KB}$ per SM on Hopper).
- Prefill Quantization Degradation: Quantizing KV states to FP8 during the ingestion of large contexts often introduces numerical instability, leading to underflows in attention score computation ($mathbf{S} = mathbf{Q}mathbf{K}^T / sqrt{d}$) without block-wise scaled accumulation.
To resolve these limitations, modern production runtimes leverage custom Triton kernels that process prompts using Chunked Prefill with Online Softmax Rescaling. This approach enables KV blocks to be computed, compressed to FP8, and streamed directly into pinned host/RDMA buffers without materializing the full tensor in GPU memory.
The following production-grade Triton kernel demonstrates chunked block attention executing FP8 matrix loading, dynamic dequantization on SRAM, and numerically stable online softmax updating:
import triton.language as tl
import torch
@triton.jit
def _chunked_prefill_kernel_fp8(
Q, K, V, sm_scale,
Out,
stride_qz, stride_qh, stride_qm, stride_qk,
stride_kz, stride_kh, stride_kn, stride_kk,
stride_vz, stride_vh, stride_vn, stride_vk,
stride_oz, stride_oh, stride_om, stride_ok,
Z, H, N_CTX,
BLOCK_M: tl.constexpr,
BLOCK_DMODEL: tl.constexpr,
BLOCK_N: tl.constexpr,
):
"""
Chunked Prefill Kernel processing long-sequence attention blocks.
Supports FP8 Key/Value ingestion with high-precision (FP32) accumulation.
"""
# Block identification
start_m = tl.program_id(0)
offs_hz = tl.program_id(1)
# Base offsets for query pointers
offs_m = start_m * BLOCK_M + tl.arange(0, BLOCK_M)
offs_d = tl.arange(0, BLOCK_DMODEL)
q_ptrs = Q + (offs_hz * stride_qh) + (offs_m[:, None] * stride_qm) + (offs_d[None, :] * stride_qk)
# Initialize online softmax statistics in FP32
m_i = tl.zeros([BLOCK_M], dtype=tl.float32) – float("inf")
l_i = tl.zeros([BLOCK_M], dtype=tl.float32)
acc = tl.zeros([BLOCK_M, BLOCK_DMODEL], dtype=tl.float32)
# Load Query block (kept in SRAM across the loop)
# Mask out-of-bounds tokens
q_mask = offs_m[:, None] < N_CTX
q = tl.load(q_ptrs, mask=q_mask, other=0.0)
# Compute loop over sequence length in chunks of BLOCK_N
for start_n in range(0, (start_m + 1) * BLOCK_M, BLOCK_N):
offs_n = start_n + tl.arange(0, BLOCK_N)
# Pointer calculations for K and V
k_ptrs = K + (offs_hz * stride_kh) + (offs_n[None, :] * stride_kn) + (offs_d[:, None] * stride_kk)
v_ptrs = V + (offs_hz * stride_vh) + (offs_n[:, None] * stride_vn) + (offs_d[None, :] * stride_vk)
k_mask = offs_n[None, :] < N_CTX
v_mask = offs_n[:, None] < N_CTX
# Load FP8 matrices (cast dynamically to float32/bfloat16 for math)
k_fp8 = tl.load(k_ptrs, mask=k_mask, other=0.0)
v_fp8 = tl.load(v_ptrs, mask=v_mask, other=0.0)
k = k_fp8.to(tl.float32)
v = v_fp8.to(tl.float32)
# S = Q * K^T * sm_scale
qk = tl.zeros([BLOCK_M, BLOCK_N], dtype=tl.float32)
qk += tl.dot(q, k)
qk *= sm_scale
# Apply causal masking if within current diagonal block
if start_n + BLOCK_N > start_m * BLOCK_M:
mask = offs_m[:, None] >= offs_n[None, :]
qk = tl.where(mask, qk, float("-inf"))
# Numerically stable Online Softmax Update
m_ij = tl.maximum(m_i, tl.max(qk, 1))
p = tl.exp(qk – m_ij[:, None])
alpha = tl.exp(m_i – m_ij)
l_i = l_i * alpha + tl.sum(p, 1)
# Rescale existing accumulator
acc = acc * alpha[:, None]
# Update accumulator with new chunk: acc += P * V
acc += tl.dot(p.to(tl.float32), v)
m_i = m_ij
# Final normalization
acc = acc / l_i[:, None]
# Output offsets and write back to HBM
offs_out = (offs_hz * stride_oh) + (offs_m[:, None] * stride_om) + (offs_d[None, :] * stride_ok)
tl.store(Out + offs_out, acc.to(tl.bfloat16), mask=q_mask)
This Triton kernel implements three critical optimizations tailored for extreme sequence handling:
- SRAM Preservation via Block Partitioning: By constraining tile dimensions (e.g.,
BLOCK_M=64,BLOCK_N=64), register pressure within the SM remains under control, maintaining 66% occupancy on compute units while avoiding local memory spills to off-chip DRAM. - Online Numeric Correction: Softmax normalizers ($m_i$ and $l_i$) update dynamically per block via safe multiplication ($alpha = e^{m_{text{old}} – m_{text{new}}}$). This prevents underflow when calculating attention distributions across $10^7$ values.
- Native FP8 Dequantization on Ingestion: By retaining the raw global KV cache as FP8 and casting to FP32 only when loads enter register registers, memory bus utilization is reduced by 50% relative to standard 16-bit ingestion.
4. Networking Infrastructure: Ultra Ethernet Consortium (UEC) RDMA & RoCEv2
Transferring a single request’s KV cache states at 10M tokens means moving between 80 GB and 320 GB of data between Prefill instances and the serving Decode tier. If network transit stalls, Decode units idle, destroying total token output efficiency. Traditional Infiniband networks, while providing low latency, are proprietary, expensive, and historically difficult to scale across global multi-tenant cloud fabrics.
Consequently, infrastructure engineers are shifting towards standard-based, high-radix Ethernet fabrics governed by specifications from the Ultra Ethernet Consortium (UEC), replacing classical RoCEv2 (RDMA over Converged Ethernet) with modernized transport layers (such as the UEC Transport Specification – UET).
Why Classical RoCEv2 Degrades at 10M-Token Scale
RoCEv2 relies fundamentally on Priority Flow Control (PFC) to deliver a lossless fabric. At 10M token ingestion, multi-gigabyte bursts of KV memory trigger severe Incast congestion at egress top-of-rack (ToR) switches. PFC link-pause frames ripple backward across switches (PFC Deadlock), halting unrelated traffic flows across the cluster. Furthermore, Equal-Cost Multi-Pathing (ECMP) hashes flows based on packet headers, funneling massive single-stream KV transfers into individual paths and creating persistent hash collisions and link hot-spots.
The UEC Transport layer solves these constraints through three architectural primitives:
- Packet Spraying across Multiple Paths: Instead of binding an entire RDMA queue pair (QP) to a static hashed path, the UEC transport layer distributes individual packets across all available physical paths between leaf-spine networks. This balances utilization across 800Gbps links and eliminates head-of-line blocking.
- Out-of-Order Execution via Packet Trimming: If a downstream switch experiences buffer saturation, packets are not discarded blindly. Switches trim payload data while preserving protocol metadata, returning a congestion notification back to the sender. The receiving Network Interface Card (NIC) handles packet reordering directly in hardware, shielding the application runtime.
- Fine-Grained Congestion Notification and Dynamic Credit Allocation: Replaces slow, reactive Explicit Congestion Notification (ECN) loops with rate pacing driven by measured round-trip time (RTT) telemetry, preventing queues from accumulating before packet loss occurs.
5. Systems Integration: The 2026 Autonomous AI Agent Paradigm
The infrastructural drive to maintain 10M tokens concurrently is not merely an exercise in benchmarking—it is a foundational requirement for autonomous ai agents 2026. Historical multi-agent implementations suffered from context drift, compounding hallucinations, and fragmented long-term planning, primarily caused by brittle vector search (RAG) and arbitrary context-truncation windows.
When serving ultra-long context windows natively, multi-agent frameworks shift from stateless message brokers to deeply grounded, persistent execution environments. Systems no longer rely on localized embeddings to guess past actions. Instead, agents preserve their complete operational history: compiled binary traces, execution environment states, visual logs, and tool interaction outputs remain readily accessible in the KV cache across days of execution.
Recent evaluations of multi-agent orchestration frameworks demonstrate this performance gap. As documented in benchmarks on autonomous execution architectures, multi-agent pipelines lose coordination accuracy when context states are aggressively pruned between tool-use loops. By pairing full-context persistence with structured tool-grounding platforms—such as the enterprise agent standards detailed in the Google AI Enterprise Multimodal Agent Documentation—autonomous workflows can parse and reference entire software codebases without losing track of downstream dependencies.
Operating agents over a continuous 10M-token canvas introduces specific runtime challenges that require active mitigations:
- Attention Sink Saturation: Models often place disproportionately high attention scores on the initial $4 text{ to } 32$ tokens of a context window regardless of their semantic content. If these tokens are accidentally dropped during paging, decoding fidelity collapses. Production systems designate these as non-evictable “Streaming Attention Sinks.”
- Cumulative Numerical Drift in FP8 Softmax: When processing 10 million tokens under FP8 quantization, rounding artifacts can cause the softmax denominator $sum e^{S_{ij}}$ to accumulate significant drift. Systems must run intermediate rescale steps using 32-bit accumulators within the Triton execution blocks to prevent token generation from degrading into non-terminating loops.
- KV-Cache Fragmentation and Thrashing: Disaggregated memory fabrics face high memory thrashing when many agent processes allocate dynamically sized context blocks simultaneously. Systems must use strict page allocation layers (e.g., Virtual Block Allocation across 2MB page boundaries) to keep physical memory contiguity transparent to GPU drivers.
6. Production Engineering Blueprint: The Reference Architecture
Deploying a 10M-token inference architecture requires aligning the orchestration system, memory fabrics, and execution kernels into a unified software-defined pipeline. The reference deployment below details the full configuration for an enterprise production cluster capable of sustaining 1,000 concurrent 10M-token contexts.
Enterprise 10M-Token Reference Stack
Cluster Orchestration & Scheduling Tier
Kubernetes cluster equipped with custom scheduling plugins; separates ingress pods into isolated prefill-pool and decode-pool nodegroups. The scheduler dynamically balances incoming prompts across prefill instances based on GPU memory fragmentation, continuous sequence lengths, and NVLink domain affinity.
Prefill Hardware & Execution Tier
Clusters of 8x NVIDIA HGX H100/H200 nodes interconnected via 3.2 Tbps NVLink/NVSwitch. Execution is managed by custom Triton chunked kernels configured to BLOCK_M=64, BLOCK_N=64, consuming FP8 inputs and outputting compressed cache allocations directly to system memory without local intermediate materialization.
Unified Memory Fabric (RDMA Subsystem)
High-radix switch fabric built with Ultra Ethernet Consortium-compliant interfaces operating at 800GbE per port. PagedAttention cache handles are transferred via one-sided non-blocking RDMA Read/Write operations managed through custom Unified Communication X (UCX) endpoints, completely bypassing the host CPU.
Decode Engine Tier
High continuous-batching decode workers running a specialized vLLM/TensorRT-LLM core. These nodes bypass prompt processing entirely, dynamically mapping remote KV pointers from the storage fabric into local threadblocks for auto-regressive generation, sustaining steady-state Inter-Token Latencies below $25text{ ms}$.
Operating production LLMs at the 10-million-token horizon demands a fundamental shift in infrastructure design. Moving beyond monolithic GPU configurations, this operational regime requires treated inference as an end-to-end distributed systems challenge: decoupling prefill from decoding, optimizing attention routines with custom low-level Triton kernels, and moving memory blocks across reliable Ultra Ethernet RDMA backbones. As models continue to expand, mastering these disaggregated memory and networking patterns becomes the defining differentiator in running ultra-long context windows stably, sustainably, and at production scale.

