Inference-Efficient LLMs: How Sparse Compute and Compressed Memory Work
Summary
Long-horizon agents repeatedly process enormous cached histories while generating comparatively little new text, making prefill compute, KV-cache storage, memory traffic, and persistent cache capacity first-class costs that conventional decoder-only scaling does not solve.
- • Traced the architecture path from DeepSeek-V3 and R1 through V3.1, V3.2, V4, and V4.1 rather than treating each release as an isolated model.
- • Separated total model capacity from activated compute: V4.1-Flash has a 552B backbone plus 196B of Engram memory while activating about 8B parameters during prefill and 16B during decode.
- • Examined how CED, CSA2, FP4 KV caching, bounded replay, Single-Pass mHC, Engram, and DSpark attack different terms of the inference-cost equation.
- • Distinguished architectural savings from provider pricing and from benchmark claims that still require independent reproduction.
- • 40-layer causal encoder-decoder split: 20 causal encoder layers plus 20 decoder layers
- • Compressed Sparse Attention 2 with cross-layer KV reuse, index reuse, and hierarchical sparse indexing
- • FP4 global KV cache at about 890 bytes per token and bounded reconstruction of short-lived SWA state
- • 196B-parameter sparsely accessed Engram conditional memory
- • Single-Pass manifold-constrained Hyper-Connections and DSpark five-token speculative drafting
- • Native multimodal pre-training with a DeepSeek-ViT vision encoder
- • The important V4.1 idea is asymmetry: input tokens and output tokens no longer need to pay the same computational path.
- • KV-cache compression is now a multi-dimensional systems problem spanning entry width, sequence length, layer reuse, numerical precision, and storage lifetime.
- • The low API price is an outcome of model-and-serving co-design; 8B active input parameters do not make V4.1-Flash an 8B model or a small self-hosting target.
Introduction
The expensive part of an AI agent is increasingly not the sentence it writes. It is everything the model has to read again before writing that sentence.
A coding agent may carry a repository map, conversation history, tool traces, compiler errors, retrieved files, screenshots, and previous patches through dozens of turns. A research agent may keep hundreds of thousands of tokens alive while each individual step generates only a few hundred or a few thousand new tokens. Once the context becomes large enough, prefill compute, KV-cache storage, memory bandwidth, and cache persistence become as important as raw decoding speed.
That is the problem DeepSeek-V4.1-Flash is designed around.
The headline numbers are unusual: a 552B-parameter Mixture-of-Experts backbone, another 196B parameters of sparsely accessed Engram memory, a one-million-token context window, but only about 8B activated parameters per token during prefill and 16B during decoding. Its global KV cache is reported at about 890 bytes per token, roughly one quarter of the previous V4-Flash footprint, while the persistent cache that has to survive in SSD or host memory falls to roughly one eighth.
Those numbers sound like one clever compression trick. They are not.
V4.1-Flash is better understood as a stack of orthogonal optimizations. Causal Encoder-Decoder (CED) changes how much of the network a prompt must traverse. Compressed Sparse Attention 2 (CSA2) reuses attention state across layers. FP4 KV quantization shrinks each stored entry. SWA Bounded Replay stops treating every short-range state as something worth storing permanently. Single-Pass mHC attacks residual-stream memory traffic. Engram moves a large amount of memorization into deterministic sparse lookup. DSpark reduces the serial cost of autoregressive generation.
The interesting part is not that DeepSeek made another cheaper model. It is that the architecture attacks nearly every major term in the long-context serving bill at once.
The Architecture Lineage
V4.1 makes more sense when read as the next step in a sequence of DeepSeek design decisions rather than as an isolated release.
There is an important distinction in that table: R1 was primarily a post-training breakthrough, not a new inference backbone. DeepSeek-R1-Zero and R1 were built on DeepSeek-V3-Base. GRPO made reinforcement learning cheaper by avoiding a separate critic model and demonstrated that strong reasoning behavior could emerge from large-scale RL. It changed how capability was elicited from the model much more than it changed the architecture used to serve each token.
V3.2 and V4 moved the efficiency story back into the backbone. V3.2 introduced fine-grained sparse attention. V4 then compressed long-context state more aggressively with Compressed Sparse Attention and Heavily Compressed Attention, added manifold-constrained Hyper-Connections, and adopted Muon for much of the optimization stack.
V4.1 is where those ideas become a serving architecture.
1. The Cost Problem Is No Longer Just FLOPs
The classic transformer efficiency discussion usually starts with attention complexity. Full attention over a sequence of length (N) has an (O(N^2)) interaction pattern, so sparse attention is an obvious place to optimize.
For production agents, that is only part of the bill.
A long-running agent looks more like this:
The system repeatedly pays across several dimensions:
Optimizing only the feed-forward FLOPs leaves the memory problem intact. Optimizing only sparse attention leaves duplicate KV state across layers intact. Quantizing weights does not automatically shrink a large KV cache. Shrinking the KV cache does not make a cache miss cheap to recompute.
V4.1-Flash is interesting because its components target different terms of that equation, so the savings can compound rather than overlap.
2. Causal Encoder-Decoder: Input and Output Take Different Paths
The most important architectural change is also the easiest to misread.
V4.1-Flash is still an autoregressive text generator. Its "encoder" is not a BERT-style bidirectional encoder. It is causal. The language backbone contains 40 causal Transformer layers, split into:
The purpose of this split is not sequence-to-sequence translation. It is to make prefill cheaper than decode.
In a conventional decoder-only transformer, prompt tokens pass through every layer. Each layer creates the state its own attention mechanism needs, so a million-token prefix effectively pays the full depth of the network before generation begins.
CED changes the global-attention path.
For upper decoder layers, the global KV entries are projected directly from the final encoder hidden state rather than being reconstructed from each decoder layer's own hidden states:
For a long prompt, most tokens therefore do not need a normal full pass through the upper half merely to construct global KV state.
DeepSeek gives the prefill complexity as approximately:
That is the source of the asymmetric activation number: roughly 8B active parameters per input token during prefill versus 16B during decode.
This is a more meaningful efficiency metric than saying "the model has 552B parameters." Total parameters describe stored capacity. Activated parameters describe how much of that capacity participates in a token's computation. CED goes one step further and makes activation depend on which phase of inference the token belongs to.
That matters for agents because their workloads are asymmetric too. They often ingest far more tokens than they generate.
3. Sliding-Window Attention Still Needs State — So V4.1 Replays It
CED can cheaply create the decoder's global KV state from encoder outputs, but local attention creates a complication.
V4.1 also uses sliding-window attention (SWA). Local keys and values are still layer-specific because they are derived from each layer's current hidden state. If the model tried to reproduce the complete decoder SWA history after a long prefill, much of the compute saved by CED would come back through the side door.
DeepSeek's answer is Decoder SWA Bounded Replay.
Instead of replaying a long effective history through every decoder layer, the model reconstructs SWA state only for the final local window:
For long contexts, the global path carries distant information while the local path only needs enough reconstructed state to resume nearby token interactions.
The important idea is broader than this implementation: not all model state deserves the same persistence guarantee.
Global context may be reused hours later. A tiny sliding window is valuable immediately and rapidly becomes irrelevant. Treating both as equally durable cache entries wastes storage on state with completely different reuse distributions.
V4.1 turns that observation into part of the architecture.
4. CSA2: Compressing KV Across the Layer Dimension
DeepSeek's path toward cheaper long context can be understood as progressively compressing KV cache along different axes.
Earlier methods already attacked two of them:
- Entry dimension — techniques such as GQA and DeepSeek's earlier Multi-head Latent Attention reduce how much information is stored for each position.
- Sequence dimension — V4's CSA/HCA compress groups of sequence positions and avoid attending densely to every original token.
V4.1 adds the third major axis:
- Layer dimension — do not let every layer maintain a fully independent long-range cache and independently rediscover the same sparse locations.
This is the purpose of Compressed Sparse Attention 2 (CSA2).
CSA2 has three operating modes:
All modes still compute the current layer's query and short-range SWA state. What changes is how much long-range machinery is duplicated.
Conceptually:
This is a different kind of sparsity from ordinary MoE routing. MoE asks, "which weights should this token use?" CSA2 asks, "which historical states and which previous routing decisions should this layer reuse?"
The two sparsities operate on different resources and stack cleanly.
5. Hierarchical Sparse Indexing: Sparse Attention Can Still Have an Expensive Search
Sparse attention is not automatically cheap.
If every query has to score every compressed position merely to decide which positions deserve attention, the model may replace expensive attention with an expensive indexing stage. At one million tokens, even a lightweight scorer becomes meaningful.
V4.1 uses a Hierarchical Sparse Indexer to bound this work.
An earlier Full-mode layer searches broadly and produces a candidate pool. The report gives an example of selecting 2,048 blocks with eight positions each, producing 16,384 candidates. Later Reindex layers search within that already narrowed pool instead of rescoring the entire context.
This turns "which old tokens matter?" into a hierarchy rather than a fresh global ranking at every layer.
The design is particularly important because V4.1 is optimizing a regime where the context can be enormous and the generated continuation comparatively small. The index itself has to scale like infrastructure, not like a benchmark prototype.
6. FP4 KV Cache: Quantization Moves From Weights Into Runtime State
Weight quantization gets most of the attention because model checkpoints are visibly huge. In long-context serving, runtime state can become the more immediate capacity constraint.
V4 already used low precision aggressively, including FP4 for parts of its attention indexing machinery and mixed FP4/FP8 model weights. V4.1 extends quantization to the main global KV cache.
DeepSeek applies quantization-aware training and stores the main KV using an FP4 representation with per-group scaling. Sliding-window KV remains at higher precision because it is more sensitive.
The reported global-cache footprint is approximately:
At one million tokens, simple arithmetic gives:
That is only the global KV figure before serving-system overhead, but it makes the scale tangible. DeepSeek reports it as roughly one quarter of the corresponding V4-Flash global KV footprint.
The key point is that this compression is orthogonal to CSA2:
Doing only one leaves the other dimensions untouched.
7. Persistent Cache Is a Storage-Policy Problem
A production KV cache does not live in one place.
HBM is fast and scarce. Host DRAM is larger and slower. SSD is much larger and much slower again. A long-running serving platform therefore needs a hierarchy much closer to an operating-system cache than to a single tensor allocation.
V4.1 explicitly separates global KV from SWA KV based on how likely each is to be reused.
DeepSeek reports the global state as having a long-tail reuse distribution extending to roughly 72 hours, while the useful reuse window for SWA state is measured in minutes.
That difference changes the optimal policy. Persisting both to SSD indefinitely would spend I/O and storage on local state that can be regenerated cheaply.
So V4.1 stops doing that.
The result is a reported persistent KV footprint of roughly one eighth of V4-Flash. Half of the improvement comes conceptually from no longer persisting SWA state, while the global cache itself is also substantially smaller.
This is an unusually systems-oriented architectural decision: the model is designed around the fact that HBM, DRAM, SSD, and recomputation have different prices.
8. Single-Pass mHC: Residual Connections Become a Memory-Bandwidth Problem
DeepSeek-V4 introduced manifold-constrained Hyper-Connections (mHC), an extension of the ordinary residual connection.
A normal transformer residual path is conceptually simple:
Hyper-Connections maintain multiple residual streams and learn how information should be mixed before and after sublayers. mHC constrains the mixing so signal propagation remains stable, using a doubly stochastic structure.
The modeling idea is powerful, but a sophisticated residual mechanism can create an implementation problem: more intermediate state means more activation reads and writes, and modern accelerators are often limited by memory traffic rather than arithmetic alone.
V4.1 introduces Single-Pass mHC. The design shifts one dependency between adjacent blocks so the residual mixing can be fused into a single kernel path instead of requiring repeated passes over the same activation state.
At a high level:
DeepSeek reports that this halves the activation-memory traffic associated with the original implementation.
This is easy to overlook because it does not change a benchmark-friendly model parameter count. But at scale, a layer that performs the same mathematical job with fewer HBM round trips can matter as much as a reduction in nominal FLOPs.
9. Engram: Make Memorization Sparse Too
MoE makes computation sparse: a model can own hundreds of billions of parameters while activating only a small subset for each token.
Engram applies a related idea to memory.
V4.1 adds 196B Engram parameters outside the 552B backbone. They are not another stack of transformer blocks that every token traverses. They are sparsely accessed conditional memory built around deterministic n-gram lookup.
The mechanism uses token n-grams, multiple hash functions, learned embeddings, and context-aware gates. V4.1 uses n-gram orders 2, 3, and 4 and inserts Engram modules at two points in the network.
Conceptually:
The architectural bet is that memorizing recurring lexical and local patterns does not always require spending transformer compute.
If an address is deterministic, the serving system also knows which memory rows will be needed before the model reaches the layer that consumes them. Those rows can be prefetched from slower memory or across RDMA while other computation is happening.
That makes Engram simultaneously a model idea and a storage-scheduling idea.
It also explains why parameter count must be read carefully. Saying V4.1-Flash has "8B active parameters" without mentioning the 552B backbone and 196B Engram store is misleading. The model has enormous stored capacity. It simply touches a small portion of that capacity on a given token.
Low active compute is not the same thing as a small checkpoint.
10. DSpark: Speculation Becomes Load-Aware
Autoregressive generation still has a serial dependency:
DeepSeek-V3 used Multi-Token Prediction (MTP) as an auxiliary training objective, encouraging the model to predict several future tokens. V4 retained that line of work.
V4.1 changes the arrangement. The backbone is trained without the old MTP module, and a separate DSpark speculative-decoding module is trained after backbone pre-training.
DSpark uses three small Transformer blocks and produces a five-token draft in one forward pass. The draft positions are generated semi-autoregressively, with lightweight dependencies between them, and a confidence head estimates how much of the draft is likely to survive verification.
The important systems detail is the scheduler. The ideal speculative block length depends not only on model confidence but also on current engine throughput. Verifying more draft tokens is useful when acceptance is high and verification is cheap; it can be counterproductive when the engine is saturated.
So speculative decoding is not treated as a fixed architectural constant. It is scheduled against serving conditions.
That is a recurring theme in V4.1: the boundary between "model architecture" and "inference engine" keeps getting thinner.
11. Native Multimodality Is Part of Pre-Training
V4.1-Flash also adds native visual understanding, but the interesting detail is how early multimodality enters the stack.
A DeepSeek-ViT vision encoder produces a grid of visual features. A 3 × 3 pixel-unshuffle operation reduces the spatial token count by a factor of nine, and an MLP projector maps the result into the language model's hidden dimension. Visual embeddings are then inserted into the same sequence as text embeddings.
The vision encoder uses 2D rotary positional encoding to handle variable image resolutions, and multimodal data is included from the start of language-model pre-training rather than added only as a late alignment stage.
DeepSeek also extends its auxiliary-loss-free MoE balancing with separate correction biases for image and text tokens. That matters because the two modalities can prefer different experts. Forcing them into one aggregate balancing signal could make routing look numerically balanced while one modality silently monopolizes particular experts.
This is a small detail compared with CED or CSA2, but it is a good example of the model's general philosophy: sparsity is useful only if the routing system understands the structure of the workload.
12. From V3 to V4.1: What Actually Changed
The DeepSeek line is often summarized as "MoE plus cheap training." That description is now too shallow.
V3: Sparse Weights and Compressed Heads
DeepSeek-V3 had 671B total parameters with 37B active per token. It combined DeepSeekMoE with Multi-head Latent Attention (MLA), which compressed the KV representation shared across attention heads, and used auxiliary-loss-free load balancing to keep expert utilization stable without paying a large quality penalty from balancing losses.
It also trained with Multi-Token Prediction.
The central idea was already visible: separate total capacity from per-token work.
R1: The Breakthrough Was the Training Loop
R1 used DeepSeek-V3-Base rather than introducing a fundamentally new serving topology.
R1-Zero applied large-scale reinforcement learning directly, while R1 added cold-start examples and a multi-stage SFT/RL pipeline. Group Relative Policy Optimization (GRPO) estimated advantages from groups of sampled responses rather than training a separate critic network of comparable size.
That made R1 important for reasoning, but its architectural inheritance matters: the famous reasoning model sat on the same broad V3 foundation.
V3.1: One Model, Two Inference Behaviors
V3.1 continued pre-training on another 840B tokens, extended the context to 128K, and unified thinking and non-thinking behavior in one model.
The key change here was not a radically different transformer block. It was consolidation: one checkpoint could serve a fast direct path or a deliberate reasoning path, with post-training increasingly optimized for tool use and agents.
V3.2: Do Not Read the Entire History
V3.2-Exp introduced DeepSeek Sparse Attention (DSA), a fine-grained sparse attention mechanism intended to reduce long-context training and inference cost while preserving behavior close to V3.1-Terminus.
This changes the long-context question from:
to:
That sparse-selection idea survives into the V4 family.
V4: Compress the Sequence
V4 pushed context to one million tokens and replaced the earlier attention arrangement with a hybrid of Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA).
CSA compressed KV along the sequence dimension and then applied sparse attention. HCA compressed more aggressively but used dense attention over the compressed representation. V4 also introduced mHC and adopted Muon across much of training.
DeepSeek reported that V4-Pro at one-million-token context required only 27% of the single-token inference FLOPs and 10% of the KV cache of V3.2.
V4.1: Compress the Layer, Precision, Lifetime, and Path
V4.1 removes V4's CSA/HCA hybrid in favor of CSA2 and adds new compression dimensions around it:
That is the architectural story of V4.1-Flash.
It is not one dramatic invention. It is multi-axis compression with a serving system designed around the resulting asymmetries.
13. Why the API Can Be So Cheap
Architecture and API price are not the same thing. A provider can subsidize a product, change margins, alter utilization targets, or reprice overnight.
Still, V4.1's current pricing makes more sense when viewed through the architecture.
As of September 20, 2026, DeepSeek lists these API rates per one million tokens:
The cached-input price is particularly revealing. At peak pricing, a cache hit costs 2% of an uncached input token.
That economics depends on making cached state cheap enough to retain, restore, and serve at high concurrency. V4.1 is therefore not merely reducing arithmetic. It is reducing the infrastructure cost of remembering that the arithmetic was already done.
DeepSeek currently lists a concurrency limit of 2,500 for deepseek-flash, versus 500 for V4-Pro. Pricing and quotas remain provider policy, but the direction is consistent with what the architecture is optimized to do: serve many long-running, input-heavy agent sessions without allowing KV state to consume the machine.
14. What "8B Active" Does Not Mean
The most likely misunderstanding around V4.1 is the same one that appears around large MoE models generally.
V4.1-Flash is not an 8B model.
The technical report describes:
while activating approximately:
Those are different quantities.
A dense 8B checkpoint can plausibly live on one high-memory consumer machine after quantization. V4.1 has hundreds of billions of stored parameters plus a massive conditional-memory table. The fact that only a small subset is touched for a token reduces compute, but the weights still have to exist somewhere and the serving system still has to route to them.
DeepSeek's own launch material discusses large-scale deployments in terms of thousands of GPUs plus a storage cluster. Community inference stacks are already working to make smaller deployments practical, but the open weights should not be confused with a lightweight local model.
The architectural achievement is not "552B becomes 8B."
It is:
552B of backbone capacity can expose an input path whose active computation resembles a much smaller model, while the rest of the system is organized so that retrieving the required capacity and context remains economical.
That is a much more interesting claim.
15. What the Architecture Does Not Prove
Several caveats are important.
One million tokens is a capacity limit, not a guarantee of perfect use. Sparse retrieval can select the wrong historical positions. The V4.1 report explicitly calls out selection errors in CSA2 as a robustness boundary that is not yet fully characterized.
Bounded replay is intentionally approximate. Reconstructing only recent SWA state saves enormous persistent-cache cost, but approximation creates edge cases around cache resumption. DeepSeek reports negligible degradation in its evaluations and trains the post-training stack to adapt to replay, but the mechanism is not mathematically identical to retaining all original state.
Cheap active compute does not mean cheap deployment everywhere. Engram, MoE expert weights, sparse indexing, distributed cache storage, custom kernels, and expert routing move complexity into the serving stack.
The strongest efficiency numbers are still primarily DeepSeek measurements. The weights and technical report are public, which makes independent work possible, but reproducing HBM footprint, SSD behavior, throughput, and end-to-end economics requires a mature inference implementation and comparable hardware.
API price is not an architectural constant. The current rates are evidence that DeepSeek can expose the model cheaply today, not a theorem that every V4.1 deployment costs the same amount.
DeepSeek itself notes that benchmark proximity to leading closed models does not imply parity on the hardest reasoning and edge-case workloads. That is the right framing. The architecture can be important even if the model is not uniformly the strongest model on every task.
16. The Full Serving Path
Putting the pieces together, a simplified V4.1 serving path looks like this:
And behind that model path is a cache hierarchy:
This is why describing V4.1 only as a new "LLM architecture" undersells it. The model graph, numerical formats, kernels, cache hierarchy, persistence policy, and speculative scheduler are designed together.
17. Developer Checklist
Before treating V4.1-Flash as a universal cheap replacement for another model, ask:
- Is the workload actually input-heavy, where CED and cache compression have the greatest leverage?
- What percentage of requests hit a reusable prefix cache rather than forcing fresh prefill?
- Does the application regularly approach hundreds of thousands or one million tokens, or is the long-context machinery mostly idle?
- How sensitive is the task to sparse-attention retrieval failures deep in the context?
- Does cache-resumption accuracy matter enough that bounded SWA reconstruction needs dedicated testing?
- Are current API prices being compared at the same cache-hit assumptions and the same reasoning-effort setting?
- If self-hosting, where will the 552B backbone and 196B Engram memory reside?
- Does the inference engine implement CSA2, Engram prefetching, Single-Pass mHC, FP4 KV, and DSpark efficiently, or does it merely load the checkpoint?
- Is native vision useful, and does the production stack preserve the multimodal routing behavior the model was trained with?
- Are latency measurements separating prefill, time-to-first-token, decode throughput, cache restore, and tool execution?
- Have quality tests included adversarial long-context cases rather than only short benchmark prompts?
- Are cost claims based on your traffic distribution, not a provider's ideal cache-hit workload?
The broader lesson is that frontier-model efficiency is no longer one scalar such as parameter count, FLOPs, or quantization bits.
DeepSeek's progression from V3 to V4.1 is increasingly a progression toward factorizing cost. MoE separates capacity from active weights. Sparse attention separates available history from attended history. CSA2 separates model depth from the number of independent long-range caches. CED separates the cost of reading from the cost of writing. Engram separates memorization from transformer computation. Bounded replay separates reconstructible state from persistent state. DSpark separates the number of generated tokens from the number of full serial decode steps.
V4.1-Flash is compelling because those separations line up with the workload modern agents actually create.
The model is large. The context is huge. The stored memory is enormous. But the amount of expensive work performed at each stage is deliberately much smaller than any of those headline capacities would suggest.
That is the architecture behind the low cost.
References
- DeepSeek V4.1 Flash launch: https://api-docs.deepseek.com/news/news260910/
- DeepSeek-V4.1-Flash technical report: https://arxiv.org/abs/2609.19969
- DeepSeek V4.1 Flash model and pricing documentation: https://api-docs.deepseek.com/quick_start/pricing/
- DeepSeek V4 technical report: https://arxiv.org/abs/2606.19348
- DeepSeek V3.2 release: https://www.deepseek.com/en/news/deepseek-v3-2/
- DeepSeek V3.2-Exp and DeepSeek Sparse Attention: https://api-docs.deepseek.com/news/news250929/
- DeepSeek V3.1 release: https://deepseek.com/en/news/deepseek-v3-1/
- DeepSeek-V3 technical report: https://arxiv.org/abs/2412.19437
- DeepSeek-R1 technical report: https://arxiv.org/abs/2501.12948
- DeepSeek-R1 peer-reviewed Nature paper: https://doi.org/10.1038/s41586-025-09422-z
- DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation: https://arxiv.org/abs/2607.05147
- Engram / Conditional Memory via Scalable Lookup: https://arxiv.org/abs/2601.07372