DeepSeek V4: 1M Token Context, Hybrid Attention, and What Actually Matters

DeepSeek V4 has officially landed, and it’s not just another parameter bump. The new release brings two Mixture-of-Experts (MoE) models, a claimed 1 million token context window, and a hybrid attention architecture designed to make ultra-long context actually usable on high-end consumer hardware.

Two New DeepSeek V4 Models: Flash and Pro

DeepSeek V4 comes in two main variants, both using a Mixture-of-Experts design. In simple terms, MoE models have a very large total parameter count, but only a subset of those parameters are “active” for any given token. That lets you pack more knowledge into the model without paying the full compute cost every time.

DeepSeek V4 Flash is the lighter option. It has around 284 billion total parameters with 13 billion active per token. You still need serious VRAM or system memory to run it locally, but with only 13B active parameters, it should be relatively fast while still offering strong reasoning and language capabilities.

DeepSeek V4 Pro is the heavyweight. It weighs in at about 1.6 trillion parameters with 49 billion active per token. This is the version aimed at maximum capability and benchmark performance, assuming you have the hardware to match.

Both models are positioned as strong tool-calling and reasoning models, with explicit support for different “thinking modes.”

Reasoning Modes: No-Think, Medium, and Max

DeepSeek V4 is framed as a reasoning-focused model. Instead of a single fixed behavior, it exposes three reasoning modes:

No-think is for fast, lightweight answers where you don’t need deep chains of thought. This is ideal for simple queries or when latency and token usage matter more than nuance.

Medium thinking is a middle ground, giving more structured reasoning without going all-in on long deliberation.

Max thinking is for complex, multi-step problems where you want the model to really dig in. Across benchmarks, the max mode generally scores the highest, as you’d expect from a deeper reasoning pass.

Benchmarks in the technical report are run at “pass@1” (one run per test), which keeps costs down but can make results noisier. Even so, the jump from no-think to max thinking is clearly visible across most tasks, suggesting that the reasoning modes are doing real work rather than being a cosmetic toggle.

How DeepSeek V4 Stacks Up to Other Models

On paper, DeepSeek V4 Pro with max reasoning competes with top-tier proprietary models like Claude Opus, GPT-4.6/4.5, and other Chinese frontier models such as GLM and Kimi. In the reported benchmarks, Pro Max only clearly wins on a handful of benchmarks and is competitive—but not dominant—on others.

This lines up with early impressions: V4 is a solid step up from earlier DeepSeek versions, but it’s not a “Claude Mythos is over” moment. If you want a deeper dive into how V4’s specs and design compare to other open models, check out this breakdown of the 1.6T-parameter Pro and Flash models.

The more important story isn’t that V4 instantly beats every premium cloud model. It’s that it introduces a new attention architecture that could reshape what’s possible for local, long-context models.

The Real Breakthrough: Hybrid Attention and 1M Token Context

The headline feature of DeepSeek V4 is the claimed 1 million token context window. This isn’t the first time we’ve seen million-token claims, but the key difference here is how DeepSeek approaches the memory problem that usually makes such windows impractical.

Under the hood, the model uses a hybrid attention architecture built around two techniques:

Compressed Sparse Attention (CSA) selectively keeps more detailed information for the most relevant parts of the context while compressing or sparsifying less important parts. This reduces how much key-value (KV) cache memory is needed as the sequence grows.

Heavily Compressed Attention (HCA) goes even further, aggressively compressing long-range context so the model can still “remember” it without storing everything at full resolution.

Why does this matter? In standard transformers, KV cache memory grows linearly with sequence length. At very long contexts—hundreds of thousands of tokens or more—the KV cache becomes enormous, often requiring hundreds of gigabytes of VRAM or RAM. That’s why million-token contexts are usually more marketing than something you can actually run at home.

DeepSeek V4’s hybrid attention reportedly cuts KV cache usage by around 90% compared to DeepSeek V3.2 at the same sequence length. A graph in the technical report shows the old model’s KV cache usage climbing in a straight line as context grows, while V4’s curves stay 9–14x lower at the million-token mark.

In practice, that means a million-token context becomes plausible on a high-end 128 GB machine, instead of needing some exotic multi-GPU setup. It’s still not “everyone’s laptop,” but it’s a big step toward making truly long-context local models realistic.

It’s also important that this is done at the architecture level. Unlike external tricks like KV cache compression layers or runtime hacks (for example, approaches like TurboQuant that can be bolted onto many models), DeepSeek’s hybrid attention is built into the model itself. That makes it more robust and potentially more effective—and something other model families can adopt in future generations.

Why This Matters for RAG, Agents, and Real Work

The technical report doesn’t just focus on benchmarks; it also highlights real-world use cases where DeepSeek V4 is meant to shine.

RAG and Long-Context Search

Retrieval-Augmented Generation (RAG) is one of the biggest practical uses for LLMs today. The more context a model can handle, the more documents, notes, and search results you can feed it in a single shot—and the more grounded and specific its answers can be.

DeepSeek V4 Pro reportedly outperforms previous DeepSeek models on RAG-style search tasks. That makes sense: a reasoning-focused model with a huge, efficient context window is a natural fit for reading large document sets, knowledge bases, or codebases.

Agentic Search and Tool Use

The report also calls out agentic search—setups where the model uses tools like web search, CLI commands, or APIs to gather information and then reason over it. This is essentially RAG plus tools, and it depends heavily on reliable tool calling and planning.

DeepSeek V4 is positioned as strong at tool use, which is important if you’re building AI agents or workflows that chain together multiple steps. Combined with long context, that means an agent can keep more of its working memory “in mind” as it plans, calls tools, and refines answers.

White-Collar and Knowledge Work

One unusual and interesting part of the report is a section on white-collar tasks—things like finance, law, education, and general knowledge work. These are hard to benchmark with standard datasets because they’re open-ended and subjective.

To evaluate this, DeepSeek’s team built an internal suite of professional tasks based on real work done by Chinese knowledge workers across multiple industries. Outputs from different models were then graded in a blind study for content quality, structure, style, formatting, and nuance.

The exact judging setup isn’t fully detailed (for example, how many evaluators were involved), so you should treat the results with some caution. But it’s still a useful signal that the model is being tested on realistic, messy tasks that look more like actual jobs than like exam questions.

Coding Across the Stack

Coding is another major focus. The report describes an internal workload of around 200 tasks contributed by 50+ engineers, covering bug fixes, new features, PRDs, diagnostics, and more. Importantly, the tasks include harder languages like CUDA, Rust, and C++, not just Python.

Claude Opus still performs very well in these tests, as expected, but DeepSeek V4 Pro with max reasoning is competitive and shows promising results. For a more critical look at how V4 behaves in hands-on coding scenarios, you can read this preview of DeepSeek V4’s real-world coding performance.

What to Expect Next

Right now, there’s a gap between the model’s capabilities and ecosystem support. Popular local tooling like vLLM, llama.cpp, SG Lang, LM Studio, and oobabooga don’t yet support DeepSeek V4’s hybrid attention out of the box. Until they do, running V4 locally will be more experimental and limited to custom setups.

Over time, though, the hybrid attention ideas behind V4 are likely to spread. If other model families adopt similar techniques—and if they’re combined with quantization advances like 1-bit models or runtime KV compression—we could see a new generation of local LLMs that handle hundreds of thousands to a million tokens on high-end consumer hardware.

DeepSeek V4 itself is a strong, iterative model upgrade. But the real story is the architecture: a practical path toward ultra-long context that doesn’t require datacenter-scale hardware. For anyone building local RAG systems, AI agents, or serious knowledge-work tools, that’s a development worth watching closely.