DeepSeek published the inference engine that serves its own API on October 20, under Apache 2.0. It runs one model on two kinds of hardware, and three of the systems that make it fast stayed private inside the company.
What DeepSeek put on GitHub
The repository is deepseek-ai/DeepInfer, and it's the framework behind DeepSeek's production traffic rather than a research prototype. The license is Apache 2.0, which means companies can use it commercially without asking.
It supports exactly one model: DeepSeek-V4.1-Flash. The default checkpoint is the 1017 build released the weekend before.
That's a narrow target by design. Serving code gets fast by assuming things about the model it serves, and every assumption that gets baked in makes the code less general.
For anyone who rents GPUs by the hour, the release answers a question that used to require guesswork. This is how a company serving its own API at scale squeezes throughput out of hardware it owns.
What got tuned inside DeepInfer
The kernel work is the most portable part. DeepInfer ships FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and a set of custom TileLang kernels, several of which DeepSeek had already released on their own over the past year.
Parallelism covers attention and experts separately. Attention uses data parallelism with balanced context parallelism. The mixture-of-experts layers, where each token routes to a few specialist sub-networks, use expert parallelism with EPLB, a balancer that shifts experts between GPUs when the load drifts.
DeepSeek also added asynchronous scheduling, structured output through regex, JSON schema, and EBNF grammars, plus deterministic sampling and batch-invariant inference. Those last two work only on NVIDIA hardware.
Batch-invariant inference is the quiet one. It means the same prompt returns the same numbers regardless of how requests get grouped into a batch, which matters if you're debugging a model rather than serving a chat window.
DSpark and the cost of checking a guess
Speculative decoding is the trick where a small draft model guesses several tokens ahead and the big model checks them. The usual problem is that checking costs almost as much as generating.
DSpark, DeepSeek's version, has the draft model propose five tokens at a time. A confidence head predicts which of those the main model will accept, and only the likeliest candidates get verified. DeepSeek says that cuts the amount checked per step by about 30%, which it estimates as a 4% to 7% throughput gain under production load.
Four to seven percent sounds small until you price it. At the scale DeepSeek operates, a few points of throughput is a data center's worth of GPUs.
The company hasn't published independent measurements, and these numbers are its own.
Sliding-window replay and the prefill bill
The second speed trick targets prefill, the stage where the model reads the prompt before it writes anything. Long prompts make prefill expensive, and in agent workloads the prompt grows with every tool call, so the bill climbs as the session runs.
DeepInfer's bounded replay pushes only the last 128 to 140 tokens of the prompt back through the decoder. DeepSeek says that nearly halves prefill compute.
Nearly half is a big claim, and the mechanism explains why it's plausible. Most of a long prompt gets reused from cache; only the tail needs recomputation when the window slides. Whether the approximation holds for your workload depends on what your prompts look like.
The three pieces DeepSeek kept
Not everything shipped. The KV cache system stayed in-house, as did prefill and decode running as separate deployments, and the load balancer that spreads work across replicas.
DeepSeek says those parts are tightly coupled to its internal infrastructure. A company blog post on Zhihu explains how they work without releasing code.
So who can actually run it well? DeepSeek's own guidance puts the minimum efficient deployment at 64 GPUs for prefill and 64 for decode. Below that scale, the missing pieces matter less because you weren't going to need cross-replica balancing anyway.
Blackwell and Ascend, with no path back to Hopper
The hardware list is short and forward-looking. DeepInfer runs on NVIDIA SM100 chips, the Blackwell generation, and on Huawei's Ascend 950 NPU. Hopper and older NVIDIA cards aren't supported.
No fallback.
That's a deliberate cutoff. Kernel code written for Blackwell's memory system doesn't map cleanly onto Hopper, and supporting both would mean maintaining two engines.
Every published benchmark ran on NVIDIA: four SM100 chips for prefill, 64 for decode. There are no Ascend numbers, so the Huawei path is supported but unmeasured in public.
The serving math DeepSeek published
One cluster serves 155 trillion tokens a day, according to the deployment notes that ship with the repository. At off-peak pricing, that works out to roughly $38 of revenue per GPU-hour.
Those two figures together are the most useful thing in the release, because they let an operator sanity-check a build-out. If your cost per GPU-hour runs above $38, the configuration isn't paying for itself at DeepSeek's rates.
Where the code came from
DeepInfer started as a fork of vLLM, with ideas borrowed from SGLang and TensorRT-LLM. DeepSeek is upfront about the lineage, which is unusual for a company shipping a flagship system.
The release also includes deepseek-recipe, a small library that turns requests into token sequences using the same logic as the public API. That's the piece that makes a local deployment match hosted behavior, and mismatches there are a common source of confusing bugs.
DeepSeek says inference code for future models will land in the same repository.
Why the timing is awkward
The company declined to open-source its full engine in April 2025, citing tight coupling with cluster management tools and a small team with no appetite for maintaining a large open project. Components came out in pieces instead.
Since then it has moved closer to Huawei, porting TileLang to Ascend on September 30. Bloomberg reported in September that DeepSeek plans at least 160,000 Ascend 950DT chips for a new data center in Inner Mongolia.
One day before the release, the US Department of Homeland Security told federal civilian agencies to remove DeepSeek products, including third-party services built on its technology.
So the code is public in a way that matters to engineers, and restricted in a way that matters to procurement officers. Both things are true at once. If you run inference on your own hardware, the kernels and the recipe library are worth reading this week. If you were hoping for a full serving stack you could lift wholesale, three missing pieces will send you back to vLLM with a shorter list of ideas.






