DeepSeek open-sources DeepInfer inference framework under Apache 2.0

DeepSeek on 20 October released DeepInfer, the inference framework it uses for its production services, as open-source software under the Apache 2.0 licence. The code is on GitHub.

DeepSeek said in the README that DeepInfer is "the high-performance inference framework that powers DeepSeek's large-scale production services." It targets DeepSeek-V4.1-Flash and supports Nvidia SM100 GPUs and Huawei Ascend 950 NPUs.

The repository lists five kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect. It also lists TileLang.

Features include attention data parallelism with balanced context parallelism, and mixture-of-experts expert parallelism with EPLB. The framework also has draft-token acceleration with confidence-scheduled verification, called DSpark, an asynchronous zero-overhead batch scheduler and a batch-invariant mode.

DeepInfer also supports constrained generation by regex, JSON schema or EBNF grammar, according to the repository.

DeepSeek claims the framework handles about 155 trillion tokens per day per cluster, with a 96% input cache hit rate. The figure comes from DeepSeek and has not been independently verified.

The README says the framework was "originally built primarily on vLLM", with ideas drawn from SGLang and TensorRT-LLM. The previous week, DeepSeek said it plans to pre-train on Huawei Ascend 950 chips by the end of 2026.

Reports differ on the repository's popularity, with star counts ranging from 5.5k to more than 8,000 within hours of release. One report said the launch came a day after a US Department of Homeland Security ban on DeepSeek in federal agencies. That claim is unconfirmed. DeepSeek has not said what it will release next.