DeepSeek-V4-Flash-MXFP4-GGUF

⚠️ WORK IN PROGRESS: these GGUF files are newly converted and still under validation.

This repo packages the official DeepSeek-V4-Flash weights as GGUF. The goal is plain: keep the file as close as practical to the official checkpoint, so people can test DeepSeek V4 Flash locally without first turning the whole model into a lower-precision quant.

Files

file	size	notes
DeepSeek-V4-Flash-MXFP4-FP8-BF16.gguf	156.1 GB	Source-dtype-faithful file. Keeps official routed experts as MXFP4, official dense FP8 as FP8, official BF16 as BF16, and official F32 as F32.
DeepSeek-V4-Flash-MXFP4-F16-Q8Compat.gguf	158.1 GB	Compatibility file tested with a V4-capable llama.cpp fork. Dense FP8 is decoded for that runtime path.

Precision And Compatibility

Both files are converted directly from the official DeepSeek checkpoint.

• Routed experts: MXFP4. The official checkpoint stores routed expert matrices in packed FP4 with one E8M0 scale per 32 values. This conversion rearranges that packed data into GGUF MXFP4 blocks without re-quantizing it.
• Dense FP8 weights: native FP8. The FP8-BF16 file keeps the official FP8 payload bytes and E8M0 scale bytes in a custom row-local GGUF tensor type. This preserves the official FP8 values without decoding them to F16, Q8_0, or another lower-precision format.
• BF16/F32 tensors: source dtype. The FP8-BF16 file stores official BF16 tensors as GGUF BF16 and official F32 tensors as GGUF F32. This is slightly more compact than widening BF16 tensors to F32, while still preserving the original checkpoint values.

The Q8Compat file is a companion for V4-capable llama.cpp forks that expect some dense paths in older GGUF tensor types. It stores the same MXFP4 routed experts, but decodes dense FP8 tensors to F16 and stores the attention-output and shared-expert tensors as Q8_0 for that compatibility path. BF16-source tensors use F16/F32 compatibility storage because the tested llama.cpp fork does not currently run the source-BF16-preserving layout.

The FP8-BF16 file requires a runtime that implements the custom dense-FP8 GGUF tensor type and DeepSeek V4 architecture support. In practice today, use a V4-capable llama.cpp/ds4 fork rather than mainline llama.cpp.

Runtime Compatibility

The Q8Compat file has been smoke-tested with antirez/llama.cpp-deepseek-v4-flash at commit 2f2d44052b7d. It passed short factual, JSON, and tool-call-shaped JSON smoke tests, plus a small Python code-generation smoke. On Apple Silicon, use --no-repack to avoid an extra full-model repack allocation:

llama-cli \
  -m DeepSeek-V4-Flash-MXFP4-F16-Q8Compat.gguf \
  -p "The capital of France is" \
  -n 8 -c 32768 -ngl 0 \
  --no-repack --single-turn -r "<|im_end|>"

The structured smoke tests used the same runtime path and produced parseable JSON for a simple record and a tool-call-shaped object. This file still requires DeepSeek V4 architecture support and GGUF MXFP4 support; it is not intended for mainline llama.cpp builds that do not have V4 support.