Instructions to use deepseek-ai/DeepSeek-V4-Flash-0731 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-V4-Flash-0731")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731", device_map="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V4-Flash-0731"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash-0731

SGLang

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V4-Flash-0731" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V4-Flash-0731" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V4-Flash-0731 with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash-0731
```

Community implementation: native DeepSeek-V4-Flash-0731 inference in pure C on CPU

#66

by shyringo - opened 25 days ago

Discussion

shyringo

25 days ago

I maintain an Apache-2.0 community CPU runtime for DeepSeek-V4-Flash-0731:

https://github.com/shyringo/deepseek-v4-flash-0731-in-c

It runs the native checkpoint directly with C99 and OpenMP, without a GPU, CUDA, PyTorch, or weight conversion. Cold MoE experts stream from NVMe while active experts, hot weights, caches, and working state use RAM.

The full checkpoint is about 166.9 GB on disk. The 8 GB figure is the minimum runnable system-RAM plan, not a claim that all weights fit in memory. The reference measurements used an 18 GiB inference plan on Windows 11 + WSL2 with an Intel Core i5-1340P, 31.65 GiB installed RAM, and a Samsung NVMe SSD:

Ordinary open-ended generation: 20.203 s TTFT and 1.705 s/token TPOT (0.59 token/s).
Best measured prompt-lookup case with a reusable continuation: 26.748 s TTFT and 0.892 s/token TPOT (1.12 token/s). Every displayed speculative token was verified by the full target model.

For correctness, a four-layer tiny model runs 130 positions and must match an independent Python implementation with maxdiff=0.000000. The full checkpoint also has a fixed 16-token oracle. The repository includes reproducible benchmark records, CI, implementation provenance, and exact quick-start commands.

The project builds on and credits the expert-streaming approach demonstrated by kimi-k3-in-c; reused code, model adaptations, and project-specific optimizations are documented separately in NOTICE and the provenance documentation.

Would this be useful to link under "How to Run Locally" or a community implementations section? Compatibility reports from other CPUs, RAM sizes, and storage devices would also be very welcome.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment