Instructions to use deepseek-ai/DeepSeek-V4-Flash-0731 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-V4-Flash-0731")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731", device_map="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V4-Flash-0731"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash-0731

SGLang

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V4-Flash-0731" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V4-Flash-0731" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V4-Flash-0731 with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash-0731
```

Hermes-Agent and Silly Tavern speed Improvements by preventing context corruption

#62

by Testererer - opened 30 days ago

Discussion

Testererer

30 days ago

•

edited 30 days ago

Just wanted to leave a note here on how to prevent DeepSeek V4 to dump its context at every other prompt.

I'm running DeepSeek V4 Flash 0731 locally with ik_llama.cpp on an old server CPU.
It works fine actually. But it's slow enough to see what's happening in the background.

First I noticed in Silly Tavern that the context gets dumped and recreated at every message. Then I noticed that Hermes-Agent has the same problem, especially at tool calls.

I think it has to do with the aggressive context compression of DSV4 and the handling of it, not entirely sure.
You'll find my assumptions below, but first the fix:

Silly Tavern Fix:

Click on the large letter "A" in the top menu for the formatting.
Activate the "Instruct Template" (via power button).
Click on each fields "Story String Suffix", "User Message Suffix" and "Assistant Message Suffix" and press enter. That adds a newline (/n).

Done. Now the context won't get reloaded at every prompt. DeepSeek V2.5 templates work fine btw.

Hermes-Agent fix:

I basically told hermes to fix itself. ;-)
To change its code in a way that makes sure, that every prompt and tool call that gets sent to the LLM ends with a newline (/n) as suffix.
If it doesn't end on a newline, it's added. The change was about 3 lines and hermes did it by itself in a jiffy.

My assumption:

My knowledge is wonky, but here we go:
DSV4 has a new-ish type of context compression. But if existing context gets corrupted, the entire context has to be dumped and reloaded.
While that should not be a problem, it often is.

The problem:
Prompts by Silly Tavern and Hermes-Agent don't always end on a space or a newline. The next context of a prompt or a tool call might just gets stuck to the previous without newline and spacing.

But Tokens are not letters.
So if you stick a new word directly onto the last word of the previous context without clear spacing, the last few letters of the existing context and the new letters might create a different token. And just like that the existing context is corrupted, causing the entire context to get dumped and reloaded.
Especially tool calls in hermes seem to call cause this very frequently. That really starts to hurt once you're above 100k tokens of context.

Hope this helps someone and it isn't just me. ;-)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment