Instructions to use FLAIR-Gatech/Qwen3-4B-MDraft-b16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FLAIR-Gatech/Qwen3-4B-MDraft-b16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="FLAIR-Gatech/Qwen3-4B-MDraft-b16", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("FLAIR-Gatech/Qwen3-4B-MDraft-b16", trust_remote_code=True)
model = AutoModel.from_pretrained("FLAIR-Gatech/Qwen3-4B-MDraft-b16", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FLAIR-Gatech/Qwen3-4B-MDraft-b16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FLAIR-Gatech/Qwen3-4B-MDraft-b16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FLAIR-Gatech/Qwen3-4B-MDraft-b16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/FLAIR-Gatech/Qwen3-4B-MDraft-b16

SGLang

How to use FLAIR-Gatech/Qwen3-4B-MDraft-b16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FLAIR-Gatech/Qwen3-4B-MDraft-b16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FLAIR-Gatech/Qwen3-4B-MDraft-b16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FLAIR-Gatech/Qwen3-4B-MDraft-b16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FLAIR-Gatech/Qwen3-4B-MDraft-b16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use FLAIR-Gatech/Qwen3-4B-MDraft-b16 with Docker Model Runner:
```
docker model run hf.co/FLAIR-Gatech/Qwen3-4B-MDraft-b16
```

Qwen3-4B-MDraft-b16

MDraft is a diffusion drafter for speculative decoding with Qwen/Qwen3-4B. It is fine-tuned from z-lab/Qwen3-4B-DFlash-b16 using Multi-Draft On-Policy Distillation (MDraft). The architecture and inference procedure are unchanged from DFlash: replace the DFlash draft checkpoint with this checkpoint in a compatible inference setup.

This repository contains the drafter, which requires the separate Qwen3-4B target model and its tokenizer. The 4B in the name identifies the target model size; the drafter itself has 537,427,200 parameters.

Paper: Multi-Draft On-Policy Distillation for Diffusion Drafters
Authors: Petr Molodyk, Yuchen Zhu, Jaemoo Choi, Molei Tao, and Yongxin Chen, Georgia Institute of Technology

Model details

Property	Value
Repository	`FLAIR-Gatech/Qwen3-4B-MDraft-b16`
Required target and tokenizer	`Qwen/Qwen3-4B`
Initialization	`z-lab/Qwen3-4B-DFlash-b16`
Architecture	DFlash parallel block diffusion drafter; five layers
Block layout	16 slots: one committed anchor and 15 drafted positions
Weight precision	FP32; example inference uses BF16

Method

A parallel drafter predicts several positions without conditioning on earlier tokens sampled within the same block. The autoregressive verifier does condition on those tokens, so different sampled prefixes can provide conflicting training targets for the same drafter prediction.

MDraft samples multiple draft blocks from one context, averages the verifier's full-vocabulary conditional distributions across their prefixes, and blends that average with each prefix-specific distribution in log-probability space. It trains the drafter against the resulting targets using reverse KL. Multiple drafts are used during training; MDraft adds no multi-draft requirement at inference.

Usage

MDraft checkpoints are compatible with DFlash pipelines. The Transformers inference example follows the upstream DFlash model card, with only the draft checkpoint ID changed. Both models run on one CUDA GPU.

Installation

pip install transformers==4.57.3 torch==2.9.1 accelerate

The upstream utils.py also imports datasets; if it is not already installed, run pip install datasets==5.0.1.

Inference

from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer

# 1. Load the DFlash Draft Model
# Note: trust_remote_code=True is required for DFlash. We recommend run on one GPU currently.
model = AutoModel.from_pretrained(
    "FLAIR-Gatech/Qwen3-4B-MDraft-b16", 
    trust_remote_code=True, 
    dtype="auto", 
    device_map="cuda:0"
).eval()

# 2. Load the Target Model
target = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B", 
    dtype="auto", 
    device_map="cuda:0"
).eval()

# 3. Load Tokenizer and Prepare Input
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")
prompt = "How many positive whole-number divisors does 196 have?"
messages = [
    {"role": "user", "content": prompt}
]
# Note: this draft model is used for thinking mode disabled
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# 4. Run Speculative Decoding
# The 'spec_generate' function is a custom method provided by the DFlash model
generate_ids = model.spec_generate(
    input_ids=model_inputs["input_ids"], 
    max_new_tokens=2048, 
    temperature=0.0, 
    target=target, 
    stop_token_ids=[tokenizer.eos_token_id]
)

print(tokenizer.decode(generate_ids[0], skip_special_tokens=True))

To load a local checkpoint, replace FLAIR-Gatech/Qwen3-4B-MDraft-b16 in the first from_pretrained call with the absolute path of its model directory.

trust_remote_code=True loads the DFlash architecture included with the checkpoint. The drafter requires the corresponding Qwen target model.

For an existing DFlash deployment in vLLM or SGLang, use this repository as the draft model and keep Qwen/Qwen3-4B as the target model. Use a backend version with DFlash support.

Training

The paper's Qwen3-4B configuration uses eight proposals per context (K=8), blending coefficient alpha=0.9, and survival-based position weights.

Post-training data

The post-training corpus combines math, code, and chat prompts from Nemotron-Post-Training-Dataset-v2, with completions generated by Qwen3-4B. Samples are prompt-completion pairs, not unique questions; multiple samples can share the same prompt.

Data type	Prepared training samples	Training share
Math	410,448	61.52%
Code	128,372	19.24%
Chat	128,372	19.24%
Total	667,192	100%

These counts describe the prepared mix before block-length filtering, which leaves 665,156 usable pairs.

Evaluation

The tables report results from the MDraft paper. Values are mean acceptance lengths, including one correction/bonus token per verifier pass. Mean is the unweighted average across the six benchmarks; higher is better. Bold marks the highest value within each temperature and column.

Temperature 0

Method	GSM8K	MATH500	AIME25	MBPP	HumanEval	MT-Bench	Mean
DFlash	6.469	8.023	7.379	6.023	6.581	4.119	6.432
AdaFlash	6.793	8.339	7.709	6.141	6.713	4.329	6.671
Draft-OPD	6.604	7.844	7.113	5.882	6.438	4.198	6.347
RKL	6.688	7.899	7.108	5.940	6.513	4.247	6.399
MDraft	6.907	8.322	7.640	6.187	6.740	4.317	6.685

Temperature 0.6

Method	GSM8K	MATH500	AIME25	MBPP	HumanEval	MT-Bench	Mean
DFlash	6.168	7.365	6.044	5.599	6.214	3.964	5.892
AdaFlash	6.487	7.747	6.413	5.792	6.469	4.101	6.168
Draft-OPD	6.397	7.446	6.448	5.616	6.273	4.065	6.041
RKL	6.518	7.564	6.385	5.711	6.384	4.109	6.112
MDraft	6.739	7.905	6.480	6.001	6.593	4.205	6.321

Temperature 1

Method	GSM8K	MATH500	AIME25	MBPP	HumanEval	MT-Bench	Mean
DFlash	5.655	6.489	4.953	5.145	5.684	3.720	5.274
AdaFlash	5.988	6.914	5.264	5.285	5.980	3.843	5.546
Draft-OPD	6.067	6.854	5.214	5.343	5.916	3.919	5.552
RKL	6.300	7.041	5.374	5.541	6.039	4.062	5.726
MDraft	6.424	7.258	5.649	5.672	6.330	4.087	5.903

Compared with DFlash, MDraft increases the overall mean acceptance length by 3.9%, 7.3%, and 11.9% at temperatures 0, 0.6, and 1, respectively.

Citation

@misc{molodyk2026mdraft,
  title = {Multi-Draft On-Policy Distillation for Diffusion Drafters},
  author = {Molodyk, Petr and Zhu, Yuchen and Choi, Jaemoo and Tao, Molei and Chen, Yongxin},
  year = {2026}
}