jeongseokoh/Qwen3-8B_SPEED-24-BoS

Qwen3-8B SPEED-24-BoS

This repository contains a SPEED LoRA adapter for Qwen/Qwen3-8B.

It is not a standalone full-weight checkpoint. At load time, Transformers/PEFT will need access to the base model Qwen/Qwen3-8B, then apply the adapter in this repository.

The repository also bundles the minimal Qwen3 SPEED remote-code runtime needed for inference:

• modeling_speed.py
• modeling_speed_qwen3.py
• speed_inference.py
• speed_decoder_modeling.py
• speed_modeling_common.py
• prompt/tokenizer helper files

You do not need the original SPEED source repository on the inference server, but you must load with trust_remote_code=True.

SPEED Configuration

This checkpoint was trained and exported with:

Setting	Value
Base model	Qwen/Qwen3-8B
Adapter type	LoRA
Lower SPEED layers	24
Prompt prefill mode	lower layers only
Upper prompt targets	bos,assistant
Prefill attention	causal
Decode tokens	full-depth

For faithful inference, use lower_k=24 and speed_upper_targets=("bos", "assistant").

Installation

Use a recent CUDA/PyTorch environment suitable for Qwen3-8B. The checkpoint was created with these library versions:

pip install "transformers==4.57.6" "peft==0.19.1" huggingface_hub accelerate safetensors

Install PyTorch separately if your server needs a specific CUDA wheel.

Basic SPEED Inference

import sys
import torch
from huggingface_hub import snapshot_download

model_id = "jeongseokoh/Qwen3-8B_SPEED-24-BoS"
repo_dir = snapshot_download(model_id)
sys.path.insert(0, repo_dir)

from speed_inference import load_speed_model

model, tokenizer = load_speed_model(
    repo_dir,
    dtype=torch.bfloat16,
    device_map="auto",
    speed_generate=True,
    speed_layers=24,
    speed_attn="causal",
    speed_upper_targets=("bos", "assistant"),
)
model.eval()

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]

with torch.inference_mode():
    outputs = model.generate(
        speed_generate=True,
        messages=messages,
        lower_k=24,
        speed_upper_targets=("bos", "assistant"),
        max_new_tokens=256,
        do_sample=True,
        temperature=0.6,
        top_p=0.95,
        top_k=20,
        return_dict_in_generate=True,
    )

prompt_len = outputs["prompt_lengths"][0]
generated_ids = outputs["sequences"][0, prompt_len:]
print(tokenizer.decode(generated_ids, skip_special_tokens=True))

Document or Long-Context Inference

The SPEED hook accepts chat-style messages and an optional context. The context span is processed as prompt context according to the SPEED prefill rule.

question = "What are the key claims in the document?"
document = "..."  # long document text

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": question},
]

with torch.inference_mode():
    outputs = model.generate(
        speed_generate=True,
        messages=messages,
        context=document,
        lower_k=24,
        speed_upper_targets=("bos", "assistant"),
        max_new_tokens=512,
        do_sample=False,
        return_dict_in_generate=True,
    )

prompt_len = outputs["prompt_lengths"][0]
print(tokenizer.decode(outputs["sequences"][0, prompt_len:], skip_special_tokens=True))

Returning Text Directly

For convenience, the bundled runtime can return decoded text directly. Keep return_dict_in_generate=True so the prompt length is handled correctly.

text = model.generate(
    speed_generate=True,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Give me a short answer."},
    ],
    lower_k=24,
    max_new_tokens=128,
    return_dict_in_generate=True,
    return_text=True,
)
print(text)

Important Notes

• This repository is a PEFT adapter repo. Do not pass SPEED-only arguments such as speed_generate directly to AutoModelForCausalLM.from_pretrained(model_id, ...); Transformers/PEFT may route that call through the base Qwen3ForCausalLM class, which does not accept those arguments.
• Use snapshot_download() and the bundled speed_inference.load_speed_model() entrypoint as shown above. The original SPEED source repository is not needed on the inference server.
• Always pass speed_generate=True for SPEED inference. Calling ordinary generate() without speed_generate=True uses the normal generation path.
• This is a PEFT adapter. The base model Qwen/Qwen3-8B must be downloadable from the inference server.
• pipeline("text-generation", ...) is not recommended because the SPEED API needs structured arguments such as messages, context, and lower_k.
• vLLM serving is not covered by this adapter repository. Use the Transformers path above unless you have a separate SPEED-vLLM runtime.

Files

Only the Qwen3 SPEED modeling file is included. Mistral/Ministral modeling files are intentionally omitted from this upload artifact.