Qwen2.5-3B-Instruct-FP8-W8A8

Model Overview

This model is a quantized version of Qwen/Qwen2.5-3B-Instruct using FP8 W8A8 quantization with the compressed-tensors format.

• Quantization Method: FP8 (float-quantized), W8A8 (8-bit weights, 8-bit activations)
• Quantization Strategy: Per-tensor, symmetric
• Calibration Dataset: CNN/DailyMail (512 samples, max sequence length 2048)
• Format: compressed-tensors
• Model Size: ~3.8 GB (vs ~6.0 GB for the original BF16 model)

Quantization Details

Component	Precision	Strategy	Observer	Symmetric
Weights	FP8 (8-bit float)	Per-tensor	MinMax	Yes
Activations	FP8 (8-bit float)	Per-tensor	MinMax	Yes

• The lm_head layer is not quantized to preserve output quality.

How to Use

With vLLM

from vllm import LLM, SamplingParams

model = LLM(model="JongYeop/Qwen2.5-3B-Instruct-FP8-W8A8")

sampling_params = SamplingParams(max_tokens=512, temperature=0.7)
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello! Can you introduce yourself?"},
]
outputs = model.chat(messages, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "JongYeop/Qwen2.5-3B-Instruct-FP8-W8A8"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello! Can you introduce yourself?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True))

Creation

This model was created using llmcompressor with the following recipe:

quant_stage:
  quant_modifiers:
    QuantizationModifier:
      config_groups:
        group_0:
          targets: [Linear]
          weights:
            num_bits: 8
            type: float
            symmetric: true
            strategy: tensor
            dynamic: false
            observer: minmax
          input_activations:
            num_bits: 8
            type: float
            symmetric: true
            strategy: tensor
            dynamic: false
            observer: minmax
      targets: [Linear]
      ignore: [lm_head]

Evaluation

For evaluation results, please refer to the original Qwen2.5-3B-Instruct model card. FP8 W8A8 quantization typically preserves most of the original model's quality.