Instructions to use tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ")
model = AutoModelForMultimodalLM.from_pretrained("tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ

SGLang

How to use tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ with Docker Model Runner:
```
docker model run hf.co/tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ
```

Swift-1.5-Qwen3.8-27B — AWQ W4A16 (compressed-tensors)

Weights: model.safetensors (18.6 GB, int4 + BF16 parts) and model_mtp.safetensors (0.85 GB, BF16 MTP head).

Unofficial community quantization. This is a 4-bit AWQ (weight-only int4, group size 128) derivative of ukisai/Swift-1.5-Qwen3.8-27b. It was not made, reviewed or endorsed by UkisAI. It was produced for internal evaluation on vLLM with a 48 GB NVIDIA card (e.g. RTX 6000 Ada).

Status: NOT yet tested on vLLM or on any NVIDIA GPU. The checkpoint was produced and checked on AMD hardware only (Radeon 8060S via PyTorch ROCm on Windows): config sanity checks, perplexity and greedy-generation comparisons against the BF16 original using transformers + compressed-tensors. Please report back if vllm serve fails.

What is in here


Base	ukisai/Swift-1.5-Qwen3.8-27b (BF16, `Qwen3_5ForConditionalGeneration`: 64 decoder layers = 48 Gated-DeltaNet + 16 full-attention, 1 MTP layer, 27-block vision encoder)
Scheme	W4A16, symmetric int4, group size 128, `pack-quantized` compressed-tensors format (vLLM: `CompressedTensorsWNA16` → Marlin kernel on Ampere/Ada/Hopper)
Algorithm	AWQ (activation-aware scaling, `duo_scaling=True`, `n_grid=10`) followed by round-to-nearest int4 with min-max group scales
Tool	llm-compressor 0.14.0 (Apache-2.0), compressed-tensors 0.19.0, transformers 5.17.0, torch 2.11.0+rocm7.13.0
Size	19.5 GB on disk (BF16 original: 55.6 GB). The int4 decoder weights (packed + scales) are 12.5 GB; the rest is BF16 `lm_head` + embeddings (2 × 2.5 GB), vision encoder (~0.9 GB) and MTP head (0.85 GB)

Quantized vs. kept in BF16

Quantized to int4 (all 64 decoder layers): self_attn.{q,k,v,o}_proj, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, mlp.{gate,up,down}_proj.

Kept in BF16 (ignore list in config.json):

lm_head, embed_tokens
the whole vision encoder and merger (model.visual.*)
linear_attn.in_proj_a / linear_attn.in_proj_b (output dim = 48 heads; vLLM fuses them as in_proj_ba, too small for int4 kernels)
the MTP head (mtp.*, copied unchanged from the BF16 checkpoint so that MTP speculative decoding can work)
all norms, conv1d, A_log, dt_bias (not Linear layers)

AWQ smoothing mappings (llm-compressor's built-in Qwen3.5 hybrid mappings): input_layernorm → q/k/v (full-attention layers), input_layernorm → in_proj_qkv/z/b/a (linear-attention layers), post_attention_layernorm → gate/up, up → down. See recipe.yaml.

Calibration data

128 chat-formatted samples (rendered with the model's own chat template, thinking disabled), truncated to 2048 tokens, 99,780 tokens in total:

Source	License	Samples	Content
HuggingFaceH4/ultrachat_200k (test_sft)	MIT	36	English multi-turn chat
shareAI/ShareGPT-Chinese-English-90k (computer_zh)	Apache-2.0	27	Chinese programming / technical chat
shareAI/ShareGPT-Chinese-English-90k (common_zh)	Apache-2.0	19	Chinese general chat
theblackcat102/evol-codealpaca-v1	Apache-2.0	35	English coding instructions
glaiveai/glaive-function-calling-v2	Apache-2.0	11	tool / function calling (converted to the model's tool-call format)

Quality check (AMD / transformers, not vLLM)

24 held-out chat samples from the same five sources (not used for calibration), each ≤2048 tokens, evaluated with transformers + compressed-tensors (weights dequantized at load, so numerically the same int4 values vLLM uses). "Assistant-only" counts only the 10,706 tokens of the assistant turns (what the model actually generates); "full sequence" also scores the user turns (17,416 tokens). A data-free round-to-nearest (RTN) W4A16 g128 checkpoint with the identical ignore list was made as a baseline.

Metric	BF16 original	AWQ (this repo)	RTN baseline
Assistant-only PPL, all 24 samples	3.849	4.318 (+12.2%)	4.301 (+11.7%)
Assistant-only PPL, excluding sample #4 (outlier, see below)	3.111	3.088 (-0.7%)	3.442 (+10.6%)
Median per-sample change in assistant NLL (nats/token)	0	+0.018	+0.033
Full-sequence PPL (incl. user turns)	23.45	29.89	26.58
Top-1 next-token agreement with BF16 (full sequence)	100%	89.1%	88.7%

Assistant-only PPL by source (BF16 → AWQ / RTN): Chinese coding 2.649 → 2.758 / 2.993; Chinese general 3.820 → 3.751 / 4.010; English code 1.904 → 1.929 / 1.952; tool calling 1.810 → 1.621 / 1.656; English chat (UltraChat) 7.126 → 9.933 / 8.803.

Caveats, stated plainly:

One English essay sample (#4, 1,575 assistant tokens) degrades a lot with AWQ (PPL 13.2 → 30.2; RTN 15.7) and dominates the "all samples" number. On the other 23 samples AWQ is on par with BF16 and clearly better than RTN.
Full-sequence PPL (scoring user-written text) is worse for AWQ than for RTN. AWQ was calibrated on chat data and optimises the layer outputs; predicting user text is not what the model is used for, but the number is reported for completeness.
24 samples / ~10k tokens is a small sanity check, not a benchmark. Please run your own task evaluations.

Greedy generations (thinking disabled, 256 new tokens) for a Chinese coding prompt (merge_intervals with Chinese comments) and an English one (log parser with type hints) were correct and well-formed for both BF16 and AWQ; the AWQ outputs differ in wording/implementation details (e.g. regex instead of split) but not in quality.

vLLM usage (48 GB card, e.g. RTX 6000 Ada) — untested

vllm serve <path-or-repo> \
  --served-model-name swift-1.5-27b-awq \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.90 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Weights take about 19.5 GB of VRAM; the rest goes to the KV / linear-attention state cache. With 48 GB, --max-model-len 65536 should fit comfortably; try 131072 or the full 262144 if you need it (only 16 of 64 layers keep a growing KV cache). Add --language-model-only to skip the vision encoder and save memory if you do not need images.
MTP speculative decoding (to be verified; the MTP head is included in BF16): --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' (newer vLLM versions may also accept "method":"mtp").
vLLM should detect the format automatically (quant_method: compressed-tensors); do not pass --quantization awq.

How it was made

python scripts/build_calib.py          # builds calib.jsonl / eval.jsonl from the five datasets above
python scripts/quantize.py <bf16-dir> <out> --n 128 --seqlen 2048 --grid 10 --device-map auto
python scripts/validate.py <out>        # static config / tensor checks
python scripts/eval_ppl.py <dir> out.json [--ref bf16.json]

(scripts/dist_shim.py only adds single-process stubs because the Windows ROCm build of PyTorch has no torch.distributed.) llm-compressor oneshot with the sequential (layer-by-layer) pipeline on a Radeon 8060S (ROCm 7.13, PyTorch 2.11 on Windows), recipe = AWQModifier(duo_scaling=True, n_grid=10) + QuantizationModifier(targets=["Linear"], scheme="W4A16", ignore=[...]). Runtime about 94 minutes (Radeon 8060S, 64 layers). A first attempt with n_grid=20 was stopped after two layers for time reasons; the layer-0 AWQ search errors with 10 vs 20 grid points differed by <0.5%.

License

The model weights are a derivative of Swift 1.5 Qwen3.8-27B by UkisAI, licensed under the Swift Open License v1.0 (see LICENSE). In short: use, including commercial use, is free for individuals and organizations with gross annual revenue (including affiliates) below US$1,000,000; above that threshold, commercial use requires a separate Swift Enterprise License from UkisAI (https://ukisai.com/contact). Read the full LICENSE; this summary is not legal advice.
Swift 1.5 is itself a derivative of Qwen3.8-27B, Copyright 2026 Alibaba Cloud, licensed under the Apache License 2.0 (see LICENSE-APACHE-2.0). Nothing here limits your rights in Qwen3.8-27B under Apache 2.0.
NOTICE reproduces UkisAI's attribution notice, with an addendum describing the changes made for this quantization.
"Swift" and "UkisAI" are names of the original author and are used here only to describe the origin of the weights.

Change notice (Swift Open License §4(b) / Apache-2.0 §4(b)): model-*.safetensors, model.safetensors.index.json, config.json (added quantization_config), recipe.yaml and README.md were created or modified by this quantization. model_mtp.safetensors contains the original BF16 MTP tensors unchanged. Tokenizer, chat template, generation and processor configs are copied unchanged from the original repository.

Downloads last month: 68

Safetensors

Model size

27B params

Tensor type

I32

BF16

Model tree for tsaipifong/Swift-1.5-Qwen3.8-27B-AWQ

Base model

Qwen/Qwen3.8-27B

Finetuned

ukisai/Swift-1.5-Qwen3.8-27b

Quantized

(91)

this model