Instructions to use Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF

SGLang

How to use Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF
```

需要llama.cpp的编译方法和启动参数

by traddo - opened Jun 13

Discussion

traddo

Jun 13

Model Card中洋洋洒洒写一大堆，就不写怎么编译，怎么启动是吧？

Jacob1970

Jun 13

jackrong做了很多模型，但是量化的q5基本都有输出突然中断的问题，这个也不例外。你需要不停的提示“continue”，才能勉强继续。还是感谢他的尝试。

jasonshao1983

Jun 13

This Model Card is just a model manual. The rest falls under llama.cpp's scope. The author already provided the context tuning parameters in the card, so you can find the general compilation and usage guide anywhere (e.g., on their GitHub).

hyzx86

Jun 14

•

edited Jun 14

git pull https://github.com/TheTom/llama-cpp-turboquant.git
cd  llama-cpp-turboquant

#RTX3090 CPU :i9-9900
rm -rf build

export CC="ccache gcc"
export CXX="ccache g++"

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_NATIVE=ON \
  -DGGML_LTO=ON \
  -DBUILD_SHARED_LIBS=OFF

cmake --build build --config Release -j$(($(nproc) - 1))

/home/hyzx86/ai/TheTom/llama-cpp-turboquant/build/bin/llama-server
-m /home/hyzx86/ai/models/Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF/Qwopus3.6-27B-Coder-MTP-Q5_K_M.gguf
--mmproj /home/hyzx86/ai/models/Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF/mmproj-F32.gguf
-a "Qwopus3.6-27B-Coder-text"
--host 0.0.0.0 --port 8088
-t 8 --parallel 1 -fa on -ngl 999 --jinja
--spec-type draft-mtp --spec-draft-n-max 2
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0
--reasoning on
--chat-template-kwargs '{"preserve_thinking":true}'
--cache-type-k q8_0 --cache-type-v turbo3
-b 256-c 122000

fb700

Jun 14

https://github.com/BoFan-tunning/llama.cpp-MTP-TurboQuant
You can use this project to run mtp with TurboQuan。

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment