Subscribe and Support

This is meta-models/Muse-Glimmer-30B quantized with AutoRound using a target bitwidth set at 3.5. Model packed by llm-compressor.

The model is compatible with vLLM (tested: v0.26+muse glimmer). Made and Tested with an RTX Pro 6000 (Google Colab's G4). Currently under evaluation: Subscribe to be notified when I publish the results.

Made by: The Kaitchup

Instructions

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer kaitchup/Muse-Glimmer-30B-autoscheme-3.5bit \
  --generation-config auto \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Downloads last month: 48

Safetensors

Model size

7B params

Tensor type

BF16

I32

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for kaitchup/Muse-Glimmer-30B-autoscheme-3.5bit

Base model

meta-models/Muse-Glimmer-30B

Quantized

(147)

this model

Collection including kaitchup/Muse-Glimmer-30B-autoscheme-3.5bit

Quantized Muse Glimmer

Collection

Compatible with vLLM 0.27+ • 5 items • Updated 7 days ago