Instructions to use yeasah/Muse-Glimmer-30B-exl3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yeasah/Muse-Glimmer-30B-exl3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="yeasah/Muse-Glimmer-30B-exl3")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("yeasah/Muse-Glimmer-30B-exl3", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yeasah/Muse-Glimmer-30B-exl3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yeasah/Muse-Glimmer-30B-exl3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yeasah/Muse-Glimmer-30B-exl3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/yeasah/Muse-Glimmer-30B-exl3

SGLang

How to use yeasah/Muse-Glimmer-30B-exl3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yeasah/Muse-Glimmer-30B-exl3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yeasah/Muse-Glimmer-30B-exl3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yeasah/Muse-Glimmer-30B-exl3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yeasah/Muse-Glimmer-30B-exl3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use yeasah/Muse-Glimmer-30B-exl3 with Docker Model Runner:
```
docker model run hf.co/yeasah/Muse-Glimmer-30B-exl3
```

EXL3 quantizations of meta-models/Muse-Glimmer-30B

These quantizations can be used with exllamav3 or vLLM via vllm-exl3-plugin.

They are calibrated broadly, on what models read and write while working (documents and tool output in conversations, code and diffs, the model's own replies) rather than on web text alone.

Bits are allocated per tensor, from the model's measured sensitivity on its own replies, at the same total size as uniform bits.

Actual consumed VRAM may be significantly lower than the total checkpoint size, depending on configuration: see the VRAM columns below.

Which size?

Each bitrate is available as a separate revision of this repository. When planning available VRAM for kv space, refer to the appropriate VRAM column.

The dashed line marks Muse-Glimmer-30B-FP8-block, scored on the same evals as these quants: a familiar point to read them against. It is scored with its weights dequantized under bf16 compute, so it measures the checkpoint's weights, not a deployment (the activation quantization it is served with is not emulated).

BPW	Size on disk	VRAM (exllamav3)	VRAM (vLLM)	Excess KLD	Δ perplexity
2.50	14.21 GiB	11.70 GiB	8.83 GiB	0.0792	+9.56%
3.00	15.67 GiB	13.17 GiB	10.29 GiB	0.0386	+4.38%
3.50	17.14 GiB	14.63 GiB	11.76 GiB	0.0188	+2.25%
4.00	18.76 GiB	16.25 GiB	13.38 GiB	0.0095	+1.00%
5.00	21.69 GiB	19.18 GiB	16.31 GiB	0.0025	+0.24%
6.00	24.62 GiB	22.11 GiB	19.24 GiB	0.0005	+0.04%
(base)	55.46 GiB

BPW: Average bits per weight across tensors (not including input embeddings, encoders, MTP draft layers, or output head)
Size on disk: Total aggregate size of tensors in checkpoint, in GiB
VRAM (exllamav3): Total VRAM consumption as typically used in exllamav3, in GiB (minus embeddings and MTP draft layers)
VRAM (vLLM): Total VRAM consumption as typically used in vLLM, in GiB (blockq embeddings, offloaded/omitted encoders, no MTP draft layers)
Excess KLD: how far each quant's next-token distribution is from the unquantized model's, minus the unquantized model's own rounding noise, so 0 would be indistinguishable from the original (lower is better)
Δ perplexity: change in perplexity on the same text, same weights (lower is better)

More detail

What each eval slice is, every slice compared on its own, how the composite is built, the calibration, and how to reproduce these quants: TECHNICAL.md.

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for yeasah/Muse-Glimmer-30B-exl3

Base model

meta-models/Muse-Glimmer-30B

Quantized

(199)

this model

Collection including yeasah/Muse-Glimmer-30B-exl3

EXL3 quantizations

Collection

Select EXL3 checkpoints • 5 items • Updated 19 days ago