UraionLabs/Ling-3.0-tiny-oQ8e

Uraion Labs
_{High-performance MLX model optimizations for Apple Silicon.}

---

Ling-3.0-tiny — oQ8e (MLX)

Ling-3.0-tiny-oQ8e is an 8-bit outlier-aware quantized release of inclusionAI/Ling-3.0-tiny, optimized specifically for Apple Silicon Macs (M1/M2/M3/M4) via MLX and oMLX.

⚡ 8B-class MoE reasoning at 1.3B active parameter speeds (~86–105+ tokens/s) in only ~8.4 GB RAM.

---

🌟 Full Credits & Upstream Attribution

This quantized repository is built upon the breakthrough work of the Ant Ling Team at Ant Group (@inclusionAI).

All credit for the novel hybrid-linear MoE architecture, pre-training corpus, reasoning distillation, thinking mode alignment, and agentic capabilities belongs to the Ant Ling team:

• 🏛️ Original Model Creator: Ant Ling Team (@inclusionAI)
• 📦 Official Base Model: inclusionAI/Ling-3.0-tiny
• 🤖 ModelScope: inclusionAI on ModelScope
• 🐙 OpenRouter API: Ling-3.0-tiny on OpenRouter
• 📖 SGLang Cookbook: Official Ling-3.0-tiny Launch Recipes

Please show your support by starring and checking out the official base model at inclusionAI/Ling-3.0-tiny!

---

⚡ Why Ling-3.0-tiny-oQ8e for Apple Silicon?

Ling-3.0-tiny is one of the most efficient open reasoning models ever created. It couples a hybrid linear-attention architecture (3:1 KDA-MLA) with a sparse 128-expert MoE FFN where only 8 routed experts + 1 shared expert activate per token (only 1.3B active parameters out of 7.9B total).

Highlights of the oQ8e MLX Quantization:

• 🚀 Blazing Fast on Mac: Delivers 86–105+ tokens/second on Apple Silicon (tested on M4 Pro and Apple M-series chips).
• 🧠 Zero Degradation with oQe Quantization: Quantized using oMLX oQe (Outlier-aware Quantization with importance matrix calibration) across all 8,928 MoE expert matrices (oqe_code_multilingual calibration set), preserving complex mathematical, coding, and multi-step reasoning abilities.
• 💾 Low Memory Footprint: Weighs only ~8.39 GB, running comfortably on 16 GB, 24 GB, 36 GB, 48 GB, 64 GB, and 128 GB Unified Memory Macs with headroom for your operating system and IDE.
• 🎯 Native Hybrid Reasoning: Switch seamlessly between instantaneous direct answers and deep <think> chain-of-thought exploration.
• 📚 131K–256K Context: Built for long-context document analysis, multi-file code review, and sustained agentic workflows.

---

🚀 Quickstart: Run in Seconds on Apple Silicon

Option 1: CLI with mlx-lm

Install mlx-lm:

pip install --upgrade mlx-lm

Run immediate text generation:

python -m mlx_lm.generate \
  --model UraionLabs/Ling-3.0-tiny-oQ8e \
  --prompt "<role>HUMAN</role>Explain how quantum computing works in simple terms.<|role_end|><role>ASSISTANT</role>\n<think>" \
  --max-tokens 1024 \
  --temp 1.0 \
  --top-p 0.95

---

Option 2: Python API (mlx_lm)

from mlx_lm import load, generate

# Load model and tokenizer directly from Hugging Face
model, tokenizer = load("UraionLabs/Ling-3.0-tiny-oQ8e")

# Format prompt with Ling's native chat template
prompt = (
    "<role>SYSTEM</role>detailed thinking on<|role_end|>"
    "<role>HUMAN</role>Write a Python script to calculate Fibonacci numbers using dynamic programming.<|role_end|>"
    "<role>ASSISTANT</role>\n<think>"
)

# Generate with recommended sampling parameters
response = generate(
    model=model,
    tokenizer=tokenizer,
    prompt=prompt,
    max_tokens=1024,
    temp=1.0,
    top_p=0.95,
    verbose=True
)

print(response)

---

Option 3: Local OpenAI-Compatible API Server

With omlx (High Performance Apple Silicon Server):

omlx serve UraionLabs/Ling-3.0-tiny-oQ8e --port 8000

With standard mlx-lm:

python -m mlx_lm.server --model UraionLabs/Ling-3.0-tiny-oQ8e --port 8000

Query via curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "UraionLabs/Ling-3.0-tiny-oQ8e",
    "messages": [
      {"role": "user", "content": "What are 3 practical use cases for sparse MoE models?"}
    ],
    "temperature": 1.0,
    "top_p": 0.95
  }'

Query with Python openai client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="UraionLabs/Ling-3.0-tiny-oQ8e",
    messages=[
        {"role": "system", "content": "detailed thinking on"},
        {"role": "user", "content": "Compare KDA attention with traditional multi-head attention."}
    ],
    temperature=1.0,
    top_p=0.95,
)

print(response.choices[0].message.content)

---

🧠 Thinking Mode & Prompt Formatting

Ling-3.0-tiny supports native hybrid reasoning. You can control thinking mode directly inside the system prompt:

• Thinking Mode ON (Deep Reasoning):

  <role>SYSTEM</role>detailed thinking on<|role_end|><role>HUMAN</role>{prompt}<|role_end|><role>ASSISTANT</role>
  <think>
  

• Thinking Mode OFF (Fast Response):

  <role>SYSTEM</role>detailed thinking off<|role_end|><role>HUMAN</role>{prompt}<|role_end|><role>ASSISTANT</role>
  <think></think>
  

Recommended Sampling Parameters:

• Temperature: 1.0
• Top-P: 0.95
• Top-K: 20

---

💻 Hardware Requirements & Apple Silicon Compatibility

Thanks to the ultra-efficient 1.3B active parameter footprint and 8-bit oQe quantization, Ling-3.0-tiny-oQ8e runs smoothly on virtually any Apple Silicon Mac:

Apple Silicon Hardware	Unified Memory	Status	Estimated Speed
M1 / M2 / M3 / M4 (Base)	16 GB	✅ Recommended	50 – 75 tok/s
M1 / M2 / M3 / M4 Pro	18 GB / 24 GB / 36 GB / 48 GB	🚀 Ideal Setup	80 – 100 tok/s
M1 / M2 / M3 / M4 Max	32 GB / 36 GB / 64 GB / 128 GB	⚡ Blazing Fast	95 – 120+ tok/s
M1 / M2 Ultra	64 GB / 128 GB / 192 GB	🏎️ Peak Performance	100 – 130+ tok/s

Requires macOS Sonoma or Sequoia with MLX installed.

---

📊 Quantization & Model Specifications

Property	Value
Original Architecture	BailingMoeV3ForCausalLM (Ling-3.0 Hybrid)
Upstream Base Model	inclusionAI/Ling-3.0-tiny
Original Authors	Ant Ling Team (Ant Group / inclusionAI)
Quantized By	Uraion Labs
Quantization Format	oQ8e (oMLX Importance-Matrix Calibrated 8-bit)
Quantization Mode	Affine (Group Size: 64, Bits: 8)
Calibration Dataset	oqe_code_multilingual (Metal hardware-accelerated)
Total Parameters	7,893,392,800 (~7.9B)
Activated Parameters	1,300,000,000 (~1.3B)
Total Experts	128 routed experts + 1 shared expert (8 active per token)
Attention Mechanism	3:1 KDA-MLA (3 Kimi Delta Attention + 1 Multi-Head Latent Attention)
Context Length	131,072 tokens (up to 262,144 tokens with YaRN)
Storage / Disk Size	~8.39 GB
Target Runtime	Apple Silicon macOS (mlx-lm, omlx, mlx)
License	MIT License

---

🔬 Benchmark Performance (Base Ling-3.0-tiny)

Evaluated by the Ant Ling team and Artificial Analysis:

• Artificial Analysis Intelligence Index v4.1.1: 25
• Artificial Analysis Agentic Index: 16
• Inference Speed: Reaches over 160 tokens/s on high-end hardware, with ~86–90 tokens/s measured locally on an M4 Pro MacBook.
• Terminal-Bench 2.1: Strong agentic coding performance under the Artificial Analysis protocol.

---

📜 Citation & Credits

If you use this model in your research or applications, please cite both the original Ant Ling Team and the MLX quantization:

@misc{antling2025ling3tiny,
  title={Ling-3.0-tiny: Efficient Hybrid-Linear Attention Mixture-of-Experts Architecture},
  author={Ant Ling Team},
  organization={Ant Group, inclusionAI},
  year={2025},
  url={https://huggingface.co/inclusionAI/Ling-3.0-tiny}
}

@misc{uraionlabs2025ling3tinyoq8e,
  title={Ling-3.0-tiny-oQ8e: High-Performance 8-Bit MLX Quantization for Apple Silicon},
  author={Uraion Labs},
  year={2025},
  url={https://huggingface.co/UraionLabs/Ling-3.0-tiny-oQ8e}
}

---

_{Optimized with ❤️ for the Apple Silicon MLX community by Uraion Labs.}