deepsky-ia/TinyLlama-1.1B-Chat-v1.0-GGUF-MacQuantized

🦙 TinyLlama 1.1B Chat — GGUF (Imatrix Quantized)

A GGUF imatrix-quantized build of TinyLlama/TinyLlama-1.1B-Chat-v1.0, optimized for fast, private, on-device inference on Apple Silicon and other consumer hardware.

🚀 Optimized with an Importance Matrix (imatrix). Unlike standard quantizations that calibrate on random data, this build was processed with a dense text corpus (The Adventures of Sherlock Holmes) to compute a high-fidelity Importance Matrix. This preserves the model's most influential weights, yielding lower perplexity and better reasoning than plain K-quants at the same bit width.

---

✨ Why this build

• Runs fully offline / on-device — no data ever leaves the machine. Useful where privacy, data residency or regulatory constraints rule out cloud APIs.
• Small footprint — ~700 MB at 4-bit; runs comfortably on a laptop.
• Imatrix-calibrated — better quality retention than standard K-quants.
• Apple Silicon friendly — built and tested for Metal / llama.cpp on macOS, and portable to Linux and Windows.

---

📦 Available Files

Filename	Quant Type	Size	Use Case
TinyLlama-1.1B-Chat-v1.0-Q4_K_M.gguf	Q4_K_M	~700 MB	🌟 Recommended. Best balance of speed and quality.

---

🛠️ How to Use

Option 1 — llama.cpp (command line)

# Point -m at the downloaded .gguf file
./llama-cli -m TinyLlama-1.1B-Chat-v1.0-Q4_K_M.gguf \
  -p "Hello, how are you?" \
  -n 400 -e

Or pull it directly from the Hub:

llama-cli -hf deepsky-ia/TinyLlama-1.1B-Chat-v1.0-GGUF-MacQuantized:Q4_K_M

Option 2 — llama-cpp-python

# pip install llama-cpp-python
from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="deepsky-ia/TinyLlama-1.1B-Chat-v1.0-GGUF-MacQuantized",
    filename="TinyLlama-1.1B-Chat-v1.0-Q4_K_M.gguf",
)

llm.create_chat_completion(
    messages=[{"role": "user", "content": "What is the capital of France?"}]
)

Option 3 — Ollama

ollama run hf.co/deepsky-ia/TinyLlama-1.1B-Chat-v1.0-GGUF-MacQuantized:Q4_K_M

Option 4 — LM Studio / Jan

Search for deepsky-ia/TinyLlama-1.1B-Chat-v1.0-GGUF-MacQuantized inside the app and download the Q4_K_M file.

---

💬 Prompt format

TinyLlama-Chat uses the Zephyr-style chat template:

<|system|>
You are a helpful assistant.</s>
<|user|>
{your message}</s>
<|assistant|>

---

📋 Model details

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Parameters: 1.1B
• Architecture: Llama
• Quantization: Q4_K_M (4-bit) with importance matrix
• Format: GGUF (for llama.cpp and compatible runtimes)
• License: Apache-2.0 (inherited from the base model)

---

⚠️ Limitations

This is a 1.1B-parameter model. It is well suited to lightweight, on-device and edge use cases, fast prototyping and privacy-sensitive settings, but it is not comparable to larger models on complex reasoning, factual accuracy or long-context tasks. Outputs should be reviewed before use in any high-stakes setting.

---

📚 Citation

If you use this model, please cite:

@misc{salmeron_tinyllama_imatrix,
  author    = {Salmeron, Jose L.},
  title     = {TinyLlama 1.1B Chat GGUF (Imatrix Quantized)},
  year      = {2026},
  publisher = {Hugging Face},
  doi       = {10.57967/hf/7849},
  url       = {https://huggingface.co/deepsky-ia/TinyLlama-1.1B-Chat-v1.0-GGUF-MacQuantized}
}