Instructions to use qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M

Use Docker

docker model run hf.co/qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF with Ollama:
```
ollama run hf.co/qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF with Docker Model Runner:
```
docker model run hf.co/qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M
```

Lemonade

How to use qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Llama-3-8B-Instruct-Coder-v2-iMat-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

Quant Infos

Quantized with recent bpe pre-tokenizer fixes https://github.com/ggerganov/llama.cpp/pull/6920
quants done with an importance matrix for improved quantization loss
0, K & IQ quants in basically all variants from Q8 down to IQ1_S
Quantized with llama.cpp commit 04976db7a819fcf8bfefbfc09a3344210b79dd27 (master from 2024-05-07)

Imatrtix generated with this dataset.

./imatrix -c 512 -m $model_name-f16.gguf -f $llama_cpp_path/groups_merged.txt -o $out_path/imat-f16-gmerged.dat

Original Model Card

Llama-3-8B-Instruct-Coder-v2

How is this model diffrent from rombodawg/Llama-3-8B-Instruct-Coder? Well the first model was trained on a dataset that had some major flaws that I originally had missed, with version 2 all of those flaws are fixed, and the model is fully retrained so it performs much better than the previous iteration.

This model is llama-3-8b-instruct from Meta (uploaded by unsloth) trained on the full 150k Code Feedback Filtered Instruction dataset. You can find that dataset linked below. This AI model was trained with the new Qalore method developed by my good friend on Discord and fellow Replete-AI worker walmartbag.

The Qalore method uses Qlora training along with the methods from Galore for additional reductions in VRAM allowing for llama-3-8b to be loaded on 14.5 GB of VRAM. This allowed this training to be completed on an RTX A5000 24GB in 50 hours for less than $15.

Dataset used for training this model:

https://huggingface.co/datasets/Replete-AI/CodeFeedback-Filtered-Instruction-Simplified-Pairs

Qalore notebook for training:

https://colab.research.google.com/drive/1bX4BsjLcdNJnoAf7lGXmWOgaY8yekg8p?usp=sharing

Quantizations for easier inference:

Downloads last month: 293

GGUF

Model size

8B params

Architecture

llama

Hardware compatibility

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for qwp4w3hyb/Llama-3-8B-Instruct-Coder-v2-iMat-GGUF

Base model

rombodawg/Llama-3-8B-Instruct-Coder-v2

Quantized

(2)

this model