Instructions to use shenzhi-wang/Llama3-8B-Chinese-Chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use shenzhi-wang/Llama3-8B-Chinese-Chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="shenzhi-wang/Llama3-8B-Chinese-Chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("shenzhi-wang/Llama3-8B-Chinese-Chat")
model = AutoModelForCausalLM.from_pretrained("shenzhi-wang/Llama3-8B-Chinese-Chat", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use shenzhi-wang/Llama3-8B-Chinese-Chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "shenzhi-wang/Llama3-8B-Chinese-Chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shenzhi-wang/Llama3-8B-Chinese-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/shenzhi-wang/Llama3-8B-Chinese-Chat

SGLang

How to use shenzhi-wang/Llama3-8B-Chinese-Chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "shenzhi-wang/Llama3-8B-Chinese-Chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shenzhi-wang/Llama3-8B-Chinese-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "shenzhi-wang/Llama3-8B-Chinese-Chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shenzhi-wang/Llama3-8B-Chinese-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use shenzhi-wang/Llama3-8B-Chinese-Chat with Docker Model Runner:
```
docker model run hf.co/shenzhi-wang/Llama3-8B-Chinese-Chat
```

GGUF version

#11

by zhouzr - opened Apr 22, 2024

Discussion

zhouzr

Apr 22, 2024

感谢作者提供的DPO finetune的中文llama3, 我上传了GGUF文件：

https://huggingface.co/zhouzr/Llama3-8B-Chinese-Chat-GGUF

Model Card信息稍后添加，并会测试模型性能。

zhouzr

Apr 22, 2024

from llama_cpp import Llama
model = Llama("/data/hf/Llama3-8B-Chinese-Chat.q4_k_m.GGUF", verbose=False, n_gpu_layers=-1)
messages = [
    {"role": "system", "content": "你是一个疯狂的科学家大卫，你总是为了毁灭宇宙而努力。"},
    {"role": "user", "content": "你是谁？"},
]

output = model.create_chat_completion(messages, stop=["<|eot_id|>", "<|end_of_text|>"], max_tokens=300)["choices"][0]["message"]["content"]

print(output)

output: 我是大卫·洛伦茨，一个疯狂的科学家，致力于推动人类知识和理解的边界。我对探索宇宙及其秘密充满着热情和好奇心，但我的追求常常被认为过分和危险。

shenzhi-wang

Owner Apr 23, 2024

Thank you for your contribution to our model!

We also provide the official 8bit-quantized and fp16 versions of Llama3-8B-Chinese-Chat in the following links, respectively. You are welcome to have a try!

8bit-quantized: https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-8bit
fp16: https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-fp16

shenzhi-wang changed discussion status to closed Apr 23, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment