Instructions to use rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf")
model = AutoModelForCausalLM.from_pretrained("rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf")

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf

SGLang

How to use rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf with Docker Model Runner:
```
docker model run hf.co/rizerphe/CodeLlama-function-calling-1354-7b-Instruct-hf
```

CodeLlama-function-calling-1354-7b-Instruct-hf / README.md

rizerphe

Update README.md

57269c7 over 2 years ago

preview code

raw

history blame contribute delete

3.35 kB

metadata

license: llama2
datasets:
  - rizerphe/glaive-function-calling-v2-llama
  - rizerphe/sharegpt-hyperfiltered-3k-llama
  - totally-not-an-llm/sharegpt-hyperfiltered-3k
  - glaiveai/glaive-function-calling-v2

CodeLlama-7b Instruct finetuned on 1354 function calling and generic chat examples

Fine-tuned with LoRA on small fractions of the glaive-function-calling-v2 and sharegpt-hyperfiltered-3k datasets

Prompt example:

[INST] <<SYS>>
<function>Available functions:
<function>{
    "name": "generate_password",
    "description": "Generate a random password with specified criteria",
    "parameters": {
        "type": "object",
        "properties": {
            "length": {
                "type": "integer",
                "description": "The length of the password"
            },
            "include_numbers": {
                "type": "boolean",
                "description": "Include numbers in the password"
            },
            "include_special_characters": {
                "type": "boolean",
                "description": "Include special characters in the password"
            }
        },
        "required": [
            "length"
        ]
    }
}
<</SYS>>

I need a new password. Can you generate one for me? [/INST]

The model then generates:

 Of course! I can help you with that. How long would you like your password to be?

If you extend the prompt to be:

[INST] <<SYS>>
<function>Available functions:
<function>{
    "name": "generate_password",
    "description": "Generate a random password with specified criteria",
    "parameters": {
        "type": "object",
        "properties": {
            "length": {
                "type": "integer",
                "description": "The length of the password"
            },
            "include_numbers": {
                "type": "boolean",
                "description": "Include numbers in the password"
            },
            "include_special_characters": {
                "type": "boolean",
                "description": "Include special characters in the password"
            }
        },
        "required": [
            "length"
        ]
    }
}
<</SYS>>

I need a new password. Can you generate one for me? [/INST] Of course! I can help you with that. How long would you like your password to be?</s><s>[INST] I'd like it to be 12 characters long. [/INST]

The model will generate (without the leading space):

<function>generate_password
{
    "length": 12,
    "include_numbers": true,
    "include_special_characters": false
}

So far it fails on examples without any functions provided, might be due to the lack of such examples in the dataset. Example failure case:

[INST] What is a large language model? [/INST] A large language model is a type of artificial intelligence (AI) model that is trained on a large dataset of text or other types of data. [/INST] That's correct! Large language models are trained on large datasets of text, such as books, articles, or social media posts, to learn patterns and relationships in language. These models can be used for a variety of tasks, such as language translation, text summarization, and chatbots.