florianvoss commited on
Commit
b7d71de
Β·
verified Β·
1 Parent(s): 28faea6

adapt model card to neat

Browse files
Files changed (1) hide show
  1. README.md +24 -47
README.md CHANGED
@@ -35,33 +35,25 @@ This repository contains the **LFM2-2.6B-a16w4** model, optimized and compiled f
35
  | LFM2-2.6B | A16W8/A16W4 | Modalix | 512 | 37.87 tokens/sec | 0.18 sec |
36
  | LFM2-2.6B | A16W8/A16W4 | Modalix | 1024 | 37.17 tokens/sec | 0.38 sec |
37
 
38
-
39
  ## Prerequisites
40
 
41
  To run this model, you need:
42
 
43
  1. **SiMa.ai Modalix Device**
44
- 2. **SiMa.ai CLI**: [Installed](https://docs.sima.ai/pages/sima_cli/main.html#installation) on your Modalix device.
45
- 3. **Hugging Face CLI**: For downloading the model.
 
46
 
47
  ## Installation & Deployment
48
 
49
  Follow these steps to deploy the model to your Modalix device.
50
 
51
- ### 1. Install LLiMa Demo Application
52
- > **Note:** This is a **one-time setup**. If you have already installed the LLiMa demo application (e.g. for another model), you can skip this step and continue with model download.
53
 
54
- On your Modalix device, install the LLiMa demo application using the `sima-cli`:
55
 
56
- ```bash
57
- # Create a directory for LLiMa
58
- cd /media/nvme
59
- mkdir llima
60
- cd llima
61
- # Install the LLiMa runtime code
62
- sima-cli install -v 2.1.0 tools/llima -t selection
63
- ```
64
- > **Note:** To only download the LLiMa runtime code, select **Demo Web App** when prompted.
65
 
66
  ### 2. Download the Model
67
 
@@ -84,65 +76,50 @@ scp -r LFM2-2.6B-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/
84
 
85
  ```text
86
  /media/nvme/llima/
87
- β”œβ”€β”€ run.sh
88
  └── models/
89
  └── LFM2-2.6B-a16w4/ # The compiled model
90
  ```
91
 
92
  ## Usage
93
 
94
- ### Run the Application
95
 
96
- Navigate to the demo directory and start the application:
97
 
98
  ```bash
99
- cd /media/nvme/llima/
100
- ./run.sh
101
  ```
102
 
103
- The script will detect the installed model(s) and prompt you to select one.
104
 
105
- Once the application is running, open a browser and navigate to:
106
- ```text
107
- https://<modalix-ip>:5000/
108
  ```
109
- *Replace \<modalix-ip\> with the IP address of your Modalix device.*
110
 
111
- ### API Usage
112
 
113
- To use OpenAI-compatible API, run the model in API mode:
114
- ```bash
115
- llima run LFM2-2.6B-a16w4 --mode web
116
- ```
117
 
118
- You can interact with it using `curl` or Python.
119
 
120
- **Example: Chat Completion**
121
 
122
- ```bash
123
- curl -N -k -X POST "https://<modalix-ip>:9998/v1/chat/completions" \
124
- -H "Content-Type: application/json" \
125
- -d '{
126
- "messages": [
127
- { "role": "user", "content": "Why is the sky blue?" }
128
- ],
129
- "stream": true
130
- }'
131
- ```
132
- *Replace \<modalix-ip\> with the IP address of your Modalix device.*
133
 
134
  ## Limitations
135
 
136
  - **Quantization**: This model is quantized (A16W4/A16W8) for optimal performance on embedded devices. While this maintains high accuracy, minor deviations from the full-precision model may occur.
137
 
138
-
139
  ## Troubleshooting
140
 
141
- - **`sima-cli` not found**: Ensure that sima-cli is installed on your Modalix device.
 
142
  - **Model can't be run**: Verify the model directory is exactly inside `/media/nvme/llima/models/` and not nested (e.g., `/media/nvme/llima/models/LFM2-2.6B-a16w4/LFM2-2.6B-a16w4`).
143
  - **Permission Denied**: Ensure you have read/write permissions for the `/media/nvme` directory.
144
 
145
  ## Resources
146
 
147
- - [SiMa.ai Documentation](https://docs.sima.ai)
148
- - [SiMa.ai Hugging Face Organization](https://huggingface.co/simaai)
 
 
 
35
  | LFM2-2.6B | A16W8/A16W4 | Modalix | 512 | 37.87 tokens/sec | 0.18 sec |
36
  | LFM2-2.6B | A16W8/A16W4 | Modalix | 1024 | 37.17 tokens/sec | 0.38 sec |
37
 
 
38
  ## Prerequisites
39
 
40
  To run this model, you need:
41
 
42
  1. **SiMa.ai Modalix Device**
43
+ 2. **SiMa.ai CLI**: Installed on your Modalix device.
44
+ 3. **SiMa.ai Neat Runtime**: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
45
+ 4. **Hugging Face CLI**: Optional, for downloading the model on a host before copying it to Modalix.
46
 
47
  ## Installation & Deployment
48
 
49
  Follow these steps to deploy the model to your Modalix device.
50
 
51
+ ### 1. Install or Update Neat Runtime
52
+ > **Note:** This is a **one-time setup**. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.
53
 
54
+ Follow the [SiMa.ai Neat getting started guide](https://developer.sima.ai/software/getting-started/) to install or update the Neat Library on your Modalix device.
55
 
56
+ The `llima` CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under `/media/nvme/llima/models` by default. Set `LLIMA_MODELS_PATH` to use a different model directory.
 
 
 
 
 
 
 
 
57
 
58
  ### 2. Download the Model
59
 
 
76
 
77
  ```text
78
  /media/nvme/llima/
 
79
  └── models/
80
  └── LFM2-2.6B-a16w4/ # The compiled model
81
  ```
82
 
83
  ## Usage
84
 
85
+ ### Validate with LLiMa CLI
86
 
87
+ Run the model directly on Modalix:
88
 
89
  ```bash
90
+ llima run LFM2-2.6B-a16w4
 
91
  ```
92
 
93
+ For all runtime options, run:
94
 
95
+ ```bash
96
+ llima run -h
 
97
  ```
 
98
 
99
+ ### GenAI Demo Application
100
 
101
+ The GenAI demo application is separate from LLiMa installation. Use the [GenAI Multimodal Assistant](https://developer.sima.ai/examples/app/genai%2Fmultimodal-assistant) page to install and run the demo app. Once installed, the demo app can use precompiled models such as this one.
 
 
 
102
 
103
+ ### API Usage
104
 
105
+ To serve this model with OpenAI- or Ollama-compatible APIs and send requests to it, use the GenAI server workflow in [Serve GenAI Models](https://developer.sima.ai/software/tutorials/serve-genai-models).
106
 
107
+ For direct LLM calls without setting up a server, see [Run an LLM](https://developer.sima.ai/software/tutorials/run-an-llm).
 
 
 
 
 
 
 
 
 
 
108
 
109
  ## Limitations
110
 
111
  - **Quantization**: This model is quantized (A16W4/A16W8) for optimal performance on embedded devices. While this maintains high accuracy, minor deviations from the full-precision model may occur.
112
 
 
113
  ## Troubleshooting
114
 
115
+ - **`sima-cli` not found**: Ensure that `sima-cli` is installed on your Modalix device.
116
+ - **`llima` not found**: Install or update the Neat Library. See [Getting Started](https://developer.sima.ai/software/getting-started/).
117
  - **Model can't be run**: Verify the model directory is exactly inside `/media/nvme/llima/models/` and not nested (e.g., `/media/nvme/llima/models/LFM2-2.6B-a16w4/LFM2-2.6B-a16w4`).
118
  - **Permission Denied**: Ensure you have read/write permissions for the `/media/nvme` directory.
119
 
120
  ## Resources
121
 
122
+ - [GenAI with LLiMa](https://developer.sima.ai/software/genai-llima/)
123
+ - [Serve GenAI Models](https://developer.sima.ai/software/tutorials/serve-genai-models)
124
+ - [Run an LLM](https://developer.sima.ai/software/tutorials/run-an-llm)
125
+ - [GenAI Multimodal Assistant](https://developer.sima.ai/examples/app/genai%2Fmultimodal-assistant)