Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -10,8 +10,59 @@ tags:
|
|
| 10 |
|
| 11 |
# llama.cpp b10355 CUDA runtime for Colab T4 (SM75)
|
| 12 |
|
| 13 |
-
This public repository stores a reproducible
|
|
|
|
|
|
|
|
|
|
| 14 |
|
| 15 |
-
|
| 16 |
|
| 17 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
# llama.cpp b10355 CUDA runtime for Colab T4 (SM75)
|
| 12 |
|
| 13 |
+
This public repository stores a reproducible `llama-server` runtime, not model
|
| 14 |
+
weights. It was cross-built on an actual Google Colab G4 with CUDA 12.8 and
|
| 15 |
+
`CMAKE_CUDA_ARCHITECTURES=75`, then downloaded without authentication and
|
| 16 |
+
validated on an actual Colab Tesla T4.
|
| 17 |
|
| 18 |
+
## Provenance
|
| 19 |
|
| 20 |
+
- Source: `ggml-org/llama.cpp`
|
| 21 |
+
- Release: `b10355`
|
| 22 |
+
- Commit: `dd1ea524333b1e697489067d7a4c39c60d32beee`
|
| 23 |
+
- Build host: NVIDIA RTX PRO 6000 Blackwell Server Edition (Colab G4)
|
| 24 |
+
- Target: Linux x86-64, CUDA 12.8, SM75, Colab Tesla T4
|
| 25 |
+
- Archive: `llama-cpp-b10355-cuda128-sm75-colab.tar.gz`
|
| 26 |
+
- Archive SHA-256: `4469c6f18805f2cf2917df6bb8c0dbd37937344ab6c2e77c312cccdf7b145a9a`
|
| 27 |
+
- `llama-server` SHA-256: `f4f421a690bc949174752111bd20ac861fcdc2ba65fca361955edf7c67b53252`
|
| 28 |
+
|
| 29 |
+
## Verified T4 result
|
| 30 |
+
|
| 31 |
+
The public archive was restored at revision
|
| 32 |
+
`5738166d5979c9d39d0b3cacbaa1c5c2a3d44b16` with no HF token. It loaded
|
| 33 |
+
`unsloth/Qwen2.5-VL-7B-Instruct-GGUF` revision
|
| 34 |
+
`68bb8bc4b7df5289c143aaec0ab477a7d4051aab`, using
|
| 35 |
+
`UD-Q4_K_XL` plus `mmproj-BF16`.
|
| 36 |
+
|
| 37 |
+
- GPU memory after load: 6,647 MiB
|
| 38 |
+
- Model load to healthy server: 4.01 seconds on the verified rerun
|
| 39 |
+
- Fixed workload: uncached 512 prompt tokens to fixed 128 output tokens,
|
| 40 |
+
batch 1, concurrency 1, five repetitions
|
| 41 |
+
- Client end-to-end median: 34.53 tok/s
|
| 42 |
+
- llama.cpp decode median: 39.20 tok/s
|
| 43 |
+
- Five output hashes were identical
|
| 44 |
+
- Multimodal smoke: 1024x768 candy photo, 4.00 seconds; correctly reported five
|
| 45 |
+
candies and the teal, orange, and green colors
|
| 46 |
+
|
| 47 |
+
See `t4_validation.json` and `t4_setup_receipt.json` for raw evidence.
|
| 48 |
+
|
| 49 |
+
## Restore
|
| 50 |
+
|
| 51 |
+
```python
|
| 52 |
+
from huggingface_hub import hf_hub_download
|
| 53 |
+
import hashlib, tarfile
|
| 54 |
+
|
| 55 |
+
path = hf_hub_download(
|
| 56 |
+
repo_id="WJO/llama-cpp-b10355-colab-t4-sm75",
|
| 57 |
+
filename="llama-cpp-b10355-cuda128-sm75-colab.tar.gz",
|
| 58 |
+
revision="5738166d5979c9d39d0b3cacbaa1c5c2a3d44b16",
|
| 59 |
+
)
|
| 60 |
+
assert hashlib.sha256(open(path, "rb").read()).hexdigest() == (
|
| 61 |
+
"4469c6f18805f2cf2917df6bb8c0dbd37937344ab6c2e77c312cccdf7b145a9a"
|
| 62 |
+
)
|
| 63 |
+
with tarfile.open(path, "r:gz") as archive:
|
| 64 |
+
archive.extractall("/content", filter="data")
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
The runtime expects the CUDA and standard system libraries included in the
|
| 68 |
+
current Colab CUDA 12.8 image. Verify the archive and binary hashes before use.
|