Intel Arc B70 results: 32 T/s with UD-Q6_K, SYCL, MTP
UD-Q6_K MTP Prompt processing: ~750 T/s
UD-Q6_K MTP Prompt generation: 28 ~ 32 T/s
I'm running a single Asrock Intel Arc B70 in an old HP Z6 G4 Workstation.
PCI-E 3.0, ReBAR on.
OS is Fedora Server 44.
I'm using the "llama-toolboxes-cockpit" by kyuz0: https://github.com/kyuz0/llama-toolboxes-cockpit
It packs intel drivers and and a full llama.cpp setup into Podman containers. No further install needed.
BUT: Use Fedora. It won't work with Ubuntu. I tried.
I usually run the UD-Q4_M model at 200k context. Fits well into the 32GB. Token generation ~35 T/s.
Direct start command without llama-cockpit:
podman run --rm -it --name llama-cockpit-server --device /dev/dri --group-add keep-groups --security-opt seccomp=unconfined --security-opt label=disable --userns=keep-id -v /home/loe/LLM:/models:ro -p 8080:8080 docker.io/kyuz0/intel-b70-ai-toolboxes:sycl llama-server -m /models/Qwen3.8-27B/Qwen3.8-27B-Q4_K_M.gguf -c 200000 -ngl 999 --host 0.0.0.0 --port 8080 --load-mode none -fa 1 --cache-type-k q8_0 --cache-type-v q8_0 --jinja --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --spec-type draft-mtp --spec-draft-n-max 2 -np 1
Here's the llama-server while running UD-Q6_K in Hermes-Agent, reading some documents:
Thank you! What about higher MTP draft level, like 4 instead of 2?
Higher MTP draft level does practically nothing. It might improve for some coding tasks though.
The thing is:
As long as you only run one B70, you can use the llama-toolboxes-cockpit by kyuz0 with SYCL or Vulcan.
There is no limitation to the models you can run, it's normal llama.cpp.
The performance is as good as you could ever hope for a card with 608 GB/s memory bandwidth at 230 W.
It is just 2/3 the bandwidth of a RTX3090.
But to my knowledge:
For using multiple B70s, you'd need to run vLLM and model support limitations by Intel apply.
Overall:
Getting the B70 to run LLMs is way more involved than with a Geforce.
But it's 32GB. And cheap.
With models like Qwen3.8 it's perfectly usable for running hermes-agent and things like that.
Run Fedora Server and llama-toolboxes-cockpit for quick success.
Are you using the Dynamic 3.0 GGUF or the older Dynamic 2.0. I see a sharp drop like 20-40% in tokens/sec with these new quants and on top of that n-gram does not seem to be working well with these new 3.0 quants. I basically switched to LM-Studio's native Q4_K_M gguf file, and the speed is back up to 28t/s on my intel arc B60 and n-gram works too (the unsloth 3.0 drops this 18t/s, I know this is an issue with 3.0 because i had downloaded the 2.0 version before and did not notice any such performance drop).
I had to switch to LM-studio gguf because I had deleted the 2.0 while replacing it with the 3.0 version thinking it is an upgrade. And now I cannot find the older one online.
Update** Ignore this I found the old version from HF. the old version q4_K_M was not a UD model. The UD models is what was showing the significant performance drop, saw the same performance with the older UD_Q4_K_XL.
Asrock Intel Arc B70 cant be combined under win11? You need to run Linux crap for that to get multi-gpu setups working?
Asrock Intel Arc B70 cant be combined under win11? You need to run Linux crap for that to get multi-gpu setups working?
I'm not sure if there's some kind of support for that by now?
But running them on Win 11 doesn't make much sense. It wastes a lot of GPU memory and also:
These things are loud! B70s use server grade cooling.
I would advise against putting it into your living space. I've put mine into an old HP Z6 G4 workstation and set it up in my boiler room.
I use x3 3090 Turbo same blower cooler it's not loud at 80% power limit. I use HP Z8G4 Xeon x2 2630R. At 266K context I get 5tok/s no MTP at 50% power limit, need to service 2 GPU's
Given these llama.cpp bugs I do not want to use MPT:
- The Direct Verification Flaw (Issue #23335)
- Multi-Context Prompt Corruption (Issue #27296)
- Concurrent Prompt Leaking (Issue #28286)
- Text Repetition Loops (Issue #23577)
- Inter-Request State Contamination (Issue #26425)
I wonder what best way is to run Qwen3.8-27B-GGUF, I use Dirk Q8-XL version KV Q8 too. I use Win 11.