Running this model on Apple Silicon Macs (unofficial port, MPS and MLX)

#1
by jonnyArt - opened

Thank you for releasing Karpo Decision. We wanted to run it on Macs and found that the first decide() call fails there, because the sliding-window layers use a flash-attention call that PyTorch only provides for CUDA (aten::_flash_attention_forward is not implemented for MPS or CPU).

We wrote a small unofficial port and are sharing it in case it helps other Mac users:

The 12B loads and answers on a Mac with this patch (about 0.95 s per short request as shipped, about 0.44 s in float16 batches of 16, on the machine we used). Its answers were not compared with published 12B results; the comparison we ran was on the 31B, where 1 of 1,606 top answers changed against your published results. Details and limits are in the README.

The port is not affiliated with or endorsed by MachinePulse; if you would like it named or described differently, tell us and we will change it.

Sign up or log in to comment