38 points JonSchneider 54 minutes ago 10 comments
kamranjon 41 minutes ago | parent
Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!
kadoban 14 minutes ago | parent
You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
spijdar 7 minutes ago | parent
I run Unsloth's UD-Q4_K_S on 20 GB of VRAM (RX 7900 XT) and I get ~90k tokens of context without quantizing KV cache. With 8-bit quantization, I get about a 134k token context window. That's with only one slot, but for me, it works pretty darn well, with 20-35 tok/s depending on how full that window is.
abraxas 40 minutes ago | parent
I'm not following the local mdoel scene too closely but this seems quite amazing. Is this able to be run on Apple silicon too?
kamranjon 39 minutes ago | parent
"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."
pizza234 14 minutes ago | parent
Their mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though!
Havoc 20 minutes ago | parent
Their first 27B bonsai was able to run on an iphone.
Aurornis 11 minutes ago | parent
These are small enough that you can run them entirely in the browser https://huggingface.co/spaces/webml-community/ternary-bonsai...
Remember to clear the downloaded weights afterward.
Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
outofpaper 4 minutes ago | parent
So you have some fun examples?
z2 7 minutes ago | parent
I'd love to see a Bonsai model start with a 100B+ parameter model and get that down to <30 GB. But maybe at that point we call it Topiary?