Yet another Hacker News

Janas: Run LLMs larger than RAM by streaming experts from SSD

31 points khalidkhair 3 hours ago 13 comments

3eb7988a1663 1 hour ago | parent

If anyone needs to see what 62 tokens/second looks like: https://token.qusai.pro/?speed=62

Which seems sufficient for mortals who are not running gas town, but want a local model.

kolinko 52 minutes ago | parent

60 toks/sec may be ok for pure chat or token generation, but not for token preprocessing nor any sort of agentic work.

serf 17 minutes ago | parent

>.. nor any sort of agentic work.

may I ask why?

oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?

brcmthrowaway 48 minutes ago | parent

Good luck when you need to prefill context..

khalidkhair 22 minutes ago | parent

Not sure if the measurement method used for this is actually valid or reliable enough, but for reference, Codex 6.1 Sol Standard runs at 19.9 tok/s according to a benchmark I recently saw: https://www.reddit.com/r/codex/s/b4mUszSOv1

InsidiousWombat 1 hour ago | parent

I wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes.

Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")

kadoban 1 hour ago | parent

I think what we're going to see first is essentially gpus with nvme (or at least flash) bolted on, not sure if that's also what you're saying or not.

KerrAvon 1 hour ago | parent

There are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.

purplecats 1 hour ago | parent

would this work on a virtual disk, say w s3 or dropbox

doublerabbit 47 minutes ago | parent

Not online services. The overhead of TCP and latency I imagine would really lag performance. A SAN over SFP, maybe.

trollbridge 27 minutes ago | parent

Sure, it’ll just be absurdly slow.

Maybe next you can run an LLM off of some old MFM drives

kadoban 14 minutes ago | parent

Why are the models being used so old (in llm years anyway). Why no Qwen 3.8 ?

droidjj 6 minutes ago | parent

Based on how vibe-coded this whole thing is, I’m guessing the newer models weren’t in the training data and the agent just picked what it knew existed.