YHK - yet another hacker news clone

Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

46 points albinstman 3 hours ago 17 comments

bozdemir 1 hour ago | parent

Model list is a bit limited, and whole harness affects the speed, so the libs used in harness and their versions might be useful to show for each individual model setup.

albinstman 1 hour ago | parent

I’ll look into adding harnesses. The benchmark sources usually don’t contain harness so the problem is limited data. What models would you like to see?

cyanydeez 51 minutes ago | parent

He's exagerating, I hope. The only bottleneck a harness attachs is pre-context fill and whatever tooling is installed. A proper harness reduces these to negligable startup costs.

Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.

bozdemir 19 minutes ago | parent

Exagerating? I setup llm endpoints on corporate level in a company on H200 gpus, that serves hundreds of software engineers, I have observed differences, even on different versions of nvidia drivers. I shared my day to day experience. It is more and more important if you have very limited VRAM like 24gb on a 4090 RTX desktop or RTX 5090 Laptop gpu. I set up and use llms on my personal devices also, when I can't use cloud models on classified company materials. Here is an example problem what I daily report and deal with.

https://github.com/ggml-org/llama.cpp/issues/27330

jimnotgym 15 minutes ago | parent

I can believe different drivers make a difference, but that wasn't the point under debate

bozdemir 17 minutes ago | parent

For example you might add Strata & Qwen 3.8 Flash Next with some quantizations. Nowadays it is a go to selection and very sought after MoE model.

pil0u 1 hour ago | parent

I don't know what basis these numbers are estimated on, but my M2 Ultra 128GB runs almost twice as fast as what's reported for the M3 Ultra 512GB.

I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.

albinstman 1 hour ago | parent

Thank you for the feedback. What model/quant did you test so I can look in it?

hbogert 1 hour ago | parent

Shame they are not including old crap like my RX580 which is doing some impressive stuff with Qwen 9B Models.

internet_points 31 minutes ago | parent

that's 8G vram? you're using it for coding?

dotancohen 31 minutes ago | parent

Tell us more about what your RX580 can do with the small Qwen models. What workflows are you doing, and what kind of performance are you seeing?

friendly_chap 55 minutes ago | parent

This is a great idea but I'm far less concerned about output token speed and way more about the ability to parallelize prefill - for agentic coding that's pretty much way more important than output speed.

Perenti 40 minutes ago | parent

Doesn't provide for hybrid setups RTX5070 + RTX5080 for example.

hizlikovboy27 37 minutes ago | parent

around this time, i was thinking about running a local LLM for the company i work for. i'm trying to figure out what kind of hardware i'd need and what level of performance i could expect. i tried a few things on your website but how much do the results reflect real world usage?

in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?

good job.

gitowiec 15 minutes ago | parent

I would like to see in such simulators:

- multi GPU configuration (hetero and homo)

- PCIe gen chooser (3.0/4.0/5.0)

- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)

- is PCIe bifurcated or CPU native (threadripper pro for example)

kristopolous 3 minutes ago | parent

just go rent one for a few minutes and try it out ... it'll cost you well under $0.50 if you're just trying to get a feel for if it's right for you.