YHK - yet another hacker news clone

Jev in 25 Lines of Python

90 points bashbjorn 1 hour ago 32 comments

heaney-555 1 hour ago | parent

Latency and compute comparison needed.

thephyber 44 minutes ago | parent

Is benchmarking Jev still a ToS violation?

ricardobeat 41 minutes ago | parent

Was it? That would make it unusable in any corporate setting.

dhsysusbsjsi 1 hour ago | parent

Whilst I do like reading these things for technical know how, I can sympathise with the creator of jev who now presumably has to apply an order of magnitude effort to explain why the 100 smaller things done better than this add up to a much better product.

jpnc 53 minutes ago | parent

Replace 'explain' with 'sell'. Don't forget that it's a gold rush. There's no reason to sympathize with corporations in their rush for the slice of the pie.

ramon156 16 minutes ago | parent

IT's the infamous "OneDrive in 10 lines of code (SFTP)"

While technically correct, it's not the same thing

no-name-here 1 hour ago | parent

Beyond the missing latency and compute comparisons that Heaney commenter mentioned, also nothing about its error rate compared to Jev (nor if it even always outputs in a format the app can parse, not sure how solved that is).

But then at the end it says it’s parody. Maybe HN title should say it’s a joke.

est 53 minutes ago | parent

latency and compute comparisons highly depends on your local setup.

you can swith to a better model for lower error rate.

ricardobeat 45 minutes ago | parent

Which massively slows down the output. Doing this with Qwen 9B already takes you into seconds per answer territory, and Jev is supposedly frontier level intelligence.

sigmoid10 56 minutes ago | parent

Going directly for the logprobs is always icky when you use a chat model as base, because they are trained to write prose as output. So your "choice" tokens and thus their probabilities might get diluted in whatever else it wanted to say. If you have to do it in the same way as this post, at least add clear system instructions and a carefully worded beginning to the assistant output section of the prompt to lower the chances of it wandering off immediately.

I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).

_flux 46 minutes ago | parent

Seems like all normal english words could risk the same, so would using short but random strings be even better?

Actually to me it sounds it could be benchmarked if this kind of effect exists in the first place.

sigmoid10 41 minutes ago | parent

Best option would be reasoning + clear system instructions + constrained output. That is, if you have to use a chat model. Which works well enough to be sure, but hey I haven't tried raising millions of dollars when I did that 3 years ago. But perhaps I was the stupid one.

_davide_ 45 minutes ago | parent

Agreed, it's a real issue, but it can probably be vastly reduced by having the schema in the system prompt and by giving the model an expectation of a fixed value: no decent modern would pick a prose ligament over a provided value.

To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.

wongarsu 30 minutes ago | parent

Sure, you can fix that in a couple lines. Then a couple more lines for evaluating multiple questions on the same answer in parallel. Then a couple more lines for the confidence score (which is trivial to compute from all we have, but missing regardless). Then a harness to fine-tune an existing model to perform better on this specific task, and a collection of training data to use for that

I think we can all agree that Jev is not rocket science. It's a good idea executed well, with marketing that might have been a tad too bold

porridgeraisin 9 minutes ago | parent

The confidence score is not trivial to compute. That is the whole point of the model. Even if you are using a proper scoring function such as NLL, it is not enough to ensure calibration in deep nets. So you have to do good post training to ensure it. These are all known techniques, but they are far from trivial, especially on large scale datasets.

ainch 42 minutes ago | parent

In my experience as well using logprobs to try to quantify uncertainty, LLMs are a poor fit. Neural nets in general struggle with 'calibration' --- ie. if a prediction is truly 50/50, neural nets are often prone to predicting overconfidently [0].

I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.

0: https://arxiv.org/pdf/1706.04599

nautilus50 42 minutes ago | parent

+1, llama.cpp has a --grammar parameter which you can pass a BNF style grammar file to constrain generation. It can be used in Python llama.cpp wrapper

https://til.simonwillison.net/llms/llama-cpp-python-grammars

porridgeraisin 10 minutes ago | parent

Yes. But even then, the probabilities are not calibrated. In jev/laya, they are (well, relatively anyways).

teaonly 55 minutes ago | parent

The principle is this.

ricardobeat 47 minutes ago | parent

Now, can you do it in <200ms for 45 questions at once, have 0% malformed output, and any kind of meaningful benchmark? We’ll wait!

_davide_ 24 minutes ago | parent

> <200ms for 45 questions at once

Considering your own question length: ~120 characters x 45 divided by 4.1 ~= 1317 tokens.

So question processing at 5.5k PP(around the actual PP speed of GPT5.6 Sol) it would take around ~0.24 seconds + the context processing.

Computing the output should be around ~20ms (at 50 tok/s), computing 45 tokens in parallel.

> have 0% malformed output

Pretty trivial; only the allowed output is selectable :)

So, I keep repeating myself: Jev was a low-hanging fruit all along; no one cared, and probably no one will in a few weeks?

onion2k 47 minutes ago | parent

It's fast.

If you're comparing with something, you need to state 'fast' in relative terms. Jev is definitely fast, and if this Python takes the same time to get a decision then it's also fast. If it's 100* slower than Jev though, you shouldn't be calling it 'fast', because relatively speaking it's really, really slow.

_davide_ 39 minutes ago | parent

By design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.

So, fast in the LLM space and comparable with Jev.

brap 33 minutes ago | parent

What I don’t understand is, why would you not want “reasoning” in a classifier?

Speed and cost are obvious reasons, but isn’t this a tradeoff?

ph1l337 10 minutes ago | parent

not sure if true, but if you look at laya they use BERT type models. If jev is also using a BERT-type model it is autoregressive and therefore can't reason in the way that GPT-type models can. However, you get the advantage of being able to attend in both directions.

cupofjoakim 27 minutes ago | parent

I wonder if this could be a good stepping stone to write a local prompt router to optimise what model get what prompt. I.e. if the prompt is just a lookup, send it to haiku, if it's reasoning, send it to opus and if it's implementation send it to sonnet.

v18a 20 minutes ago | parent

I was thinking the same. Haven't tried it out.

iLoveOncall 25 minutes ago | parent

Nothing I hate more than bullshit articles claiming X in Y lines of code, only to use libraries abstracting hundreds of thousands of lines of code.

shawabawa3 24 minutes ago | parent

strong "You can build dropbox quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" vibes

You have built something like jev but not jev (for starters, the output of what you've built will be absolutely worthless, the whole reason Jev is getting so much hype is because the output is good enough)