YHK - yet another hacker news clone

Jev introduces a new shape of LLM

25 points benwerd 3 hours ago 5 comments

aszen 1 hour ago | parent

About jev being a black box and the potential for bias, I think it boils down to what questions you are asking the model.

Broad questions like Is this resume good / score this city will ofcourse be biased but I think jev encourages more granular focused questions like Score this candidates Python experience / Rate this city for its food which then allows you to introduce your own biases in which questions you ask and how you combine their answers.

In this way I think jev like models can be easier to reason about for critical decisions.

tipsytoad 35 minutes ago | parent

I’m not sure I understand the hype around this model. Isn’t this just an llm with a chat template, with the options prefix cached?

  <option>option A</option> <option>option B</option><endofoptions>userprompt<eos>
Then the llm is constrained to a few special tokens indicating the possibilities? e.g. <option1> <option2>

rene_d 27 minutes ago | parent

To get calibrated probabilities sounds like a very good feature, if they are indeed well calibrated.

And in my experiments even Qwen 3.8 has a hard time to consistenly conform to a schema, requiring retries, JSON cleanup etc, so to have a model of similar quality (SemIf et al) that simply cannot deviate from the schema by construction could be very helpful.

But I still need to experiment with either Jev/SemIf myself.

faragon 29 minutes ago | parent

Is it 100% deterministic?

hresvelgr 24 minutes ago | parent

I think "Black boxes are back in fashion" is missing the point. I think LLMs are still largely black boxes, and I don't think chain of thought is representative of any degree of inner machination. Asking it questions to justify itself is at best a facsimile, and for the most part it's useful, but it's fundamentally a facsimile.

Where I understand Jev to be a significant jump is that afaik the confidence scoring is actually derived from the normalised probabilities, and not a continuation in a chain of prediction masquerading as "confidence."