Yet another Hacker News

73 points softwaredoug 1 hour ago 9 comments

bellajbadr 1 hour ago | parent

Is this only about getting fix json output?

howunfortunate 56 minutes ago | parent

As an MLE who has been failing to get anyone interested in classifiers for many years, the hype around Jev makes me scream internally.

Yes, I get that a zero-shot classifier is more convenient than the traditional kind, it's very cool. Kind of. But then again plain LLMs have been perfectly cheap and serviceable as zero-shot classifiers for quite some time now, so again I'm back to my internal screaming.

copperx 41 minutes ago | parent

I want to share the rage. Can you expound on what makes you scream?

howunfortunate 29 minutes ago | parent

Idk, imagine you worked on Skype's B2B sales team for years and then COVID happens and Zoom blows up.

Is it a better thing? Yeah. Does it affect me in any tangible way? No.

But come on, really people? All you needed was like one tiny bell & whistle to take this from nothing to the hottest thing of all time?

firasd 34 minutes ago | parent

I think part of what made Jev catch on is that the API is like an if(...) or switch statement

People are just so used to the chat style APIs that they didn't even consider doing things like sending a bunch of emojis to a chat model and then asking for the optimal one in this context etc. Also chat models are pricier for the same behavior and can also output something random like a refusal

But yeah ironically I think in the initial breakthrough LLM paper on GPT-3 in 2020 some of the multiple choice questions were answered by comparing token probabilities of specific continuations rather than fill in the blank

howunfortunate 13 minutes ago | parent

You know, it's a good point.

Maybe I should have spent less time pitching to PMs and more time pitching ground up to devs, who have the right foundation to intuitively understand the usefulness.

jofzar 26 minutes ago | parent

> But then again plain LLMs have been perfectly cheap and serviceable as zero-shot classifiers for quite some time now, so again I'm back to my internal screaming.

It's the scale of "perfectly cheap", jev (specifically) is so dirt cheap and fast that you can throw it at things that should not be justifiable in the past and you barely have to do any work other then quick testing.

demibabs 16 minutes ago | parent

Is simply changing the temperature so that the model appears calibrated over a particular benchmark after the fact “allowed”? Feels p-hacking esque.

ursaguild 9 minutes ago | parent

This is really cool to see. Being able to play the token generation was awesome. Amazing job with breaking down how to think about these models. This made the idea of Jev/decision models really easy to grasp for me. The idea of calibrating the model was helpful. I thought this was a great overview.