YHK - yet another hacker news clone

DishwasherBench – Can Frontier Models Stack a Dishwasher?

53 points mlukewizard 2 hours ago 25 comments

mlukewizard 2 hours ago | parent

Inspired by Joma Tech's new YouTube video

Stacked loads are run against a dishwasher simulation calibrated to my own Bosch dishwasher

Playable by both humans and AI

Currently a WIP, simulations need refining before running a full benchmark, but keen to hear some feedback!

nastrofa 1 hour ago | parent

This is so cool!

Imagine creating real simulated env for every physical task and using RL to train robots. That will be a game changer.

mlukewizard 1 hour ago | parent

yeaaa perhaps this is the future

embedding-shape 37 minutes ago | parent

It was the future many years ago, today it's standard practice :)

edh649 1 hour ago | parent

This is already pretty well established in the ML robotics world! e.g. this paper from 2019: https://publica-rest.fraunhofer.de/server/api/core/bitstream... and also for waymo in 2016: https://waymo.com/blog/2016/12/reliving-past-how-these-data-...

_puk 1 hour ago | parent

The obvious fork is "can frontier models stack a dishwasher to my wife's exacting standards".

I can't. So that would surely be proof of AGI.

jojogeo 1 hour ago | parent

Never have truer words been spoken, I too score poorly on DishwasherBench wife mode. Drives her up the wall.

teekert 1 hour ago | parent

At our place it's opposite, I guess when you do it often enough you get a preferred method though a relentless optimization loop, which in the end ensures it's all clean, the plates are well separated only in 1 particular order, the glasses are placed to the left so all small pans fit at the right, deep plates are placed half over cutlery rack so they are supported and more fit in, small plates in front so all lids fit before them... etc etc etc.

stoneman24 1 hour ago | parent

Yep, I have a more optimised model of stacking than my wife’s ideas. This has lead to “discussions” (still at a friendly level) and the dish washer being placed in my list of responsibilities.

I think optimal configuration is very much a NP complete problem.

mvdwoord 1 hour ago | parent

Try explaining this to a raccoon.. on meth.

I did, once ;)

mrweasel 1 hour ago | parent

"In every partnership, there is a person who stacks the dishwasher like a Scandinavian architect and a person who stacks the dishwasher like a raccoon on meth." - Someone online.

At home I'm the architect, my wife's the raccoon. I don't actually believe that you can stack the dishwasher wrong as such, but you can make it hard for yourself to unload, if you don't sort the items correctly.

fxwin 1 hour ago | parent

> I don't actually believe that you can stack the dishwasher wrong as such

There are definitely more and less correct ways, but personally i think it's "wrong" to load it in a way that leaves the arms blocked or (not) tilt pieces that get excessive water pooling on top

freecodeio 51 minutes ago | parent

shouldn't stacking like a raccoon on meth be the most optimal way though? since if they're too well stacked you risk water being unable to enter tight places

embedding-shape 38 minutes ago | parent

> I don't actually believe that you can stack the dishwasher wrong as such

You most definitely can, they're not magic! The water must hit the right spots and obviously stacking three plates without any space between leads to stuff not getting properly cleaned.

If you try to teach a kid to put the dishwasher you'll learn a ton of wrong ways of doing it! :)

notrealyme123 5 minutes ago | parent

I think putting cups with the opening facing upward is very close to "technically correct, but very wrong"

ce4 29 minutes ago | parent

You're already loading it perfectly according to The Oatmeal :-D

https://theoatmeal.com/comics/dishwasher

AlecSchueler 28 minutes ago | parent

Are our wives' standards impossible to reach or are our husbands weaponising incompetence?

judetechdevs 53 minutes ago | parent

Both models’ loads came out 84-100% clean, so is the score mostly just how much they fit?

mlukewizard 45 minutes ago | parent

Totally the opposite, there's a detailed dishwasher simulation under the hood: ray tracing, splash simulation, soak, heat, enzyme action

I calibrated it against my actual dishwasher by doing some test runs with baked egg

judofyr 14 minutes ago | parent

As we know from physics, even the most sophisticated mechanisms (e.g. friction) often ends up having a very simple model (proportional to the normal force).

Sooo, the score is maybe mostly just how much they fit?

dist-epoch 48 minutes ago | parent

The real story of how Navier Stokes was solved invoices a dishwasher:

https://www.youtube.com/watch?v=ofdMfljL360

kolinko 30 minutes ago | parent

Came here to post this :)

comrade1234 46 minutes ago | parent

Years ago after my wife filled the dishwasher I commented that it looked like a 12-year-old filled it. I said it as a joke but it really hurt her and I still feel bad about it. That said, it no longer looks like that when she fills it...

walrus01 30 minutes ago | parent

For added difficulty, have it 'stack' a 3D rendered volumetric plastic model of a pelican riding a bicycle, like if it were putting a child's toy in the dishwasher.