21 points tkgally 1 hour ago 2 comments
In November and December 2025, inspired by Simon Willison’s pelican-riding-a-bicycle benchmark, I had some then-current LLMs create SVGs from thirty similar prompts, such as “Generate an SVG of an octopus operating a pipe organ.” Simon mentioned that experiment on his blog [1].
Nine months have passed and much stronger models have been released, so I tried the experiment again today. The linked site shows the results.
Running ten of the prompts through six models at OpenRouter cost about twenty dollars, so I stopped there for now.
steinvakt2 29 minutes ago | parent
Feels like google has a different training set than the others?
samayashar 14 minutes ago | parent
All models are pretty good now at generating these images. Back in the day, I remember experimenting with the pelican images and most of the models couldn't align the legs with the wheels. Right now as well, GPT messed up an octopus leg by originating it through the instrument rather than the octopus itself.
I think that intertwining two entities (living/non-living) is still challenging but overall they're pretty sound.