37 points qtqtqt 1 hour ago 20 comments

rafram 50 minutes ago | parent

> We have 13 tests at this time.

Does GIMP have a very restrictive AI policy or something? I get not wanting to ship a gigantic pile of UI tests without thorough review, but basically any model released in the last six months would be able to write UI tests covering every single control in an afternoon.

samuria 41 minutes ago | parent

It’s frustrating to create tests with models, not sure if it’s everyone’s experience, but every time I asked it to create tests, it makes the tests conform to the sometimes broken code. Rather than calling out what the logic/code is wrong, it will make the tests conform to whatever is already broken.

Waterluvian 38 minutes ago | parent

It can’t magically know expectations. If the code is well documented or there’s decent associated documentation, I’ve found it has no issue writing tests.

forgotmypw17 36 minutes ago | parent

I recommend iterating on your prompts.

Prod your agent first until it can explain back to you what's wrong with the tests.

Then ask your agent to write a better prompt for writing tests (and save it to an .md file)

Then reference that .md file and iterate on the tests again. It may take a couple of tries, but I've had good results with this strategy.

Ended up with a whole prompting framework.

skydhash 4 minutes ago | parent

Prodding an agent vs doing interesting things with my time? What should I choose?

To be fair, some people like to prod. But, for me, I'd rather take hours reading manuals than do trial and error prodding for 30 minutes (even if I knew it would be that short). Yes it's not optimal in terms of time, but it's better for my soul and well being.

deskglass 40 minutes ago | parent

locusofself 22 minutes ago | parent

Not using AI to at least writes tests in this day and age is crazy work. I started a desktop application 2 months ago and it has almost 3,000 tests now, MANY of them UI tests.

timcambrant 9 minutes ago | parent

AI is a bit too excellent at writing tests. The number of tests doesn't really convey any information on test coverage. At least I've found that agents that naively improve codebases will write tests for pretty much anything, including irrelevant behavior that has little value in ensuring stability, security or correct behavior. It would probably help to run a test review stage with a strong model, where agents attempt to correct and consolidate tests to guard behavior rather than ensuring that theoretical code paths are followed.

hulitu 4 minutes ago | parent

> any model released in the last six months would be able to write UI tests covering every single control in an afternoon.

Microsoft also seems to use "models" to "test" its programs. Look how shitty they are.

joegibbs 21 minutes ago | parent

They made a blog post to announce that they’d written thirteen unit tests? Sure, there’s value in handmade software and all that but GIMP is not really a particularly excellent piece of software - if anything was to benefit from unleashing the agents it would be GIMP.

Jolter 11 minutes ago | parent

I’d go so far as to say that in general, adding tests for legacy software before a refactor or rewrite is the perfect time to put the agent to work.

hulitu 7 minutes ago | parent

> if anything was to benefit from unleashing the agents

AI can write tests now ? For its own software ? Can it evaluate the results ? Correctly ? /s

altmanaltman 5 minutes ago | parent

The blog article doesn't say they have written thirteen unit tests.

> *GIMP already has unit tests*. A unit test checks one small part of the code, for example how GIMP reads an XCF file. But our unit tests were not opening a real window. So it coult not tell us if a dialog opens, or if a menu item did its job.

> Note that GIMP *already had tests of GUI code*, usually interacting directly with the core code, and sometimes even simulating input devices’ events (pointer movements, clicks, drags…). But nothing can beat interacting with the real interface. Furthermore such legacy tests directly hijacking core code come with their own problems, such as breaking all too often (and not because they detected a bug), all while not being able to detect whole categories of bugs.

> UI tests check the program as a whole, in the same way as you use it.

qtqtqt 5 minutes ago | parent

They must have set up a reusable harness, and it's the most important part.

WD-42 5 minutes ago | parent

Did you read the article? These aren’t unit tests. They are UI tests which are pretty hard to do well with desktop apps.

qtqtqt 3 minutes ago | parent

Especially given it isn't as easy as using Cypress.

slopinthebag 30 seconds ago | parent

we forget, but before AI this would have been at least 6 months of work, at least 50 meetings, and at least 7 separate sprints.

dllu 11 minutes ago | parent

Funny how GIMP spawned the Gimp Toolkit, aka GTK, which is widely used in Linux, and yet GIMP's own UI seems to have long languished as a rather outdated design. Glad that they have tests, though, and hope they continue to improve it.

alexey-salmin 2 minutes ago | parent

Ah! I always thought it was Gnome Toolkit, didn't even bother to check. Thanks

Aozora7 2 minutes ago | parent

GIMP really doesn't even have a reason to exist at this point, and it's only going to fall further behind.