YHK - yet another hacker news clone

That being said, I'm sure a frontier LLM could make some improvements to your CNN.

It's comparing a fine-tuned CNN to general purpose models on a tiny dataset.

Just use the LLM to make a thing that's actually good at what you need it to do.

Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version

Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.

Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).

I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.

>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.

This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.

You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.

Llms have their applications. Other models still do.

This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).

I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?

It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.

General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.

However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?

The problem is the narrow models are always super fragile. You change the lighting, zoom level, skin tone, or give it a picture of a cat (or more dangerously a misaligned image) instead of a skin lesion, and it breaks completely.

None of this is new, it’s why narrow classifiers peaked almost 10 years ago and why large general transformer models took over. If you have a task where the data distribution is always identical and has no long tail or outliers then inception or resnet or whatever is great. Problem is these are vanishingly rare.

I'm surprised he's surprised.

> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.

Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).