Old CNN beats GPT-6 and Claude at skin cancer ID
Original: A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
Why This Matters
Specialized fine-tuned models still outperform general LLMs on high-stakes clinical image tasks.
A dermatologist tested GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-confirmed skin lesions; both scored ~42% accuracy. A fine-tuned 2015-era Inception v3 CNN scored 62%, correctly flagging melanomas the LLMs missed.
Dr. Magnus Lynch, a dermatologist, ran a structured 200-lesion diagnostic test using MILK10k, a 2025 public dataset from the International Skin Imaging Collaboration. Each case included a clinical photo and a dermoscopic image, all histopathology-confirmed. The split: 100 lesions requiring treatment (melanomas, BCCs, SCCs, actinic keratoses) and 100 safe to discharge (moles and seborrhoeic keratoses). Both GPT-6 Astra and Claude Fable 5.1 were given identical prompts at default settings and asked to pick from six diagnoses or say "unsure," provide confidence levels, and recommend discharge or not. They scored 84 and 85 correct respectively—about 42%. A fine-tuned Inception v3 CNN, Google's 2015 architecture, retrained on 68,011 ISIC Archive images on a single H100 GPU in roughly 20 minutes, scored 125 correct (62%). The CNN never saw the test images during training. In one striking case, both LLMs called an invasive melanoma under 0.8mm thick a harmless seborrhoeic keratosis and recommended discharge. The CNN correctly flagged it as melanoma from the dermoscopic view. Lynch notes the CNN is "deliberately not state of the art"—the point being that task-specific fine-tuning on domain data still outperforms general-purpose frontier models on narrow clinical classification.