Medical AI Produces Visually Appealing Heatmaps That Often Misalign with Actual Diagnoses

Image: Illustrative image · .edu.edits · CC BY-SA 4.0 · Source

Researchers at IIIT Hyderabad evaluated four chest X-ray AI models generating heatmaps that highlight areas of interest. They found these visualizations frequently correspond more to statistical patterns than true pathological locations, raising concerns about clinical reliability.

A team from the International Institute of Information Technology Hyderabad (IIIT Hyderabad) conducted an audit of four specialized computer vision models designed for chest X-ray analysis. The study focused on how well the attention heatmaps—visual overlays indicating regions important to the AI’s diagnosis—matched real pathological locations noted by radiologists. The models evaluated were MAIRA-2, MedGemma-4B, LLaVA-Med-1.5, and LLaVA-1.5, tested across three public datasets and an additional sample of pneumothorax patients.

Their automatic comparison ranked MAIRA-2 highest in heatmap-to-pathology alignment, followed by MedGemma and then the two LLaVA versions. However, a critical finding was that, in typical cases, the models did not outperform simple anatomical templates based on average statistical distributions. Essentially, the AI tended to highlight areas statistically associated with certain pathologies rather than the exact locations found in individual X-rays.

Only in atypical cases—where actual findings significantly diverged from common patterns—did MAIRA-2 and MedGemma produce truly image-specific localization, pinpointing pathology more accurately. This limitation raises concerns about the dependability of such AI-generated heatmaps as clinical tools.

Further experiments involving 124 radiologists showed that experts interpret heatmap plausibility differently from automated metrics. Radiologists often found broader, clinically meaningful heatmaps more convincing, even if these did not precisely match rigid statistical boundaries.

The researchers especially emphasized the need to develop evaluation metrics incorporating anatomical priors and recommended involving radiologists closely in assessing AI tools. Their work highlights an important gap between AI model visual explanations and real-world clinical usefulness, cautioning against overreliance on visually appealing but potentially misleading AI outputs for medical diagnosis.