Reliable quality assessment of synthetic medical images remains one of the key unresolved barriers to the safe clinical deployment of generative models. Generative models for medical image synthesis have achieved impressive visual quality in recent years and are increasingly used in applications such as adaptive radiotherapy, treatment planning, and completion of incomplete imaging protocols. In these scenarios, synthetic images are no longer merely visual aids for clinicians, but are increasingly used as direct inputs to downstream diagnostic and therapeutic processes – from dosimetric calculations to organ-at-risk segmentation. Undetected generative errors, such as anatomical hallucinations, misplaced bone boundaries, or unrealistic textures, can have direct consequences for patient safety. Classical image quality assessment (IQA) metrics, such as PSNR and SSIM, were developed primarily for natural images and often fail to reflect the perceptual quality relevant to clinical use; studies have repeatedly shown that these metrics correlate poorly with expert ratings, particularly for synthetic medical images containing complex combinations of artifacts. Visual assessment by experienced specialists remains the gold standard, but is too costly, time-consuming, and poorly reproducible for systematic use. This gap motivates the development of automated, transparent, and explainable evaluation methods that can adequately approximate expert visual judgment.
This work presents a framework for automated prediction of visual quality scores for synthetic medical images from a set of objective IQA metrics, using ensemble regression models trained on consensus expert ratings. Synthetic images were obtained using the adversarial diffusion model SynDiff [1], which combines the strengths of generative adversarial networks (GANs) and denoising diffusion probabilistic models (DDPMs), operating on unpaired data without requiring aligned image pairs. Experiments were conducted on four cross-modality translation tasks – T1 → T2 and T2 → T1 on the BraTS2020 dataset, CBCT → CT on SynthRAD2023, and FLAIR → DIR on a private MR dataset of multiple sclerosis patients – covering a total of N = 287 subjects and ensuring diversity in anatomy, modality, and artifact type. A dedicated web application was developed for blind and standardized visual assessment of volumetric image pairs on a six-point Likert scale. Thirteen trained raters evaluated the quality of all synthesized volumes under a randomized blinded protocol; each rating was supported by a written justification and a localized annotation of the detected artifact. Consensus scores derived from these ratings served as the target variable for regression modeling. Eighteen IQA metrics were computed for each synthesized image: ten reference-based (PSNR, SSIM, MS-SSIM, IW-SSIM, FSIM, VSI, GMSD, HaarPSI, LPIPS and DISTS) and eight no-reference (NIQE, entropy, CPBD, BE, BEW, VL, MTV and JNB). Two separate ensemble regression models were trained using the AutoML framework Auto-Sklearn and evaluated with four-fold cross-validation.
Results demonstrated that automated prediction of visual quality from IQA metrics is feasible and robust across all four translation tasks. Predicted scores closely matched expert consensus ratings, typically within ± 0,5 points on the scale. The reference-based model achieved a mean R2 = 0,752 and mean absolute error MAE = 0,374, while the no-reference model achieved R2 = 0,589 and MAE = 0,478. Neither model showed statistically significant differences between predicted and actual score distributions (Wilcoxon signed-rank test, p = 0,86 and p = 0,64, respectively) nor systematic bias at the level of individual subsets. Visual scores varied substantially across tasks: CBCT → CT proved the most challenging (median 2,6), consistent with the high heterogeneity of CBCT acquisitions from different clinical centers and non-standardized Hounsfield unit calibration, while FLAIR → DIR was by far the easiest (median 4,2), as all images in that dataset originate from a single imaging system.
Explainability analysis using SHAP values and partial dependence plots revealed that predictions of the reference-based model are consistently driven by structure- and contrast-sensitive metrics, particularly IW-SSIM, PSNR, and MS-SSIM, while the no-reference model relies on statistical measures of global image properties – NIQE, entropy, and CPBD. IW-SSIM emerged as the most robust predictor across all cross-validation folds, consistent with its design: by locally weighting structural differences according to information content, it assigns greater importance to errors in anatomically rich regions that are genuinely critical for perceptual quality. Partial dependence plots further revealed non-monotonic behavior of standard SSIM, which becomes insensitive to subtle but clinically relevant artifacts at intermediate and high values and can even reward over-smoothed or locally consistent hallucinated content – highlighting the limitations of this metric when used in isolation. Taken together, the results confirm that no single metric suffices for reliable quality assessment of synthetic medical images, but that their combination within an ensemble regression framework calibrated on expert visual ratings enables transparent, reproducible, and clinically meaningful automated evaluation. The developed framework, together with the publicly available assessment application and annotation protocol, provides a practical foundation for standardized quality control of generative systems in medical imaging.
|
