Povečanje ločljivosti magnetno resonančnih slik z generativnimi nevronskimi mrežami

KOSIN, MATIJA (author), Špiclin, Žiga (mentor)

V svetu zabavne industrije se v zadnjih letih naglo razvija področje super ločljivosti, pri katerem želimo iz nizko-ločljivostne slike slabše kakovosti pridobiti visoko-ločljivostno sliko višje kakovosti. Praktično uporabna vrednost je, na primer, da preko medmrežja prenašamo nizko-ločljivostno sliko in jo pri uporabniku v realnem času prevzorčimo v visoko-ločljivostno sliko. V svetu medicinskih slik to področje še ni tako zelo razvito, vendar odpira širok nabor potencialno uporabnih aplikacij. Če bi uspeli iz nizko-ločljivostne magnetno resonančne (MR) slike pridobiti kakovostno visoko-ločljivostno MR sliko, bi to skrajšalo čas zajema slik in posredno zmanjšalo stroške slikanja. Nadaljnje analize visoko-ločljivostnih slik lahko izboljšajo slikovno podprto diagnostiko in potek zdravljenja pacienta. Pri aplikacijah povečanja ločljivosti na medicinskih slikah je, v primerjavi z aplikacijami v zabavni industriji, kritičnega pomena vsebinska kakovost visoko-ločljivostne slike, na primer natančnost in smiselnost obnovljenih podrobnosti v sliki. Za povečanje ločljivosti ali super-ločljivostno nadvzorčenje slik smo uporabili konvolucijske nevronske mreže. Učili smo jih s pomočjo generativnih nasprotniških mrež (GAN), ki so v zadnjih letih postale zelo uspešna strategija učenja. Osnovni princip je hkratno učenje dveh mrež: generatorja G in diskriminatorja D. Prvi poskuša ustvariti čim bolj realno sliko, medtem ko drugi skuša ločiti med umetno ustvarjeno in dejansko sliko. Tako nasprotniško učenje je kot tekmovanje med dvema mrežama in ob ustreznem načrtovanju privede do zelo realističnih umetno ustvarjenih slik. V nalogi smo uporabili različne zasnove mrež generatorjev G, kot so EDSR, mDCSRN, MRDG in RWMAN, pri čemer je bil EDSR zasnovan za naravne slike. Kot diskriminatorske mreže D pa smo uporabili PPD, SRGAN in MedSRGAN. Sistematično smo z eksperimenti na 2D slikah preverili različne kombinacije generatorskih in diskriminatorskih mrež, strategije predobdelave slik in učenja, optimizirali kritične komponente kot je kriterijska funkcija za učenje G in D, in nastavitve hiperparametrov. Mreže smo implementirali tudi za uporabo na 3D slikah in preverili uporabnost postopkov povečanja ločljivosti pri razgradnji možganskih struktur v MR slikah glave. V eksperimentih na 2D rezinah MR slik glave smo ugotavljali vpliv na kakovost obnove, pri čemer smo za metrike kakovosti uporabili PSNR, SSIM in NRMSE. Postopki povečanja ločljivosti so v splošnem izboljšali vrednosti vseh metrik. Ugotovili smo, da je nujna uporaba realnih učnih podatkov, saj s sintetičnimi slikami nismo dosegli po kakovosti obnove primerljivih rezultatov. Pomembna je tudi normalizacija sivinskih vrednosti v vhodnih slikah; z naskokom najboljša je bila normalizacija na intervalu [0, 1]. Ugotovili smo tudi, da je normalizacija znotraj skritih slojev G nepotrebna, medtem ko je le-ta ključnega pomena v skritih slojih D. Pri učenju GAN je bistveno, da je učenje regularizirano z neko gradientno kaznijo. Kot zelo uspešno se je izkazalo tudi pred-učenje, kjer smo ločeno za G pred-nastavili uteži in jih nato uporabili pri GAN učenju. S tem smo pridobili na hitrosti učenja in se izognili lokalnim optimumom. Najbolj pomembna komponenta pri kakovosti povečanja ločljivosti slik je bila kriterijska funkcija in pa zasnova G, kjer z izboljšanimi rezultati izstopata arhitekturi EDSR in RWMAN, ki spadata v družino residualnih nevronskih mrež. Pri eksperimentih na 3D MR slikah glave smo ugotovili, da se, z izjemo SSIM, izboljšajo vse metrike kakovosti obnove. Preverili smo tudi koliko h kakovosti razgradnje možganskih struktur z avtomatsko metodo doprinese obnovljena visoko-ločljivostna slika v primerjavi z razgradnjo realno zajete nizko-ločljivostne slike. Rezultate smo preverili z metrikama DSC in SurfDSC, pri čemer je bila referenčna razgradnja opravljena na realno zajeti visoko-ločljivostni sliki. Ugotovili smo, da se obe metriki na obnovljeni visoko-ločljivostni sliki, primerjalno glede na razgradnjo nizko-ločljivostne slike, izboljšata za 2-4%. Preverili smo tudi statistično signifikanco rezultatov obeh metrik z Wilcoxon-ovim testom in ugotovili, da se razgradnja statistično signifikantno izboljša na skoraj vseh možganskih strukturah obnovljene slike v primerjavi z razgradnjo nizko-ločljivostne slike.

• Language: Slovenian
• Keywords: Super ločljivost, magnetno resonančno slikanje, nevronske mreže, generativno nasprotniško učenje, kakovost razgradnje.
• Work type: Master's thesis/paper
• Organization: FE - Faculty of Electrical Engineering
• Year: 2021
• PID: 20.500.12556/RUL-131998
• COBISS.SI-ID: 79977731
• Publication date in RUL: 08.10.2021

Secondary language

• Language: English
• Title: Super-resolution of magnetic resonance images using generative adversarial networks

Abstract

In recent years, many applications in the entertainment industry adopted the so-called super resolution approaches, which aim to reconstruct a high-resolution high quality image from input low-resolution low quality image. For instance, the low-resolution image may be quickly transferred through the network and, at the user side, reconstructed into its high-resolution version in real time. In the domain of medical images this area is not as adopted, but it bears potential for a wide set of applications. Namely, if we could reconstruct a high-resolution magnetic resonance (MR) image from low-resolution MR image acquisition, it would reduce the time and associated costs of MR imaging. Furthemore, the adoption of such high-resolution images could improve image-based diagnostics and the overall patient treatment. In applications of super resolution on medical images, compared to those used in the entertainment industry, the quality of the resulting image content needs to be reconstructed in a very precise and sensible manner for practical use. Our approach to image super resolution was based on the use of convolutional neural networks. The networks were designed as generative adversarial networks (GANs), which represent a very efficacious training strategy. The basic principle is to simultaneously train two networks: a generator G and discriminator D. The first aims to generate as realistic images as possible, while the second aims to classify between the real and artificially generated images. This type of adversarial learning is mimicking a competition between the two networks and was shown to produce very realistic images. In this work, we applied several generator architectures, such as EDSR, mDCSRN, MRDG and RWMAN, where the EDSR was designed for natural images, and several discriminator architectures such as PPD, SRGAN and MedSRGAN. We have systematically evaluated combinations of generator and discriminator architectures, learning strategies and image preprocessing approaches, optimized critical components like the loss function of G and D networks, and tuned hyperparameters. Next, the networks were extended for use with 3D images and the practical value of super resolution for improving brain MR structure segmentation was verified. In the experiments involving 2D images, obtained as slices of 3D MR images, we verified the impact of super resolution on the quality of reconstruction based on PSNR, SSIM and NRMSE quality metrics. The super resolution approaches generally improved all the metrics. We found that it is very important to train with real images, because we could not reproduce similar results with synthetically generated low- and high-resolution MR images. Next, image intensity normalization was crucial, with [0, 1] interval normalization giving the best results. Applying normalization in hidden layers of G was found to be unnecessary, whereas it was crucial in D. In GAN training the use of regularization with some sort of gradient penalty gave best results. Using pre-trained G networks as initial network weights in the entry point of GAN training showed very good quality super resolution results. Namely, avoiding the local optima shortened the training time. By far the most important component was the loss function, which critically determined the quality of our reconstructed high-resolution image. We also noticed the difference with the applied architectures of network G, where EDSR and RWMAN, both belonging to a family of residual networks, gave best results. In the experiments on 3D MR images, we observed improved values of the PSNR and NRMSE quality metrics, but not SSIM. Next, we compared the automated brain segmentations on reconstructed high-resolution and real low resolution image. The comparison was based on the DSC and SurfDSC segmentation quality metrics, where the segmentation of real high-resolution image was used as a reference. We found that in general both metrics improved with the use of reconstructed high-resolution image by 2-4%. The Wilcoxon test showed that, compared to the segmentation of low-resolution image, the segmentation of the reconstructed high-resolution MR image was significantly better for nearly all brain structures.

• Keywords: Super resolution, magnetic resonance imaging, neural networks, generative adversarial networks, quality of segmentation.