Generative adversarial model based augmentation of angiographic images for improved detection of intracranial aneurysms

KRIŽAJ, DAVID (author), Špiclin, Žiga (mentor)

The risks associated with intracranial aneurysms have motivated research in the past and still continue to do so today, this thesis being no exception. With a reported incidence of around 3.2% (in the absence of risk factors) and with the associated risk of rupture rising steeply in correlation with aneurysm size, the value of early detection, evaluation and potential treatments is crucial. This thesis focuses on computer-aided detection of intracranial aneurysms, which is a computer vision task. Computer vision as a field of research has progressed immensely in the past 10 years or so as a result of the availability and development of deep convolutional neural networks and associated hardware technology. It has become the default approach in most detection and segmentation tasks where non-trivial analytical approaches were employed prior to this era. However, as machine learning approaches require adequately sized training data, the same methods may excel at common problems, where learning sets are easily obtainable and plentiful, but show poor performance in niche fields where training data is scarce. Multiple approaches can be taken in order to tackle this issue, such as fine-tuning the existing models and/or augmentation of available data through the deep learning based creation of artificial data. The lack of publicly available medical image datasets is a problem that must be frequently tackled when applying deep learning machine vision principles in medical image analysis. This is due to multiple reasons such as policies on preserving patient privacy, cost of expert's work when annotating/manually segmenting images and a general lack of initiative to produce large public datasets for scientific use just to name a few. We try to address this issue by proposing a problem-specific dataset augmentation technique, which is itself based on generative convolutional neural networks. Our proposal was to take a scarce dataset of 3D magnetic resonance images with corresponding vasculature and aneurysm segmentations, create several 2D projections at locations of interest from multiple view and train a general adversarial network (GAN) on that dataset so that the network would learn to impaint a randomly shaped and textured aneursym into the regions of healthy vasculature. This would allow us to transform a large number of healthy datasets into datasets depicting the pathology of interest, which would as such be suitable for training a standard U-net based detector network. We also opted for a U-net detector variation that employed a combination of the Tversky index and focal loss concepts in order to increase the performance for highly unbalanced datasets, a property inherent to our aneurysm datasets, where we are normally interested in a very small portion of an input image. Our GAN architecture of choice was an of the shelf Cycle-GAN with some customized cost function terms. Learning adversarial networks for a given problem has proven to be very challenging. In the case of the generative network, the impainted aneurysms were meaningful, but their fusion with the vessels was not optimal in all cases. Therefore, we applied additional processing of the input grayscale image of the aneurysm-free vessel and the synthetically generated aneurysm image to ultimately produce useful images depicting aneurysms and vessels for the synthetic training set. Concretely, the problem turned out to be that the images produced in this way varied greatly depending on the extracted patch of the vascular images at the input to the network. With standard morphological operations, Poisson fusion of the input and the synthetic images and manual visual evaluation, we managed to discard all textural and anatomical irregularities and thus obtain the final synthetic set. The process of creating the final set of training images limited the ability to scale the size of the training set due to involved manual inspection. We limited ourselves to just over 2000 synthetic images versus around 6000 real projections and trained our detector network on combinations of both synthetic and real images at various ratios to determine the contribution of synthetic images to the learning convergence and performance of the aneurysm detector model. From the convergence curves of the loss functions during learning and the results of the evaluation of the detector model, we concluded that learning on only synthetic images is insufficient for adequate detector learning. From the results of further experiments, we concluded that the addition of synthetic images to the real ones in the training set did not degrade the performance of the detector and in some cases even improved the convergence and generalization of the detector model.

• Language: English
• Keywords: aneurysm detection, computer vision, convolutional neural networks, generative adversarial networks, dataset augmentation
• Work type: Master's thesis/paper
• Organization: FE - Faculty of Electrical Engineering
• Year: 2023
• PID: 20.500.12556/RUL-144387
• COBISS.SI-ID: 142485251
• Publication date in RUL: 17.02.2023

Secondary language

• Language: Slovenian
• Title: Bogatenje angiografskih slik z generativnimi nasprotniškimi modeli za izboljšano zaznavanje intrakranialnih anevrizem

Abstract

Možganske anevrizme in z njimi povezana zdravstvena tveganja so v preteklosti dali razloge in motivacijo za številne raziskave in pričujoče delo v tem pogledu ni izjema. Naj zadošča dejstvo, da je v obči populaciji že brez prisotnosti dejavnikov tveganja incidenca anevrizem 3.2%, tveganje za pok anevrizme pa strmo narašča z večanjem njihove velikosti. Zato je razvoj metod in procesov za čim prejšnje odkritje in čim boljše vrednotenje te patologije ključnega pomena. To delo se osredotoča na strojno podprto detekcijo možganskih anevrizem, torej problem s področja strojnega vida. Slednje je bilo v zadnjih desetih letih zaznamovano z izjemnimi preboji na račun razvoja tehnologij konvolucijskih nevronskih mrež in strojne opreme, ki jih je naredila v splošnem mnogo dostopnejše za uporabo številnim posameznikom in organizacijam. Strojni vid na osnovi nevronskih mrež se je uveljavil kot primarni pristop in osnovno merilo uspešnosti za številne probleme, kjer so se prej uporabljale nalogam prilagojene analitične rešitve. Skupna lastnost pristopov s konvolucijskimi nevronskimi mrežami je seveda potreba po primerno velikih in označenih učnih množicah. Mreža z izbrano arhitekturo lahko izjemno dobro reši problme, kjer se število učnih slik zadostno, medtem kot taista mreža lahko da neuporabne rezultate na nišnih področjih, kjer je učnih (in testnih) slik malo. Obstaja več načinov spopadanja s tem problemom, med drugim prenos uteži modela za sorodni problem in/ali doučenje modelov in umetno bogatenje obstoječe učne množice. Pri analizi medicinskih slik je pomanjkanje ustreznih, tudi javno dostopnih baz pogost in pereč problem. Javni dostop do kakršnih koli zdravstvenih podatkov običajno ščiti zakonodaja, ki ureja pacientove pravice do zasebnosti in varovanja podatkov, potrebna so predhodna soglasja in postopki anonimizacije. Za probleme detekcije struktur zanimanja v medicinskih slikah vedno potrebujemo slikam pridružene oznake, ki so običajno določene ročno s strani izkušenih radiologov, kar je časovno zahteven in tudi drag proces. K reševanju tega problema smo pristopili z metodo bogatenja podatkov, ki sama temelji na rabi generativnih konvolucijskih nevronskih mrež. Namenili smo se razširiti maloštevilno učno množico tridimenzionalnih magnetno resonančnih slik možganskega ožilja s pripadajočimi razgradnjami žilja in anevrizem z uporabo generativnih nasprotniških modelov. Uporabljali smo dvodimenzionalne projekcije omenjenih slik, učenje mreže pa je bilo zastavljen tako, da se je generativna nevronska mreža učila prevajati izseke slik zdravega ožilja v izseke z dodanimi anevrizmami. Naš cilj je bil omogočiti avtomatsko ustvarjanje velikega števila sintetičnih učnih slik, s katerimi bi pozneje lahko naučili konvolucijsko nevronsko mrežo za detekcijo anevrizem. Generativni nasprotniški model je imel arhitekturo uveljavljene Cycle-GAN mreže, ki smo jo nadgradili z dodatnimi kazenskimi členi v izgubni funkciji. Za detektor smo uporabili arhitekturo U-mreže s Focal-Tversky izgubno funkcijo, pri čemer je izbiro arhitekture motiviralo izrazito neravnovesje med številom vzorcev anevrizme in ozadja slike, in vrednotili kvaliteto detekcije intrakranialnih anevrizem v dvodimenzionalnih projekcijah. Takšen pristop se je izkazal za smiselnega pri iskanju anevrizem, ki predstavljajo le majhen del celotne vaskulature. Učenje nasprotniških mrež za dani problem se je izkazalo za zelo zahtevno. Pri generativni mreži so bile vrisane anevrizme smiselne, njihovo zlivanje z žiljem pa ni bilo v vseh primerih optimalno. Zato smo uporabili dodatno obdelavo vhodne sivinske slike z ožiljem brez anevrizme in sintentično generirane slike z anevrizmo in ožiljem, da smo na koncu proizvedli uporabne slike z vrisanimi anevrizmami za sintetično učno množico. Konkretno se je za težavo izkazalo, da so proizvedene slike močno variirale v odvisnosti od izseka slike ožilja na vhodu v mrežo. S standardnimi morfološkimi operacijami, Poissonovim zlivanjem vhodne in sintentične slike in vizualnim vrednotenjem smo uspeli zavreči vse slike s teksturnimi in anatomskimi nepravilnostmi v končni sintetični množici. Proces ustvarjanja končne množice učnih slik je zaradi ročnega vrednotenja omejil zmožnosti skaliranja velikosti učne množice. Omejili smo se na nekaj več kot 2000 sintetičnih slik proti okoli 6000 dejanskim projekcijam in našo detektorsko mrežo učili na kombinacijah tako sintentičnih kot dejanskih slik v različnih razmerjih, da bi ugotovili doprinos sintetičnih slik k učenju in zmogljivosti modela detektorja anevrizem. Iz potekov izgubnih funkcij med učenjem in rezultatov vrednotenja modela detektorja smo ugotovili, da je učenje na zgolj sintetičnih slikah nezadostno za ustrezno učenje detektorja. Iz rezultatov nadaljnjih poskusov smo zaključili, da dodatek sintetičnih slik k dejanskim v učni množici ni poslabšal zmogljivosti detektorja in je v nekaterih primerih celo izboljšal konvergenco in generalizacijo modela detektorja.

• Keywords: detekcija anevrizem, strojni vid, konvolucijske nevronske mreže, generativne nasprotniške mreže, bogatenje učne množice