Destilacija znanja globokih modelov za biometrijo beločnice

Bizjak, Matic (author), Peer, Peter (mentor), Štruc, Vitomir (co-mentor), Vitek, Matej (co-mentor)

Destilacija znanja je pristop izdelave lahkih modelov s prenosom znanja iz globokih modelov, ki imajo veliko število parametrov, so časovno zahtevni in imajo zelo visoko natančnost. V magistrskem delu ovrednotimo pristop destilacije znanja na področju biometrije očesa. Izdelamo nov postopek pridobitve lahkega modela za segmentacijo beločnice s kombinacijo dveh pristopov, destilacije znanja in rezanja filtrov, ter pokažemo, da sta oba pristopa ključna za uspeh našega postopka. S predstavljenim izvirnim postopkom pridobitve lahkega modela odstranimo 74 % operacij s plavajočo vejico za eno sklepanje in 73,2 % parametrov ter izgubimo 1,27 % natančnosti, poleg tega pa odstranimo 2-krat toliko parametrov kot najsodobnejši model in v primerjavi izgubimo le 1,74 % natančnosti. V luči te primerjave na koncu identificiramo možne nadgradnje, ki imajo potencial za izboljšanje našega pristopa.

• Language: Slovenian
• Keywords: destilacija znanja, rezanje filtrov, konvolucijske nevronse mreže, beločnica, segmentacija
• Work type: Master's thesis/paper
• Typology: 2.09 - Master's Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2020
• PID: 20.500.12556/RUL-121899
• COBISS.SI-ID: 37218051
• Publication date in RUL: 06.11.2020

Secondary language

• Language: English
• Title: Knowledge distillation of deep learning models for sclera biometrics

Abstract

Knowledge distillation is a technique for the development of lightweight models by transferring knowledge from a deep model with high memory footprint and high computational complexity. In this work we evaluate knowledge distillation for eye biometrics. We propose a new algorithm for creating a lightweight model for sclera segmentation by combining knowledge distillation with filter pruning and show that both techniques are key to achieving good results. With the presented algorithm we remove 74% floating point operations needed for one inference and 73.2% parameters and sacrifice 1.27% of the accuracy. In addition, we remove twice as many parameters as the current state-of-the-art filter pruning approach and in comparison sacrifice 1.74% of the accuracy. In the light of this comparison, we identify possible improvements that have a potential to further increase the accuracy of our algorithm.

• Keywords: knowledge distillation, filter pruning, convolutional neural networks, sclera, segmentation