Visual ear detection and recognition in unconstrained environments

Emeršič, Žiga (avtor), Peer, Peter (mentor), Štruc, Vitomir (komentor)

Automatic ear recognition systems have seen increased interest over recent years due to multiple desirable characteristics. Ear images used in such systems can typically be extracted from profile head shots or video footage. The acquisition procedure is contactless and non-intrusive, and it also does not depend on the cooperation of the subjects. In this regard, ear recognition technology shares similarities with other image-based biometric modalities. Another appealing property of ear biometrics is its distinctiveness. Recent studies even empirically validated existing conjectures that certain features of the ear are distinct for identical twins. This fact has significant implications for security-related applications and puts ear images on a par with epigenetic biometric modalities, such as the iris. Ear images can also supplement other biometric modalities in automatic recognition systems and provide identity cues when other information is unreliable or even unavailable. In surveillance applications, for example, where face recognition technology may struggle with profile faces, the ear can serve as a source of information of the identity of people in the surveillance footage. The importance and potential value of ear recognition technology for multi-modal biometric systems are also evidenced by the number of research studies on this topic. Today, ear recognition represents an active research area for which new techniques are developed regularly, and several datasets needed for the training and testing of the technology are publicly available. Nevertheless, despite the research efforts directed at ear biometrics, to the best of our knowledge, there exist only a few commercial systems based on ear biometrics. We conjecture that the limited availability of commercial ear recognition technology is a consequence of the open challenges that have still not been appropriately addressed. This thesis attempts to meet some of these challenges and provide the community with new solutions and insights that can be used to advance the field further. Most of the early research work on ear biometrics focused on laboratorylike settings, where the variability in ear appearance was limited. Usually, there were no major variations in pose, occlusion, etc. However, in real-life applications, this is not the case. Development and research hit a wall by using images captured in constrained environments. Methods developed for these settings were difficult to translate into real-world scenarios and would typically perform poorly. In unconstrained acquisition environments, ear recognition techniques are confronted with large pose variations, illumination changes, etc. We surmise that this significant difference in the data is the main culprit for performance differences and the stagnation of the field. Furthermore, the lack of commercial solutions could also be attributed to the lack of unconstrained ear data available. This issue pervades recognition and detection tasks, meaning that annotated datasets of ear images are needed, together with images with annotated ear positions for ear detection tasks. Existing ear recognition solutions do not suffice and under-perform in unconstrained settings. More powerful solutions are needed to address ear detection and recognition in these environments successfully. Another key issue to be addressed is identifying the most important weaknesses of existing ear recognition techniques. This can be done through a performance analysis taking into account various covariates that can help answer the following research questions: How do ear recognition techniques perform across different image resolutions in unconstrained settings? How sensitive are existing techniques to the presence of occlusions and ear accessories? Do existing recognition approaches exhibit a performance bias when presented with images of either male or female subjects? How do recognition techniques generalize to ear image data with different characteristics? Answering these questions may help identify open problems with existing ear recognition approaches and help focus research efforts and resources in the right direction. To address the research problems discussed above, we make several contributions in this thesis, i.e., 1. We present novel techniques for ear detection that work in unconstrained environments and consider contextual information for the detection procedure. The techniques frame ear detection as a semantic segmentation task and are shown to yield state-of-the-art performance on public ear datasets. 2. We develop novel recognition approaches that consider local and global information and ensure robust recognition in unconstrained environments. The developed approaches are also shown to enable explainable decision-making by focusing only on the most important ear regions. 3. We introduce new ear datasets captured in unconstrained environments to train and test the developed detection and recognition techniques. While we make contributions targeting ear recognition systems specifically, many of the solutions presented are also applicable to other biometric modalities and therefore have implications for other areas of biometrics as well.

• Jezik: Angleški jezik
• Ključne besede: ear recognition, ear detection, deep neural networks, unconstrained environment
• Vrsta gradiva: Doktorsko delo/naloga
• Tipologija: 2.08 - Doktorska disertacija
• Organizacija: FRI - Fakulteta za računalništvo in informatiko
• Leto izida: 2021
• PID: 20.500.12556/RUL-134234
• COBISS.SI-ID: 91890691
• Datum objave v RUL: 03.01.2022

Sekundarni jezik

• Jezik: Slovenski jezik
• Naslov: Vizualna detekcija in razpoznavanje uhljev v nenadzorovanih okoljih

Izvleček

Sistemi za samodejno razpoznavanje uhljev so v zadnjih letih vse bolj priljubljeni, saj v primerjavi z drugimi rešitvami k biometričnemu razpoznavanju oseb izkazujejo vrsto privlačnih lastnosti. Slike uhljev, ki se uporabljajo v sistemih za samodejno razpoznavanje uhljev, je običajno mogoče pridobiti na enostaven način iz profilnih slik ali video posnetkov oseb. Postopki pridobivanja slikovnih podatkov so brezstični in nevsiljivi, in hkrati ne zahtevajo eksplicitnega sodelovanje oseb pri postopku zajema. V teh pogledih je tehnologija za prepoznavanje uhljev podobna drugimi načinom biometričnega razpoznavanja ki temeljijo, npr. na slikah obrazov ali načinu hoje. Druga privlačna lastnost biometrije uhljev je visoka sposobnost razlikovanja. Nedavne študije so empirično potrdile, da se nekatere značilnosti ušesa razlikujejo celo pri enojajčnih dvojčkih. To dejstvo ima pomembne posledice za aplikacije, povezane z varnostjo, saj omogoča izredno učinkovito razpoznavanje oseb na podlagi slik uhljev, ki se lahko kosa z drugimi (epigenetičnimi) biometričnimi modalnostmi, kot je na primer šarenica. Slike uhljev lahko služijo tudi kot dodatek drugim biometričnim modalnostim v več modalnih sistemih, kadar so drugi viri informacij nezanesljivi ali celo nerazpoložljivi. Kot primer navedimo nadzorne aplikacije, kjer je tehnologija za prepoznavanje obrazov zaradi delne prekritosti obraza ali kota zajema slike pogosto nezanesljiva. V takšnih aplikacijah lahko uhlji služijo kot dodatni vir informacij o identiteti ljudi na nadzornih posnetkih. O pomembnosti in potencialni vrednosti tehnologije prepoznavanja uhljev za več-modalne biometrične sisteme priča tudi število raziskovalnih študij na to temo. Danes prepoznavanje uhljev predstavlja aktivno raziskovalno področje, za katerega je na voljo več javno dostopnih zbirk podatkov. Kljub raziskovalnim prizadevanjem, usmerjenim v biometrijo uhljev, pa obstaja le malo komercialnih sistemov, ki temeljijo na biometriji ušes. Domnevamo, da je omejena razpoložljivost komercialne tehnologije za prepoznavanje uhljev posledica odprtih problemov, ki do danes še vedno niso ustrezno rešeni. V doktorski disertaciji naslovimo nekaj teh problemov in izzivov ter predlagamo izvirne postopke, s katerimi zagotovimo učinkovito detekcijo in razpoznavanje uhljev v realnih okoljih. Večina zgodnjih raziskovalnih del na področju biometrije uhljev je bila osredotočena na laboratorijsko okolje, kjer je bila variabilnost videza uhljev omejena. V slikovnih podatkih tako ni bilo večjih razlik v zornem kotu zajema, svetlobnih razmerah, prekrivanju uhljev itd. Metode, razvite za takšna okolja, je težko aplicirati v realnih okoljih, saj se postopki za razpoznavanje uhljev tam soočajo z velikimi spremembami v izgledu ušes zaradi spremenljivih osvetlitev, orientacij glave, prisotnosti okluzij, in drugih podobnih dejavnikov. Ta razlika v karakteristikah slikovnih podatkih je eden od glavnih vzrokov za manjšo učinkovitost obstoječih rešitev v nenadzorovanih okoljih. Manjšo učinkovitost obstoječih rešitev lahko pripišemo tudi pomanjkanju razpoložljivih zbirk uhljev, zajetih v nenadzorovanih okoljih, ki so potrebne za izgradnjo postopkov za detekcijo uhljev kot tudi za razvoj postopkov razpoznavanja. Poleg naprednejših rešitev za detekcijo in razpoznavanje uhljev, ki učinkovito delujejo tudi v nenadzorovanih okoljih, je eden od ključnih problemov na tem področju tudi razumevanje najpomembnejših slabosti obstoječih rešitev. Ključna raziskovalna vprašanja, ki zahtevajo podrobno analizo so: Kako postopki za razpoznavanje uhljev delujejo pri različnih ločljivostih slik v nenadzorovanih okoljih? Kako občutljive so obstoječi postopki na prisotnost okluzij in različnih dodatkov v uhljih, kot so npr. uhani in slušalke? Ali obstoječi pristopi k prepoznavanju kažejo pristranskost, če so predstavljeni s podobami moških ali žensk? Kako postopki prepoznavanja posplošujejo na slikovne podatke z različnimi lastnostmi? Odgovori na ta vprašanja lahko pripomorejo k prepoznavanju ključnih odprtih težav z obstoječimi pristopi prepoznave uhljev in usmerjanju raziskovalnih aktivnosti v pravo smer. V doktorskem delu predlagamo več izvirnih prispevkov, ki naslavljajo zgornje probleme in raziskovalna vprašanja in jih lahko povzamemo v naslednjih treh skupinah: 1. Predstavljamo nove postopke za detkecijo uhljev, ki delujejo v nenadzorovanih okoljih in v postopku detekcije upoštevajo tudi slikovni kontekst. 2. Predlagamo nov model za razpoznavanje uhljev, ki upošteva tako lokalno, kot tudi globalno informacijo o izgledu uhljev. 3. Predstavimo novo zbirko slik uhljev, zajeto v nenadzorovanih razmerah. Čeprav so prispevki namenjeni predvsem sistemom za razpoznavanje uhljev, so številne predstavljene rešitve uporabne tudi za druge biometrične modalnosti.

• Ključne besede: razpoznava uhljev, detekcija uhljev, globoke nevronske mreže, nenadzorovano okolje