Incorporation of Quality Information into Face Recognition Systems

Babnik, Žiga (author), Štruc, Vitomir (mentor)

Face Recognition (FR), through its excellent performance and reliability, has become an integral part of modern life, ranging from everyday entertainment to important security and authentication applications. Recognition models thrive when the variation of the samples’ quality components is minimized. However, in certain scenarios, due to the limitations connected with the capture process, no assurance can be made about the quality of samples. This is often the case for mobile and security systems, where changes in the capture environment and noncooperation of the subject can result in samples unfit for recognition. Using recognition models in low-quality domains often leads to a decrease in performance and reliability, which, depending on the end application, can result in monetary loss and breaches of the privacy of end users. To address this issue, we can either enhance the recognition models or eliminate problematic samples. Improving FR models is a difficult task, reliant on improvements to the model’s architecture, training data, or loss function. On the other hand, given accurate estimates of sample quality, rejecting samples unfit for recognition is trivial. The fitness of samples in FR is measured by the biometric utility (quality) of samples, and is estimated by Face Image Quality Assessment (FIQA) techniques. Modern methods can be divided into supervised and unsupervised techniques. Supervised techniques rely on pseudo-quality labels used for training quality-regression models. Unsupervised techniques predict utility through analysis of the samples’ characteristics and their interaction with a target FR model. The two groups of techniques exhibit both desirable and undesirable characteristics. Supervised techniques, due to their computationally heavy pseudo-label generation and consequent supervised training, are quick at inferring the quality estimate of samples. This is because only a single forward pass through the final trained model is required for this task. For the same reason, they are extremely hard to adapt to unseen target FR models, where the whole generation and supervision process would need to be redone. On the contrary, unsupervised techniques require minimal setup and instead offload the processing to inference, making them both easily adaptable yet slow for the actual estimation process. Ideally, a given FIQA technique would combine positive aspects across both groups, being easily adaptable without much effort while being able to generate the quality estimate as quickly as existing supervised techniques. Biometric utility as estimated by FIQA techniques is primarily used as a simple discarding tool, where scores are compared to a system-wide threshold value. For this reason, the provided quality (utility) estimates must be as informative as possible. Additionally, quality information in existing systems, as presented, is used only passively. Direct use of FIQA estimates inside recognition-model training or inference remains comparatively limited. For these reasons, the main goal of this work is twofold: (i) to improve the predictive accuracy and functionality of existing quality estimation techniques, and (ii) to establish direct incorporation of quality information into existing as well as newly developed recognition systems. To this end, we present the following original contributions to science: 1. We present FaceQAN, a novel unsupervised FIQA technique based on adversarial noise. Through visual comparison, we discover that adversarial noise, a type of image perturbation, exhibits desirable traits for quality estimation. By using adversarial noise, applicable to any deep model, FaceQAN imposes no architectural limitations on the target FR model, improving the applicability of previous unsupervised techniques. Extensive experimental evaluation shows that FaceQAN is able to achieve state-of-the-art performance in most covered situations. 2. We propose a new quality estimation paradigm using the FROQ approach. Here, desirable aspects of supervised and unsupervised techniques are combined, allowing for easy adaptation to any FR model and a quick inference time. FROQ estimates the utility by observing specific layers of the target recognition model. Uncovering layers fit for the quality estimation task is done using a semi-supervised approach leveraging the power of a calibration set of images and their pseudo-quality labels. This process leverages only information already present in the recognition model; it does not introduce any new information into the system. Through experimental evaluation, we show that this approach outperforms all existing unsupervised estimation techniques. 3. We present eDifFIQA, a powerful supervised FIQA technique, combining recognition and quality estimation performance into a single model. The proposed knowledge distillation approach combines information from two distinct sources of utility information: DifFIQA pseudo-labels and a pretrained FR model. Through a separate loss term, the final regression model is able to retain the recognition performance of the baseline recognition model used in the distillation process. Experimental evaluation of eDifFIQA shows that it achieves state-of-the-art performance for the task of quality estimation, all while retaining excellent recognition performance. 4. We propose FunFace, an adaptive loss function that focuses on improving performance in low-quality domains. Extending the popular AdaFace framework, which uses feature norms as a source of margin variability, with an additional utility estimate, calculated using the certainty ratio, allows the recognition model to differentiate between samples of varying utility scores. Evaluating FunFace against state-of-the-art recognition models shows its prowess in low-quality domains where it outperforms all competitors. 5. We present UVFace, an extension to the classical recognition training paradigm, designed to excel in video-based recognition scenarios. Here, feature norms act as video-template weights. The proposed approach presents a new training branch that focuses on aligning the samples’ feature norms to their utility estimates, provided by an auxiliary FIQA technique. Bridging feature norms and utility scores allows UVFace to outperform state-of-the-art models on video-based benchmarks, without sacrificing performance in standard image-based scenarios.

• Language: English
• Keywords: Deep Learning, Biometrics, Face Recognition, Face Image Quality Assessment
• Work type: Doctoral dissertation
• Typology: 2.08 - Doctoral Dissertation
• Organization: FE - Faculty of Electrical Engineering
• Year: 2026
• PID: 20.500.12556/RUL-190021
• COBISS.SI-ID: 301733379
• Publication date in RUL: 20.10.2026

Secondary language

• Language: Slovenian
• Title: Vključevanje informacij o kakovosti v sisteme razpoznavanja obrazov

Abstract

Razpoznavanje obrazov je zaradi svoje učinkovitosti in zanesljivosti pomemben del sodobnega življenja ter se uporablja tako v preprostih razvedrilnih kot tudi naprednih varnostnih in avtentikacijskih sistemih. Ti sistemi delujejo najbolje, ko je spremenljivost dejavnikov, povezanih s kakovostjo obrazne slike, kot so na primer obrazna poza, osvetlitev ter ločljivost, majhna. V primeru mobilnih in varnostnih sistemov zagotavljanje zadostne kakovosti obraznih vzorcev za natančno razpoznavanje identitete ni vedno mogoče. To je najpogosteje posledica sprememb v okolju, v katerem je slika zajeta in nesodelovanja subjektov med zajemom slike. Nizkokakovostne obrazne slike v sistemih za razpoznavanje obrazov pogosto povzročijo nepravilne odločitve, ki lahko vodijo do hujših kršitev zasebnosti ali finančne škode, odvisno od obravnavanega sistema. Število nepravilnih odločitev lahko zmanjšamo z izboljšanjem zanesljivosti razpoznavalnih sistemov ali s predčasno zavrnitvijo problematičnih obraznih vzorcev. Izboljšanje zanesljivosti je kompleksno raziskovalno vprašanje, ki zahteva spremembe v arhitekturi, učnih podatkih ali izgubni funkciji sistema. Zavrnitev neprimernih vzorcev pa je preprosta, če imamo na voljo ocene kakovosti vhodnih vzorcev. Biometrično korist (kakovost) slike ocenjujejo metode za oceno kakovosti obraznih vzorcev. Razdelimo jih v dve skupini: nadzorovane in nenadzorovane pristope. Nadzorovani pristopi s pomočjo psevdo oznak kakovosti učijo regresijske modele za napovedovanje biometrične koristi. Nenadzorovani pristopi ocenijo kakovost vzorcev z analizo tako njihovih značilnosti kot njihove interakcije s ciljnim razpoznavalnim sistemom. Obe skupini imata z vidika uporabnika prednosti in slabosti. Nadzorovani pristopi potrebujejo ogromno računskega vložka za ustvarjanje psevdo oznak ter posledično nadzorovano učenje regresorja. Zaradi tega delujejo hitro med inferenco, vendar jih je izredno težko prilagoditi novemu razpoznavalnemu sistemu. Prilagoditev bi tako zahtevala ponovno ustvarjanje oznak kot tudi naknadno učenje regresijske mreže. Popolno nasprotje predstavljajo nenadzorovani pristopi, ki ne potrebujejo računsko zahtevne vzpostavitve in jih zato zlahka uporabimo s poljubnim ciljnim sistemom. Njihova časovna zahtevnost pri inferenci je zaradi načina delovanja mnogo večja kot pri nadzorovanih pristopih. V idealnem primeru bi želeli, da metoda za oceno kakovosti obraznih slik deluje hitro in je hkrati prilagodljiva ciljnim razpoznavalnim sistemom. Sistemi za razpoznavanje obrazov uporabijo ocene biometrične koristi za zavrnitev vzorcev tako, da jih primerjajo s sistemsko določenim pragom kakovosti. Posledično želimo izboljšati zanesljivost ocen kakovosti, saj te neposredno določajo uspešnost zavrnitve problematičnih vzorcev. Prav tako želimo utrditi medsebojno delovanje metod za oceno kakovosti in metod za razpoznavanje obrazov. Trenutno se ocene koristi uporabljajo le za zavračanje vzorcev na podlagi praga kakovosti. Z drugimi besedami, metode za oceno kakovosti obraznih slik trenutno niso neposredno vključene v sisteme razpoznavanja. Zaradi tega se v okviru disertacije osredotočimo na: (i) izboljševanje natančnosti in funkcionalnih lastnosti metod za oceno kakovosti obraznih slik, ter (ii) neposredno vključevanje ocen kakovosti v obstoječe in nove razpoznavalne sisteme. Glavna cilja sta: • (O1) Izboljšanje zanesljivosti in učinkovitosti metod za oceno kakovosti obraznih slik. V okviru cilja se osredotočamo na odpravljanje glavnih omejitev obstoječih nadzorovanih in nenadzorovanih metod za oceno kakovosti. Natančneje se osredotočamo na: (i) zmanjšanje arhitekturne odvisnosti nenadzorovanih metod, ter (ii) združevanje pozitivnih lastnosti nadzorovanih in nenadzorovanih pristopov, tj. hitro izvajanje ter prilagodljivost ciljnim razpoznavalnikom. • (O2) Vključevanje informacij o kakovosti vzorcev v postopek razpoznavanja obrazov. V okviru cilja se osredotočamo na integracijo ocen kakovosti neposredno v postopek razpoznavanja. Natančneje se osredotočamo na: (i) razširitev obstoječih razpoznavalnikov za potrebe ocenjevanja kakovosti ter (ii) izboljšavo učnega postopka razpoznavalnikov za doseganje boljših rezultatov. Zastavljene cilje, s katerimi želimo izboljšati natančnost in razširiti uporabnost obstoječih pristopov ocenjevanja kakovosti ter okrepiti medsebojno delovanje ocenjevanja kakovosti in procesa razpoznavanja, obravnavamo z naslednjimi izvirnimi prispevki k znanosti: 1. Prvi prispevek je FaceQAN, nenadzorovan pristop za ocenjevanje kakovosti, ki zaradi uporabe nasprotniškega šuma arhitekturno ne omejuje ciljnega razpoznavalnika. 2. Drugi prispevek je FROQ, ki z uvedbo opazovalca kakovosti dosega hitro inferenco brez nadzorovanega učenja in tako združuje pozitivne lastnosti nadzorovanih in nenadzorovanih pristopov. 3. Tretji prispevek je eDifFIQA, ki z razširitvijo standardnih postopkov učenja nadzorovanih metod dosega vrhunske rezultate tako pri ocenjevanju kakovosti kot pri razpoznavanju. 4. Četrti prispevek je FunFace, ki razširi pogosto uporabljeno razpoznavalno funkcijo izgube AdaFace z ocenami kakovosti vzorcev in tako izboljša delovanje na nizkokakovostnih slikah. 5. Zadnji, peti prispevek je UVFace, ki s poravnavo norm značilk z ocenami kakovosti vzorcev izboljša delovanje razpoznavalnikov v domeni videoposnetkov.

• Keywords: globoko učenje, biometrija, razpoznavanje obrazov, ocenjevanje kakovosti obraznih slik