Incorporation of Quality Information into Face Recognition SystemsBabnik, Žiga (Avtor) Štruc, Vitomir (Mentor) Deep LearningBiometricsFace RecognitionFace Image Quality AssessmentFace Recognition (FR), through its excellent performance and reliability, has become an integral part of modern life, ranging from everyday entertainment to important security and authentication applications. Recognition models thrive when the variation of the samples’ quality components is minimized. However, in certain scenarios, due to the limitations connected with the capture process, no assurance can be made about the quality of samples. This is often the case for mobile and security systems, where changes in the capture environment and noncooperation of the subject can result in samples unfit for recognition. Using recognition models in low-quality domains often leads to a decrease in performance and reliability, which, depending on the end application, can result in monetary loss and breaches of the privacy of end users. To address this issue, we can either enhance the recognition models or eliminate problematic samples. Improving FR models is a difficult task, reliant on improvements to the model’s architecture, training data, or loss function. On the other hand, given accurate estimates of sample quality, rejecting samples unfit for recognition is trivial. The fitness of samples in FR is measured by the biometric utility (quality) of samples, and is estimated by Face Image Quality Assessment (FIQA) techniques. Modern methods can be divided into supervised and unsupervised techniques. Supervised techniques rely on pseudo-quality labels used for training quality-regression models. Unsupervised techniques predict utility through analysis of the samples’ characteristics and their interaction with a target FR model. The two groups of techniques exhibit both desirable and undesirable characteristics. Supervised techniques, due to their computationally heavy pseudo-label generation and consequent supervised training, are quick at inferring the quality estimate of samples. This is because only a single forward pass through the final trained model is required for this task. For the same reason, they are extremely hard to adapt to unseen target FR models, where the whole generation and supervision process would need to be redone. On the contrary, unsupervised techniques require minimal setup and instead offload the processing to inference, making them both easily adaptable yet slow for the actual estimation process. Ideally, a given FIQA technique would combine positive aspects across both groups, being easily adaptable without much effort while being able to generate the quality estimate as quickly as existing supervised techniques. Biometric utility as estimated by FIQA techniques is primarily used as a simple discarding tool, where scores are compared to a system-wide threshold value. For this reason, the provided quality (utility) estimates must be as informative as possible. Additionally, quality information in existing systems, as presented, is used only passively. Direct use of FIQA estimates inside recognition-model training or inference remains comparatively limited. For these reasons, the main goal of this work is twofold: (i) to improve the predictive accuracy and functionality of existing quality estimation techniques, and (ii) to establish direct incorporation of quality information into existing as well as newly developed recognition systems. To this end, we present the following original contributions to science: 1. We present FaceQAN, a novel unsupervised FIQA technique based on adversarial noise. Through visual comparison, we discover that adversarial noise, a type of image perturbation, exhibits desirable traits for quality estimation. By using adversarial noise, applicable to any deep model, FaceQAN imposes no architectural limitations on the target FR model, improving the applicability of previous unsupervised techniques. Extensive experimental evaluation shows that FaceQAN is able to achieve state-of-the-art performance in most covered situations. 2. We propose a new quality estimation paradigm using the FROQ approach. Here, desirable aspects of supervised and unsupervised techniques are combined, allowing for easy adaptation to any FR model and a quick inference time. FROQ estimates the utility by observing specific layers of the target recognition model. Uncovering layers fit for the quality estimation task is done using a semi-supervised approach leveraging the power of a calibration set of images and their pseudo-quality labels. This process leverages only information already present in the recognition model; it does not introduce any new information into the system. Through experimental evaluation, we show that this approach outperforms all existing unsupervised estimation techniques. 3. We present eDifFIQA, a powerful supervised FIQA technique, combining recognition and quality estimation performance into a single model. The proposed knowledge distillation approach combines information from two distinct sources of utility information: DifFIQA pseudo-labels and a pretrained FR model. Through a separate loss term, the final regression model is able to retain the recognition performance of the baseline recognition model used in the distillation process. Experimental evaluation of eDifFIQA shows that it achieves state-of-the-art performance for the task of quality estimation, all while retaining excellent recognition performance. 4. We propose FunFace, an adaptive loss function that focuses on improving performance in low-quality domains. Extending the popular AdaFace framework, which uses feature norms as a source of margin variability, with an additional utility estimate, calculated using the certainty ratio, allows the recognition model to differentiate between samples of varying utility scores. Evaluating FunFace against state-of-the-art recognition models shows its prowess in low-quality domains where it outperforms all competitors. 5. We present UVFace, an extension to the classical recognition training paradigm, designed to excel in video-based recognition scenarios. Here, feature norms act as video-template weights. The proposed approach presents a new training branch that focuses on aligning the samples’ feature norms to their utility estimates, provided by an auxiliary FIQA technique. Bridging feature norms and utility scores allows UVFace to outperform state-of-the-art models on video-based benchmarks, without sacrificing performance in standard image-based scenarios.20262026-10-20 14:25:05Doktorsko delo/naloga190021sl