Privacy-preserving face analytics using deep learning methodsRot, Peter (Avtor) Štruc, Vitomir (Mentor) Peer, Peter (Komentor) analiza obrazazasebnostmehke biometrične značilnostiidentifikacijagloboko učenjeThe rapid advancements in face analytics have enabled a wide range of applications, from identity verification to expression analysis and personalized services. In addition to identity information, these technologies can extract various soft-biometric attributes such as gender, ethnicity, age, and body mass index (BMI), providing valuable insights for different domains. However, the extraction of these attributes raises significant privacy and data protection concerns. To achieve a balance between the benefits and risks of face analytics, it is essential to develop privacy-enhancing techniques that safeguard individuals' privacy rights while responsibly utilizing facial data. De-identification methods, known for concealing identity-related features, are well-recognized examples of such approaches. However, a relatively lesser-studied group of privacy-enhancing methods has recently emerged, known as soft-biometric privacy-enhancing techniques (SB-PETs). In contrast to de-identification methods, SB-PETs aim to retain identity information while preventing the extraction of soft-biometric attributes. In recent years, several SB-PET methods have been proposed to eliminate information about soft-biometrics at different levels of the biometric pipeline, including the image- and representation-level. However, there are still many open and crucial research questions regarding these techniques. For instance, how can the evaluation of SB-PETs be conducted efficiently and comprehensively? How can their effects on images be detected? To what degree can information about individual soft-biometric characteristics be eliminated without critically affecting recognition performance? Can the privacy of multiple soft-biometrics be efficiently enhanced simultaneously? In this work, we address these questions and make the following scientific contributions: 1. We present a novel evaluation methodology, PrivacyProber, to assess existing image-level SB-PETs, revealing their vulnerability to reconstruction attacks. We investigate the robustness of soft-biometric privacy techniques against attribute recovery attempts, highlighting variations in their effectiveness. Using the newly proposed attribute-recovery robustness (ARR), we evaluate the robustness of SB-PETs by comparing attribute-classification accuracy between original and reconstructed images. 2. We propose a new detection approach, APEND (Evidence Aggregation for Privacy-Enhancement Detection), which effectively identifies privacy-enhanced images and outperforms existing detectors. The main advantage of APEND is that it operates under a black-box assumption that requires no knowledge or information about the specific SB-PET being employed. 3. We present PriDSS (Privacy through Fusion of Disentangled Spatial Segments), a novel image-level SB-PET, that effectively protects the gender attribute while maintaining the visual quality of facial images. PriDSS employs image fusion to combine identity-related facial parts (i.e., eyes, mouth, nose, etc.) with contextual information from the opposite gender that result in images that confuse machine classifiers and minimize modification traces. The generated images outperform the competing PrivacyNet model in term of photorealism. 4. We propose PFRNet (Privacy-enhancing Face-representation Learning Network), a representation-level method that effectively disentangles binary-defined facial attributes from face verification templates obtained using state-of-the-art face recognition models. PFRNet can be applied to various face verification templates and, in this regard, is not limited to a specific method for extrating these templates from faces. Feature disentanglement proves to be an effective approach for soft-biometric privacy enhancement. 5. We also introduce ASPECD (Adaptable Soft-biometric Privacy–Enhancement using Centroid Decoding), a representation-level SB-PET that enables controllable privacy-enhancement of multiple soft-biometric characteristics. ASPECD is built on multiple components, each dedicated to an individual soft-biometric modality. We extend PFRNet to be suitable for the privacy-enhancement of categorical soft-biometric characteristics. Finally, we evaluate ASPECD on gender and ethnicity, compare it with the state-of-the-art approach MultiIVE, and report promising results.20242025-01-27 13:15:01Doktorsko delo/naloga166838VisID: 35296COBISS_ID: 224463107sl