Deep Face Decoder : towards understanding the embedding space of convolutional networks through visual reconstruction of deep face templates
ID Križaj, Janez (Author), ID Plesh, Richard O. (Author), ID Banavar, Mahesh (Author), ID Schuckers, Stephanie (Author), ID Štruc, Vitomir (Author)

.pdfPDF - Presentation file, Download (6,68 MB)
MD5: 06824F9A044039547890B396A6E81DDE
URLURL - Source URL, Visit https://www.sciencedirect.com/science/article/pii/S095219762400099X This link opens in a new window

Advances in deep learning and convolutional neural networks (ConvNets) have driven remarkable face recognition (FR) progress recently. However, the black-box nature of modern ConvNet-based face recognition models makes it challenging to interpret their decision-making process, to understand the reasoning behind specific success and failure cases, or to predict their responses to unseen data characteristics. It is, therefore, critical to design mechanisms that explain the inner workings of contemporary FR models and offer insight into their behavior. To address this challenge, we present in this paper a novel template-inversion approach capable of reconstructing high-fidelity face images from the embeddings (templates, feature-space representations) produced by modern FR techniques. Our approach is based on a novel Deep Face Decoder (DFD) trained in a regression setting to visualize the information encoded in the embedding space with the goal of fostering explainability. We utilize the developed DFD model in comprehensive experiments on multiple unconstrained face datasets, namely Visual Geometry Group Face dataset 2 (VGGFace2), Labeled Faces in the Wild (LFW), and Celebrity Faces Attributes Dataset High Quality (CelebA-HQ). Our analysis focuses on the embedding spaces of two distinct face recognition models with backbones based on the Visual Geometry Group 16-layer model (VGG-16) and the 50-layer Residual Network (ResNet-50). The results reveal how information is encoded in the two considered models and how perturbations in image appearance due to rotations, translations, scaling, occlusion, or adversarial attacks, are propagated into the embedding space. Our study offers researchers a deeper comprehension of the underlying mechanisms of ConvNet-based FR models, ultimately promoting advancements in model design and explainability.

Keywords:deep face templates, template inversion, face recognition, deep learning, explainability, interpretability
Work type:Article
Typology:1.01 - Original Scientific Article
Organization:FE - Faculty of Electrical Engineering
Publication status:Published
Publication version:Version of Record
Number of pages:20 str.
Numbering:Vol. 132, art. 107941
PID:20.500.12556/RUL-156219 This link opens in a new window
ISSN on article:1873-6769
DOI:10.1016/j.engappai.2024.107941 This link opens in a new window
COBISS.SI-ID:184095491 This link opens in a new window
Publication date in RUL:14.05.2024
Copy citation
Share:Bookmark and Share

Record is a part of a journal

Title:Engineering applications of artificial intelligence
Publisher:Elsevier, International Federation of Automatic Control
COBISS.SI-ID:23000325 This link opens in a new window


License:CC BY 4.0, Creative Commons Attribution 4.0 International
Description:This is the standard Creative Commons license that gives others maximum freedom to do what they want with the work as long as they credit the author.

Secondary language

Keywords:globoke obrazne predloge, inverzija predlog, prepoznavanje obrazov, globoko učenje, razložljivost, interpretabilnost


Funder:ARRS - Slovenian Research Agency
Project number:P2-0250
Name:Metrologija in biometrični sistemi

Funder:ARRS - Slovenian Research Agency
Project number:J2-2501
Name:Globoki generativni modeli za lepotno in modno industrijo (DeepBeauty)

Funder:Other - Other funder or multiple funders
Funding programme:Center for Identification Technology Research

Funder:NSF - National Science Foundation
Project number:1650503

Similar documents

Similar works from RUL:
Similar works from other Slovenian collections: