Interpreting Face Recognition Templates using Symbolic RepresentationsManojlovska, Anastasija (Avtor) Štruc, Vitomir (Mentor) Grm, Klemen (Komentor) Interpretable Artificial IntelligenceSymbolic RepresentationsFace TemplatesAttribute RecognitionDeep LearningNatural Language ProcessingComputer VisionCLIPAdaFaceSwinFaceAs the field of Artificial Intelligence (AI) is gaining popularity, there is an increasing demand for making the decisions of AI systems transparent and understandable. Explainable Artificial Intelligence (XAI) is an emerging field, which aims to address the "black box" challenge in deep learning architectures and make the processes that lead to the decisions more explainable/interpretable to humans. It is particularly important to make the AI systems more transparent due to various legal regulations, such as the General Data Protection Regulation (GDPR), which requires the AI systems to be not only accurate, but also explainable/interpetable. This is specifically relevant in scenarios where the AI system has to make decisions about a person's identity, since the wrong verdict might have huge consequences. In this thesis, we use symbolic representations to interpret the encoded facial attribute information in face templates, which are easier to understand by non-experts in this field. To achieve this goal, two strategies are developed. First, we employ the CLIP model to generate natural language descriptions of the extracted face templates. The face templates are initially generated with CLIP's Image Encoder. We further implement state-of-the-art face recognition and face analysis models AdaFace and SwinFace as the backbones to extract face templates, which we later interpret using CLIP's Text Encoder. The differences in these architectures allow us to analyze the impact they have on the encoded information content. The second strategy involves implementing the AdaFace and SwinFace models as the backbones to binary and multi-label classifiers to predict the presence of the facial attributes, such as "Male", "Young", "Attractive", "Brown_Hair", "Wearing_Hat", etc. in the extracted face templates. By using this approach we aim to represent the encoded information content using very basic symbolic representatons. Moreover, this strategy serves as a baseline for the CLIP-based models, to which we compare the performance. We further fine-tune and evaluate various model variants using the VGGFace2 dataset and the annotated attribute labels from the MAADFace dataset. The results indicate that fine-tuning the CLIP model on a domain-specific task improves its ability to better represent the information encoded in face templates and align encoded text descriptions with these templates, which allows for natural language interpretation. Furthermore, the results show that SwinFace outperforms AdaFace both in the CLIP-based and classification approaches, indicating that SwinFace is more effective at encoding the attribute information in the extracted face templates. Moreover, neither of the CLIP-based models outperform the baseline classifiers.20242024-09-12 11:15:01Magistrsko delo/naloga161560sl