Visual Realism Assessment for Deepfake Videos

Dragar, Luka (author), Emeršič, Žiga (mentor), Batagelj, Borut (co-mentor)

In this thesis, we tackle the issues of artificial intelligence and DeepFake technology, which in the era of rapid digitalization, pose significant security and privacy concerns. We focus on the assessment of quality and visual realism of DeepFakes, a key factor for the impact of a forged video. We introduce an effective approach for quantifying the visual realism of DeepFake videos, using an ensemble of ConvNext, a Convolutional Neural Network (CNN), and Eva, a vanilla Vision Transformer (ViT). These models were trained on a subset of the DeepFake Game Competition 2022 (DFGC 2022) dataset to regress to Mean Opinion Scores (MOS) from DeepFake videos. Our work yielded successful results, securing third place in the DeepFake Game Competition on Visual Realism Assessment (DFGC-VRA 2023). The thesis provides a detailed presentation of the employed models, data preprocessing procedures, and training, as well as a comparison of our results with other competitors.

• Language: English
• Keywords: deepfake, visual realism, deep learing
• Work type: Bachelor thesis/paper
• Typology: 2.11 - Undergraduate Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2023
• PID: 20.500.12556/RUL-150088
• COBISS.SI-ID: 168011011
• Publication date in RUL: 13.09.2023

Secondary language

• Language: Slovenian
• Title: Ocena kakovosti ponarejenih posnetkov

Abstract

V diplomski nalogi obravnavamo problematiko umetne inteligence in tehno- logijo globokih ponaredkov (angl. DeepFake), ki sta v dobi hitre digitalizacije ključni za varnost in zasebnost. Osredotočili smo se na ocenjevanje kakovosti in vizualnega realizma globoko ponarejenih videposnetkov, kar je ključnega pomena za njihov vpliv. Predstavljamo učinkovit pristop za kvantifikacijo vizualnega realizma globokih ponaredkov z uporabo ansambla dveh napred- nih globokih nevronskih mrež imenovanih ConvNext in Eva. Modela smo natrenirali na podmnožici podatkovne množice DeepFake Game Competition (DFGC) 2022, s ciljem napovedati povprečno oceno mnenja (MOS) ponare- jenega videoposnetka. Rezultati našega dela so se izkazali za uspešne, saj je naš pristop na tekmovanju DFGC-VRA 2023 zasedel tretje mesto. V diplom- ski nalogi so podrobno predstavljeni uporabljeni modeli, postopki predhodne obdelave podatkov in treniranja modelov, ter primerjava naših rezultatov s sotekmovalci.

• Keywords: globoki ponaredek, vizualni realizem, globoko učenje