Ensuring Face Consistency and Image Naturalness in Multi-Person Image Generation with Diffusion Models

Žakelj, Mark (author), Marolt, Matija (mentor)

Diffusion models have been widely used for consistent subject generation, but current methods are mostly focused on consistency of a single subject in an image, while consistent multi-subject generation remains an unexplored problem. We propose a method that combines diffusion models with IP-Adapters for facial consistency, ControlNet for ensuring image variability, and facial inpainting for improved facial quality and consistency. We introduce our own module for facial matching, which improves prompt adherence in cases where the age of the subjects varies significantly or their gender is different. Our method produces images of great quality with facial consistency limited only by the underlying IP-Adapter methods.

• Language: English
• Keywords: Diffusion models, Image generation, Facial consistency
• Work type: Master's thesis/paper
• Typology: 2.09 - Master's Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2024
• PID: 20.500.12556/RUL-165288
• COBISS.SI-ID: 218108675
• Publication date in RUL: 29.11.2024

Secondary language

• Language: Slovenian
• Title: Zagotavljanje doslednosti obrazov in naravnosti slik pri generiranju večih oseb z difuzijskimi modeli

Abstract

Difuzijski modeli se pogosto uporabljajo za dosledno generiranje subjektov, vendar so trenutne metode večinoma osredotočene na doslednost enega subjekta na sliki, medtem ko dosledno generiranje več subjektov ostaja neraziskan problem. Predlagamo metodo, ki združuje difuzijske modele z IP-Adapterji za doslednost obrazov, ControlNet-om za zagotavljanje variabilnosti slik ter Face Inpaint-om za izboljšanje kakovosti in doslednosti obraza. Uvedli smo tudi lasten modul za ujemanje obrazov, ki izboljša ujemanje slik in pripadajočega teksta v primerih, ko se starost referenčnih subjektov bistveno razlikuje ali je spol subjektov različen. Naša metoda generira slike visoke kakovosti, kjer je skladnost obrazov omejena zgolj z uporabljenimi metodami IP-Adapter.

• Keywords: Difuzijski modeli, Generiranje slik, Doslednost obrazov