Generative models for sketch synthesis

Lin, Aimee (avtor), Čehovin Zajc, Luka (mentor)

Freehand sketches represent a human abstraction of visual content and have the potential to be used for communication with advanced computer systems. However, the systematic collection of sketches required for training models to understand them is time-consuming and expensive. Generative models trained to convert images into synthetic sketches with a human-like level of abstraction can help address the challenge of understanding abstraction. We propose a generative diffusion model from the ControlNet family, conditioned on depth information extracted from the source image for sketch synthesis. The model is compared with reference sketch-generation approaches, including GAN-based models and advanced edge-detection methods, with a particular focus on semantic similarity to real sketches. The quality of the generated sketches is therefore evaluated through the downstream task of Sketch-Based Image Retrieval (SBIR), where the primary measure of quality is the ability to transfer knowledge to human sketches without additional training (zero-shot transfer). The FS-COCO dataset is used for evaluation. Both the aggregate results and additional category-holdout experiments reveal a consistent hierarchy of abstraction: sketches generated by diffusion models achieve the best retrieval performance, followed by those generated by GAN-based models, while those obtained using edge-detection methods perform the worst. The category holdout experiments further show that the advantage of higher levels of abstraction is greatest for categories with high intra-class shape variability, such as animals, and smallest for the category of persons. We believe these findings will contribute to more efficient model training for sketch understanding.

• Jezik: Angleški jezik
• Ključne besede: Deep learning, computer vision, generative models, sketch synthesis, diffusion models, sketch-based image retrieval
• Vrsta gradiva: Magistrsko delo
• Organizacija: FRI - Fakulteta za računalništvo in informatiko
• Leto izida: 2026
• PID: 20.500.12556/RUL-190103
• Datum objave v RUL: 22.10.2026

Sekundarni jezik

• Jezik: Slovenski jezik
• Naslov: Generativni modeli za sintezo skic

Izvleček

Prostoročne slike predstavljajo človeško abstrakcijo vizualnih vsebin, s potencialom uporabe v komunikaciji z naprednimi računalniškimi sistemi. Vendar pa je sistematičen zajem skic, potreben za učenje modelov za njihovo razumevanje, zamuden in drag. Pri naslavljanju problema razumevanja abstrakcije lahko pomagajo generativni modeli, ki so naučeni, da slike pretvorijo v sintetične skice s človeku podobno stopnjo abstrakcije. V tem delu za generiranje skic predlagamo generativni difuzijski model iz družine ControlNet, pogojen z globinsko informacijo iz izvorne slike. Primerjan je z referenčnimi modeli za generiranje skic kot so GAN in napredna detekcija robov. Pri tem nas zanima predvsem semantična podobnost z realnimi skicami. Zato je kvaliteta generiranih skic ovrednotena preko končnega scenarija iskanja po zbirki slik (Sketch-Based Image Retrieval, SBIR), pri katerem je primarno merilo kakovosti prenos znanja brez dodatnega učenja (zero-shot) na človeške skice. Za ovrednotenje je uporabljena zbirka FS-COCO. Tako pri agregatnih ocenah kot pri dodatnih poizkusih z izpuščanjem kategorij se pokaže dosledna hierarhija abstrakcije: skice, ustvarjene z difuzijskimi modeli, dosegajo najboljše rezultate pri iskanju, sledijo jim skice, ustvarjene z modeli tipa GAN, nato pa še skice, pridobljene z metodami detekcije robov. Eksperimenti izpuščanja kategorij kažejo, da je prednost višje stopnje abstrakcije največja pri kategorijah z veliko znotraj-razredno variabilnostjo oblik, kot so živali, in najmanjša pri kategoriji oseb. Verjamemo, da bodo ta dognanja prispevala k lažjemu učenju modelov za razumevanje skic.

• Ključne besede: Globoko učenje, računalniški vid, generativni modeli, sinteza skic, difuzijski modeli, iskanje slik na podlagi skic