Evaluating Model Tradeoffs for Click Prediction

KALC, MATEJ (author), Demšar, Jure (mentor), Kopič, Davorin (co-mentor), Hartman, Jan (co-mentor)

In the context of online advertising, Click-Through Rate (CTR) is the probability that a user clicks on an ad. CTR prediction is done using machine learning methods, such as Factorization machines (FM) and neural networks. Various improved versions of these traditional approaches have been proposed in the last decade, the main goal of this thesis is to evaluate these upgrades. We evaluated the models in two phases: using different combinations of parameters and using Bayesian optimization for parameter tuning. In the first phase, results showed that the group of models that use neural networks achieves a higher Area Under the ROC Curve (AUC). Kernel-extended Factorization Machine, a new proposed model during the Data Science project competition at the Faculty of Computer Science, performed worse than the FM model. In the second phase, we applied Bayesian optimization to the models to achieve an even higher AUC. The second-generation of the Deep&Cross model surprisingly surpassed Deep Factorization Machine with a higher AUC, which had the highest AUC in the first phase. During the evaluation, we also tested the degree of FM and concluded that there is no need for a degree higher than two.

• Language: English
• Keywords: CTR prediction, Factorization machines, Deep learning, Online learning, Bayesian optimization
• Work type: Bachelor thesis/paper
• Typology: 2.11 - Undergraduate Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2021
• PID: 20.500.12556/RUL-129844
• COBISS.SI-ID: 76724995
• Publication date in RUL: 08.09.2021

Secondary language

• Language: Slovenian
• Title: Ocenjevanje modelov za napovedovanje verjetnosti klika

Abstract

CTR je verjetnost, da uporabnik klikne nek oglas. CTR se napoveduje z uporabo metod strojnega učenja, kot sta Factorization machine (FM) in nevronska mreža. V zadnjem desetletju so bile predlagane različne izboljšane različice FM-ja in modeli z izboljšanimi nevronskimi mrežami. Glavni cilj diplomske naloge je ovrednotiti modele na isti množici podatkov. Te modele smo ovrednotili v dveh fazah: z uporabo različnih kombinacij parametrov in z uporabo Bayesove optimizacije za nastavljanje parametrov. V prvi fazi so rezultati pokazali, da je skupina modelov, ki uporabljajo nevronske mreže, dosegla višjo Ploščino pod ROC krivuljo (AUC). Kernel-extended Factorization machine, novi predlagani model med Data Science projektom na Fakulteti za računalništvo, je bil slabši od modela FM. V drugi fazi smo za modele uporabili Bayesovo optimizacijo, s katero smo dosegli še višji AUC. Model Deep&Cross V2 je z višjim AUC-jem presenetljivo presegel Deep Factorization Machine, model z višjim AUC-jem v prvi fazi. Med testiranjem smo ovrednotili razne stopnje reda za FM in ugotovili, da ni potrebe po stopnji, višji od dveh.

• Keywords: napovedovanje verjetnosti klika, Factorization machine, globoko učenje, sprotno učenje, Bayesova optimizacija