Evaluation of prediction of single cell transcriptome based on biological knowledge transfer between samples

Kukenberger, Ana (author), Jakše, Jernej (mentor), Theis, Fabian (comentor)

This study explores the evaluation of perturbation prediction models in single-cell RNA sequencing through the application of alternative metrics, beyond the commonly used correlation and R-squared. Nine evaluation metrics, cosine distance, Euclidean distance, log-likelihood, maximum mean discrepancy, mean absolute error, mean squared error, Pearson’s R, R-squared, and root mean squared error, were applied to both real and simulated datasets to assess their ability to capture differences between control, perturbed, and predicted conditions. We observed that while alternative metrics showed some variation in behavior, they did not consistently outperform traditional metrics. Furthermore, the study investigates the impact of gene expression variability, using a binning strategy, and identifies biases introduced by grouping genes with similar expression strengths. These findings highlight the importance of data variability and the potential limitations of the binning approach in perturbation prediction evaluations. Our research suggests that traditional metrics, such as R-squared and Pearson’s R, are adequate for evaluating scRNA-seq perturbation models in controlled settings. However, future work could benefit from exploring more diverse datasets, alternative models, and refined binning strategies to further examine this statement.

• Language: Slovenian
• Keywords: scRNA-seq, prediction models, out of distribution prediction, evaluation, metrics
• Work type: Master's thesis/paper
• Typology: 2.09 - Master's Thesis
• Organization: BF - Biotechnical Faculty
• Year: 2025
• PID: 20.500.12556/RUL-169147
• COBISS.SI-ID: 236011267
• Publication date in RUL: 15.05.2025

Secondary language

• Language: English
• Title: Vrednotenje napovedovanja transkriptoma posameznih celic na podlagi biološkega prenosa znanja med vzorci

Abstract

V sklopu te raziskave smo želeli ugotoviti ali lahko izboljšamo vrednotenje modelov za napovedovanje perturbacij na podatkih transkriptoma posameznih celic z uporabo metrik, ki presegajo običajno uporabljeni korelacijo in koeficient determinacije. Uporabili smo devet metrik: kosinusno razdaljo, evklidsko razdaljo, logaritem verjetja, maksimalno srednjo diskrepanco, srednjo absolutno napako, srednjo kvadratno napako, korelacijo, koeficient determinacije in koren srednje kvadratne napake. Metrike smo uporabili tako na resničnih, kot tudi na simuliranih podatkih, da bi ocenili njihovo zmožnost zajetja razlik med kontrolo, perturbiranim stanjem in napovedanim perturbiranim stanjem. Opazili smo, da so alternativne metrike sicer pokazale nekoliko drugačne rezultate, vendar niso konsistentno pokazale boljših rezultatov v primerjavi s pogosto uporabljenimi metrikami. Poleg tega je to delo raziskovalo tudi vpliv spremenljivosti izražanja genov, z uporabo razporejanja genov s podobnim izražanjem v razrede in raziskovalo pristranskosti, ki jih prinaša taka strategija. Te ugotovitve poudarjajo pomen variabilnosti podatkov in morebitne omejitve pristopa razporejanja genov s podobnim izražanjem v razrede. To delo kaže, da so pogosto uporabljene metrike, kot sta korelacija in koeficient determinacije, dovolj primerne za ocenjevanje modelov za napovedovanje perturbacij. Vendar pa bi bilo pri prihodnjem delu koristno raziskati bolj raznolike nabore podatkov, alternativne modele in izpopolnjene strategije razporejanja genov s podobnim izražanjem v razrede, da bi to trditev dodatno preverili.

• Keywords: scRNA-seq, napovedni modeli, napovedovanje izven distribucije, evalvacija, metrike