Bayesian models for multivariate count data

Pirš, Gregor (author), Štrumbelj, Erik (mentor)

Multivariate normal distribution offers a convenient approach to several multivariate problems due to its mathematical simplicity. However, often the data can not be modeled with the multivariate normal distribution well, and one such example are multivariate counts. Some approaches, which can be considered trivial for continuous, can be very difficult for count data. Due to that, some options are not explored well. For example, in factor analysis, it is relatively simple to alleviate the assumption of conditional independence for continuous data, but it is not that straight-forward for count data. By using suitable multivariate count distributions we are able to extend some established methods in a new direction of modeling the uncertainty. In this thesis, we focus on two such extensions: a) alleviating the conditional independence assumption in count factor analysis, and b) combining count predictions by fitting their structure with a suitable multivariate count distribution. At the beginning, we explored how the assumption of conditional independence in static count factor analysis methods affects their out-of-sample probabilistic predictions. We implemented several Bayesian factor analysis methods and paired them with a Gaussian copula in a two-stage fitting process. The copula serves to find the covariance not found by the latent structure of the model. We compared the methods on a toy and 5 real-world data sets in terms of out-of-sample probabilistic predictions. The results indicate that the assumption of conditional independence is very restrictive in terms of probabilistic prediction power. Additionally, we provide a normalization step for finding interpretable latent structure with count factor analysis. As the next step, we focused on problems that include continuous covariates, for example time. In this setting, the factor scores are not assumed static, but we enforce a smooth structure on their values, depending on the covariate – resulting in smooth latent trajectories. We developed a new Bayesian model for latent trajectory extraction and prediction for count data, without the assumption of conditional independence, based on Gaussian process factor analysis. We extended count-likelihood Gaussian process factor analysis by modeling the residual covariance with a Gaussian copula. Contrary to the two stage fitting process described in the previous paragraph, we incorporated both elements into a single model. We provide a fully Bayesian implementation of the model and use augmented likelihood for inference with Hamiltonian Monte Carlo. We compared the proposed method to other Gaussian process factor analysis models on 20 toy data sets, finding latent qualities of NBA teams, and forecasting disease counts. The results show that the proposed method is useful for latent structure extraction and out-of-sample prediction of multivariate counts. In the second part of this thesis we explored the use of multivariate count distributions in developing count ensembles based on modeling the structure of candidate predictions. Combining classifiers proved a rich source of models, which combine the knowledge of candidate models by learning the latent structure of their predictions. As such, they are especially useful for combining biased models, or models with systematic errors. Their performance is directly affected by how well we are able to model the structure of predictions. First, we developed a new method for combining classifiers, based on modeling the latent structure of predictions with multivariate normal mixtures, to alleviate some of the drawbacks of the state-of-the-art related methods. The method proved to be very flexible while remaining robust. As the next step, we extended the classifier combination method to combining counts, by truncating the response counts and using suitable multivariate count distributions to model the candidate predictions. Results suggest that using this extension provides better probabilistic predictions than the original classifier model, even though it is less flexible.

• Language: English
• Keywords: latent structure, conditional independence, copula, negative binomial distribution, ensemble learning
• Work type: Doctoral dissertation
• Typology: 2.08 - Doctoral Dissertation
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2021
• PID: 20.500.12556/RUL-128112
• COBISS.SI-ID: 69314051
• Publication date in RUL: 02.07.2021

Secondary language

• Language: Slovenian
• Title: Bayesovski modeli za multivariatne števne podatke

Abstract

Multivariatna normalna porazdelitev predstavlja priročen pristop do modeliranja multivariatnih problemov zaradi matematične preprostosti. Velikokrat pa podatkov ne moremo dobro modelirati z multivariatno normalno porazdelitvijo. En tak primer so multivariatni števni podatki. Nekateri pristopi, ki so trivialni za zvezne podatke, so lahko veliko kompleksnejši za števne podatke. Zaradi tega so te možnosti slabše raziskane. Na primer, pri faktorski analizi je relativno preprosto zaobiti predpostavko o pogojni neodvisnosti pri zveznih podatkih, pri števnih pa to predstavlja večji izziv. Z uporabo ustreznih multivariatnih števnih porazdelitev lahko razširimo nekatere uveljavljene metode v nove smeri ocenjevanja negotovosti. V tem delu se osredotočamo na dve takšni razširitvi: a) odpraviti predpostavko o pogojni neodvisnosti v števni faktorski analizi in b) kombiniranje števnih napovedi na podlagi prileganja ustrezne multivariatne števne porazdelitve tem napovedim. Raziskavo smo začeli z vprašanjem, kako predpostavka pogojne neodvisnosti v statični faktorski analizi vpliva na kvaliteto verjetnostnih napovedi. Implementirali smo več metod bayesovske faktorske analize in jih združili z Gaussovo kopulo v dvostopenjskem prileganju. Kopula služi iskanju kovarianc, ki jih latentna strutura ne zajame. Metode smo primerjali na sintetični in petih realnih podatkovnih množicah. Kot kritetij za primerjavo smo izbrali verjetnostne napovedi izven vzorca. Rezultati nakazujejo, da predpostavka o pogojni neodvisnosti zelo omejuje metodo v smislu njene napovedne moči. Poleg tega smo predstavili normalizacijski korak, ki nam omogoča najti razumljive latentne dimenzije v števni faktorski analizi. V naslednji fazi smo se osredotočili na naloge, ki so vsebovale zvezne kovariate, na primer čas. V tem primeru vrednosti latentnih faktorjev niso statične, ampak jih prikažemo z gladko krivuljo, ki je odvisna od kovariata. S tem pridemo do gladkih latentnih trajektorij. Razvili smo nov bayesovski model za iskanje latentnih trajektorij in napovedovanje, primeren za uporabo s števnimi podatki. Model ne predpostavi pogojne neodvisnosti in temelji na števni faktorski analizi z Gaussovimi procesi. Slednjo smo razširili z Gaussovo kopulo, ki modelira preostalo kovarianco, ki ni zajeta v latentni strukturi. V nasprotju z dvostopenjskim prileganjem iz prejšnjega odstavka smo v tem delu združili oba elementa v enoten model. Podali smo popolnoma bayesovsko implementacijo predlaganega modela in uporabili razširjeno verjetje za sklepanje s hamiltonskim Monte Carlom. Razvito metodo smo primerjali s preostalimi faktorskimi analizami z Gaussovimi procesi na 20 sintetičnih podatkovnih množicah, iskanju latentnih moči ekip v NBA in napovedovanju števila bolezni. Rezultati kažejo, da je razvita metoda uporabna za iskanje latentne strukture in verjetnostno napovedovanje izven vzorca za multivariatne števne podatke. V drugem delu te disertacije smo raziskali uporabo multivariatnih števnih porazdelitev pri razvoju ansambla števnih napovedi, ki temelji na modeliranju strukture napovedi, ki jih podajo različni viri. Kombiniranje klasifikatorjev je bogat vir modelov, ki kombinirajo znanje večih modelov tako, da se naučijo latentne strukture napovedi. S tem so še posebej uporabni pri kombiniranju pristranskih modelov, ali modelov s sistematičnimi napakami. Njihova učinkovitost je direktno pogojena s tem, kako dobro smo sposobni modelirati to strukturo. Najprej smo razvili novo metodo za kombiniranje klasifikatorjev, kjer smo latentno strukturo napovedi modelirali z multivariatnimi normalnimi mešanicami. Ta model je odpravil nekatere pomanjkljivosti uveljavljenih metod na področju. Izkazal se je za zelo fleksibilnega, kljub temu da je tudi robusten. Kot naslednji korak smo ta model razširili na kombiniranje števnih napovedi, tako da smo odvisno spremenljivko omejili navzgor, strukturo napovedi pa smo modelirali z ustrezno multivariatno števno porazdelitvijo. Rezultati nakazujejo, da je takšna razširitev smiselna, saj nam števni model večinoma nudi boljše verjetnostne napovedi, kljub temu da ni tako fleksibilen.

• Keywords: latentna struktura, pogojna neodvisnost, kopula, negativna binomska porazdelitev, ansambelsko učenje