Non-boundary covariance matrix estimation in generalized linear mixed effects models using data augmentation priors

Košuta, Tina (author), Langerholc, Erik (author), Blagus, Rok (author)

Boundary estimates of random effects covariance matrices commonly arise when using maximum likelihood (ML) estimation in generalized linear mixed effects models, leading to numerical challenges and affecting statistical inference. To mitigate this, we introduce penalties to the likelihood function derived from conditionally conjugate priors for the covariance or precision matrices of the random effects. Our choice of penalties (priors) allows representation through pseudo-observations, enabling implementation of the proposed penalized estimator within the existing ML software by augmenting the original data. We derive a procedure for constructing these pseudo-observations, a non-trivial task because their likelihood contribution must match the functional form of the penalty and depend only on the covariance or precision matrix of the random effects. Our method includes penalty parameters that can be set using existing prior knowledge or, when no reliable prior information is available, via a novel fully data-driven procedure that eliminates the need for prior specification. Through simulation studies under realistic scenarios, we illustrate that the proposed approach can provide improved estimates of random-effects covariance matrices compared with competing methods in the settings considered. The approach is further illustrated on real-world data.

• Language: English
• Keywords: data-driven priors, (inverse) Wishart prior, maximum a posteriori estimate, penalized maximum likelihood, pseudo-observations
• Work type: Article
• Typology: 1.01 - Original Scientific Article
• Organization: MF - Faculty of Medicine
• Publication status: Published
• Publication version: Version of Record
• Year: 2026
• Number of pages: 10 str.
• Numbering: Vol. 82, no. 1, art. ujag013
• PID: 20.500.12556/RUL-184320
• UDC: 311
• ISSN on article: 0006-341X
• DOI: 10.1093/biomtc/ujag013
• COBISS.SI-ID: 270806019
• Publication date in RUL: 03.07.2026

Record is a part of a journal

• Title: Biometrics
• Shortened title: Biometrics
• Publisher: Oxford
• ISSN: 0006-341X
• COBISS.SI-ID: 5381383

Licences

• License: CC BY 4.0, Creative Commons Attribution 4.0 International
• Link: http://creativecommons.org/licenses/by/4.0/
• Description: This is the standard Creative Commons license that gives others maximum freedom to do what they want with the work as long as they credit the author.

Secondary language

• Language: Slovenian
• Keywords: apriorne ocene, ki temeljijo na podatkih, (inverzne) Wishartove apriorne ocene, maksimalna aposteriorna ocena, penalizirana maksimalna verjetnost, psevdo-opazovanja

Projects

• Funder: ARIS - Slovenian Research and Innovation Agency
• Funding programme: Young Researchers

• Funder: ARIS - Slovenian Research and Innovation Agency
• Project number: P3-0154
• Name: Metodologija za analizo podatkov v medicini