20.500.12556/RUL-186056 Assessing reliability of BERT-based models on question answering tasks Reliability estimation of large language models is in many cases as crucial as their accuracy, as reliable models are more trustworthy, robust, and suitable for practical applications. Recent advancements in natural language processing (NLP), particularly those based on transformer architectures, have significantly accelerated progress across various NLP tasks. This study focuses on the reliability of transformer-based question answering (QA) models, specifically BERT models and its variants (RoBERTa, ALBERT, DistilBERT). These encoder-only pretrained transformers have demonstrated remarkable accuracy in QA tasks that can be treated as classification tasks. However, their reliability remains underexplored. This study evaluates the reliability of four BERT-based models by assessing response stability under two conditions: (1) internal model variations induced via Monte Carlo Dropout (MCD) and (2) input perturbations through paraphrasing. Using the SQuAD and QuAC datasets, we investigate how dropout rates affect prediction consistency and whether lexical changes impact answer stability. Our findings reveal that RoBERTa maintains higher reliability, whereas AlBERT and DistilBERT exhibit significant inconsistencies. Statistical analyses confirm that enabling MCD during prediction does not disrupt inference dynamics, validating its effectiveness as a reliability metric. These findings underscore the importance of evaluating both accuracy and stability in QA models to ensure stability in real-world applications. natural language processing large language models reliability estimation BERT models question answering obdelava naravnega jezika veliki jezikovni modeli ocenjevanje zanesljivosti modeli BERT odgovarjanje na vprašanja true false true Angleški jezik Slovenski jezik Članek v reviji 2026-08-26 10:32:33 2026-08-26 10:32:35 2026-08-27 04:35:51 0000-00-00 00:00:00 2026 0 0 21 str. no. Vol. 2026 0000-00-00 Zaloznikova Objavljeno NiDoloceno 0000-00-00 0000-00-00 0000-00-00 004.85:004.912:81'322 0952-813X 10.1080/0952813X.2026.2716084 288277507 RAZ_Yadav_Pooja_2026.pdf RAZ_Yadav_Pooja_2026.pdf 1 9CAEE05F93DFB84F202CAB2319D9A59C 052831de4f7bd415417b2ee099781b03cc9b2b16990a1a5cbb7320d392cdf32f b62711f3-a128-11f1-9b0d-0050569b8976 https://repozitorij.uni-lj.si/Dokument.php?lang=slv&id=245109 https://www.tandfonline.com/doi/full/10.1080/0952813X.2026.2716084 1 479c527b-a128-11f1-9b0d-0050569b8976 https://repozitorij.uni-lj.si/Dokument.php?lang=slv&id=245108 Fakulteta za računalništvo in informatiko 0 0 0