Question-answering from old sources with large language models

Tavchioski, Ilija (author), Robnik Šikonja, Marko (mentor)

With the expansion of artificial intelligence and the field of natural language processing, researchers and corporations trained their models on huge text corpora. However, a large portion of historical sources and knowledge remain underutilized due to significant differences in vocabulary, linguistic structure, and writing styles. In this work, we address historical texts and documents in less-resourced languages, focused on Slovenian. By using a digitized corpus of documents in historical Slovenian, we generated a QA (question-answering) dataset using Slovene large language model GaMS (Generative Model for Slovene). We supported our research on historical Slovenian with several methodologies such as fine-tuned large language models, PageIndex RAG (Retrieval-Augmented Generation) and a RAG approach with a hybrid retriever, expanded with different embeddings such as Sentence BERT, F2LLM and our own fine-tuned model. The results show that the GaMS3 model is the most suited for generating a good QA dataset from historical data and is the best performing model for question answering, while the hybrid retriever enhanced with embeddings calculated from our own finetuned model was best suited for retrieval tasks.

• Language: English
• Keywords: Natural Language Processing, Historical language, Machine Learning, Large Language Models, Question Answering
• Work type: Master's thesis/paper
• Typology: 2.09 - Master's Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2026
• PID: 20.500.12556/RUL-182246
• COBISS.SI-ID: 277475587
• Publication date in RUL: 05.05.2026

Secondary language

• Language: Slovenian
• Title: Odgovarjanje na vprašanja iz starih virov z velikimi jezikovnimi modeli

Abstract

Z razmahom umetne inteligence in področja obdelave naravnega jezika so modeli naučeni na ogromnih zbirkah besedil. Vendar pa velik del zgodovinskih virov in znanja ostaja neizkoriščen zaradi pomembnih razlik v besedišču, jezikovni strukturi in slogu pisanja. V tem delu obravnavamo zgodovinska besedila in dokumente v manj razvitih jezikih, s poudarkom na slovenščini. Z uporabo digitaliziranega korpusa dokumentov v zgodovinski slovenščini smo generirali podatkovno množico vprašanj in odgovorov z uporabo GaMS (Generativni Model Slovenščine). Našo raziskavo zgodovinske slovenščine smo podprli z več metodologijami, kot so prilagojeni veliki jezikovni modeli, PageIndex RAG (Retrieval-Augmented Generation) ter pristop RAG z uporabo hibridnega iskalnika, razširjenega z različnimi vektorskimi vložitvami, kot so Sentence BERT, F2LLM in naš lastni prilagojen model. Rezultati so pokazali, da je model GaMS3 najprimernejši za generiranje kakovostne množico vprašanj in odgovorov iz zgodovinskih podatkov ter najboljši za odgovarjanje na vprašanja, medtem ko je hibridni iskalnik, nadgrajen z vektorskimi vložitvami, izračunanimi z lastnim prilagojenim modelom, najprimernejši za naloge iskanja.

• Keywords: obdelava naravnega jezika, zgodovinski jezik, strojno učenje, veliki jezikovni modeli, odgovarjanje na vprašanja