Knowledge inspection and injection in deep neural networks for text processing

Klemen, Matej (avtor), Robnik Šikonja, Marko (mentor), Kosem, Iztok (komentor)

In the field of natural language processing, the aim is to develop methods for language understanding and text-based reasoning. Although latest approaches still do not truly understand language, large language models based on deep neural networks are increasingly successful at reaching these goals judging by their achieved predictive accuracy. While the models are accurate, considerably less is known about the knowledge contained within them. In our work, we focus on knowledge in language models and examine it from three perspectives: inspection of existing knowledge, injection of additional knowledge, and improvement of evaluation benchmarks to improve the assessment of the knowledge contained in the models. In the first part, we present a novel method for introducing additional morphological knowledge into LSTM and BERT models. Through experiments, we demonstrate differences in the knowledge contained in investigated models: additional knowledge often benefits LSTM models, whereas BERT models benefit only in cases where the knowledge is of high quality. In the second part, we first present a knowledge-enhanced method for spelling correction. We develop a method for the synthetic generation of a training dataset that incorporates additional knowledge in the form of rules describing how humans produce spelling errors. Using this method, we create a large synthetic training dataset to train the first Slovenian neural spell checker (SloNSpell) which achieves higher accuracy than existing methods. Next, we present an agent-based system augmented with procedural knowledge (UDagent) for grammar analysis based on corpus resources. We test the system on word order analysis problems and demonstrate improvements compared to a baseline system without additional knowledge. In the third part, we present an improved procedure for data collection and annotation for the task of natural language inference. Using this procedure, we create the Slovenian dataset SI-NLI, which contains 5,937 examples and is subsequently also translated into English. We use the dataset to train and evaluate a variety of models and demonstrate differences in their performance, where the highest performance is achieved by the largest language models. Our work highlights the need for greater emphasis on the knowledge contained in language models. Across specific tasks and a broad set of languages, we identify concrete challenges, propose solutions, and conclude that despite increasingly powerful language models, the need for incorporating additional knowledge remains.

• Jezik: Angleški jezik
• Ključne besede: natural language processing, knowledge injection, knowledge inspection, evaluation improvements, language model
• Vrsta gradiva: Doktorsko delo/naloga
• Tipologija: 2.08 - Doktorska disertacija
• Organizacija: FRI - Fakulteta za računalništvo in informatiko
• Leto izida: 2026
• PID: 20.500.12556/RUL-185488
• COBISS.SI-ID: 290995203
• Datum objave v RUL: 06.08.2026

Sekundarni jezik

• Jezik: Slovenski jezik
• Naslov: Pregled in vpeljevanje znanja v globoke nevronske mreže za obdelavo besedil

Izvleček

Na področju obdelave naravnega jezika stremimo k razvoju metod, ki omogočajo razumevanje jezika in sklepanje na podlagi besedila. Čeprav pristopi jezika še vedno ne razumejo, cilje, sodeč po doseženi napovedni točnosti, vse bolj dosegajo veliki jezikovni modeli na podlagi globokih nevronskih mrež. Kljub odličnemu delovanju pa o znanju, vsebovanem v jezikovnih modelih, vemo precej manj. V delu se osredotočimo na znanje v jezikovnih modelih, ki ga obravnavamo na tri načine: pregled obstoječega znanja, vpeljevanje dodatnega znanja in izboljševanje evalvacijskih množic za izboljšan pregled znanja. V prvem delu predstavimo novo metodo za vpeljevanje dodatnega morfološkega znanja v modele LSTM in BERT. Z eksperimenti pokažemo razlike v vsebovanem znanju v analiziranih modelih: modelom LSTM dodatno znanje pogosto koristi, medtem ko modelom BERT koristi zgolj v primerih, ko je znanje visoke kakovosti. V drugem delu najprej predstavimo z znanjem obogateno metodo za popravljanje črkovalnih napak. Razvijemo metodo za sintetično generiranje učne množice, ki vsebuje dodatno znanje v obliki pravil človeškega ustvarjanja črkovalnih napak. Z metodo ustvarimo veliko sintetično učno množico za učenje prvega slovenskega nevronskega črkovalnika SloNSpell, ki deluje natančneje od obstoječih metod. Nato predstavimo s postopkovnim znanjem obogaten agentni sistem (UDagent) za slovnično analizo, temelječo na korpusnih virih. Sistem testiramo na problemih analize besednega reda in pokažemo izboljšave v primerjavi z izhodiščnim sistemom brez dodatnega znanja. V tretjem delu predstavimo izboljšan postopek pridobivanja in označevanja podatkov za nalogo ugotavljanja pomenskega sosledja. S postopkom ustvarimo slovensko množico SI-NLI, ki vsebuje 5937 primerov in je naknadno prevedena tudi v angleščino. Množico uporabimo za učenje in evalvacijo številnih modelov in pokažemo razlike v njihovi uspešnosti, kjer najvišjo uspešnost dosegajo največji jezikovni modeli. V delu prikažemo potrebo po večjem poudarku na vsebovanem znanju v jezikovnih modelih. Na specifičnih nalogah in širokem naboru jezikov izpostavimo konkretne težave, predlagamo rešitve in ugotovimo, da kljub vse bolj zmogljivim jezikovnim modelom potreba po dodatnem znanju ostaja.

• Ključne besede: obdelava naravnega jezika, vpeljevanje dodatnega znanja, pregled obstoječega znanja, izboljševanje evalvacije, jezikovni model