Knowledge inspection and injection in deep neural networks for text processingKlemen, Matej (Avtor) Robnik Šikonja, Marko (Mentor) Kosem, Iztok (Komentor) natural language processingknowledge injectionknowledge inspectionevaluation improvementslanguage modelIn the field of natural language processing, the aim is to develop methods for language understanding and text-based reasoning. Although latest approaches still do not truly understand language, large language models based on deep neural networks are increasingly successful at reaching these goals judging by their achieved predictive accuracy. While the models are accurate, considerably less is known about the knowledge contained within them. In our work, we focus on knowledge in language models and examine it from three perspectives: inspection of existing knowledge, injection of additional knowledge, and improvement of evaluation benchmarks to improve the assessment of the knowledge contained in the models. In the first part, we present a novel method for introducing additional morphological knowledge into LSTM and BERT models. Through experiments, we demonstrate differences in the knowledge contained in investigated models: additional knowledge often benefits LSTM models, whereas BERT models benefit only in cases where the knowledge is of high quality. In the second part, we first present a knowledge-enhanced method for spelling correction. We develop a method for the synthetic generation of a training dataset that incorporates additional knowledge in the form of rules describing how humans produce spelling errors. Using this method, we create a large synthetic training dataset to train the first Slovenian neural spell checker (SloNSpell) which achieves higher accuracy than existing methods. Next, we present an agent-based system augmented with procedural knowledge (UDagent) for grammar analysis based on corpus resources. We test the system on word order analysis problems and demonstrate improvements compared to a baseline system without additional knowledge. In the third part, we present an improved procedure for data collection and annotation for the task of natural language inference. Using this procedure, we create the Slovenian dataset SI-NLI, which contains 5,937 examples and is subsequently also translated into English. We use the dataset to train and evaluate a variety of models and demonstrate differences in their performance, where the highest performance is achieved by the largest language models. Our work highlights the need for greater emphasis on the knowledge contained in language models. Across specific tasks and a broad set of languages, we identify concrete challenges, propose solutions, and conclude that despite increasingly powerful language models, the need for incorporating additional knowledge remains.20262026-08-06 14:20:03Doktorsko delo/naloga185488sl