Segmentacija in postavljanje končnih ločil v slovenskih stavkih z modeli tipa BERT

VELIKONJA, NEJC (author), Robnik Šikonja, Marko (mentor)

V diplomski nalogi se osredotočamo na postavljanje končnih ločil v povedih. Modeli za postavljanje ločil so uporabni pri urejanju besedil, generiranih s prepoznavanjem govora in potencialno tudi pri popravljanju različnih besedil. Ugotoviti želimo, ali je poved pripovedna, vprašalna ali vzklična in kje se ta poved konča. Končna implementacija za vsako besedo v besedilu napove, ali in katero ločilo ji sledi. Uporabili smo slovenski različici modela BERT, ki sta uspešni pri obdelavi naravnega jezika. Model CroSloEngual BERT, ki je bil naučen na podlagi slovenskega, hrvaškega in angleškega jezika, in model SloBERTa, ki je bil naučen na izključno slovenskem jeziku, smo izpopolnili na pripravljeni učni množici. Rezultati kažejo, da model SloBERTa ločila napoveduje bolje od modela CroSloEngual BERT. Ugotovili smo tudi, da je težko napovedovati klicaje, saj jih v učni množici ni dovolj.

• Language: Slovenian
• Keywords: globoke nevronske mreže, obdelava naravnega jezika, model BERT, model RoBERTa, napovedovanje končnih ločil, segmentacija stavkov, transformerji, jezikovni model
• Work type: Bachelor thesis/paper
• Typology: 2.11 - Undergraduate Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2021
• PID: 20.500.12556/RUL-130323
• COBISS.SI-ID: 77868291
• Publication date in RUL: 13.09.2021

Secondary language

• Language: English
• Title: Slovene sentence segmentation and punctuation using BERT-like models

Abstract

The thesis focuses on the prediction of final punctuation in sentences. Punctuation prediction models are useful in speech recognition and potentially in correcting various texts. We want to predict where sentences end and whether they end with an exclamation, a period or a question mark. Our implementation predicts whether and what punctuation to place after each word. We used two Slovene variants of BERT model, both successful in natural language processing tasks. The CroSloEngual BERT model has been pretrained on Slovenian, Croatian and English language. We compared it to SloBERTa model, trained exclusively on Slovenian corpora. We fine-tuned these models on prepared data sets. Results show that SloBERTa model is better at predicting punctuation than the CroSloEngual BERT model. Results show that predicting exclamation mark is difficult due to a low number of training instances.

• Keywords: deep neural networks, natural language processing, end of sentence punctuation prediction, model RoBERTa, model BERT, sentence segmentation, transformers, language model