Evalvacija sistema za strojno prevajanje predavanj

Krek, Luka (author), Popič, Damjan (mentor)

Ob današnjem tehnološkem napredku na področju zagotavljanja jezikovnih in prevajalskih storitev je pomembno, da temu sledi tudi univerzitetno študijsko okolje. Temu je namenjen projekt oz. proces razvoja sistema za strojno prevajanje predavanj Online Notes (ON), ki ga obravnava to magistrsko delo. Namen dela je predstaviti osnovne značilnosti delovanja sistema ON, ugotoviti, kako dobro deluje na trenutni stopnji razvoja in postaviti temelje za nadaljnje ocenjevanje njegove uspešnosti v prihodnje. Analizirano je delovanje sistema, ki je razdeljeno na tri stopnje – razpoznava govora v slovenščini, prevajanje besedila iz slovenščine v angleščino, in prikaz besedila v obliki podnapisov, ki je namenjen končnim uporabnikom. Pri prvi stopnji so analizirane napake v prepoznavanju in pretvorbi govora s pomočjo metode podrobne ročne evalvacije. Enako metodo smo nato uporabili pri evalvaciji prevodov, ki je bila poleg tega opravljena tudi strojno, in sicer z uporabo ocenjevalnih metrik BLEU in METEOR. Odzive končnih uporabnikov pri tretji stopnji smo preverili s pomočjo anketnega vprašalnika. Pokazalo se je, da so pri razpoznavi govora potrebne določene izboljšave, predvsem pri segmentaciji stavkov. Napake so posledično vplivale tudi na slabšo kvaliteto prevodov, kljub temu pa so rezultati prevajalnika znotraj sistema ON v primerjavi s prevajalnikoma Google in DeepL solidni. To potrjujejo tudi odzivi respondentov v anketnem vprašalniku. Rezultati raziskave kažejo, da je razvoj sistema na dobri poti in da lahko ob primernih izboljšavah pričakujemo kvaliteten sistem za strojno prevajanje predavanj, ki bo primeren za uporabo v predavalnicah.

• Language: Slovenian
• Keywords: Strojno prevajanje, ročna evalvacija, razpoznava govora, avtomatsko podnaslavljanje, analiza napak
• Work type: Master's thesis/paper
• Organization: FF - Faculty of Arts
• Year: 2021
• PID: 20.500.12556/RUL-133615
• COBISS.SI-ID: 102494211
• Publication date in RUL: 04.12.2021

Secondary language

• Language: English
• Title: Evaluation of a machine translation system for lectures

Abstract

With today's technological advances in language and translation services, it is important that the university learning environment keeps up with the times. This is the aim of the project for the development of Online Notes (ON), an automatic machine translation system for lectures. The purpose of this thesis is to present the basic features of the ON system, to determine how well it is working at its current stage of development and to lay the foundations for further evaluation of performance of the system in the future. The analysis of the system's performance is divided into three stages – automatic speech recognition (ASR) in Slovene, neural machine translation (NMT) from Slovene to English, and display of the text in subtitle form for end-users. At the first stage, errors in automatic speech recognition are analysed using a detailed manual evaluation method. The same method is applied in the evaluation of machine translation, which was also done automatically using the BLEU and METEOR evaluation metrics. Evaluations by the end-users were collected by means of a questionnaire. Results show that improvements are needed in speech recognition, and especially in sentence segmentation. Consequently, segmentation errors resulted in a lower quality of the translations, but the results of the ON system’s translator are solid compared to the Google and DeepL translators. This was also confirmed by the respondents' answers. Final results show that the development of the system is well on track and that, with appropriate improvements, we can expect an effective system for machine translation of lectures that will be suitable for use in lecture theatres.

• Keywords: Machine translation, manual evaluation, speech recognition, automatic captioning, error analysis