Improving Large Language Models for Machine Translation Using Synthetic Preference Data

Vajda, Dario (author), Robnik Šikonja, Marko (mentor), Vreš, Domen (comentor)

Large language models have emerged as effective machine translation systems. We explore how a general instruction-tuned large language model can be improved for machine translation using relatively few easily produced data resources. With Slovene as our primary use case, we improve the GaMS-9B-Instruct model using Direct Preference Optimization (DPO) training on a programmatically curated and enhanced subset of a public dataset. As DPO requires pairs of quality-ranked instances, we generated its training dataset by translating English Wikipedia articles using two LLMs, GaMS-9B-Instruct and EuroLLM-9B-Instruct. We ranked the resulting translations based on heuristics coupled with automatic evaluation metrics such as COMET. The evaluation shows that our fine-tuned model outperforms both models involved in the dataset generation. In comparison to the baseline models, the fine-tuned model achieved a COMET score gain between 0.02 and 0.04 on translating a wide variety of texts. It also avoids language and formatting errors more consistently.

• Language: English
• Keywords: machine learning, machine translation, large language models
• Work type: Bachelor thesis/paper
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2026
• PID: 20.500.12556/RUL-184701
• Publication date in RUL: 13.07.2026

Secondary language

• Language: Slovenian
• Title: Izboljšanje velikih jezikovnih modelov za strojno prevajanje z uporabo sintetičnih podatkov o preferencah

Abstract

Veliki jezikovni modeli so se izkazali kot učinkoviti strojni prevajalniki. V delu raziskujemo, kako lahko splošno namenski veliki jezikovni model prilagojen za sledenje navodilom izboljšamo za strojno prevajanje z uporabo relativno majhne količine enostavno pridobljenih podatkovnih virov. Na primeru slovenščine izboljšamo model GaMS-9B-Instruct z uporabo neposredne optimizacije preferenc (DPO), s katero ga učimo na strojno generirani in izboljšani podmnožici javno dostopnih podatkov. Ker metoda DPO zahteva pare primerov, razvrščenih po kakovosti, smo njen nabor podatkov za učenje ustvarili s prevajanjem angleških člankov z Wikipedije z uporabo dveh modelov: GaMS-9B-Instruct in EuroLLM-9B-Instruct. Prevode smo razvrstili na podlagi hevristik v kombinaciji z avtomatskimi metrikami, kot je COMET. Evalvacija kaže, da naš prilagojen model presega zmogljivost obeh modelov, uporabljenih v procesu generiranja učne množice. V primerjavi z izhodiščnima modeloma je prilagojen model pri prevajanju raznolikih besedil dosegel izboljšanje vrednosti metrike COMET za približno 0,02 do 0,04. Prav tako se dosledneje izogiba jezikovnim in oblikovnim napakam.

• Keywords: strojno učenje, strojno prevajanje, veliki jezikovni modeli