This master’s thesis examines whether moral judgment and value alignment in large language models vary depending on the language of the prompt. The research focuses on a comparison between English and Slovene prompts in morally relevant tasks from the ETHICS dataset, which includes five ethical subsets: commonsense morality, deontology, justice, virtue ethics, and utilitarianism. Since ETHICS was originally developed in English and annotated by English-speaking annotators, the thesis establishes a Slovene human baseline in addition to the original English labels. The Slovene baseline is based on the majority judgments of Slovene annotators.
The empirical analysis includes two large language models: Llama 3.3 and GaMS3. The models were tested on a selected subset of ETHICS in English and Slovene. English model responses were evaluated against the original ETHICS labels, while Slovene responses were evaluated against the Slovene majority labels. The results show that both models achieve lower agreement with human reference judgments in Slovene than in English. The differences between the two languages are not evenly distributed across all ethical subsets but are most pronounced in utilitarianism and virtue ethics. Further analysis also revealed a bias toward label 1 in the models, particularly in the deontological subset and the GaMS3 model.
The thesis shows that results of moral evaluation in English cannot be directly generalized to Slovene. It also highlights the importance of localized human baselines in the evaluation of large language models in smaller languages, since a direct translation of an evaluation dataset does not necessarily provide an equivalent reference point in the target linguistic context.
|
