Mono- and cross-lingual evaluation of representation language models on less-resourced languages

Ulčar, Matej (author), Žagar, Aleš (author), Armendariz, Carlos S. (author), Repar, Andraž (author), Pollak, Senja (author), Purver, Matthew (author), Robnik Šikonja, Marko (author)

The current dominance of large language models in natural language processing is based on their contextual awareness. For text classification, text representation models, such as ELMo, BERT, and BERT derivatives, are typically fine-tuned for a specific problem. Most existing work focuses on English; in contrast, we present a large-scale multilingual empirical comparison of several monolingual and multilingual ELMo and BERT models using 14 classification tasks in nine languages. The results show, that the choice of best model largely depends on the task and language used, especially in a cross-lingual setting. In monolingual settings, monolingual BERT models tend to perform the best among BERT models. Among ELMo models, the ones trained on large corpora dominate. Cross-lingual knowledge transfer is feasible on most tasks already in a zero-shot setting without losing much performance.

• Language: English
• Keywords: monolingual models, multilingual models, ELMo, BERT, corpus, cross-lingual datasets, language models, contextual embeddings, less-resourced languages, BERT, ELMo
• Work type: Article
• Typology: 1.01 - Original Scientific Article
• Organization: FRI - Faculty of Computer and Information Science
• Publication status: Published
• Publication version: Version of Record
• Year: 2026
• Number of pages: 29 str.
• Numbering: Vol. 95, art. 101852
• PID: 20.500.12556/RUL-182550
• UDC: 004.8
• ISSN on article: 1095-8363
• DOI: 10.1016/j.csl.2025.101852
• COBISS.SI-ID: 241622275
• Publication date in RUL: 15.05.2026

Record is a part of a journal

• Title: Computer speech & language
• Shortened title: Comput. speech lang.
• Publisher: Elsevier
• ISSN: 1095-8363
• COBISS.SI-ID: 203927043

Licences

• License: CC BY 4.0, Creative Commons Attribution 4.0 International
• Link: http://creativecommons.org/licenses/by/4.0/
• Description: This is the standard Creative Commons license that gives others maximum freedom to do what they want with the work as long as they credit the author.

Secondary language

• Language: Slovenian
• Keywords: korpusi, večjezični veliki modeli