Measuring catastrophic forgetting in cross-lingual classification : transfer paradigms and tuning strategies

Koloski, Boshko (author), Škrlj, Blaž (author), Robnik Šikonja, Marko (author), Pollak, Senja (author)

Cross-lingual transfer leverages knowledge from a resource-rich source language, commonly English, to enhance performance in less-resourced target languages. Two widely used strategies are: Cross-Lingual Validation (CLV), which involves training on the source language and validating on the target language, and Intermediate Training (IT), where models are first fine-tuned on the source language and then further trained on the target language. While both strategies have been studied, their effects on encoder-based models for classification tasks remain underexplored. In this paper, we systematically compare these strategies across six multilingual classification tasks, evaluating downstream performance, catastrophic forgetting, and both zero-shot and full-shot scenarios. Additionally, we contrast parameter-efficient adapter methods with full-parameter fine-tuning. Our results show that IT generally performs better in the target language, whereas CLV more effectively preserves source-language knowledge across multiple cross-lingual transfers. These findings underscore the trade-offs between optimizing target performance and mitigating catastrophic forgetting.

• Language: English
• Keywords: cross-lingual transfer, cross-lingual learning, catastrophic-forgetting, document classification
• Work type: Article
• Typology: 1.01 - Original Scientific Article
• Organization: FRI - Faculty of Computer and Information Science
• Publication status: Published
• Publication version: Version of Record
• Year: 2025
• Number of pages: Str. 33509-33520
• Numbering: Vol. 13
• PID: 20.500.12556/RUL-178163
• UDC: 004.8
• ISSN on article: 2169-3536
• DOI: 10.1109/ACCESS.2025.3543608
• COBISS.SI-ID: 227115523
• Publication date in RUL: 20.01.2026

Record is a part of a journal

• Title: IEEE access
• Publisher: Institute of Electrical and Electronics Engineers
• ISSN: 2169-3536
• COBISS.SI-ID: 519839513

Licences

• License: CC BY 4.0, Creative Commons Attribution 4.0 International
• Link: http://creativecommons.org/licenses/by/4.0/
• Description: This is the standard Creative Commons license that gives others maximum freedom to do what they want with the work as long as they credit the author.

Secondary language

• Language: Slovenian
• Keywords: medjezikovno učenje, medjezikovni prenos, globoko učenje, strojno učenje