Cross-Lingual Transfer Learning for Joint Intent Classification and Slot FillingOsterman, Janez (Avtor) Dobrišek, Simon (Mentor) cross-lingual transfer learninglanguage modelsnatural language understandingintent classificationslot fillingIn today's world, ubiquitous voice assistants have seamlessly integrated into our daily lives. From Amazon's Alexa to Apple's Siri, people engage with these systems daily. Voice assistants are representatives of Natural Language Processing (NLP) systems, with Natural Language Understanding (NLU) forming a vital core component. In the context of in-car voice assistants like 'Hey Mercedes,' these systems play a crucial role in providing drivers with information spanning various domains. They facilitate communication on everyday topics and empower users to execute many different tasks simply by issuing voice commands, eliminating the need for manual intervention, such as closing windows. NLU enables the system to extract important information from the user's request which allows the system to provide a desired answer or take a desired action. There are many different NLU tasks, however, for systems like voice assistants intent classification and slot filling are two crucial ones. Intent classification primarily concerns identifying the underlying purpose or intention behind a user's request, while slot filling is dedicated to extracting specific slots from the user's request. These slots contribute detailed, specific information that complements the overall intent, resulting in a more precise and comprehensive system response. Language models form the foundation of natural language understanding. Traditionally, these models require a large amount of training data with annotations to become proficient in specific tasks. When we aim to enable a system to effectively comprehend and operate in a specific language, the presence of annotated data in that language becomes a necessity. The challenge arises when we face a shortage of annotated training datasets for a particular language. English, being widely prevalent on the internet, boasts abundant datasets, as do a handful of other high-resource languages. However, for the majority of languages, often referred to as low-resource languages, the scarcity of annotated datasets presents a significant hurdle. Developing datasets in a new language can be a costly and time-intensive task. This is why cross-lingual transfer learning techniques have gained significant traction in recent years. These methods utilize annotated datasets in high-resource languages to make the model perform well in some particular low-resource target language. The successful implementation of efficient cross-lingual transfer learning techniques not only mitigates costs but also expedites the incorporation of new languages into voice assistant systems. The 'Hey Mercedes' voice assistant encompasses numerous domains, many of which are accessible in multiple languages. When the objective is to integrate a new language into the voice assistant, the preference is to leverage pre-existing training datasets in high-resource languages rather than embarking on the creation of a brand-new dataset tailored to the target language. The primary objective of this thesis is to delve into a variety of cross-lingual transfer learning approaches and identify the most effective one. Our investigation encompasses four distinct groups of approaches: direct transfer, translation-based, code-switching, and in-context learning methods. The first three approaches involve supervised fine-tuning of pre-trained BERT-based language models for our NLU task - joint intent classification and slot filling. In contrast, the in-context learning approach is grounded in the concept of learning from analogy through the examination of a few annotated data examples. Our research encompasses a comprehensive set of experiments conducted in two separate domains - ATIS and Mercedes Car Functions. In the ATIS domain, we designate English as our high-resource language and strive to attain robust performance on test languages, including German, Spanish, French, and Portuguese. Meanwhile, in the Mercedes Car Functions domain, we select German as our high-resource language and aim to achieve good model performance in English, French, Spanish, and Portuguese. The findings clearly illustrate that the code-switching cross-lingual transfer learning approach yields the most favorable results. Nevertheless, it's crucial to note that this approach is heavily dependent on the availability of high-quality translators, which may not always be guaranteed. Therefore, while code-switching is a promising approach, it may not always be the most suitable solution, prompting the need for consideration of other approaches.20232023-11-24 12:00:00Magistrsko delo/naloga152441sl