Sarcasm detection with transfer learning from multiple sources

Đoković, Lazar (author), Robnik Šikonja, Marko (mentor)

Sarcasm detection is a natural language processing task of classifying whether an utterance is sarcastic or not. It is closely related to sentiment analysis since it often inverts surface sentiment. Despite the great interest and research done by the sentiment analysis community, it remains a challenging problem. This is because sarcastic sentences are highly dependent on context, and they are often accompanied by various non-verbal cues. Recent work in sarcasm detection mostly focuses on the Transformer architecture of neural networks and its application in high-resourced languages like English. To build a sarcasm detection dataset for Slovene, we leverage two modern techniques in machine translation and language modeling. The first approach uses a medium-size Transformer model trained specifically for neural machine translation, while the second method utilizes a very large generative model. We explore the viability of such datasets and how the size of a pretrained Transformer affects its ability to detect sarcasm. We use this data to train ensembles of Transformer-based models. We evaluate model performance using established methodologies. Our results show that larger models generally outperform smaller ones, and that ensembling can slightly improve sarcasm detection performance. Our best ensemble approach achieves an $\text{F}_1$-score of 0.765.

• Language: English
• Keywords: natural language processing, large language models, sarcasm detection, neural machine translation, BERT model, GPT model, Llama model
• Work type: Bachelor thesis/paper
• Typology: 2.11 - Undergraduate Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2024
• PID: 20.500.12556/RUL-160384
• COBISS.SI-ID: 208260867
• Publication date in RUL: 27.08.2024

Secondary language

• Language: Slovenian
• Title: Zaznavanje sarkazma s prenosom znanja iz več virov

Abstract

Zaznavanje sarkazma je naloga obdelave naravnega jezika, pri kateri ugotavljamo, ali je izjava sarkastična ali ne. Tesno je povezana z analizo mnenj, saj pogosto spremeni površinsko razumljeno mnenje. Kljub mnogim raziskavam ostaja sarkazem izziv za avtomatsko detekcijo, saj so sarkastični stavki odvisni od konteksta in jih pogosto spremljajo neverbalni znaki. Nedavni pristopi k zaznavanju sarkazma večinoma uporabljajo arhitekturo nevronskih mrež transformer v jezikih z veliko viri, predvsem v angleščini. Za izdelavo učne množice za zaznavanje sarkazma v slovenščini smo uporabili dve sodobni tehniki strojnega prevajanja in jezikovnega modeliranja. Prvi pristop uporablja srednje velik model transformer, učen posebej za nevronsko strojno prevajanje, medtem ko druga metoda uporablja zelo velik generativni jezikovni model. Raziskali smo uporabnost teh učnih množic in kako velikost modelov vpliva na njihovo sposobnost zaznavanja sarkazma. Z generiranimi podatki smo ustvarili več modelov in napovedni ansambel, sestavljen iz več jezikovnih modelov. Pristope smo ovrednotili z uporabo uveljavljenih metod. Rezultati kažejo, da večji modeli presegajo manjše, ansambli pa nekoliko izboljšajo uspešnost zaznavanja sarkazma. Naš najboljši ansambel doseže $\text{F}_1$-oceno 0,765.

• Keywords: obdelava naravnega jezika, veliki jezikovni modeli, zaznavanje sarkazma, strojno prevajanje, model BERT, model GPT, model Llama