Due to their effectiveness, large pre-trained language models are currently one of the main approaches to natural language processing. These models are trained on extensive collections of texts that contain various forms of biases and stereotypes. In this thesis, we explore how these biases and stereotypes are reflected in pre-trained language models for the Slovenian language, particularly in the SloBERTa model, pretrained mostly on standard language. To detect stereotypes, we employ a language model to predict missing words in sentences describing the characteristics of individual social groups. Using this approach, we identify salient attributes for a wide range of social groups, analyse their sentiment and statistically evaluate the results. The results of the analysis indicate statistically significant differences among the considered groups in terms of detected sentiment. It also turns out that the models reflect numerous negative stereotypes, which are more pronounced in models trained on colloquial language, such as SloBERTa-SlEng and SlEng-BERT. This work contributes to a better understanding of how large-scale language models work and the challenges associated with their use in practice. In addition, it has the potential to be used to uncover social stereotypes and attitudes towards individual social groups.
|
