20.500.12556/RUL-155112 Quantifying the impact of context on the quality of manual hate speech annotation The quality of annotations in manually annotated hate speech datasets is crucial for automatic hate speech detection. This contribution focuses on the positive effects of manually annotating online comments for hate speech within the context in which the comments occur. We quantify the impact of context availability by meticulously designing an experiment: Two annotation rounds are performed, one in-context and one out-of-context, on the same English YouTube data (more than 10,000 comments), by using the same annotation schema and platform, the same highly trained annotators, and quantifying annotation quality through inter-annotator agreement. Our results show that the presence of context has a significant positive impact on the quality of the manual annotations. This positive impact is more noticeable among replies than among comments, although the former is harder to consistently annotate overall. Previous research reporting that out-of-context annotations favour assigning non-hate-speech labels is also corroborated, showing further that this tendency is especially present among comments inciting violence, a highly relevant category for hate speech research and society overall. We believe that this work will improve future annotation campaigns even beyond hate speech and motivate further research on the highly relevant questions of data annotation methodology in natural language processing, especially in the light of the current expansion of its scope of application. hate speech manual annotation inter-annotator agreement impact of context true false true Angleški jezik Ni določen Članek v reviji 2024-03-20 14:21:59 2024-03-20 14:22:00 2024-03-21 04:07:10 0000-00-00 00:00:00 2023 0 0 Str. 1481-1494 iss. 6 Vol. 29 2023 0000-00-00 Zaloznikova Objavljeno NiDoloceno 0000-00-00 0000-00-00 0000-00-00 004.8 1351-3249 10.1017/S1351324922000353 118777859 quantifying-the-impact-of-context-on-the-quality-of-manual-hate-speech-annotation.pdf quantifying-the-impact-of-context-on-the-quality-of-manual-hate-speech-annotation.pdf 1 7019AF32FD10551FF7B3C1301583B20C dab404bea3da2dc1442a9decf0d8ce6fb36a398f5c9898936b449696d4ad93ef 4ce1c120-e6bd-11ee-8d8c-0050569b8976 https://repozitorij.uni-lj.si/Dokument.php?lang=slv&id=181893 https://www.cambridge.org/core/journals/natural-language-engineering/article/quantifying-the-impact-of-context-on-the-quality-of-manual-hate-speech-annotation/B6E813E528CE094DBE489ABD3A047D8A 0 341c22b2-e6bd-11ee-8d8c-0050569b8976 https://repozitorij.uni-lj.si/Dokument.php?lang=slv&id=181892 Fakulteta za računalništvo in informatiko 0 0 0