Learning with noisy labels [~re]visited

Hudovernik, Valter (author), Rot, Žiga (author), Vovk, Klemen (author), Škodnik, Luka (author), Čehovin Zajc, Luka (author)

Learning with noisy labels (LNL) is a subfield of supervised machine learning investigating scenarios in which the training data contain errors. While most research has focused on synthetic noise, where labels are randomly corrupted, real-world noise from human annotation errors is more complex and less understood. Wei et al. (2022) introduced CIFAR-N, a dataset with human-labeled noise and claimed that real-world noise is fundamentally more challenging than synthetic noise. This study aims to reproduce their experiments on testing the characteristics of human-annotated label noise, memorization dynamics, and benchmarking of LNL methods. We successfully reproduce some of the claims but identify some quantitative discrepancies. Notably, our attempts to reproduce the reported benchmark reveal inconsistencies in the reported results. To address these issues, we develop a unified framework and propose a refined benchmarking protocol that ensures a fairer evaluation of LNL methods. Our findings confirm that real-world noise differs structurally from synthetic noise and is memorized more rapidly by deep networks. By open-sourcing our implementation, we provide a more reliable foundation for future research in LNL.

• Language: English
• Keywords: noisy learning, deep learning, repeatability of experiments
• Work type: Article
• Typology: 1.01 - Original Scientific Article
• Organization: FRI - Faculty of Computer and Information Science
• Publication status: Published
• Publication version: Version of Record
• Year: 2026
• Number of pages: 22 str.
• Numbering: Vol. 11, iss. 1, Art. 2
• PID: 20.500.12556/RUL-179046
• UDC: 004.85
• ISSN on article: 2430-3658
• DOI: 10.5281/zenodo.18401497
• COBISS.SI-ID: 267165699
• Publication date in RUL: 04.02.2026

Record is a part of a journal

• Title: The Rescience journal
• Publisher: The ReScience journal
• ISSN: 2430-3658
• COBISS.SI-ID: 525809177

Licences

• License: CC BY 4.0, Creative Commons Attribution 4.0 International
• Link: http://creativecommons.org/licenses/by/4.0/
• Description: This is the standard Creative Commons license that gives others maximum freedom to do what they want with the work as long as they credit the author.

Secondary language

• Language: Slovenian
• Keywords: učenje na šumnih podatkih, globoko učenje, ponovljivost eksperimentov