Position : explainable AI cannot advance without better user studies

Pičulin, Matej (author), Petek, Bernarda (author), Ograjenšek, Irena (author), Štrumbelj, Erik (author)

In this position paper, we argue that user studies are key to understanding the value of explainable AI methods, because the end goal of explainable AI is to satisfy societal desiderata. We also argue that the current state of user studies is detrimental to the advancement of the field. We support this argument with a review of general and explainable AI-specific challenges, as well as an analysis of 607 explainable AI papers featuring user studies. We demonstrate how most user studies lack reproducibility, discussion of limitations, comparison with a baseline, or placebo explanations and are of low fidelity to real-world users and application context. This, combined with an overreliance on functional evaluation, results in a lack of understanding of the value explainable AI methods, which hinders the progress of the field. To address this issue, we call for higher methodological standards for user studies, greater appreciation of high-quality user studies in the AI community, and reduced reliance on functional evaluation.

• Language: English
• Keywords: explainable artificial intelligence, explanation of artificial intelligence, evaluation of artificial intelligence, users, social aspects
• Typology: 1.08 - Published Scientific Conference Contribution
• Organization: FRI - Faculty of Computer and Information Science
• Publication status: Published
• Publication version: Version of Record
• Year: 2025
• Number of pages: 18 str.
• PID: 20.500.12556/RUL-183622
• UDC: 004.8
• COBISS.SI-ID: 255898115

Copyright

Informacija o licenci CC BY 4.0 je bila pridobljena s strani avtorjev članka. (Datum opombe: 22. 6. 2026)

• Publication date in RUL: 22.06.2026

Record is a part of a monograph

• Title: International Conference on Machine Learning, 13-19 July 2025, Vancouver Convention Center, Vancouver, Canada
• Editors: Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, Jerry Zhu
• Place of publishing: Cambridge (MA)
• Publisher: PMLR
• Year: 2025
• COBISS.SI-ID: 255897859
• Collection title: Proceedings of machine learning research
• Collection numbering: Vol. 267
• Collection ISSN: 2640-3498

Licences

• License: CC BY 4.0, Creative Commons Attribution 4.0 International
• Link: http://creativecommons.org/licenses/by/4.0/
• Description: This is the standard Creative Commons license that gives others maximum freedom to do what they want with the work as long as they credit the author.

Secondary language

• Language: Slovenian
• Keywords: razlaga umetne inteligence, razložljiva umetna inteligenca, vrednotenje umetne inteligence, uporabniki, družbeni vidiki

Projects

• Funder: ARRS - Slovenian Research Agency
• Project number: P2-0442
• Name: Podatkovne vede in digitalna preobrazba