Meeting the machines half-way : testing LLMs for qualitative coding of values in texts

Moats, David (avtor), Pretnar Žagar, Ajda (avtor)

This article reports a small-scale experiment comparing human and Large Language Model (LLM) performance on a difficult qualitative coding task: assigning values to short text snippets. When annotators compared human and model outputs, they consistently preferred the human annotations. However, agreement among human coders was low. To explain this contradiction, we conducted follow-up interviews inspired by Cicourel’s work. The interviews revealed that human coders drew on linguistic nuance, cultural experience, and implicit assumptions about values and about the experiment. These assumptions differed from those underlying LLM outputs. We argue this is not simply a failure of LLMs. Rather, assigning predetermined categories to decontextualized text is inherently difficult for both humans and machines. Human annotators already ‘meet the machines halfway’ by being consistent and formal. Rather than treating LLMs as annotation machines, we propose three collaborative roles for LLMs: generating explanations, retrieving examples from data, and assisting with codebook development.

• Jezik: Angleški jezik
• Ključne besede: values, large language models, qualitative coding, ecological validity
• Vrsta gradiva: Članek v reviji
• Tipologija: 1.01 - Izvirni znanstveni članek
• Organizacija: FRI - Fakulteta za računalništvo in informatiko
• Status publikacije: Objavljeno
• Različica publikacije: Objavljena publikacija
• Leto izida: 2026
• Št. strani: 13 str.
• Številčenje: Vol. , no.
• PID: 20.500.12556/RUL-185416
• UDK: 004.8:81'322
• ISSN pri članku: 0018-7259
• DOI: 10.1080/00187259.2026.2703871
• COBISS.SI-ID: 286571779
• Datum objave v RUL: 04.08.2026

Gradivo je del revije

• Naslov: Human organization : journal of the Society for Applied Anthropology
• Skrajšan naslov: Human organ.
• Založnik: Ta
• ISSN: 0018-7259
• COBISS.SI-ID: 25575936

Licence

• Licenca: CC BY-NC 4.0, Creative Commons Priznanje avtorstva-Nekomercialno 4.0 Mednarodna
• Povezava: http://creativecommons.org/licenses/by-nc/4.0/deed.sl
• Opis: Licenca Creative Commons, ki prepoveduje komercialno uporabo, vendar uporabniki ne rabijo upravljati materialnih avtorskih pravic na izpeljanih delih z enako licenco.

Sekundarni jezik

• Jezik: Slovenski jezik
• Ključne besede: vrednote, veliki jezikovni modeli, kvalitativno kodiranje, ekološka veljavnost