Meeting the machines half-way : testing LLMs for qualitative coding of values in texts

Moats, David (author), Pretnar Žagar, Ajda (author)

This article reports a small-scale experiment comparing human and Large Language Model (LLM) performance on a difficult qualitative coding task: assigning values to short text snippets. When annotators compared human and model outputs, they consistently preferred the human annotations. However, agreement among human coders was low. To explain this contradiction, we conducted follow-up interviews inspired by Cicourel’s work. The interviews revealed that human coders drew on linguistic nuance, cultural experience, and implicit assumptions about values and about the experiment. These assumptions differed from those underlying LLM outputs. We argue this is not simply a failure of LLMs. Rather, assigning predetermined categories to decontextualized text is inherently difficult for both humans and machines. Human annotators already ‘meet the machines halfway’ by being consistent and formal. Rather than treating LLMs as annotation machines, we propose three collaborative roles for LLMs: generating explanations, retrieving examples from data, and assisting with codebook development.

• Language: English
• Keywords: values, large language models, qualitative coding, ecological validity
• Work type: Article
• Typology: 1.01 - Original Scientific Article
• Organization: FRI - Faculty of Computer and Information Science
• Publication status: Published
• Publication version: Version of Record
• Year: 2026
• Number of pages: 13 str.
• Numbering: Vol. , no.
• PID: 20.500.12556/RUL-185416
• UDC: 004.8:81'322
• ISSN on article: 0018-7259
• DOI: 10.1080/00187259.2026.2703871
• COBISS.SI-ID: 286571779
• Publication date in RUL: 04.08.2026

Record is a part of a journal

• Title: Human organization : journal of the Society for Applied Anthropology
• Shortened title: Human organ.
• Publisher: Ta
• ISSN: 0018-7259
• COBISS.SI-ID: 25575936

Licences

• License: CC BY-NC 4.0, Creative Commons Attribution-NonCommercial 4.0 International
• Link: http://creativecommons.org/licenses/by-nc/4.0/
• Description: A creative commons license that bans commercial use, but the users don’t have to license their derivative works on the same terms.

Secondary language

• Language: Slovenian
• Keywords: vrednote, veliki jezikovni modeli, kvalitativno kodiranje, ekološka veljavnost