The use of artificial intelligence tools to create synthetic texts raises questions about the reliability of their automated detection. The aim of the study was to determine how effectively the selected detectors distinguish between authentic abstracts in the field of librarianship, synthetically generated abstracts, and humanized versions of synthetically generated texts. Three detectors capable of detecting Slovenian-language texts were analyzed: AIDetector.com, ZeroGPT.com, and MyDetector.ai.
The analysis included 44 original abstracts of scientific articles written in Slovenian and published in the journal Knjižnica between 2012 and 2017. The titles of these articles were used as the thematic basis for prompts entered into ChatGPT, on the basis of which synthetic abstracts were generated. The third group consisted of humanized texts, additionally transformed in ChatGPT from the synthetic abstracts in order to make them more closely resemble natural human writing and to reduce their synthetic-content detection scores.
The results showed that the detectors did not distinguish entirely reliably between human-written and synthetically generated texts. Synthetic abstracts were generally more frequently identified as artificially generated; however, both false-positive and false-negative results occurred. After humanization, the texts were assessed as less likely to have been artificially generated, in some cases even less so than the original human-written abstracts. The findings underscore the need for cautious and critical use of such tools, particularly in academic procedures where incorrect assessments can have serious negative consequences.
|