Numerous modern companies collect significant quantities of client-interaction
data, such as call transcripts, e-mails, and meeting notes. Such data collections hold a large potential for generating data insights that can help
the company run its business. Because the understanding of adopting the
Retrieval-Augmented Generation (RAG) approach for knowledge extraction
in production environments is still limited, in this thesis we develop a method
for evaluating the impact of introducing a conversational analytics tool that
is based on RAG and enables business users to query the content of past
client interactions in natural language. The method is designed to work
independently of the chosen technological platform. We evaluate the impact quantitatively through a questionnaire based on the Technology Acceptance Model (TAM), administered to the stakeholders before and after
implementing RAG for conversational analytics in the organization. We also
evaluate the impact qualitatively through the interviews with the key stakeholders conducted after the implementation. To test the developed method,
we perform a typical case study. The results show statistically significant
improvements in certain variables of TAM, while the remaining ones remain
unchanged and do not deteriorate. The qualitative and quantitative results
corroborate each other. This thesis contributes to the evaluation of RAG
adoption in industrial conversational analytics applications.
|
