Retrieval-augmented text generation with domain-specific large language models fine-tuningIvanovski, Marko (Avtor) Žitnik, Slavko (Mentor) Ustvarjanje besedilodgovarjanje na vprašanjaiskanjekodirnikdekodirniktransformatorjiiskanje nestrukturiranih podatkovnatančna nastavitevmodeli vgrajevanjakvantizacijaDeveloping an automated question-answering system to streamline customer support email handling presents an effective solution to reduce manual response times and effort. Currently, support agents respond to emails manually, which is time-consuming and labor-intensive. We tested several configurations to build a system capable of automatically answering these emails. The data used was provided by Zebra BI and consisted of a structured corpus of support emails and an unstructured corpus derived from product documentation. We compared two main approaches for tackling such problems. The first one fine-tuned a Large Language Model to answer emails directly. This approach contained a generator component only. The second one utilized the Retrieval-Augmented Generation (RAG) architecture which contained both the retriever and generator. The retriever, retrieved similar emails and append them to the context of an off-the-shelf LLM tasked with answering the question given the context from the previous similar emails. For the latter, we implemented a custom dual-decoder retriever model using the LoRA training technique and quantization. The dual-decoder retriever model generated embeddings for both the email and email answer passages using separate decoders and ranked them based on the cosine similarity. We developed six different question-answering system configurations. Some configurations utilized both the retriever and generator component, while others had only the generator. The best-performing configuration featured our custom dual-decoder retriever model, which improved the system's ability to retrieve relevant information from the domain-specific email corpus. This also showed that training a retriever model and utilizing the RAG architecture is more effective in comparison to fine-tuning an LLM, in cases where there is lower amount of data which is of lower quality. The most optimal retriever, built on the Llama-2-7B architecture using LoRA and 4-bit quantization, achieved a 0.53 Accuracy@100 and 0.032 MRR@100. In comparison, the state-of-the-art BGE-large-en retriever model scored 0.282 Accuracy@100 and 0.009 MRR@100 on the same domain. When paired with GPT-4o as the generator, the dual-decoder retriever received a 1282 ELO rating in our manual evaluation, while the BGE-large-en retriever scored 1256 with the same generator, and the fine-tuned LLM model scored 1071. The experiments showed that the dual-decoder retriever configuration provided the most accurate and contextually relevant answers, outperforming the state-of-the-art configuration. It also showed that decoder LLM models can be utilized for building a retriever model with relatively small amount of data. The final system, integrated into a Chrome extension, had a significant impact on Zebra BI's support workflows by automating a large portion of the response process. This not only reduced response times but also improved the accuracy and consistency of answers provided to customers. The extension is now used semi-automatically by Zebra BI's support team, freeing up agents to focus their attention on more complex issues.20242024-12-24 10:51:39Magistrsko delo/naloga166209VisID: 37117COBISS_ID: 219818243sl