Slovenian legislation is extensive, causing legal professionals to spend a significant amount of time each day searching for relevant literature. To address this, we explored the effectiveness of large language models (LLMs) as legal assistants. LLMs have been successful in various tasks, but handling complex domain-specific questions remains one of their major weaknesses; often producing hallucinations. Retrieval-Augmented Generation (RAG) is a technique that bypasses the lack of domain knowledge in LLMs by retrieving content from legislation based on the question, allowing for accurate responses. With the retrieved knowledge, the LLM can correctly answer the question without hallucinating. We explored and implemented several different RAG techniques. All methods were tested on a manually crafted test set containing four test scenarios to evaluate how successful the methods are in various situations. More advanced versions of RAG, such as advanced and modular RAG, show good performance in direct questions but lower success in more general questions, such as real-world examples.
|
