Retrieval-augmented generation (RAG) systems reduce the risk of hallucinations in large language models by using external knowledge. However, a lack of relevant material in trusted sources limits the range of questions these systems can answer. Content from online communities can fill this gap, but it is not necessarily reliable. We designed and developed a system with separate indexes for two sources with different levels of credibility. The system prioritizes passages from a scientific corpus. When sufficient evidence is unavailable, it relies on claims from an online forum and assesses their trustworthiness. If neither source provides a reliable basis for an answer, the system abstains. We evaluated the system in the health domain by examining source routing, claim classification, trust assessment and response grounding. We demonstrated the feasibility of an approach that adapts source selection to the available material and its credibility.
|
