Empowering NPC dialogue with environmental context using LLMs and panoramic images

Radež, Grega (avtor), Bohak, Ciril (avtor)

We present an approach for enhancing non-playable characters (NPCs) in games by combining large language models (LLMs) with computer vision to provide contextual awareness of their surroundings. Conventional NPCs typically rely on pre-scripted dialogue and lack spatial understanding, which limits their responsiveness to player actions and reduces overall immersion. Our method addresses these limitations by capturing panoramic images of an NPC's environment and applying semantic segmentation to identify objects and their spatial positions. The extracted information is used to generate a structured JSON representation of the environment, combining object locations derived from segmentation with additional scene graph data within the NPC's bounding sphere, encoded as directional vectors. This representation is provided as input to the LLM, enabling NPCs to incorporate spatial knowledge into player interactions. As a result, NPCs can dynamically reference nearby objects, landmarks, and environmental features, leading to more believable and engaging gameplay. We describe the technical implementation of the system and evaluate it in two stages. First, an expert interview was conducted to gather feedback and identify areas for improvement. After integrating these refinements, a user study was performed, showing that participants preferred the context-aware NPCs over a non-context-aware baseline, confirming the effectiveness of the proposed approach.

• Jezik: Angleški jezik
• Ključne besede: context-aware NPCs, immersive gameplay, environmental perception, dynamic interaction systems, LLM, NPC, development, games
• Vrsta gradiva: Drugo
• Tipologija: 1.25 - Drugi sestavni deli
• Organizacija: FRI - Fakulteta za računalništvo in informatiko
• Status publikacije: Objavljeno
• Različica publikacije: Objavljena publikacija
• Leto izida: 2026
• Št. strani: 24 str.
• PID: 20.500.12556/RUL-182224
• UDK: 004.8:004.93:795
• ISSN pri članku: 2331-8422
• DOI: 10.48550/arXiv.2604.19192
• COBISS.SI-ID: 276852995
• Datum objave v RUL: 04.05.2026

Gradivo je del revije

• Naslov: ArXiv.org
• Skrajšan naslov: ArXiv.org
• Založnik: Cornell University Library
• ISSN: 2331-8422
• COBISS.SI-ID: 10638614

Licence

• Licenca: CC BY 4.0, Creative Commons Priznanje avtorstva 4.0 Mednarodna
• Povezava: http://creativecommons.org/licenses/by/4.0/deed.sl
• Opis: To je standardna licenca Creative Commons, ki daje uporabnikom največ možnosti za nadaljnjo uporabo dela, pri čemer morajo navesti avtorja.

Sekundarni jezik

• Jezik: Slovenski jezik
• Ključne besede: zavedanje okolja, dinamični pogovori, kontekstualno obči NPC-ji, LLM, NPC, razvoj, igre