Empowering NPC dialogue with environmental context using LLMs and panoramic images

Radež, Grega (author), Bohak, Ciril (author)

We present an approach for enhancing non-playable characters (NPCs) in games by combining large language models (LLMs) with computer vision to provide contextual awareness of their surroundings. Conventional NPCs typically rely on pre-scripted dialogue and lack spatial understanding, which limits their responsiveness to player actions and reduces overall immersion. Our method addresses these limitations by capturing panoramic images of an NPC's environment and applying semantic segmentation to identify objects and their spatial positions. The extracted information is used to generate a structured JSON representation of the environment, combining object locations derived from segmentation with additional scene graph data within the NPC's bounding sphere, encoded as directional vectors. This representation is provided as input to the LLM, enabling NPCs to incorporate spatial knowledge into player interactions. As a result, NPCs can dynamically reference nearby objects, landmarks, and environmental features, leading to more believable and engaging gameplay. We describe the technical implementation of the system and evaluate it in two stages. First, an expert interview was conducted to gather feedback and identify areas for improvement. After integrating these refinements, a user study was performed, showing that participants preferred the context-aware NPCs over a non-context-aware baseline, confirming the effectiveness of the proposed approach.

• Language: English
• Keywords: context-aware NPCs, immersive gameplay, environmental perception, dynamic interaction systems, LLM, NPC, development, games
• Work type: Other
• Typology: 1.25 - Other Component Parts
• Organization: FRI - Faculty of Computer and Information Science
• Publication status: Published
• Publication version: Version of Record
• Year: 2026
• Number of pages: 24 str.
• PID: 20.500.12556/RUL-182224
• UDC: 004.8:004.93:795
• ISSN on article: 2331-8422
• DOI: 10.48550/arXiv.2604.19192
• COBISS.SI-ID: 276852995
• Publication date in RUL: 04.05.2026

Record is a part of a journal

• Title: ArXiv.org
• Shortened title: ArXiv.org
• Publisher: Cornell University Library
• ISSN: 2331-8422
• COBISS.SI-ID: 10638614

Licences

• License: CC BY 4.0, Creative Commons Attribution 4.0 International
• Link: http://creativecommons.org/licenses/by/4.0/
• Description: This is the standard Creative Commons license that gives others maximum freedom to do what they want with the work as long as they credit the author.

Secondary language

• Language: Slovenian
• Keywords: zavedanje okolja, dinamični pogovori, kontekstualno obči NPC-ji, LLM, NPC, razvoj, igre