Multimodal planning in robotics

Pesjak, Drejc (author), Žabkar, Jure (mentor)

Planning in robotics represents an ongoing research challenge, as it requires the integration of sensing, reasoning, and execution. Although large language models (LLMs) provide a high degree of flexibility in planning, they often introduce hallucinated goals and actions and consequently lack the formal reliability of deterministic methods. In this thesis, we address this limitation by proposing a hybrid Sense–Plan–Code–Act (SPCA) framework that combines perception, LLM-based reasoning, and symbolic planning. Within the proposed approach, sensory information is first transformed into a symbolic description of the world in PDDL using an LLM. A heuristic planner is then used to generate a valid plan, which is subsequently converted to code by a second LLM. The generated code is first validated syntactically through compilation and then semantically in simulation. When errors are detected, local corrections can be applied and the process is repeated as necessary. The proposed method is evaluated in the OpenAI Gym MiniGrid reinforcement learning environment and in a Gazebo simulation on a UR5 robotic arm using a curriculum of tasks with increasing complexity. The system achieves a high level of performance and successfully completes approximately 70% of tasks with a relatively low number of simulation iterations.

• Language: English
• Keywords: Large language models, OpenAI gymnasium, Robot operating system (ROS), Planning domain definition language (PDDL)
• Work type: Master's thesis/paper
• Typology: 2.09 - Master's Thesis
• Organization: FRI - Faculty of Computer and Information Science
• Year: 2025
• PID: 20.500.12556/RUL-173357
• COBISS.SI-ID: 254171139
• Publication date in RUL: 16.09.2025

Secondary language

• Language: Slovenian
• Title: Večmodalno planiranje v robotiki

Abstract

Planiranje v robotiki ostaja pomemben raziskovalni izziv, zlasti pri integraciji kompleksnih zaznav in fleksibilnega odločanja. V zadnjih letih so veliki jezikovni modeli (LLM) postali priljubljeni zaradi svoje fleksibilnosti in zmožnosti reševanja različnih nalog, vendar pri planiranju pogosto prihaja do halucinacij ciljev in akcij ter izgube formalne zanesljivosti, ki jo ponujajo deterministični pristopi, kot je klasično planiranje. V magistrskem delu naslavljamo to težavo s kombinacijo LLM-jev in formalnih/determinističnih mehanizmov. Predlagamo zanko Sense–Plan–Code–Act (SPCA), v kateri prvi LLM pretvori zaznavanja v formalni opis sveta z uporabo jezika PDDL, hevristični algoritem za planiranje najde izvedljiv načrt, drugi LLM načrt pretvori v programsko kodo, prevajalnik preveri sintaktično pravilnost, simulacijsko okolje pa preveri semantično pravilnost izvedbe. V primeru napak se postopek ponovi, dokler ne dobimo delujoče končne rešitve. Predlagani pristop smo ovrednotili v okolju za spodbujevalno učenje OpenAI Gym MiniGrid in simulacijskem okolju Gazebo na robotski roki UR5. Vsako okolje ima svoj učni načrt različnih nalog, ki se stopnjujejo v težavnosti. Rezultati kažejo, da sistem doseže približno 70% uspešnost pri razmeroma majhnem številu simulacijskih ponovitev.

• Keywords: Veliki jezikovni modeli, OpenAI gymnasium, Robot operating system (ROS), Planning domain definition language (PDDL)