Multimodal planning in roboticsPesjak, Drejc (Avtor) Žabkar, Jure (Mentor) Large language modelsOpenAI gymnasiumRobot operating system (ROS)Planning domain definition language (PDDL)Planning in robotics represents an ongoing research challenge, as it requires the integration of sensing, reasoning, and execution. Although large language models (LLMs) provide a high degree of flexibility in planning, they often introduce hallucinated goals and actions and consequently lack the formal reliability of deterministic methods. In this thesis, we address this limitation by proposing a hybrid Sense–Plan–Code–Act (SPCA) framework that combines perception, LLM-based reasoning, and symbolic planning. Within the proposed approach, sensory information is first transformed into a symbolic description of the world in PDDL using an LLM. A heuristic planner is then used to generate a valid plan, which is subsequently converted to code by a second LLM. The generated code is first validated syntactically through compilation and then semantically in simulation. When errors are detected, local corrections can be applied and the process is repeated as necessary. The proposed method is evaluated in the OpenAI Gym MiniGrid reinforcement learning environment and in a Gazebo simulation on a UR5 robotic arm using a curriculum of tasks with increasing complexity. The system achieves a high level of performance and successfully completes approximately 70% of tasks with a relatively low number of simulation iterations.20252025-09-16 14:35:01Magistrsko delo/naloga173357sl