20.500.12556/RUL-173357 Multimodal planning in robotics Večmodalno planiranje v robotiki Planning in robotics represents an ongoing research challenge, as it requires the integration of sensing, reasoning, and execution. Although large language models (LLMs) provide a high degree of flexibility in planning, they often introduce hallucinated goals and actions and consequently lack the formal reliability of deterministic methods. In this thesis, we address this limitation by proposing a hybrid Sense–Plan–Code–Act (SPCA) framework that combines perception, LLM-based reasoning, and symbolic planning. Within the proposed approach, sensory information is first transformed into a symbolic description of the world in PDDL using an LLM. A heuristic planner is then used to generate a valid plan, which is subsequently converted to code by a second LLM. The generated code is first validated syntactically through compilation and then semantically in simulation. When errors are detected, local corrections can be applied and the process is repeated as necessary. The proposed method is evaluated in the OpenAI Gym MiniGrid reinforcement learning environment and in a Gazebo simulation on a UR5 robotic arm using a curriculum of tasks with increasing complexity. The system achieves a high level of performance and successfully completes approximately 70% of tasks with a relatively low number of simulation iterations. Planiranje v robotiki ostaja pomemben raziskovalni izziv, zlasti pri integraciji kompleksnih zaznav in fleksibilnega odločanja. V zadnjih letih so veliki jezikovni modeli (LLM) postali priljubljeni zaradi svoje fleksibilnosti in zmožnosti reševanja različnih nalog, vendar pri planiranju pogosto prihaja do halucinacij ciljev in akcij ter izgube formalne zanesljivosti, ki jo ponujajo deterministični pristopi, kot je klasično planiranje. V magistrskem delu naslavljamo to težavo s kombinacijo LLM-jev in formalnih/determinističnih mehanizmov. Predlagamo zanko Sense–Plan–Code–Act (SPCA), v kateri prvi LLM pretvori zaznavanja v formalni opis sveta z uporabo jezika PDDL, hevristični algoritem za planiranje najde izvedljiv načrt, drugi LLM načrt pretvori v programsko kodo, prevajalnik preveri sintaktično pravilnost, simulacijsko okolje pa preveri semantično pravilnost izvedbe. V primeru napak se postopek ponovi, dokler ne dobimo delujoče končne rešitve. Predlagani pristop smo ovrednotili v okolju za spodbujevalno učenje OpenAI Gym MiniGrid in simulacijskem okolju Gazebo na robotski roki UR5. Vsako okolje ima svoj učni načrt različnih nalog, ki se stopnjujejo v težavnosti. Rezultati kažejo, da sistem doseže približno 70% uspešnost pri razmeroma majhnem številu simulacijskih ponovitev. Large language models OpenAI gymnasium Robot operating system (ROS) Planning domain definition language (PDDL) Veliki jezikovni modeli OpenAI gymnasium Robot operating system (ROS) Planning domain definition language (PDDL) true false false Angleški jezik Slovenski jezik Magistrsko delo/naloga 2025-09-16 14:35:01 2025-09-16 14:35:07 2025-11-08 03:42:25 0000-00-00 00:00:00 2025 0 0 0000-00-00 NiDoloceno NiDoloceno NiDoloceno 0000-00-00 0000-00-00 0000-00-00 37840 254171139 Pesjak_Drejc_-_Vecmodalno_planiranje_v_robotiki.pdf Pesjak_Drejc_-_Vecmodalno_planiranje_v_robotiki.pdf 1 F8E1A93E6C030EBC9A507BDD201B95FF b0f68817b87ff31f37a8d21596f3e014df66903c5b0a6b3b02145c7cb6213a0e 68140e12-92f9-11f0-9328-0050569b8976 https://repozitorij.uni-lj.si/Dokument.php?lang=slv&id=217749 Fakulteta za računalništvo in informatiko 0 0 0