Autonomy in robotic surgery is an increasingly active research area, with growing focus on the automation of individual manipulation tasks. An important aspect of this field is the development of methods that enable robotic systems to learn task execution without explicitly programming individual motions. Two important approaches are reinforcement learning and behavior cloning. In reinforcement learning, a policy learns to perform a task through interaction with the environment and feedback in the form of a reward, whereas in behavior cloning, a policy is learned from demonstrations.
This master's thesis investigates the learning of robotic manipulation tasks in a simulation environment and the transfer of learned policies to a physical robotic system. A simulation environment with a model of the robotic system, equipped with two robotic arms and surgical instruments, was developed using Isaac Sim simulator and the Isaac Lab framework. The primary task was needle picking, in which a robotic arm must correctly grasp a surgical needle from different initial positions and lift it to a specified position. In addition, a more challenging needle handover task between the two robotic arms was considered.
For learning the needle-pickup task using reinforcement learning, the \textit{Proximal Policy Optimization} algorithm was used. The learned policy successfully completed 7 out of 10 trials in the simulation environment, achieving a success rate of 70 %. After transfer to the physical MOPS robotic system, it successfully completed 4 out of 10 trials, corresponding to a success rate of 40 %. Using behavior cloning, a success rate of 50 % was achieved for the needle-pickup task with the best-performing combination of training data, while the same success rate of 50 % was achieved for the needle-handover task.
The influence of the number of demonstrations and the number of samples per demonstration on the performance of the learned policies was analysed. For both the needle-pickup and needle-handover tasks, the highest success rate was achieved using 50 demonstrations with 300 samples per demonstration. Within the evaluated range of training data, the highest measured success rate for both tasks was therefore 50 %. The results show that the amount of training data affects the performance of the learned policies.
The thesis therefore addresses the complete process, from establishing the simulation environment and collecting demonstrations through teleoperation to learning robotic policies and experimentally evaluating them in both simulated and physical environments. The findings provide a basis for further research into the use of reinforcement learning and behavior cloning for learning more complex robotic manipulation tasks and for improving the reliability of transferring learned policies from simulation to the real environment.
|
