Motion prediction in human-robot collaboration using deep recurrent neural networks

MAVSAR, MATIJA (author), Ude, Aleš (mentor)

Collaboration between humans and robots has become increasingly popular in the last decade, since it enables complex tasks while alleviating human workers of stressful and demanding labor. Safe and efficient human-robot cooperation requires an effective system for supervision and control of the collaborative workspace. A number of deep neural network architectures have been developed that are suitable for analysis and prediction of dynamic processes, which often occur in collaborative environments. Furthermore, data augmentation methods, such as simulation, data randomization and synthetic data generation, can additionally improve the performance of motion prediction systems by incorporating diverse information into training data. In this dissertation, a number of approaches to enable robot and human motion prediction in collaborative tasks are proposed, where various neural network architectures and data augmentation methods are designed and tested. In the first part of the dissertation, the focus lies on optimizing task fluency by designing a collaboration supervision system that comprises automatic motion detection and motion classification, where the latter is used for categorizing observed human motions during human-robot collaboration (HRC) tasks. Firstly, a recurrent neural network system for human motion classification from RGB-D videos is compared to a system that makes predictions based on input marker positions, showing that classification accuracy is comparable for both input types. Secondly, a recently more popular architecture, namely a transformer network, is employed and compared to a custom recurrent network, as well as to an adapted existing architecture used for action recognition in HRC. The proposed networks outperform the existing model, while the use of one-dimensional convolutional and pooling layers further increases accuracy of motion classification. The use of third-order DMPs is proposed for description of robot tasks and for enabling smooth robot motion adaptation in real time when new predictions of observed motion are computed using motion classification networks. The developed methods are implemented in a real collaborative use case and result in a more fluent and safe task sharing between a human and a robot. In the second part, the use of generative adversarial networks (GANs) for supplementation of real data for motion classification tasks is explored. A training methodology based on GANs that utilizes a recurrent architecture for generation of synthetic robot and human motion videos during a collaborative task is introduced. The architecture is trained in a semi-supervised manner, with the output classification networks predicting one of the possible labels for the observed motion, while the recurrent generator networks produce synthetic RGB videos that are leveraged in the training process. Results show that utilization of synthetic data during the semi-supervised training increases the accuracy and generalization capability of the trained motion classification models. In the final part of the dissertation, recurrent architectures for motion prediction during object handover tasks are presented. First, an end-to-end recurrent neural network for predicting robot motion during a robot-to-robot object handover task is developed. The network processes input color-depth (RGB-D) videos of a giver robot, passing an object to another, receiving robot, and outputs the desired trajectory of the receiving robot, or the predicted trajectory that the giver robot will perform. This enables adaptive control of the receiving robot, which can start moving towards the predicted exchange location as soon as the giver begins its motion, resulting in a more fluent and dynamic handover process. Techniques for automatic generation of highly randomized simulated robot motion videos and data augmentation to increase the size of the training dataset are proposed, showing that mixing real and simulated data is beneficial for the accuracy of motion prediction. Secondly, a system for object handover location prediction from input human hand trajectories is designed and implemented in a real human-to-robot handover experiment using a humanoid robot. The developed methodologies presented in the dissertation have been tested and shown to achieve accurate motion prediction during cooperative tasks, enabling dynamic collaboration of humans and robots.

• Language: English
• Keywords: deep learning, recurrent neural networks, motion prediction, human-robot collaboration, adaptive robot control, motion recognition, dynamic movement primitives.
• Work type: Doctoral dissertation
• Organization: FE - Faculty of Electrical Engineering
• Year: 2024
• PID: 20.500.12556/RUL-162577
• Publication date in RUL: 25.09.2024

Secondary language

• Language: Slovenian
• Title: Predikcija gibanja pri sodelovanju robota s človekom z uporabo globokih rekurzivnih nevronskih mrež

Abstract

Sodelovanje med ljudmi in roboti je v zadnjem desetletju vse bolj priljubljeno, saj omogoča izvajanje kompleksnih nalog, hkrati pa delavce razbremenjuje stresnega in zahtevnega dela. Varno in učinkovito sodelovanje med človekom in robotom zahteva učinkovit sistem za nadzor in kontrolo sodelovalnega delovnega prostora. Razvite so bile številne arhitekture globokih nevronskih mrež, ki so primerne za analizo in napovedovanje dinamičnih procesov v sodelovalnih okoljih. Metode bogatenja podatkov, kot so simulacija, randomizacija in ustvarjanje sintetičnih podatkov, pa lahko dodatno izboljšajo učinkovitost sistemov za napovedovanje gibanja, saj dodajo nove informacije v učne podatke. V tej disertaciji so predlagani številni pristopi za napovedovanje gibanja robotov in ljudi pri sodelovalnih nalogah, pri čemer so zasnovane in preizkušene različne arhitekture nevronskih mrež in metode bogatenja podatkov. V prvem delu disertacije je poudarek na optimizaciji nalog z oblikovanjem sistema za nadzor sodelovanja, ki vključuje samodejno zaznavanje gibanja in klasifikacijo gibanja, s čimer želimo kategorizirati opazovane človeške gibe pri sodelovanju človeka in robota. Zasnovan je sistem z rekurzivno nevronsko mrežo za klasifikacijo človeških gibov iz barvno-globinskih videoposnetkov, in je primerjan s sistemom, ki napoveduje na podlagi položajev markerjev. Pokaže se, da je natančnost klasifikacije primerljiva za obe vrsti vhodnih podatkov. Drugič, uporabljena je nedavno bolj priljubljena arhitektura, in sicer transformatorska mreža, ki je primerjana z rekurzivno mrežo ter s prilagojeno obstoječo arhitekturo, ki se uporablja za prepoznavanje dejanj v sodelovalni robotiki. Predlagane mreže se izkažejo kot boljše od obstoječega modela, medtem ko uporaba enodimenzionalnih konvolucijskih in združevalnih slojev dodatno poveča natančnost klasifikacije gibanja. Za opis robotskih nalog in omogočanje gladkega prilagajanja gibanja robota v realnem času, ko se izračunajo nove napovedi opazovanega gibanja, so uporabljeni dinamični generatorji gibov tretjega reda. Razvite metode so preizkušene na realnem primeru sodelovanja in omogočajo bolj tekočo in varno delitev nalog med človekom in robotom. V drugem delu je raziskana uporaba generativnih nasprotniških mrež za dopolnitev realnih podatkov pri nalogah klasifikacije gibanja. Predstavljena je delno nadzorovana metoda učenja s pomočjo generativnih nasprotniških mrež, ki uporablja rekurzivno arhitekturo za generiranje sintetičnih videoposnetkov gibanja robotov in ljudi med sodelovalno nalogo. Pri tem izhodne klasifikacijske mreže napovedujejo eno od možnih oznak za opazovano gibanje, medtem ko rekurzivne generatorske mreže izdelujejo sintetične videoposnetke, ki se uporabljajo v procesu učenja. Rezultati kažejo, da uporaba sintetičnih podatkov med delno nadzorovanim učenjem poveča natančnost in sposobnost posploševanja naučenih modelov razvrščanja gibanja. V zadnjem delu disertacije so predstavljene rekurzivne arhitekture za napovedovanje gibanja med nalogami predaje predmetov. Najprej je razvita rekurzivna nevronska mreža, ki obdeluje vhodne barvno-globinske videoposnetke podajalnega robota med podajanjem predmeta drugemu, prejemnemu robotu, in napoveduje želeno trajektorijo prejemnega robota ali trajektorijo, ki jo bo izvedel podajalni robot. To omogoča prilagodljivo krmiljenje prejemnega robota, ki se lahko začne premikati proti predvideni lokaciji predaje predmeta takoj, ko podajalec začne svoje gibanje, kar omogoča dinamičen postopek predaje. Predlagane so tehnike za samodejno generiranje raznolikih simuliranih videoposnetkov gibanja robota in bogatenje podatkov, s čimer lahko povečamo bazo učnih primerov. Rezultati pokažejo, da je mešanje resničnih in simuliranih podatkov koristno za natančnost napovedovanja gibanja. Nazadnje je zasnovan sistem za napovedovanje lokacije predaje predmeta iz vhodnih trajektorij človeške roke, ki je bil preizkušen v realnem eksperimentu predaje med človekom in humanoidnim robotom. Razvite metode, predstavljene v disertaciji, so izkazale natančno napovedovanje gibanja med sodelovalnimi nalogami, kar omogoča dinamično sodelovanje ljudi in robotov.

• Keywords: globoko učenje, rekurzivne nevronske mreže, napoved gibanja, sodelovanje robota in človeka, adaptivno robotsko vodenje, prepoznavanje gibov, dinamični generatorji gibov.