Samodejni merilnik čustvenega stanja socialne interakcije v skupini

Bertalanič, Blaž (author), Meža, Marko (mentor), Blatnik, Aljaž (co-mentor)

Raziskave o človeški interakciji znotraj skupin so leta temeljile le na podatkih, ki so jih preko vprašalnikov prispevali opazovanci v študiji ali pa so jih iz ogleda posnetkov interakcij pripravili strokovnjaki na področju psihologije in pedagogike, ki so nato oblikovali končno oceno. Razvoj tehnologije in manjšanje senzorjev nam sedaj omogoča izdelavo merilnih naprav, s katerimi lahko samodejno merimo ter analiziramo človeške interakcije na neinvaziven način. Z namenom merjenja človeških interakcij sem razvil sistem imenovan Soci-Emo. Z njim merimo interakcije znotraj skupine na podlagi spremljanja pogovorov med posamezniki in določanjem njihovega čustvenega stanja. Sistem je sestavljen iz dveh delov in sicer iz nosljivih merilnih značk in modela za razpoznavo čustvenega stanja iz govora. Merilne značke so sestavljene iz mikrokrmilnika nRF52811 na katerega so priklopljeni mikrofon, microSD kartica, merilnik pospeška, infrardeč sprejemnik in oddajnik. Ti senzorji mi omogočajo samodejno merjenje bližine, gibanja in snemanje govora. Na podlagi meritev senzorjev algoritem v znački ugotovi ali sta dve osebi v bližini in ali se pogovarjata. V tem primeru značka prične s shranjevanjem zvoka na microSD kartico. Izdelane značke omogočajo vsaj 3 ure neprekinjenega delovanja. Iz zajetega zvoka lahko nato razpoznamo čustveno stanje govorcev. Klasificirati sem želel 6 različnih čustev in sicer jezo, gnus, strah, veselje, nevtralnost in žalost. Za to uporabljam s pomočjo strojnega učenja razvit model za klasificiranje čustev v govoru. Model sem razvil na podlagi 6 podatkovnih zbirk, ki vsebujejo posneti govor in podatke o klasifikaciji govora v čustveno stanje govorca. Uporabil sem zbirke CREMA-D, EMO-DB, GEMEP, MSP-IMPROV, RAVDESS in SAVEE. Model deluje na značilkah izračunanih iz govora z orodjem openSmile. Za gradnjo modela sem uporabil kombinacijo konvolucijskih in navadnih nevronskih mrež. Model izbrana čustva v govoru napoveduje z 62% pravilnostjo.

• Language: Slovenian
• Keywords: sociometer, strojno učenje, razpoznavanje čustev iz govora, merilec socialne interakcije, nRF52811, openSmile.
• Work type: Master's thesis/paper
• Organization: FE - Faculty of Electrical Engineering
• Year: 2020
• PID: 20.500.12556/RUL-119954
• Publication date in RUL: 14.09.2020

Secondary language

• Language: English
• Title: System for measurement of the emotional state during the social interaction in a group of people

Abstract

For years social interaction research was based on data gathered from surveys among participants in the study, or through video analysis, where the experts in the field of psychology and pedagogics observed the interactions and reported their final assessment. With the progress in development of technology and smaller sensors we can now make devices that can autonomously detect and analyze social interactions in an noninvasive way. With intention of measuring human interactions I developed a system called Soci-Emo. With this system we can measure interactions within a group of people and determine their emotional state by monitoring their conversations. The system consist of two parts. The first part are detection badges which are worn around a person’s neck, and the second part is emotion recognition model that works on speech. Detection badge consists of microcontroller nRF52811 connected to a microphone, microSD card, accelerometer, infrared sensor and infrared transmitter. This sensor enables me to autonomously measure proximity of other users within a group, their movement and also record their speech. With this sensor an autonomous algorithm determines if two users are in proximity and if they are talking. If that’s the case the badge starts recording sound and saving it to the microSD card. The badges can run continuously for at least three hours. Emotional state of the speaker can be determined from the recorded sound by a machine learning model, specially made for this project. I chose to classify six different emotions: anger, disgust, fear, happiness, neutral and sadness. The model was developed with the help of six different datasets, consisting of audio recordings of different emotions. I used CREMA-D, EMO-DB, GEMEP, MSP-IMPROV, RAVDESS in SAVEE datasets. Trained model works with features calculated from openSmile tool. The model is built with the help of convolutional and dense neural networks and can classify emotions with 62% accuracy.

• Keywords: sociometer, machine learning, speech emotion recognition, social interaction meter, nRF52811, openSmile.