For years social interaction research was based on data gathered from surveys among participants in the study, or through video analysis, where the experts in the field of psychology and pedagogics observed the interactions and reported their final assessment. With the progress in development of technology and smaller sensors we can now make devices that can autonomously detect and analyze social interactions in an noninvasive way.
With intention of measuring human interactions I developed a system called Soci-Emo. With this system we can measure interactions within a group of people and determine their emotional state by monitoring their conversations. The system consist of two parts. The first part are detection badges which are worn around a person’s neck, and the second part is emotion recognition model that works on speech.
Detection badge consists of microcontroller nRF52811 connected to a microphone, microSD card, accelerometer, infrared sensor and infrared transmitter. This sensor enables me to autonomously measure proximity of other users within a group, their movement and also record their speech. With this sensor an autonomous algorithm determines if two users are in proximity and if they are talking. If that’s the case the badge starts recording sound and saving it to the microSD card. The badges can run continuously for at least three hours.
Emotional state of the speaker can be determined from the recorded sound by a machine learning model, specially made for this project. I chose to classify six different emotions: anger, disgust, fear, happiness, neutral and sadness. The model was developed with the help of six different datasets, consisting of audio recordings of different emotions. I used CREMA-D, EMO-DB, GEMEP, MSP-IMPROV, RAVDESS in SAVEE datasets. Trained model works with features calculated from openSmile tool. The model is built with the help of convolutional and dense neural networks and can classify emotions with 62% accuracy.
|
