Multimodal Emotion Analysis Based on Visual, Acoustic and Linguistic Features (CROSBI ID 719700)
Prilog sa skupa u zborniku | izvorni znanstveni rad | međunarodna recenzija
Podaci o odgovornosti
Koren, Leon ; Stipancic, Tomislav ; Ricko, Andrija ; Orsag, Luka
engleski
Multimodal Emotion Analysis Based on Visual, Acoustic and Linguistic Features
In this paper, a computational reasoning framework that can interpret social signals of the person in interaction by focusing on the person’s emotional state is presented. Two distinct sources of social signals are used for this study: facial and voice emotion modalities. As a part of the first modality, a Convolutional Neural Network (CNN) is used to extract and process the facial features based on live stream video. The voice emotion analysis containing two sub-modalities is driven by CNN and Long Short-Term Memory (LSTM) networks. The networks are analyzing the acoustic and linguistic features of the voice to determine the possible emotional cues of the person in interaction. Relying on the multimodal information fusion, the system then fuses data into a single hypothesis. Results of such reasoning are used to autonomously generate the robot responses which are shown in a form of non-verbal facial animations projected on the ‘face’ surface of the affective robot head PLEA. Built-in functionalities of the robot can provide a degree of situational embodiment, self-explainability and context-driven interaction.
Nonverbal behavior ; Multimodal interaction ; Artificial intelligence ; Cognitive Robotics ; Social Signal Processing
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
Podaci o prilogu
318-331.
2022.
objavljeno
10.1007/978-3-031-05061-9_23
Podaci o matičnoj publikaciji
Social Computing and Social Media: Design, User Experience and Impact. HCII 2022. Lecture Notes in Computer Science, vol 13315.
Meiselwitz, G
Springer
978-3-031-05060-2
Podaci o skupu
International Conference on Human-Computer Interaction (HCII 2022)
predavanje
26.06.2022-01.07.2022
online
Povezanost rada
Elektrotehnika, Informacijske i komunikacijske znanosti, Računarstvo, Strojarstvo