A showcase of ÉTS researchers’ publications and other contributions

Continuous emotion recognition with spatiotemporal convolutional neural networks


Downloads per month over past year

Teixeira, Thomas, Granger, Eric and Koerich, Alessandro Lameiras. 2021. « Continuous emotion recognition with spatiotemporal convolutional neural networks ». Applied Sciences, vol. 11, nº 24.
Compte des citations dans Scopus : 3.

[thumbnail of Granger-E-2021-23853.pdf]
Granger-E-2021-23853.pdf - Published Version
Use licence: Creative Commons CC BY.

Download (705kB) | Preview


Facial expressions are one of the most powerful ways to depict specific patterns in human behavior and describe the human emotional state. However, despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot correctly handle variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task, even for humans. This paper investigates the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) to deal with long video sequences captured in the wild for continuous emotion recognition. For such an aim, several 2D CNN models that were designed to model spatial information are extended to allow spatiotemporal representation learning from videos, considering a complex and multi-dimensional emotion space, where continuous values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks, combining 2D CNNs and long short term-memory units and inflated 3D CNN models, which are built by inflating the weights of a pre-trained 2D CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.

Item Type: Peer reviewed article published in a journal
Granger, Éric
Lameiras Koerich, Alessandro
Affiliation: Génie des systèmes, Génie logiciel et des technologies de l'information
Date Deposited: 24 Jan 2022 17:21
Last Modified: 03 Mar 2022 15:37

Actions (login required)

View Item View Item