Self-Supervised Learning of Person-Specific Facial Dynamics for Automatic Personality Recognition

Siyang Song, Shashank Jaiswal, Enrique Sanchez, Georgios Tzimiropoulos, Linlin Shen, Michel Valstar

Research output: Journal PublicationArticlepeer-review

11 Citations (Scopus)


This article aims to solve two important issues that frequently occur in existing automatic personality analysis systems: 1. Attempting to use very short video segments or even single frames, rather than long-term behaviour, to infer personality traits; 2. Lack of methods to encode person-specific facial dynamics for personality recognition. To deal with these issues, this paper first proposes a novel Rank Loss which utilizes the natural temporal evolution of facial actions, rather than personality labels, for self-supervised learning of facial dynamics. Our approach first trains a generic U-net style model that can infer general facial dynamics learned from a set of unlabelled face videos. Then, the generic model is frozen, and a set of intermediate filters are incorporated into this architecture. The self-supervised learning is then resumed with only person-specific videos. This way, the learned filters' weights are person-specific, making them a valuable source for modeling person-specific facial dynamics. We then propose to concatenate the weights of the learned filters as a person-specific representation, which can be directly used to predict the personality traits without needing other parts of the network. We evaluate the proposed approach on both self-reported personality and apparent personality datasets. In addition to achieving promising results in the estimation of personality trait scores from videos, we show that the tasks conducted by the subject in the video matters, that fusion of a combination of tasks reaches highest accuracy, and that multi-scale dynamics are more informative than single-scale dynamics.

Original languageEnglish
Pages (from-to)178-195
Number of pages18
JournalIEEE Transactions on Affective Computing
Issue number1
Publication statusPublished - 1 Jan 2023
Externally publishedYes


  • Automatic personality analysis
  • CNN weights representation
  • convolution neural networks
  • facial temporal evolution
  • personalized dynamic layer
  • self-supervised learning

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Self-Supervised Learning of Person-Specific Facial Dynamics for Automatic Personality Recognition'. Together they form a unique fingerprint.

Cite this