Self-supervised Learning of Person-specific Facial Dynamics for Automatic Personality Recognition

Siyang Song, Shashank Jaiswal, Enrique Sanchez, Georgios Tzimiropoulos, Linlin Shen, Michel Valstar

Research output: Journal PublicationArticlepeer-review

8 Citations (Scopus)


This paper aims to solve two important issues that frequently occur in existing automatic personality analysis systems: 1. Attempting to use very short video segments or even single frames to infer personality traits; 2. Lack of methods to encode person-specific facial dynamics for personality recognition. Hence, we proposes a novel Rank Loss which utilizes the natural temporal evolution of facial actions, rather than personality labels, for self-supervised learning of facial dynamics. Our approach first trains a generic U-net model that can infer general facial dynamics learned from unlabelled face videos. Then, the generic model is frozen, and a set of intermediate filters are incorporated into this architecture. The self-supervised learning is then resumed with only person-specific videos. This way, the learned filters' weights are person-specific, making them a valuable source for modeling person-specific facial dynamics. We then concatenate the weights of the learned filters as a person-specific representation, which can be directly used to predict the personality traits without needing other parts of the network. We evaluate the proposed approach on both self-reported personality and apparent personality datasets. Besides achieving promising results in personality trait estimation from videos, we show that fusion of tasks reaches highest accuracy, and that multi-scale dynamics are more informative than single-scale dynamics.

Original languageEnglish
JournalIEEE Transactions on Affective Computing
Publication statusAccepted/In press - 2021
Externally publishedYes


  • Automatic personality analysis
  • CNN weights representation
  • Convolution Neural Networks
  • Face recognition
  • Faces
  • Facial temporal evolution
  • Machine learning
  • personalized dynamic layer
  • Predictive models
  • Psychology
  • self-supervised learning
  • Task analysis
  • Training

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Self-supervised Learning of Person-specific Facial Dynamics for Automatic Personality Recognition'. Together they form a unique fingerprint.

Cite this