Learning Person-Specific Cognition From Facial Reactions for Automatic Personality Recognition

Siyang Song, Zilong Shao, Shashank Jaiswal, Linlin Shen, Michel Valstar, Hatice Gunes

Research output: Journal PublicationArticlepeer-review

1 Citation (Scopus)


This article proposes to recognise the true (self-reported) personality traits from the target subject's cognition simulated from facial reactions. This approach builds on the following two findings in cognitive science: (i) human cognition partially determines expressed behaviour and is directly linked to true personality traits; and (ii) in dyadic interactions, individuals' nonverbal behaviours are influenced by their conversational partner's behaviours. In this context, we hypothesise that during a dyadic interaction, a target subject's facial reactions are driven by two main factors: their internal (person-specific) cognitive process, and the externalised nonverbal behaviours of their conversational partner. Consequently, we propose to represent the target subject's (defined as the listener) person-specific cognition in the form of a person-specific CNN architecture that has unique architectural parameters and depth, which takes audio-visual non-verbal cues displayed by the conversational partner (defined as the speaker) as input, and is able to reproduce the target subject's facial reactions. Each person-specific CNN is explored by the Neural Architecture Search (NAS) and a novel adaptive loss function, which is then represented as a graph representation for recognising the target subject's true personality. Experimental results not only show that the produced graph representations are well associated with target subjects' personality traits in both human-human and human-machine interaction scenarios, and outperform the existing approaches with significant advantages, but also demonstrate that the proposed novel strategies help in learning more reliable personality representations.

Original languageEnglish
Pages (from-to)3048-3065
Number of pages18
JournalIEEE Transactions on Affective Computing
Issue number4
Publication statusPublished - 1 Oct 2023
Externally publishedYes


  • True personality recognition
  • dyadic interaction
  • end-to-end graph representation learning
  • facial reaction generation
  • multi-dimensional edge feature
  • person-specific cognition simulation

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Learning Person-Specific Cognition From Facial Reactions for Automatic Personality Recognition'. Together they form a unique fingerprint.

Cite this