Spectral Representation of Behaviour Primitives for Depression Analysis

Siyang Song, Shashank Jaiswal, Linlin Shen, Michel Valstar

Research output: Journal PublicationArticlepeer-review

31 Citations (Scopus)


Depression is a serious mental disorder affecting millions of people all over the world. Traditional clinical diagnosis methods are subjective, complicated and require extensive participation of clinicians. Recent advances in automatic depression analysis systems promise a future where these shortcomings are addressed by objective, repeatable, and readily available diagnostic tools to aid health professionals in their work. Yet there remain a number of barriers to the development of such tools. One barrier is that existing automatic depression analysis algorithms base their predictions on very brief sequential segments, sometimes as little as one frame. Another barrier is that existing methods do not take into account what the context of the measured behaviour is. In this article, we extract multi-scale video-level features for video-based automatic depression analysis. We propose to use automatically detected human behaviour primitives as the low-dimensional descriptor for each frame. We also propose two novel spectral representations, i.e., spectral heatmaps and spectral vectors, to represent video-level multi-scale temporal dynamics of expressive behaviour. Constructed spectral representations are fed to Convolution Neural Networks (CNNs) and Artificial Neural Networks (ANNs) for depression analysis. We conducted experiments on the AVEC 2013 and AVEC 2014 benchmark datasets to investigate the influence of interview tasks on depression analysis. In addition to achieving state of the art accuracy in severity of depression estimation, we show that the task conducted by the user matters, that fusion of a combination of tasks reaches highest accuracy, and that longer tasks are more informative than shorter tasks, up to a point.

Original languageEnglish
Pages (from-to)829-844
Number of pages16
JournalIEEE Transactions on Affective Computing
Issue number2
Publication statusPublished - 2022


  • Automatic depression analysis
  • convolution neural networks
  • fourier transform
  • spectral representation
  • time-frequency analysis

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Spectral Representation of Behaviour Primitives for Depression Analysis'. Together they form a unique fingerprint.

Cite this