Adaptive Weighting of Handcrafted Feature Losses for Facial Expression Recognition

Weicheng Xie, Linlin Shen, Jinming Duan

Research output: Journal PublicationArticlepeer-review

17 Citations (Scopus)


Due to the importance of facial expressions in human-machine interaction, a number of handcrafted features and deep neural networks have been developed for facial expression recognition. While a few studies have shown the similarity between the handcrafted features and the features learned by deep network, a new feature loss is proposed to use feature bias constraint of handcrafted and deep features to guide the deep feature learning during the early training of network. The feature maps learned with and without the proposed feature loss for a toy network suggest that our approach can fully explore the complementarity between handcrafted features and deep features. Based on the feature loss, a general framework for embedding the traditional feature information into deep network training was developed and tested using the FER2013, CK+, Oulu-CASIA, and MMI datasets. Moreover, adaptive loss weighting strategies are proposed to balance the influence of different losses for different expression databases. The experimental results show that the proposed feature loss with adaptive weighting achieves much better accuracy than the original handcrafted feature and the network trained without using our feature loss. Meanwhile, the feature loss with adaptive weighting can provide complementary information to compensate for the deficiency of a single feature.

Original languageEnglish
Article number8786929
Pages (from-to)2787-2800
Number of pages14
JournalIEEE Transactions on Cybernetics
Issue number5
Publication statusPublished - May 2021
Externally publishedYes


  • Deep feature loss
  • expression recognition
  • handcrafted feature
  • loss adaptive weighting

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Adaptive Weighting of Handcrafted Feature Losses for Facial Expression Recognition'. Together they form a unique fingerprint.

Cite this