MoVE-CNNs: Model aVeraging Ensemble of Convolutional Neural Networks for Facial Expression Recognition

Facial expression is a powerful non-verbal communication that can express emotions and messages without saying a single word. In view of the prominence of facial expression, we propose a model averaging ensemble of Convolutional Neural Networks (CNN) that consolidates multiple pre-trained CNN models. Each pre-trained CNN model first undergoes transfer learning with the classification layer substituted with a multilayer perceptron. The newly formed model is then fine-tuned on the facial expression datasets and adapted to facial expression recognition. The predictions returned by all models are combined by model averaging to determine the final class probability distributions. The proposed model averaging ensemble of CNNs is evaluated on three facial expression datasets: FER-2013, modified CK+ and RAF-DB. Since the modified CK+ dataset is a small dataset, data augmentation is leveraged to increase the size and diversity of data. Apart from that, oversampling is adopted to address the class imbalance challenge in RAF-DB. The empirical results demonstrate that the proposed model averaging ensemble of CNNs outperforms the individual ensemble model at the test accuracy of 77.70%, 94.10% and 87.50% in FER 2013, modified CK+ and RAF-DB datasets, respectively.