End-To-End Audiovisual Feature Fusion for Active Speaker Detection

Fiseha B. Tesema, Zheyuan Lin, Shiqiang Zhu, Wei Song, Jason Gu, Hong Wu

Research output: Chapter in Book/Conference proceedingConference contributionpeer-review

2 Citations (Scopus)


Active speaker detection plays a vital role in human-machine interaction. Recently, a few end-to-end audiovisual frameworks emerged. However, these models' inference time was not explored and are not applicable for real-time applications due to their complexity and large input size. In addition, they explored a similar feature extraction strategy that employs the ConvNet on audio and visual inputs. This work presents a novel two-stream end-to-end framework fusing features extracted from images via VGG-M with raw Mel Frequency Cepstrum Coefficients features extracted from the audio waveform. The network has two BiGRU layers attached to each stream to handle each stream's temporal dynamic before fusion. After fusion, one BiGRU layer is attached to model the joint temporal dynamics. The experiment result on the AVA-ActiveSpeaker dataset indicates that our new feature extraction strategy shows more robustness to noisy signals and better inference time than models that employed ConvNet on both modalities. The proposed model predicts within 44.41 ms, which is fast enough for real-time applications. Our best-performing model attained 88.929% accuracy, nearly the same detection result as state-of-the-art work.

Original languageEnglish
Title of host publicationFourteenth International Conference on Digital Image Processing, ICDIP 2022
EditorsXudong Jiang, Wenbing Tao, Deze Zeng, Yi Xie
ISBN (Electronic)9781510657564
Publication statusPublished - 2022
Externally publishedYes
Event14th International Conference on Digital Image Processing, ICDIP 2022 - Wuhan, China
Duration: 20 May 202223 May 2022

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X


Conference14th International Conference on Digital Image Processing, ICDIP 2022


  • Audiovisual active speaker detection
  • Audiovisual fusion
  • BiGRU
  • MFCC
  • VGG-M

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'End-To-End Audiovisual Feature Fusion for Active Speaker Detection'. Together they form a unique fingerprint.

Cite this