Feature flow: In-network feature flow estimation for video object detection

Ruibing Jin, Guosheng Lin, Changyun Wen, Jianliang Wang, Fayao Liu

Research output: Journal PublicationArticlepeer-review

8 Citations (Scopus)

Abstract

Optical flow, which expresses pixel displacement, is widely used in many computer vision tasks to provide pixel-level motion information. However, with the remarkable progress of the convolutional neural network, recent state-of-the-art approaches are proposed to solve problems directly on feature-level. Since the displacement of feature vector is not consistent with the pixel displacement, a common approach is to forward optical flow to a neural network and fine-tune this network on the task dataset. With this method, they expect the fine-tuned network to produce tensors encoding feature-level motion information. In this paper, we rethink about this de facto paradigm and analyze its drawbacks in the video object detection task. To mitigate these issues, we propose a novel network (IFF-Net) with an In-network Feature Flow estimation module (IFF module) for video object detection. Without resorting to pre-training on any additional dataset, our IFF module is able to directly produce feature flow which indicates the feature displacement. Our IFF module consists of a shallow module, which shares the features with the detection branches. This compact design enables our IFF-Net to accurately detect objects, while maintaining a fast inference speed. Furthermore, we propose a transformation residual loss (TRL) based on self-supervision, which further improves the performance of our IFF-Net. Our IFF-Net outperforms existing methods and achieves new state-of-the-art performance on ImageNet VID.

Original languageEnglish
Article number108323
JournalPattern Recognition
Volume122
DOIs
Publication statusPublished - Feb 2022
Externally publishedYes

Keywords

  • Deep convolutional neural network (DCNN)
  • Feature flow
  • Object detection
  • Video analysis
  • Video object detection

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Feature flow: In-network feature flow estimation for video object detection'. Together they form a unique fingerprint.

Cite this