In-place metamorphic testing and exploration

Zhi Quan Zhou, Junting Zhu, Tsong Yueh Chen, Dave Towey

Research output: Chapter in Book/Conference proceedingConference contributionpeer-review

Abstract

In this paper, we consider the testing of software whose input involves a data stream, such as in video analysis applications. To address the oracle problem in testing this kind of software, existing metamorphic testing (MT) and metamorphic exploration (ME) techniques typically generate both source and follow-up input streams. In contrast, we propose a different strategy: an inplace method. Instead of examining relations among multiple input streams, our method works on a finer granularity by investigating relations among the data frames within a single input stream - such as the relations among the image frames within a video stream - treating a frame, rather than a stream, as a test case. Because our method does not generate any follow-up input stream, or change any frame values, it is also suitable for runtime error detection. We have applied this in-place method to testing video streams in Baidu Apollo, a real-life autonomous driving system. Our study identified previously-unknown obstacle perception failures in the camera perception module, including both undetected and incorrectly-detected objects. The empirical results show that our approach is practical, and readily applicable to industrial-scale systems that include (but are not limited to) the computer vision domain.

Original languageEnglish
Title of host publicationProceedings - 7th International Workshop on Metamorphic Testing, MET 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-6
Number of pages6
ISBN (Electronic)9781450393072
DOIs
Publication statusPublished - 2022
Event7th IEEE/ACM International Workshop on Metamorphic Testing, MET 2022 - Pittsburgh, United States
Duration: 9 May 2022 → …

Publication series

NameProceedings - 7th International Workshop on Metamorphic Testing, MET 2022

Conference

Conference7th IEEE/ACM International Workshop on Metamorphic Testing, MET 2022
Country/TerritoryUnited States
CityPittsburgh
Period9/05/22 → …

Keywords

  • Autonomous driving
  • Baidu Apollo
  • metamorphic exploration
  • metamorphic relation
  • metamorphic runtime checking
  • metamorphic testing
  • object detection
  • oracle problem
  • stream and frame
  • video content analysis

ASJC Scopus subject areas

  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'In-place metamorphic testing and exploration'. Together they form a unique fingerprint.

Cite this