Metamorphic testing harness for the Baidu Apollo perception-camera module

Yifan Zhang; Dave Towey; Matthew Pike; Jia Cheng Han; George Zhou; Chenghao Yin; Qian Wang; Chen Xie

doi:10.1109/MET59151.2023.00009

Metamorphic testing harness for the Baidu Apollo perception-camera module

Yifan Zhang
, Dave Towey^*
, Matthew Pike
, Jia Cheng Han
, George Zhou
, Chenghao Yin
, Qian Wang
, Chen Xie

^*Corresponding author for this work

Research output: Chapter in Book/Conference proceeding › Conference contribution › peer-review

10 Citations (Scopus)

Abstract

As the complexity of autonomous driving systems (ADSs) increases, the question of how to organize testing in an efficient manner has become a serious issue. This study investigates the potential for metamorphic testing (MT) to evaluate the perception-camera module of an open-source autonomous driving system (ADS), namely Baidu's Apollo ADS. The experiments revealed inconsistent obstacle identification results when increasing the brightness of a specific region of the driving scenarios, both in individual and sequential frames, demonstrating the ability of MT to address the oracle problem when testing the perception module of ADSs. Furthermore, this paper presents an MT harness to facilitate ADS testing, which would increase efficiency and help testers to better organize the testing procedure. We also present an industry case study to demonstrate its use in actual production phases.

Original language	English
Title of host publication	Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	9-16
Number of pages	8
ISBN (Electronic)	9798350301762
DOIs	https://doi.org/10.1109/MET59151.2023.00009
Publication status	Published - 2023
Event	8th IEEE/ACM International Workshop on Metamorphic Testing, MET 2023 - Melbourne, Australia Duration: 14 May 2023 → …

Publication series

Name	Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023

Conference

Conference	8th IEEE/ACM International Workshop on Metamorphic Testing, MET 2023
Country/Territory	Australia
City	Melbourne
Period	14/05/23 → …

Free Keywords

automated testing
Autonomous driving system (ADS)
metamorphic relation (MR)
metamorphic testing (MT)
oracle problem

ASJC Scopus subject areas

Software
Safety, Risk, Reliability and Quality
Modelling and Simulation

Access to Document

10.1109/MET59151.2023.00009

Cite this

Zhang, Y., Towey, D., Pike, M., Han, J. C., Zhou, G., Yin, C., Wang, Q., & Xie, C. (2023). Metamorphic testing harness for the Baidu Apollo perception-camera module. In Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023 (pp. 9-16). (Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MET59151.2023.00009

Zhang, Yifan ; Towey, Dave ; Pike, Matthew et al. / Metamorphic testing harness for the Baidu Apollo perception-camera module. Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 9-16 (Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023).

@inproceedings{853c3cd407a34ffb8942fcad572175d1,

title = "Metamorphic testing harness for the Baidu Apollo perception-camera module",

abstract = "As the complexity of autonomous driving systems (ADSs) increases, the question of how to organize testing in an efficient manner has become a serious issue. This study investigates the potential for metamorphic testing (MT) to evaluate the perception-camera module of an open-source autonomous driving system (ADS), namely Baidu's Apollo ADS. The experiments revealed inconsistent obstacle identification results when increasing the brightness of a specific region of the driving scenarios, both in individual and sequential frames, demonstrating the ability of MT to address the oracle problem when testing the perception module of ADSs. Furthermore, this paper presents an MT harness to facilitate ADS testing, which would increase efficiency and help testers to better organize the testing procedure. We also present an industry case study to demonstrate its use in actual production phases.",

keywords = "automated testing, Autonomous driving system (ADS), metamorphic relation (MR), metamorphic testing (MT), oracle problem",

author = "Yifan Zhang and Dave Towey and Matthew Pike and Han, \{Jia Cheng\} and George Zhou and Chenghao Yin and Qian Wang and Chen Xie",

note = "Funding Information: ACKNOWLEDGMENTS This work is supported by the Natural Science Foundation of China (project no. 61872167). The authors also acknowledge the financial support from the Artificial Intelligence and Optimisation (AIOP) and Sensors, Sensor Networks and Instrumentation (SSNI) research groups, the Faculty of Science and Engineering (FoSE), the International Doctoral Innovation Centre, Ningbo Education Bureau, Ningbo Science and Technology Bureau, and the University of Nottingham. Publisher Copyright: {\textcopyright} 2023 IEEE.; 8th IEEE/ACM International Workshop on Metamorphic Testing, MET 2023 ; Conference date: 14-05-2023",

year = "2023",

doi = "10.1109/MET59151.2023.00009",

language = "English",

series = "Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "9--16",

booktitle = "Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023",

address = "United States",

}

Zhang, Y, Towey, D , Pike, M, Han, JC, Zhou, G, Yin, C, Wang, Q & Xie, C 2023, Metamorphic testing harness for the Baidu Apollo perception-camera module. in Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023. Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023, Institute of Electrical and Electronics Engineers Inc., pp. 9-16, 8th IEEE/ACM International Workshop on Metamorphic Testing, MET 2023, Melbourne, Australia, 14/05/23. https://doi.org/10.1109/MET59151.2023.00009

Metamorphic testing harness for the Baidu Apollo perception-camera module. / Zhang, Yifan; Towey, Dave ; Pike, Matthew et al.
Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 9-16 (Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023).

Research output: Chapter in Book/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Metamorphic testing harness for the Baidu Apollo perception-camera module

AU - Zhang, Yifan

AU - Towey, Dave

AU - Pike, Matthew

AU - Han, Jia Cheng

AU - Zhou, George

AU - Yin, Chenghao

AU - Wang, Qian

AU - Xie, Chen

N1 - Funding Information: ACKNOWLEDGMENTS This work is supported by the Natural Science Foundation of China (project no. 61872167). The authors also acknowledge the financial support from the Artificial Intelligence and Optimisation (AIOP) and Sensors, Sensor Networks and Instrumentation (SSNI) research groups, the Faculty of Science and Engineering (FoSE), the International Doctoral Innovation Centre, Ningbo Education Bureau, Ningbo Science and Technology Bureau, and the University of Nottingham. Publisher Copyright: © 2023 IEEE.

PY - 2023

Y1 - 2023

N2 - As the complexity of autonomous driving systems (ADSs) increases, the question of how to organize testing in an efficient manner has become a serious issue. This study investigates the potential for metamorphic testing (MT) to evaluate the perception-camera module of an open-source autonomous driving system (ADS), namely Baidu's Apollo ADS. The experiments revealed inconsistent obstacle identification results when increasing the brightness of a specific region of the driving scenarios, both in individual and sequential frames, demonstrating the ability of MT to address the oracle problem when testing the perception module of ADSs. Furthermore, this paper presents an MT harness to facilitate ADS testing, which would increase efficiency and help testers to better organize the testing procedure. We also present an industry case study to demonstrate its use in actual production phases.

AB - As the complexity of autonomous driving systems (ADSs) increases, the question of how to organize testing in an efficient manner has become a serious issue. This study investigates the potential for metamorphic testing (MT) to evaluate the perception-camera module of an open-source autonomous driving system (ADS), namely Baidu's Apollo ADS. The experiments revealed inconsistent obstacle identification results when increasing the brightness of a specific region of the driving scenarios, both in individual and sequential frames, demonstrating the ability of MT to address the oracle problem when testing the perception module of ADSs. Furthermore, this paper presents an MT harness to facilitate ADS testing, which would increase efficiency and help testers to better organize the testing procedure. We also present an industry case study to demonstrate its use in actual production phases.

KW - automated testing

KW - Autonomous driving system (ADS)

KW - metamorphic relation (MR)

KW - metamorphic testing (MT)

KW - oracle problem

UR - https://www.scopus.com/pages/publications/85169102520

U2 - 10.1109/MET59151.2023.00009

DO - 10.1109/MET59151.2023.00009

M3 - Conference contribution

AN - SCOPUS:85169102520

T3 - Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023

SP - 9

EP - 16

BT - Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 8th IEEE/ACM International Workshop on Metamorphic Testing, MET 2023

Y2 - 14 May 2023

ER -

Zhang Y, Towey D , Pike M, Han JC, Zhou G, Yin C et al. Metamorphic testing harness for the Baidu Apollo perception-camera module. In Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 9-16. (Proceedings - 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing, MET 2023). doi: 10.1109/MET59151.2023.00009

Metamorphic testing harness for the Baidu Apollo perception-camera module

Abstract

Publication series

Conference

Free Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Validating autonomous driving: metamorphic testing for autonomous vehicles and simulators

Cite this

Metamorphic testing harness for the Baidu Apollo perception-camera module

Abstract

Publication series

Conference

Free Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Student theses

Validating autonomous driving: metamorphic testing for autonomous vehicles and simulators

Cite this