Metamorphic testing of an automated parking system: an experience report

Dave Towey; Zepei Luo; Ziqi Zheng; Peijian Zhou; Junbo Yang; Puttipatt Ingkasit; Changyang Lao; Matthew Pike; Yifan Zhang

doi:10.1109/COMPSAC57700.2023.00274

Metamorphic testing of an automated parking system: an experience report

Dave Towey^*
, Zepei Luo
, Ziqi Zheng
, Peijian Zhou
, Junbo Yang
, Puttipatt Ingkasit
, Changyang Lao
, Matthew Pike
, Yifan Zhang

^*Corresponding author for this work

Research output: Chapter in Book/Conference proceeding › Conference contribution › peer-review

4 Citations (Scopus)

Abstract

Automated Driving Systems (ADSs) have gained popularity recently. However, the unstable and unsafe ADSs have caused many traffic accidents and received widespread attention. One way to alleviate such issues is to enhance the correctness and efficiency of testing ADSs. Due to the difficulty of checking ADSs' behavior such as parking the car, confirming the correctness of the actual behavior may be non-trivial or impossible. This kind of problem is called the test oracle problem. Unlike traditional software testing, Metamorphic Testing (MT) does not focus on the correctness of the actual strategy but examines whether or not the inputs and outputs of multiple executions of a Software Under Test (SUT) satisfy certain relations of the SUT, called Metamorphic Relations (MRs). The paper also implements Mutation Analysis (MA) on Baidu Apollo ADS to evaluate our MT. MA involves small modifications to a program's source code to see if test-cases can detect these changes. This work was part of a larger endeavour to create an Open Educational Resource (OER) to support learning about how to apply MT to ADSs. This paper reports on an experience of implementing MT to test the Automated Parking System (APS) of Apollo ADS and applying MA to evaluate the MT.

Original language	English
Title of host publication	Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023
Editors	Hossain Shahriar, Yuuichi Teranishi, Alfredo Cuzzocrea, Moushumi Sharmin, Dave Towey, AKM Jahangir Alam Majumder, Hiroki Kashiwazaki, Ji-Jiang Yang, Michiharu Takemoto, Nazmus Sakib, Ryohei Banno, Sheikh Iqbal Ahamed
Publisher	IEEE Computer Society
Pages	1774-1779
Number of pages	6
ISBN (Electronic)	9798350326970
DOIs	https://doi.org/10.1109/COMPSAC57700.2023.00274
Publication status	Published - 2023
Event	47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023 - Hybrid, Torino, Italy Duration: 26 Jun 2023 → 30 Jun 2023

Publication series

Name	Proceedings - International Computer Software and Applications Conference
Volume	2023-June
ISSN (Print)	0730-3157

Conference

Conference	47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023
Country/Territory	Italy
City	Hybrid, Torino
Period	26/06/23 → 30/06/23

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being
SDG 11 Sustainable Cities and Communities

Free Keywords

Automated Parking System
Baidu Apollo
Metamorphic Relation
Metamorphic Relation Output Pattern
Metamorphic Testing
Mutation Analysis
Test Oracle Problem

ASJC Scopus subject areas

Software
Computer Science Applications

Access to Document

10.1109/COMPSAC57700.2023.00274

Cite this

Towey, D., Luo, Z., Zheng, Z., Zhou, P., Yang, J., Ingkasit, P., Lao, C., Pike, M., & Zhang, Y. (2023). Metamorphic testing of an automated parking system: an experience report. In H. Shahriar, Y. Teranishi, A. Cuzzocrea, M. Sharmin, D. Towey, AKM. J. A. Majumder, H. Kashiwazaki, J.-J. Yang, M. Takemoto, N. Sakib, R. Banno, & S. I. Ahamed (Eds.), Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023 (pp. 1774-1779). (Proceedings - International Computer Software and Applications Conference; Vol. 2023-June). IEEE Computer Society. https://doi.org/10.1109/COMPSAC57700.2023.00274

Towey, Dave ; Luo, Zepei ; Zheng, Ziqi et al. / Metamorphic testing of an automated parking system : an experience report. Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023. editor / Hossain Shahriar ; Yuuichi Teranishi ; Alfredo Cuzzocrea ; Moushumi Sharmin ; Dave Towey ; AKM Jahangir Alam Majumder ; Hiroki Kashiwazaki ; Ji-Jiang Yang ; Michiharu Takemoto ; Nazmus Sakib ; Ryohei Banno ; Sheikh Iqbal Ahamed. IEEE Computer Society, 2023. pp. 1774-1779 (Proceedings - International Computer Software and Applications Conference).

@inproceedings{c5bd46d1a0aa4bb687ede06d6ff3ae9f,

title = "Metamorphic testing of an automated parking system: an experience report",

abstract = "Automated Driving Systems (ADSs) have gained popularity recently. However, the unstable and unsafe ADSs have caused many traffic accidents and received widespread attention. One way to alleviate such issues is to enhance the correctness and efficiency of testing ADSs. Due to the difficulty of checking ADSs' behavior such as parking the car, confirming the correctness of the actual behavior may be non-trivial or impossible. This kind of problem is called the test oracle problem. Unlike traditional software testing, Metamorphic Testing (MT) does not focus on the correctness of the actual strategy but examines whether or not the inputs and outputs of multiple executions of a Software Under Test (SUT) satisfy certain relations of the SUT, called Metamorphic Relations (MRs). The paper also implements Mutation Analysis (MA) on Baidu Apollo ADS to evaluate our MT. MA involves small modifications to a program's source code to see if test-cases can detect these changes. This work was part of a larger endeavour to create an Open Educational Resource (OER) to support learning about how to apply MT to ADSs. This paper reports on an experience of implementing MT to test the Automated Parking System (APS) of Apollo ADS and applying MA to evaluate the MT.",

keywords = "Automated Parking System, Baidu Apollo, Metamorphic Relation, Metamorphic Relation Output Pattern, Metamorphic Testing, Mutation Analysis, Test Oracle Problem",

author = "Dave Towey and Zepei Luo and Ziqi Zheng and Peijian Zhou and Junbo Yang and Puttipatt Ingkasit and Changyang Lao and Matthew Pike and Yifan Zhang",

note = "Funding Information: The authors acknowledgethe financial support from the Artificial Intelligence and Optimisation Research Group (AIOP) and the Faculty of Science and Engineering (FoSE). Publisher Copyright: {\textcopyright} 2023 IEEE.; 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023 ; Conference date: 26-06-2023 Through 30-06-2023",

year = "2023",

doi = "10.1109/COMPSAC57700.2023.00274",

language = "English",

series = "Proceedings - International Computer Software and Applications Conference",

publisher = "IEEE Computer Society",

pages = "1774--1779",

editor = "Hossain Shahriar and Yuuichi Teranishi and Alfredo Cuzzocrea and Moushumi Sharmin and Dave Towey and Majumder, \{AKM Jahangir Alam\} and Hiroki Kashiwazaki and Ji-Jiang Yang and Michiharu Takemoto and Nazmus Sakib and Ryohei Banno and Ahamed, \{Sheikh Iqbal\}",

booktitle = "Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023",

address = "United States",

}

Towey, D, Luo, Z, Zheng, Z, Zhou, P, Yang, J, Ingkasit, P, Lao, C, Pike, M & Zhang, Y 2023, Metamorphic testing of an automated parking system: an experience report. in H Shahriar, Y Teranishi, A Cuzzocrea, M Sharmin, D Towey, AKMJA Majumder, H Kashiwazaki, J-J Yang, M Takemoto, N Sakib, R Banno & SI Ahamed (eds), Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023. Proceedings - International Computer Software and Applications Conference, vol. 2023-June, IEEE Computer Society, pp. 1774-1779, 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023, Hybrid, Torino, Italy, 26/06/23. https://doi.org/10.1109/COMPSAC57700.2023.00274

Metamorphic testing of an automated parking system: an experience report. / Towey, Dave; Luo, Zepei; Zheng, Ziqi et al.
Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023. ed. / Hossain Shahriar; Yuuichi Teranishi; Alfredo Cuzzocrea; Moushumi Sharmin; Dave Towey; AKM Jahangir Alam Majumder; Hiroki Kashiwazaki; Ji-Jiang Yang; Michiharu Takemoto; Nazmus Sakib; Ryohei Banno; Sheikh Iqbal Ahamed. IEEE Computer Society, 2023. p. 1774-1779 (Proceedings - International Computer Software and Applications Conference; Vol. 2023-June).

Research output: Chapter in Book/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Metamorphic testing of an automated parking system

T2 - 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023

AU - Towey, Dave

AU - Luo, Zepei

AU - Zheng, Ziqi

AU - Zhou, Peijian

AU - Yang, Junbo

AU - Ingkasit, Puttipatt

AU - Lao, Changyang

AU - Pike, Matthew

AU - Zhang, Yifan

N1 - Funding Information: The authors acknowledgethe financial support from the Artificial Intelligence and Optimisation Research Group (AIOP) and the Faculty of Science and Engineering (FoSE). Publisher Copyright: © 2023 IEEE.

PY - 2023

Y1 - 2023

N2 - Automated Driving Systems (ADSs) have gained popularity recently. However, the unstable and unsafe ADSs have caused many traffic accidents and received widespread attention. One way to alleviate such issues is to enhance the correctness and efficiency of testing ADSs. Due to the difficulty of checking ADSs' behavior such as parking the car, confirming the correctness of the actual behavior may be non-trivial or impossible. This kind of problem is called the test oracle problem. Unlike traditional software testing, Metamorphic Testing (MT) does not focus on the correctness of the actual strategy but examines whether or not the inputs and outputs of multiple executions of a Software Under Test (SUT) satisfy certain relations of the SUT, called Metamorphic Relations (MRs). The paper also implements Mutation Analysis (MA) on Baidu Apollo ADS to evaluate our MT. MA involves small modifications to a program's source code to see if test-cases can detect these changes. This work was part of a larger endeavour to create an Open Educational Resource (OER) to support learning about how to apply MT to ADSs. This paper reports on an experience of implementing MT to test the Automated Parking System (APS) of Apollo ADS and applying MA to evaluate the MT.

AB - Automated Driving Systems (ADSs) have gained popularity recently. However, the unstable and unsafe ADSs have caused many traffic accidents and received widespread attention. One way to alleviate such issues is to enhance the correctness and efficiency of testing ADSs. Due to the difficulty of checking ADSs' behavior such as parking the car, confirming the correctness of the actual behavior may be non-trivial or impossible. This kind of problem is called the test oracle problem. Unlike traditional software testing, Metamorphic Testing (MT) does not focus on the correctness of the actual strategy but examines whether or not the inputs and outputs of multiple executions of a Software Under Test (SUT) satisfy certain relations of the SUT, called Metamorphic Relations (MRs). The paper also implements Mutation Analysis (MA) on Baidu Apollo ADS to evaluate our MT. MA involves small modifications to a program's source code to see if test-cases can detect these changes. This work was part of a larger endeavour to create an Open Educational Resource (OER) to support learning about how to apply MT to ADSs. This paper reports on an experience of implementing MT to test the Automated Parking System (APS) of Apollo ADS and applying MA to evaluate the MT.

KW - Automated Parking System

KW - Baidu Apollo

KW - Metamorphic Relation

KW - Metamorphic Relation Output Pattern

KW - Metamorphic Testing

KW - Mutation Analysis

KW - Test Oracle Problem

UR - https://www.scopus.com/pages/publications/85168867208

U2 - 10.1109/COMPSAC57700.2023.00274

DO - 10.1109/COMPSAC57700.2023.00274

M3 - Conference contribution

AN - SCOPUS:85168867208

T3 - Proceedings - International Computer Software and Applications Conference

SP - 1774

EP - 1779

BT - Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023

A2 - Shahriar, Hossain

A2 - Teranishi, Yuuichi

A2 - Cuzzocrea, Alfredo

A2 - Sharmin, Moushumi

A2 - Towey, Dave

A2 - Majumder, AKM Jahangir Alam

A2 - Kashiwazaki, Hiroki

A2 - Yang, Ji-Jiang

A2 - Takemoto, Michiharu

A2 - Sakib, Nazmus

A2 - Banno, Ryohei

A2 - Ahamed, Sheikh Iqbal

PB - IEEE Computer Society

Y2 - 26 June 2023 through 30 June 2023

ER -

Towey D, Luo Z, Zheng Z, Zhou P, Yang J, Ingkasit P et al. Metamorphic testing of an automated parking system: an experience report. In Shahriar H, Teranishi Y, Cuzzocrea A, Sharmin M, Towey D, Majumder AKMJA, Kashiwazaki H, Yang JJ, Takemoto M, Sakib N, Banno R, Ahamed SI, editors, Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023. IEEE Computer Society. 2023. p. 1774-1779. (Proceedings - International Computer Software and Applications Conference). doi: 10.1109/COMPSAC57700.2023.00274

Metamorphic testing of an automated parking system: an experience report

Abstract

Publication series

Conference

UN SDGs

Free Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this