TY - GEN
T1 - Automated metamorphic-relation generation with ChatGPT
T2 - 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023
AU - Zhang, Yifan
AU - Towey, Dave
AU - Pike, Matthew
N1 - Funding Information:
ACKNOWLEDGMENTS The authors acknowledge the financial support from the Artificial Intelligence and Optimisation (AIOP) and Sensors, Sensor Networks and Instrumentation (SSNI) research groups, the Faculty of Science and Engineering (FoSE), the International Doctoral Innovation Centre, Ningbo Education Bureau, Ningbo Science and Technology Bureau, and the University of Nottingham.
Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - This paper reports on a pilot study of using ChatGPT, a language model based on GPT-3.5 architecture, for automatic generation of metamorphic relations (MRs), in the context of testing of autonomous driving systems (ADSs). The oracle problem is a major challenge in testing such systems, where it is difficult to determine whether or not the output of a system is correct. Metamorphic testing (MT) can alleviate this problem by checking the consistency of the system's outputs under various transformations. However, manual generation of MRs is often a time-consuming and error-prone process. Automated MR generation can yield several benefits, including enhanced efficiency, quality, coverage, scalability, and reusability in software testing, thereby facilitating a more comprehensive and effective testing process. In this paper, we investigate the effectiveness of using ChatGPT for automatic generation of MRs for ADSs. We provide a detailed methodology for generating MRs using ChatGPT and evaluate the generated MRs using our domain knowledge and existing MRs. The results of our study indicate that our proposed approach is effective at generating high-quality MRs, and can significantly reduce the manual effort required for MR generation. Furthermore, we discuss the practical implications and limitations of using ChatGPT for MR generation and provide recommendations for future research. Our study contributes to the advancement of automated testing of ADSs, which is crucial for ensuring their safety and reliability in real-world scenarios.
AB - This paper reports on a pilot study of using ChatGPT, a language model based on GPT-3.5 architecture, for automatic generation of metamorphic relations (MRs), in the context of testing of autonomous driving systems (ADSs). The oracle problem is a major challenge in testing such systems, where it is difficult to determine whether or not the output of a system is correct. Metamorphic testing (MT) can alleviate this problem by checking the consistency of the system's outputs under various transformations. However, manual generation of MRs is often a time-consuming and error-prone process. Automated MR generation can yield several benefits, including enhanced efficiency, quality, coverage, scalability, and reusability in software testing, thereby facilitating a more comprehensive and effective testing process. In this paper, we investigate the effectiveness of using ChatGPT for automatic generation of MRs for ADSs. We provide a detailed methodology for generating MRs using ChatGPT and evaluate the generated MRs using our domain knowledge and existing MRs. The results of our study indicate that our proposed approach is effective at generating high-quality MRs, and can significantly reduce the manual effort required for MR generation. Furthermore, we discuss the practical implications and limitations of using ChatGPT for MR generation and provide recommendations for future research. Our study contributes to the advancement of automated testing of ADSs, which is crucial for ensuring their safety and reliability in real-world scenarios.
KW - Autonomous driving system (ADS)
KW - ChatGPT
KW - large language model (LLM)
KW - metamorphic relation (MR)
KW - metamorphic testing (MT)
KW - natural language processing (NLP)
KW - oracle problem
UR - http://www.scopus.com/inward/record.url?scp=85168910241&partnerID=8YFLogxK
U2 - 10.1109/COMPSAC57700.2023.00275
DO - 10.1109/COMPSAC57700.2023.00275
M3 - Conference contribution
AN - SCOPUS:85168910241
T3 - Proceedings - International Computer Software and Applications Conference
SP - 1780
EP - 1785
BT - Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023
A2 - Shahriar, Hossain
A2 - Teranishi, Yuuichi
A2 - Cuzzocrea, Alfredo
A2 - Sharmin, Moushumi
A2 - Towey, Dave
A2 - Majumder, AKM Jahangir Alam
A2 - Kashiwazaki, Hiroki
A2 - Yang, Ji-Jiang
A2 - Takemoto, Michiharu
A2 - Sakib, Nazmus
A2 - Banno, Ryohei
A2 - Ahamed, Sheikh Iqbal
PB - IEEE Computer Society
Y2 - 26 June 2023 through 30 June 2023
ER -