Automated metamorphic-relation generation with ChatGPT: an experience report

Research output: Chapter in Book/Conference proceedingConference contributionpeer-review

8 Citations (Scopus)

Abstract

This paper reports on a pilot study of using ChatGPT, a language model based on GPT-3.5 architecture, for automatic generation of metamorphic relations (MRs), in the context of testing of autonomous driving systems (ADSs). The oracle problem is a major challenge in testing such systems, where it is difficult to determine whether or not the output of a system is correct. Metamorphic testing (MT) can alleviate this problem by checking the consistency of the system's outputs under various transformations. However, manual generation of MRs is often a time-consuming and error-prone process. Automated MR generation can yield several benefits, including enhanced efficiency, quality, coverage, scalability, and reusability in software testing, thereby facilitating a more comprehensive and effective testing process. In this paper, we investigate the effectiveness of using ChatGPT for automatic generation of MRs for ADSs. We provide a detailed methodology for generating MRs using ChatGPT and evaluate the generated MRs using our domain knowledge and existing MRs. The results of our study indicate that our proposed approach is effective at generating high-quality MRs, and can significantly reduce the manual effort required for MR generation. Furthermore, we discuss the practical implications and limitations of using ChatGPT for MR generation and provide recommendations for future research. Our study contributes to the advancement of automated testing of ADSs, which is crucial for ensuring their safety and reliability in real-world scenarios.

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023
EditorsHossain Shahriar, Yuuichi Teranishi, Alfredo Cuzzocrea, Moushumi Sharmin, Dave Towey, AKM Jahangir Alam Majumder, Hiroki Kashiwazaki, Ji-Jiang Yang, Michiharu Takemoto, Nazmus Sakib, Ryohei Banno, Sheikh Iqbal Ahamed
PublisherIEEE Computer Society
Pages1780-1785
Number of pages6
ISBN (Electronic)9798350326970
DOIs
Publication statusPublished - 2023
Event47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023 - Hybrid, Torino, Italy
Duration: 26 Jun 202330 Jun 2023

Publication series

NameProceedings - International Computer Software and Applications Conference
Volume2023-June
ISSN (Print)0730-3157

Conference

Conference47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023
Country/TerritoryItaly
CityHybrid, Torino
Period26/06/2330/06/23

Keywords

  • Autonomous driving system (ADS)
  • ChatGPT
  • large language model (LLM)
  • metamorphic relation (MR)
  • metamorphic testing (MT)
  • natural language processing (NLP)
  • oracle problem

ASJC Scopus subject areas

  • Software
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Automated metamorphic-relation generation with ChatGPT: an experience report'. Together they form a unique fingerprint.

Cite this