Metamorphic testing of Advanced Driver-Assistance System (ADAS) simulation platforms: Lane Keeping Assist System (LKAS) case studies

Context: Simulation-based testing is essential when developing Advanced Driver-Assistance Systems (ADASs) and autonomous driving (AD) systems, producing fast, high-quality test results, at relatively low cost. Simulation testing relies on the quality of the ADAS simulation platform: If the simulation platform is faulty, then the simulation results may be incorrect, and hence useless. However, because of the lack of suitable test oracles — mechanisms to determine the correctness of the software output or behavior — it can be too difficult (or expensive) to verify or validate ADAS/AD simulation platforms, a situation known as the oracle problem. Objective: To alleviate the oracle problem and better understand ADAS simulation software. Methods: We develop geometric-transformation-based metamorphic testing approaches, and report on empirical studies conducted on the verification and validation (V&V) of three popular simulation platforms for ADAS development: Simulink, CarMaker and 51Sim-One Cloud. Our examination focused on the platforms’ Lane Keeping Assist Systems (LKASs). Results: When tested with ordinary (traditional) test cases, no issues were identified on any simulation platform. However, after applying geometric-transformation-based metamorphic testing, issues were revealed, some of which were later confirmed by the MATLAB and IPG Automotive teams. To the best of our knowledge, this paper is the first to report on real bugs and issues in ADAS simulation platforms. Conclusion: Our research shows the simplicity, effectiveness and applicability of the proposed approach for ADAS simulation testing. This paper also provides successful examples of incorporating metamorphic testing into the testing of ADAS standards and protocols, and shows how practitioners can design effective metamorphic relations (MRs) inspired by using the symmetry metamorphic relation pattern.