Detecting Hidden Failures of DBMS: A Comprehensive Metamorphic Relation Output Patterns Approach

The testing of large databases faces the test oracle problem, namely, that it is difficult to verify execution results against expected outcomes. Rigger and Su applied metamorphic testing through query partitioning and ternary logic partitioning techniques to alleviate the challenge. In Part (A) of our project, we conduct an in-depth investigation and have identified a gap between the two techniques. We propose a disjoint partitioning approach to address it. In Part (B), we conduct a comprehensive investigation into the metamorphic testing of DBMS by comparing disjoint partitioning with metamorphic relation output patterns (MROPs) by Segura et al. We propose an exhaustive collection of MROPs for DBMS. To the best of our knowledge, this is the first project to integrate in-depth and comprehensive approaches to tackle the diverse challenges in DBMS testing. In Part (C), we conduct an empirical case study of their applications to OceanBase, the DBMS associated with the world's fastest online transaction processing system. Although OceanBase has been extensively tested and widely used in the industry, we have detected 12 hidden failures and 8 new crashes.