Metamorphic Testing and exploration for Machine Learning credit score models

Context: The rapid development of Machine Learning (ML) has led to the proposal of various ML models to improve credit score assessment, creating a need for effective validation methods to ensure their performance aligns with business expectations. Objective: This paper introduces a novel approach for validating credit scoring models by focusing on user-hypothesized business expectations, enabling testers to predict how input changes affect outputs and assess alignment with business intuition. Methods: The approach uses Metamorphic Testing (MT), applying Metamorphic Relations (MRs) to examine input–output relationships, and Metamorphic Exploration (ME), an advanced extension of MT that constructs MRs based on user expectations. A case study evaluates and contrasts three popular ML models, neural networks, random forests, and gradient boosting tree, using both traditional evaluation metrics in credit scoring and ME. The study investigates how models selected based on traditional metrics perform when evaluated against MRs. Results: Empirical findings reveal that all three models often violate MRs, with violations becoming more extensive as model complexity increases. Neural networks have low number of MR violations on average but tends to be less robust. Interestingly, random forests exhibit most MR violations relative to the other two models. Traditional metrics fail to capture these violations, highlighting their limitations in ensuring alignment with business expectations. Conclusions: ME is proposed as a complementary validation method for model selection and post-deployment monitoring, ensuring models adhere to business intuition. The study underscores the importance of combining traditional metrics with ME, particularly for complex models like neural networks, to improve reliability in real-world applications.