Decision Tree Models to Select High-Risk Patients for Lung Cancer Screening and Model Interpretability

Lung cancer is the most common cause of cancer deaths in the UK emphasizing the critical need for early diagnosis. Survival rates vary significantly according to the stage of diagnosis. This study aims to develop machine learning models to classify between lung cancer and non-lung cancer cases using data from the Clinical Practice Research Datalink (CPRD) which includes UK primary care records. Both interpretable and post hoc explainable approaches are explored including RuleFit, a rule-based method; decision tree, an inherently interpretable model; and random forest and eXtreme Gradient Boosting, tree-based ensemble models. The model performance is assessed using metrics such as accuracy, Area Under the Receiver Operating Characteristic Curve, sensitivity, and specificity. The models performed similarly across all measures. Additionally, SHapley Additive exPlanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME) are employed to enhance model interpretability. These insights contribute to better understanding the leading risk factors for lung cancer. Using SHAP, it is found that age and smoking status play a crucial role in lung cancer prediction for all tree-based models. Then, LIME is used to evaluate individual-level explanations and identify any discrepancies in their explanations between different models. Our study combines robust evaluation with prominent interpretability techniques to gain valuable insights into lung cancer prediction.