Deep learning-based automated and objective assessment of operative skill in surgery

Abstract

Surgical skill assessment traditionally relies on in-person observations by expert surgeons, a process that is both time-consuming and inherently subjective. As the demand for objective and automated methods to evaluate surgical skills increases, there is a growing need for systems that can accurately differentiate between surgeons of varying expertise levels, thereby enhancing training programs and improving patient outcomes. Despite recent advancements, significant research gaps remain, particularly in terms of dataset scarcity and the black box nature of many deep learning models used in this field. These gaps limit the transparency and trust required for the widespread adoption of such technologies in clinical practice.

To address these challenges, this thesis presents several novel contributions to the field of automated robotic surgical skill assessment, leveraging advanced machine learning, deep learning and Explainable Artificial Intelligence (XAI) techniques. The first study focuses on overcoming the issue of limited dataset size by employing sliding window data augmentation and transfer learning methods. Given that most existing approaches are confined to either the time domain or frequency domain, this study introduces a new dimension by exploring the time-frequency domain. Specifically, we propose a framework named Continuous Wavelet Transform-Vision Transformer (CWT-ViT) that utilises Continuous Wavelet Transform (CWT) to convert robotic surgery kinematic data into synthesized images. This method integrates prior knowledge of the da Vinci surgical system by designing a four-branch architecture, each corresponding to a robotic manipulator. Extensive experiments conducted on the benchmark JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) dataset demonstrate that our approach achieves competitive results, showcasing its effectiveness in enhancing the granularity and accuracy of surgical skill assessment.

In the second study, we address the need for an explainable and automated approach to robotic surgical skill assessment, which is crucial for surgical training and medical education. Recognising that current methods are not only time-consuming but also lack objectivity and interpretability, we collaborate with senior surgeons to provide expert annotations for a newly developed dataset, ROSMA. Building upon this, we propose an XAI framework using Multi-View Time Series Feature Fusion (M-VTSFF) and Case-Based Reasoning (CBR). Our approach integrates multiple perspectives, including shapelet, temporal, and frequency views, to capture a comprehensive understanding of surgical skills. Additionally, we introduce a novel example-based XAI method employing case-based reasoning to offer both factual and counterfactual explanations. The framework’s efficacy is demonstrated on both the JIGSAWS and ROSMA datasets, where we provide the first application-grounded XAI evaluation comparing various XAI methods for surgical skill assessment. The results show that our proposed method outperforms existing XAI techniques, aligning with the
needs of clinical environments and assisting surgeons by enhancing their understanding of the model’s decision-making process.

In the third study, we present an XAI framework that incorporates domain knowledge to construct meaningful kinematic motion features, enabling a more transparent analysis of machine learning models. A comparative analysis of different XAI techniques, such as feature importance and rule-based explanations, is conducted to interpret these black-box models effectively. Building on the insights gained from classification and XAI explanations, we further develop an automated surgical skill assessment report generation tool using Large Language Model (LLM) to assist surgeons by providing clear, concise, and actionable feedback. The proposed framework is
validated on both the JIGSAWS and ROSMA datasets using various cross-validation mechanisms, and the results are validated by senior surgeons, highlighting the balance between accuracy and interpretability.

Date of Award	17 Mar 2025
Original language	English
Awarding Institution	University of Nottingham
Supervisor	Ying Weng (Supervisor), Zhuo Chen (Supervisor), Tom Dening (Supervisor) & Akram A. Hosseini (Supervisor)