MedDistilFuse: A CLIP-Distilled Transformer with Self-Supervised Fusion for Medical Image Classification

Medical image classification plays an important role in computer-aided diagnosis by enabling automated identification and categorization of disease-related patterns in imaging data. Despite its importance, the field faces significant challenges, including data scarcity, high annotation costs, and poor model generalization. To address these issues, we propose MedDistilFuse, a novel hybrid framework that integrates knowledge distillation with contrastive representation learning. Specifically, MedDistilFuse leverages a pre-trained Contrastive Language- Image Pre-training (CLIP) model as the teacher to transfer rich visual knowledge to a lightweight student network based on Vision Transformers (ViT). To enhance representational capacity with minimal complexity, a Kolmogorov-Arnold Network (KAN)-inspired nonlinear module is incorporated into the student model. Furthermore, a contrastive learning objective is introduced to promote feature discriminability and robustness under sparse supervision. Extensive experiments across five benchmark medical imaging datasets demonstrate the proposed MedDistilFuse consistently achieves promising accuracy and top or competitive AUC scores. GitHub link: the link will be provided upon paper acceptance.