KDViT: COVID-19 diagnosis on CT-scans with knowledge distillation of vision transformer

Yu Jie Lim, Kian Ming Lim, Roy Kwang Yang Chang, Chin Poo Lee

Research output: Journal PublicationArticlepeer-review

Abstract

This paper introduces Knowledge Distillation of Vision Transformer (KDViT), a novel approach for medical image classification. The Vision Transformer architecture incorporates a self-attention mechanism to autonomously learn image structure. The input medical image is segmented into patches and transformed into low-dimensional linear embeddings. Position information is integrated into each patch, and a learnable classification token is appended for classification, thereby preserving spatial relationships within the image. The output vectors are then fed into a Transformer encoder to extract both local and global features, leveraging the inherent attention mechanism for robust feature extraction across diverse medical imaging scenarios. Furthermore, knowledge distillation is employed to enhance performance by transferring insights from a large teacher model to a small student model. This approach reduces the computational requirements of the larger model and improves overall effectiveness. Integrating knowledge distillation with two Vision Transformer models not only showcases the novelty of the proposed solution for medical image classification but also enhances model interpretability, reduces computational complexity, and improves generalization capabilities. The proposed KDViT model achieved high accuracy rates of 98.39%, 88.57%, and 99.15% on the SARS-CoV-2-CT, COVID-CT, and iCTCF datasets respectively, surpassing the performance of other state-of-the-art methods.

Original languageEnglish
Pages (from-to)1113-1126
Number of pages14
JournalAutomatika
Volume65
Issue number3
DOIs
Publication statusPublished - 2024
Externally publishedYes

Keywords

  • COVID-19 image classification
  • CT scan images
  • knowledge distillation
  • Vision transformer

ASJC Scopus subject areas

  • Control and Systems Engineering
  • General Computer Science

Fingerprint

Dive into the research topics of 'KDViT: COVID-19 diagnosis on CT-scans with knowledge distillation of vision transformer'. Together they form a unique fingerprint.

Cite this