The Role of Spatial Alignment in Multimodal Medical Image Fusion Using Deep Learning for Diagnostic Problems

Xingyue Wang, Kuang Shu, Haowei Kuang, Shiwei Luo, Richu Jin, Jiang Liu

Research output: Chapter in Book/Conference proceedingConference contributionpeer-review

Abstract

Deep learning methods have become popular in multimodal medical image fusion for diagnostic problems. Unlike conventional ways where spatial alignment is a crucial step, the deep learning methods perform the fusion at middle layers of deep neural networks and the alignment of multiple image modalities is achieved implicitly at the semantic level. Therefore, the role of spatial alignment in the fusion process using deep learning is doubted. This study tried to clarify this doubt via a series of experiments. Particularly, based on two specific clinical diagnostic problems, i.e. diagnosis of AD and AMD, performances of concatenation-based deep fusion networks with spatially aligned or misaligned inputs were compared. Moreover, modified deep fusion networks with an STN module to provide adaptive spatial alignment was proposed and tested. It was observed that there was an improvement in diagnostic results if the inputs of deep fusion networks were spatially aligned, and adaptive spatial alignment could bring additional improvement. These findings suggest that spatial alignment still works in the fusion process using deep learning and an additional adaptive spatial alignment is recommended for better fusion results.

Original languageEnglish
Title of host publicationProceedings of the 2021 3rd International Conference on Intelligent Medicine and Health, ICIMH 2021
PublisherAssociation for Computing Machinery
Pages40-46
Number of pages7
ISBN (Electronic)9781450385909
DOIs
Publication statusPublished - 13 Aug 2021
Externally publishedYes
Event3rd International Conference on Intelligent Medicine and Health, ICIMH 2021 - Virtual, Online, China
Duration: 13 Aug 202115 Aug 2021

Publication series

NameACM International Conference Proceeding Series

Conference

Conference3rd International Conference on Intelligent Medicine and Health, ICIMH 2021
Country/TerritoryChina
CityVirtual, Online
Period13/08/2115/08/21

Keywords

  • Deep Learning
  • Medical Image Analysis
  • Multimodal Medical Image Fusion
  • Spatial Alignment

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'The Role of Spatial Alignment in Multimodal Medical Image Fusion Using Deep Learning for Diagnostic Problems'. Together they form a unique fingerprint.

Cite this