Factoring 3D Convolutions for Medical Images by Depth-wise Dependencies-induced Adaptive Attention

It turns out that convolutional neural networks (CNNs) have excellent medical image processing capabilities. Hence, effectively and efficiently deploying CNNs on devices with varying computing power to make computer-aided diagnosis puts on the agenda. However, it is a dilemma to balance the limited computing resources and model complexity. Previously, we proposed factorized convolution with spectral normalization (FConvSN) to mitigate the bottleneck of deploying CNNs for 2D medical images. But due to the cube structure of 3D convolutional kernels, it does not work well for 3D medical images. Directly flattening 3D kernels to 2D weights for matrix factorization may undermine the learning ability along depth-wise, resulting in the loss of depth information and the decline of model performance. To this end, we factorize a 3D convolutional kernel to 2D weight matrices with depth-wise dimensions, then assign an attentive score for each 2D weight matrix by a depth-wise dependencies-induced adaptive attention block (AA). AA with a temperature hyper-parameter helps convolution kernel to better capture depth-wise dependencies in 3D medical images, improving its learning ability along the depth direction. We term this novel factorized convolution as FConvAA used for compressing model complexity without impairing the depth-wise expressivity. We also impose spectral normalization (SN) for FConvAA to constrain spectral norm-wise weights. We conduct extensive experiments on the public lung CT dataset LUNA16 and the private retina OCT dataset to demonstrate the effectiveness and feasibility of our FConvAA.