Nuclear cataract (NC) is a prior age-related disease for blindness and vision impairment globally. Anterior segment optical coherence tomography (AS-OCT) image is a new ophthalmology image, which can capture the lens nucleus region clearly compared with other ophthalmic images, e.g., slit lamp images. Clinical research has suggested that features e.g., mean from AS-OCT images have varying correlations with NC severity levels. However, existing convolutional neural network (CNN) based NC classification works have not incorporated the clinical features into the network design to improve the performance. To this end, we propose a novel channel-wise and spatial feature recalibration network (CSFR-Net) to predict NC severity levels automatically, which is built on a stack of channel-wise and spatial feature recalibration (CSFR) modules. In each CSFR module, we construct a channel-wise feature recalibration block and a spatial feature recalibration block to recalibrate intermediate feature maps dynamically. This feature recalibration strategy enables CSFR-Net to highlight feature representations and suppress unnecessary ones in a global-and-local manner. We conduct extensive experiments on a clinical AS-OCT image dataset and CIFAR benchmarks. The results show that our CSFR-Net achieves better performance than state-of-the-art methods with less model complexity.