Learning a discriminative dictionary with CNN for image classification

In this paper, we propose a novel framework for image recognition based on an extended sparse model. First, inspired by the impressive results of CNN over different tasks in computer vision, we use the CNN models pre-trained on large datasets to generate features. Then we propose an extended sparse model which learns a dictionary from the CNN features by incorporating the reconstruction residual term and the coefficients adjustment term. Minimizing the reconstruction residual term guarantees that the class-specific sub-dictionary has good representation power for the samples from the corresponding class and minimizing the coefficients adjustment term encourages samples from different classes to be reconstructed by different class-specific sub-dictionaries. With this learned dictionary, not only the representation residual but also the representation coefficients will be discriminative. Finally, a metric involving these discriminative information is introduced for image classification. Experiments on Caltech101 and PASCAL VOC 2012 datasets show the effectiveness of the proposed method on image classification.