Human Epithelial type 2 (HEp-2) cells are of great importance in the diagnosis of autoimmune disorder. Traditional approach requires specialists to manually observe the cells and make decisions, which is laborious, time-consuming and easily influenced by subjective experiences. Therefore, in this paper, we proposed a general framework based on Gabor Ternary Pattern (GTP) and joint sparse representation to automatically classify cell images. The method firstly searches the affine invariant key points in cell images by a multiscale canny detector, and then extracts GTP features from the local region around the points. Finally, the joint sparse representation classifier (SRC) is applied to determine the labels of the cell images. To reduce the computation costs required by the large dictionary, a k-means based approach was proposed to reduce the dictionary size. We conduct experiments on the publicly available ICPR cell image database and get a promising result. The experiments show that the approach based on GTP outperforms the SIFT-based approach and the adoption of k-means clustering not only reduce the dictionary size, but also significantly improve the classification accuracy.