Chinese sign language recognition with batch sampling ResNet-Bi-LSTM

Sign language has served as a communication medium between the Deaf community and society. Nonetheless, the practice of sign language is not common in Chinese society, along with a lack of professional sign language interpreters. Most existing studies on sign language recognition have only considered basic, simple, and static handshapes, which have not been practically implemented as real-world applications. To resolve the shortage of sign language interpreters, a sign language recognition application that interprets the sign language is required. Thus, the aim of this study was to develop and evaluate a sign language recognition framework using multi-modalities approach and spatio-temporal features that include dynamic handshapes. The proposed framework consists of three main parts, namely handshape recognition, movement tracking, and sign recognition. In this study, the use of hand skeletal data as features was also investigated, which were input to a bi-directional long short-term memory (Bi-LSTM) model for sign recognition. The proposed model was evaluated on a continuous Chinese sign language (CSL) dataset of 8 subjects with 1200 sample videos covering 100 signs. The experimental results demonstrated a true recognition rate of 98.75%, outperforming most of the state-of-the-art alternatives used for sign language recognition. The proposed sign recognition application can be deployed in public service sectors such as banks, hospitals, and police stations.