Chinese sign language recognition with batch sampling ResNet-Bi-LSTM

Wan Young Chung, Haokai Xu, Boon Giin Lee

Research output: Journal PublicationArticlepeer-review

Abstract

Sign language has served as a communication medium between the Deaf community and society. Nonetheless, the practice of sign language is not common in Chinese society, along with a lack of professional sign language interpreters. Most existing studies on sign language recognition have only considered basic, simple, and static handshapes, which have not been practically implemented as real-world applications. To resolve the shortage of sign language interpreters, a sign language recognition application that interprets the sign language is required. Thus, the aim of this study was to develop and evaluate a sign language recognition framework using multi-modalities approach and spatio-temporal features that include dynamic handshapes. The proposed framework consists of three main parts, namely handshape recognition, movement tracking, and sign recognition. In this study, the use of hand skeletal data as features was also investigated, which were input to a bi-directional long short-term memory (Bi-LSTM) model for sign recognition. The proposed model was evaluated on a continuous Chinese sign language (CSL) dataset of 8 subjects with 1200 sample videos covering 100 signs. The experimental results demonstrated a true recognition rate of 98.75%, outperforming most of the state-of-the-art alternatives used for sign language recognition. The proposed sign recognition application can be deployed in public service sectors such as banks, hospitals, and police stations.

Original languageEnglish
Article number414
Number of pages13
JournalSN Computer Science
Volume3
Issue number5
DOIs
Publication statusPublished - Sep 2022

Keywords

  • Computer vision
  • Human computer interaction
  • Machine learning
  • Sign language

ASJC Scopus subject areas

  • Computer Science (all)
  • Artificial Intelligence
  • Computer Graphics and Computer-Aided Design
  • Computational Theory and Mathematics
  • Computer Networks and Communications
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Chinese sign language recognition with batch sampling ResNet-Bi-LSTM'. Together they form a unique fingerprint.

Cite this