American sign language recognition using leap motion controller with machine learning approach

Teak Wei Chong, Boon Giin Lee

Research output: Journal PublicationArticlepeer-review

43 Citations (Scopus)

Abstract

Sign language is intentionally designed to allow deaf and dumb communities to convey messages and to connect with society. Unfortunately, learning and practicing sign language is not common among society; hence, this study developed a sign language recognition prototype using the Leap Motion Controller (LMC). Many existing studies have proposed methods for incomplete sign language recognition, whereas this study aimed for full American Sign Language (ASL) recognition, which consists of 26 letters and 10 digits. Most of the ASL letters are static (no movement), but certain ASL letters are dynamic (they require certain movements). Thus, this study also aimed to extract features from finger and hand motions to differentiate between the static and dynamic gestures. The experimental results revealed that the sign language recognition rates for the 26 letters using a support vector machine (SVM) and a deep neural network (DNN) are 80.30% and 93.81%, respectively. Meanwhile, the recognition rates for a combination of 26 letters and 10 digits are slightly lower, approximately 72.79% for the SVM and 88.79% for the DNN. As a result, the sign language recognition system has great potential for reducing the gap between deaf and dumb communities and others. The proposed prototype could also serve as an interpreter for the deaf and dumb in everyday life in service sectors, such as at the bank or post office.

Original languageEnglish
Article number3554
JournalSensors
Volume18
Issue number10
DOIs
Publication statusPublished - 19 Oct 2018
Externally publishedYes

Keywords

  • American sign language
  • Deep neural network
  • Human-computer interaction
  • Leap motion controller
  • Machine learning
  • Multi-class classification
  • Sign language recognition
  • Support vector machine

ASJC Scopus subject areas

  • Analytical Chemistry
  • Biochemistry
  • Atomic and Molecular Physics, and Optics
  • Instrumentation
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'American sign language recognition using leap motion controller with machine learning approach'. Together they form a unique fingerprint.

Cite this