Smart Wearable Hand Device for Sign Language Interpretation System with Sensors Fusion

Boon Giin Lee, Su Min Lee

Research output: Journal PublicationArticlepeer-review

149 Citations (Scopus)

Abstract

Gesturing is an instinctive way of communicating to present a specific meaning or intent. Therefore, research into sign language interpretation using gestures has been explored progressively during recent decades to serve as an auxiliary tool for deaf and mute people to blend into society without barriers. In this paper, a smart sign language interpretation system using a wearable hand device is proposed to meet this purpose. This wearable system utilizes five flex-sensors, two pressure sensors, and a three-axis inertial motion sensor to distinguish the characters in the American sign language alphabet. The entire system mainly consists of three modules: 1) a wearable device with a sensor module; 2) a processing module; and 3) a display unit mobile application module. Sensor data are collected and analyzed using a built-in embedded support vector machine classifier. Subsequently, the recognized alphabet is further transmitted to a mobile device through Bluetooth low energy wireless communication. An Android-based mobile application was developed with a text-to-speech function that converts the received textinto audible voice output. Experiment results indicate that a true sign language recognition accuracy rate of 65.7% can be achieved on average in the first version without pressure sensors. A second version of the proposed wearable system with the fusion of pressure sensors on the middle finger increased the recognition accuracy rate dramatically to 98.2%. The proposed wearable system outperforms the existing method, for instance, although background lights, and other factors are crucial to a vision-based processing method, they are not for the proposed system.

Original languageEnglish
Article number8126796
Pages (from-to)1224-1232
Number of pages9
JournalIEEE Sensors Journal
Volume18
Issue number3
DOIs
Publication statusPublished - 1 Feb 2018
Externally publishedYes

Keywords

  • Gesture recognition
  • machine learning
  • mobile application
  • sign language recognition
  • wearable computing

ASJC Scopus subject areas

  • Instrumentation
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Smart Wearable Hand Device for Sign Language Interpretation System with Sensors Fusion'. Together they form a unique fingerprint.

Cite this