Sign Language Detection Model

AIM AND OBJECTIVES

AIM

To create a Sign Language detection model which will detect the position
of human hands and then convey the message on the viewfinder of camera
in real time what the particular hand position means.

Objectives

The main objective of the project is to create a program which can
be either run on Jetson nano or any pc with YOLOv5 installed and
start detecting using the camera module on the device.
Using appropriate datasets for recognizing and interpreting data
using machine learning.
To show on the optical viewfinder of the camera module what a
particular position of hand means with respect to Sign Language.

ABSTRACT

A hand position is classified with respect to Sign Language on the basis of where the hand is placed on the body and then shown on the viewfinder of the camera what it means.
We have completed this project on jetson nano which is a very small computational device.
A lot of research is being conducted in the field of Computer Vision and Machine Learning (ML), where machines are trained to identify various objects from one another. Machine Learning provides various techniques through which various objects can be detected.
One such technique is to use YOLOv5 with Roboflow model, which generates a small size trained model and makes ML integration easier.
Sign languages are an extremely important communication tool for many deaf and hard-of-hearing people. Sign languages are the native languages of the Deaf community and provide full access to communication.
Sign Language detection model can be of great help to people who are beginners in learning sign language or to those like an elderly who have lost hearing recently.

INTRODUCTION

This project is based on Sign Language detection model. We are going to implement this project with Machine Learning and this project can be even run on jetson nano which we have done.
This project can also be used to gather information about what Sign Language a particular person is conveying through his or her hands.
Hand position can be classified into many other names based on the type of language being used based on the image annotation we give in roboflow.
Sign Language detection becomes difficult sometimes on account of people of various age groups, sizes, gender etc doing Sign Language hand position which is harder for model to detect. However, training in Roboflow has allowed us to crop images and change the contrast of certain images to match the time of day for better recognition by the model.
Neural networks and machine learning have been used for these tasks and have obtained good results.
Machine learning algorithms have proven to be very useful in pattern recognition and classification, and hence can be used for Sign Language detection as well.

LITERATURE REVIEW

Sign languages (also known as signed languages) are languages that use the visual-manual modality to convey meaning. Sign languages are expressed through manual articulations in combination with non-manual elements. Sign languages are full-fledged natural languages with their own grammar and lexicon.
Wherever communities of deaf people exist, sign languages have developed as useful means of communication, and they form the core of local Deaf cultures. Although signing is used primarily by the deaf and hard of hearing, it is also used by hearing individuals, such as those unable to physically speak, those who have trouble with spoken language due to a disability or condition (augmentative and alternative communication), or those with deaf family members, such as children of deaf adults.
As a sign language develops, it sometimes borrows elements from spoken languages, just as all languages borrow from other languages that they are in contact with. Sign languages vary in how much they borrow from spoken languages.
Although sign languages have emerged naturally in deaf communities alongside or among spoken languages, they are unrelated to spoken languages and have different grammatical structures at their core.
Some experts argue early man likely used signs to communicate long before spoken language was created.

JETSON NANO COMPATIBILITY

The power of modern AI is now available for makers, learners, and embedded developers everywhere.
NVIDIA® Jetson Nano™ Developer Kit is a small, powerful computer that lets you run multiple neural networks in parallel for applications like image classification, object detection, segmentation, and speech processing. All in an easy-to-use platform that runs in as little as 5 watts.
Hence due to ease of process as well as reduced cost of implementation we have used Jetson nano for model detection and training.
NVIDIA JetPack SDK is the most comprehensive solution for building end-to-end accelerated AI applications. All Jetson modules and developer kits are supported by JetPack SDK.
In our model we have used JetPack version 4.6 which is the latest production release and supports all Jetson modules.

Jetson Nano 2GB

PROPOSED SYSTEM

Study basics of machine learning and image recognition.
Start with implementation
Front-end development
Back-end development
Testing, analyzing and improvising the model. An application using python and Roboflow and its machine learning libraries will be using machine learning to identify which position of hand means what according to Sign Language.
Use data sets to interpret the hand position and convey what the meaning of hand position is on the viewfinder.

METHODOLOGY

The Sign Language detection model is a program that focuses on implementing real time Sign Language detection.

It is a prototype of a new product that comprises of the main module:

Hand position detection and then showing on viewfinder what the hand position means according to data fed.

Sign Language Detection Module

This Module is divided into two parts.

Hand Detection
Ability to detect the location of a person’s hand in any input image or frame. The output is the bounding box coordinates on the detected hand of a person.
For this task, initially the Data set library Kaggle was considered. But integrating it was a complex task so then we just downloaded the images from gettyimages.ae, shutterstock.com and google images and made our own data set.
This Data set identifies person’s hand in a Bitmap graphic object and returns the bounding box image with annotation of name present.
Position of Hand Detection
Recognition of the hand and what the particular position means.
Hence YOLOv5 which is a model library from roboflow for image classification and vision was used.
There are other models as well but YOLOv5 is smaller and generally easier to use in production. Given it is natively implemented in PyTorch (rather than Darknet), modifying the architecture and exporting and deployment to many environments is straightforward.
YOLOv5 was used to train and test our model for what a particular hand position means. We trained it for 149 epochs and achieved an accuracy of approximately 92%.

For Installation You can Checkout my Github:-

GitHub – nikhil0g/SIGN-LANGUAGE-DETECTION-MODELAim To create a Sign Language detection model which will detect the position of human hands and then convey the message…github.com

ADVANTAGES

As many as 90 percent of deaf children are born to hearing parents, which can make learning sign language a family affair. Hence our Sign Language Detection model can be a great help in beginning to learn it.
Sign Language detection system shows what the position of hands in viewfinder of camera module means with good accuracy.
It can then be used to help people who are just beginning to learn Sign Language or those who don’t know sign language but have a close one who is deaf.
Some children with Autism Spectrum Disorder (ASD) struggle developing verbal communication. Hence people around such children can use Sign Language detection model to understand what the child is saying.
Sign languages can be a great way to gossip without anyone else knowing, and passing on confidential information. Our model here can be used here by training it to just use hand position based on what the individuals have decided what the hand position means.

APPLICATION

Detects a person’s hand and then checks what each hand position means in each image frame or viewfinder using a camera module.
Can be used by people who wants to understand deaf people and also used in places like hospitals where the staff is not trained in Sign Language.
Can be used as a reference for other ai models based on Helmet Detection.

FUTURE SCOPE

As we know technology is marching towards automation, so this project is one of the step towards automation.
Thus, for more accurate results it needs to be trained for more images, and for a greater number of epochs.
Sign Language detection model can be very easily implemented in smart phones in the form of apps and thus make it possible for everyone to understand deaf and dumb people.
Sign Language detection model can be further improved to show a certain hand position on screen by typing a word or saying it out loud towards the smart phone.