Medical sign language translator in healthcare facility using yolo version 7 algorithm / Nurin Qistina Zaini

Sign language is a significant tool used by the impairment people as their communication tool. It employs hand articulation, face expression and body movement to convey message. Individuals who are deaf or hard of hearing experience severe communication challenges in health care facilities, restricting their access to healthcare services. In addition, people who work worked at a front desk in healthcare institutions also have a limited knowledge on this sign language. Hence, this creates barrier for these impairment people to do communication in their daily activities, especially when dealing at the healthcare facility. This project purpose is as preliminary to overcome the problem at healthcare institutions to recognize sign language and interpret it into word to let other people understand it. The system will begin to function by receiving real-time input of a sign language image. The YOLOv7 algorithm will process the image by detecting trained images in the input image. If the training image is present in the input, a bounding box with a label that covers the estimated object will be presented. For the recognized sign language gesture, the algorithm creates a bounding box with a label. The hand signs are then translated into words, allowing medical staff to clearly understand the conversation. The model’s performance is assessed using accuracy, recall, average precision and F1 score are calculated where the results for mean average precision (0.95%) for all classes are more than 0.9 accuracy and the F1 score for all classes are more than 0.8 accuracy. In the future, the system can be more well developed by using local GPU where the training phase can be done without any restriction and more classes of sign language can be added to make it more convenient to use the system.