Mobile application for real time baby sign language recognition using YOLOv8 / Siti Aishah Idris and Ahmad Firdaus Ahmad Fadzil

Nowadays, the use of baby sign language has increased among parents in recent years. This language utilizes simple hand gestures and motions as a medium to express the baby’s needs, wants, and feelings. This is particularly important because babies have not yet developed the ability to speak. However, this language is still neither widely used nor a generally recognized mode of communication, especially for caregivers. This can lead to tantrums when the baby’s needs are not understood or met promptly. Additionally, some parents have expressed difficulties in learning and remembering the sign. Hence, the development of real-time baby sign language recognition on mobile platforms could help to address these issues. This research will focus only on six basic baby sign languages that are commonly used in daily life. The model will be designed and developed using a deep learning algorithm, which is YOLOv8, the latest version of YOLO. This model can recognize and interpret baby signs based on visual input from the mobile phone’s camera in real time, and also includes a dictionary feature that can be a learning tool for parents and caregivers. In the development phase, the dataset is pre-processed before the modeling process is done using YOLOv8, and deployed on Android platforms using Java and Kotlin languages in Android Studio. Functionality testing and accuracy testing have been conducted. The functionality test produced successful results for all test cases, while for accuracy testing, the model achieved an accuracy of 99.50% for Mean Average Precision, indicating its proficiency in recognizing each of the classes.