Enhancing the Yolov8 algorithm for real-time dental segmentation

Segmentation of dental medical images is gaining importance because it enables clear visualization of anatomic details without manual intervention. In many clinical scenarios, radiographic interpretation is continuously enhanced with image segmentation techniques. Dental segmentation poses various challenges in computer vision, as this process is critical and requires high accuracy. Traditional methods, especially convolutional neural networks (CNNs), have not achieved high accuracy due to suboptimal performance and computational inefficiency. The goal of image segmentation is to group pixels based on visual properties such as color, texture, intensity, or spatial proximity to identify and delineate the boundaries of distinct objects or regions within the image. In this study, the You Only Look Once (YOLOv8) algorithm is improved to achieve real-time tooth segmentation with high accuracy and high execution speed. The increase in the number of YOLOv8 layers relied upon, as the algorithm's segmentation accuracy depends on the number of layers used to extract features from the image (backbone) and the number of layers in the head (prediction). In addition, the layer sizes are reduced to improve execution speed. The novelty of this work lies in improving the Coordinates-To-Features (C2f) module, for which its equations were derived, and in employing gradient-descent-based methods in the loss function to reduce loss and achieve the highest prediction accuracy. The enhanced model focuses more on dental features, which facilitates the efficient spread of the gradient through adaptive weights. In addition to the Proposed Activation Function (PAF), the dataset used (top view) was obtained from a dental clinic, comprising 526 images of dental patients. The highest accuracy of 99.561% was achieved when the enhanced YOLOv8 segmentation model was applied to the dental dataset. It can be concluded that the improved YOLOv8 model has increased dental segmentation accuracy compared to previous research, as it relies on a proposed PAF that enhances the distinction between features extracted from the model's layers, enabling it to separate teeth from surrounding tissues more effectively.