Loose palm fruitlets harvesting system using image segmentation and recurrent neural network based inverse kinematics for 6-DOF robotic arm

Palm oil is a vital commodity in global food and energy markets, with Malaysia contributing RM 36.19 billion to its GDP from the palm oil sector in 2023. However, due to a critical labor shortage, large quantities of palm oil fruitlets especially loose fruitlets, which contain the highest oil concentration were left uncollected. This research addresses this inefficiency by developing an AI-driven robotic system that automates the collection of loose fruitlets using a 6-degree-of-freedom (6-DOF) robotic arm integrated with a YOLO-based vision model. The YOLOv11 segmentation model was trained using two datasets: a clean dataset (fruitlets only) and a mixed dataset (fruitlets with background objects). When trained on the mixed dataset, the model achieved precision and recall of 0.999 and 1.000, respectively, and mean average precision (mAP@0.5:0.95) of 0.981, significantly outperforming the clean dataset model, which achieved precision of 0.544 and mAP@0.5:0.95 of 0.767. The model was evaluated in both PyTorch and TensorFlow Lite (TFLite) formats, with accuracy maintained across both. PyTorch delivered a faster inference speed of 84.7 ms, while TFLite required 195.4 ms per frame. For motion control, 4,000 inverse kinematics (IK) data points were generated using the Levenberg–Marquardt algorithm. These were used to train six Recurrent Neural Network (RNN) models. Among them, the Bidirectional Gated Recurrent Unit (BiGRU) model achieved the best results, with a mean squared error (MSE) of 0.506, root mean squared error (RMSE) of 0.712, and an R² score of 0.997. In testing, the robotic system completed five full pick-and-place cycles in under 30 seconds, with an average of 10 seconds for image segmentation and 15 seconds for mechanical actuation. After gripper modification, the pick-and-place success rate increased from 40% to 90%. In conclusion, the combination of YOLOv11 segmentation, BiGRU-based inverse kinematics, and robotic actuation demonstrates a robust and efficient solution for automating loose fruitlet collection. The system effectively addresses labor shortages while improving accuracy and operational speed, offering a an alternative solution, for the advancement of smart agriculture in Malaysia’s palm oil industry.