A multimodal deep learning approach for non-desctructive quality assessment of ficus carica L. / Iylia Adhwa Mazni

The rapid progress in agriculture continually increases the demand for high-quality fruits. Ficus Carica L. (figs) is considered as one of the high-value fruits that required quality assessment due to its prevalence benefits was highly demanding. However, the quality was usually assessed by a destructive method that normally led to a concern which is the loss of destructed fruits. In addition to that, this traditional approach by employing human labour is time-consuming and prone to human mistakes. In order to address these problems, an automatic non-destructive quality assessment of fig fruits using a deep learning regression model based on RGB and Laser-light Back scattering images (LLBI) is developed in this research work. To achieve this, the image of 33 matured Super Red Hybrid (SRH) fig fruits and the chemical properties (i.e., Brix value) dataset was collected for training and testing the model. The image of fig fruits was captured based on RGB and back scattering images that involve three different wavelengths of laser (650nm, 532nm, and 405nm) which represent Red, Green, and Blue, respectively. To determine the best two image modalities for this regression task, the performance of single modality deep learning models on RGB and Laser-light Backscattering images for figs fruits quality assessment is investigated before the optimization of the model takes place. The investigation is done by comparing the performance of the proposed CNN regression model with the state-of-the-art architecture which is VGG-16 and ResNet-50 for each single image modality. Then, the optimized proposed CNN regression model is extended to a novel multimodal deep learning regression model for the non-destructive quality assessment. In this model, an additional fusion layer is added before the input layer in which the two best single image modalities (RGB and red LLBI) were fused using an average fusion technique. Based on the experimental result, the proposed model shows the best prediction result and capable to assess the quality of fig fruit non-destructively where the value of RMSE and R2 achieve 0.6039, and 0.8084, respectively. Therefore, the development of a multimodal deep learning approach for non-destructive quality assessment of fig fruit was successfully achieved throughout this study.