Hotspot localization and severity classification in photovoltaic aerial thermal imagery using semantic segmentation and convolutional neural network

The rapid expansion of solar photovoltaic (PV) farms has increased the need for reliable automated fault detection systems. To achieve optimal performance and effective energy management in solar PV systems, it is crucial to detect faults in solar PV modules at the early monitoring stage. Hotspot formation is one of the most critical PV faults, often caused by shading, cell damage, or degradation, leading to power loss and potential safety hazards. While thermal imaging effectively detects hotspots, variation in emissivity, reflections, and temperature fluctuations can reduce localization accuracy and affect hotspot severity classification. Moreover, classifying hotspot severity based on temperature distribution remains a significant challenge due to the complexity of thermal colour variations and the lack of standardized reference frameworks or benchmarking criteria. A dual-input feature fusion strategy is introduced to integrate thermal image features and temperature descriptors, improving hotspot severity classification. This study proposes an automated hotspot localization and severity classification for aerial thermal imagery using Convolutional Neural Network (CNN) framework. Initially, a series of image pre-processing techniques such as image annotation and resizing are applied to adapt with the CNN framework and training process. This study proposes a two-tier semantic segmentation approach for the automatic detection of hotspot regions in aerial thermal images of solar PV systems. In the first tier, a semantic segmentation model is developed to isolate solar PV modules from the background. Subsequently, in the second tier, a fine-grained semantic segmentation model is used to segment and detect hotspot regions on the extracted solar PV modules. For this study, three CNN models namely U-Net, ResNet-18 and ResNet-50, were evaluated for hotspot segmentation and severity classification. Next, a novel dual-input feature fusion strategy is introduced by combining CNN-derived features and thermal descriptors for hotspot severity classification. This integration captures complementary thermal information and enables more reliable severity assessment. To further improve classification performance, an enhanced CNN model incorporating the proposed dual-input feature fusion strategy is developed. Specifically, convolutional layers from multiple DL models are concatenated to enhance multi-input hotspot severity classification. Finally, the model's performance is evaluated using classification metrics-based performance and compared with the solar thermographer evaluation to validate the robustness and reliability. Overall, the enhanced CNN-based framework significantly improves hotspot severity classification performance, with ResNet-18 achieving a 23% accuracy improvement compared with the single-input image-based model. The proposed approach extends existing hybrid and traditional methods by integrating pixel-level hotspot localization with structured severity classification in a fully automated framework. The developed framework provides a practical solution for automated hotspot severity classification in PV thermal inspection systems, supporting improved renewable energy monitoring and management.