Thermal and RGB Images Work Better Together in Wind Turbine Damage Detection
DOI:
https://doi.org/10.47839/ijc.23.4.3752Keywords:
unmanned aerial vehicle, image composition, multispectral images, green energy, data quality management, weighted overlayAbstract
The inspection of wind turbine blades (WTBs) is crucial for ensuring their structural integrity and operational efficiency. Traditional inspection methods can be dangerous and inefficient, prompting the use of unmanned aerial vehicles (UAVs) that access hard-to-reach areas and capture high-resolution imagery. In this study, we address the challenge of enhancing defect detection on WTBs by integrating thermal and RGB images obtained from UAVs. We propose a multispectral image composition method that combines thermal and RGB imagery through spatial coordinate transformation, key point detection, binary descriptor creation, and weighted image overlay. Using a benchmark dataset of WTB images annotated for defects, we evaluated several state-of-the-art object detection models. Our results show that composite images significantly improve defect detection efficiency. Specifically, the YOLOv8 model’s accuracy increased from 91% to 95%, precision from 89% to 94%, recall from 85% to 92%, and F1-score from 87% to 93%. The number of false positives decreased from 6 to 3, and missed defects reduced from 5 to 2. These findings demonstrate that integrating thermal and RGB imagery enhances defect detection on WTBs, contributing to improved maintenance and reliability.
References
Z. Gao and X. Liu, “An overview on fault diagnosis, prognosis and resilient control for wind turbine systems,” Processes, vol. 9, no. 2, Art. no. 2, Feb. 2021, https://doi.org/10.3390/pr9020300.
L. Mishnaevsky et al., “Repair of wind turbine blades: Costs and quality,” J. Phys.: Conf. Ser., vol. 2265, no. 3, p. 032032, May 2022, https://doi.org/10.1088/1742-6596/2265/3/032032.
M. Novak et al., “Multisensor UAV system for the forest monitoring”, in Proc. 2020 10th Int. Conf. Adv. Comput. Inf. Technol. (ACIT), Deggendorf, Germany, Sep. 16–18, 2020. New York, NY, USA: IEEE, Inc., 2020, pp. 293–296, https://doi.org/10.1109/acit49673.2020.9208993.
O. Melnychenko, O. Savenko, and P. Radiuk, “Apple detection with occlusions using modified YOLOv5-v1”, in Proc. 12th IEEE Int. Conf. Intell. Data Acquisition Adv. Comput. Syst.: Technol. Appl. (IDAACS’2023), Dortmund, Germany, Sep. 7–9, 2023. New York, NY, USA: IEEE, Inc., 2023, pp. 107–112, https://doi.org/10.1109/idaacs58523.2023.10348779.
S. Hu, Q. Wu, and X. Wang, “Energy management and trajectory optimisation for UAV-enabled legitimate monitoring systems,” IEEE Trans. Wireless Commun., vol. 20, no. 1, pp. 142–155, Jan. 2021, https://doi.org/10.1109/TWC.2020.3023816.
K. Jahani, R. G. Langlois, and F. F. Afagh, “Structural dynamics of offshore wind turbines: A review,” Ocean Eng., vol. 251, p. 111136, May 2022, https://doi.org/10.1016/j.oceaneng.2022.111136.
V. Sykes, M. Collu, and A. Coraddu, “A review and analysis of the uncertainty within cost models for floating offshore wind farms,” Renewable Sustain. Energy Rev., vol. 186, p. 113634, Oct. 2023, https://doi.org/10.1016/j.rser.2023.113634.
O. Melnychenko, L. Scislo, O. Savenko, A. Sachenko, and P. Radiuk, “Intelligent integrated system for fruit detection using multi-UAV imaging and deep learning,” Sensors, vol. 24, no. 6, p. 1913, Mar. 2024, https://doi.org/10.3390/s24061913.
M. Civera and C. Surace, “Non-destructive techniques for the condition and structural health monitoring of wind turbines: A literature review of the last 20 years,” Sensors, vol. 22, no. 4, Art. no. 4, Jan. 2022, https://doi.org/10.3390/s22041627.
C. Yang, X. Liu, H. Zhou, Y. Ke, and J. See, “Towards accurate image stitching for drone-based wind turbine blade inspection,” Renew. Energy, vol. 203, pp. 267–279, Feb. 2023, https://doi.org/10.1016/j.renene.2022.12.063.
P. Rizk, N. Al Saleh, R. Younes, A. Ilinca, and J. Khoder, “Hyperspectral imaging applied for the detection of wind turbine blade damage and icing,” Remote Sens. Appl.: Soc. Environ., vol. 18, p. 100291, Apr. 2020, https://doi.org/10.1016/j.rsase.2020.100291.
M. Car, L. Markovic, A. Ivanovic, M. Orsag, and S. Bogdan, “Autonomous wind-turbine blade inspection using lidar-equipped unmanned aerial vehicle,” IEEE Access, vol. 8, pp. 131380–131387, 2020, https://doi.org/10.1109/ACCESS.2020.3009738.
S. Lee, Y. Song, and S.-H. Kil, “Feasibility analyses of real-time detection of wildlife using UAV-derived thermal and RGB images,” Remote Sens., vol. 13, no. 11, Art. no. 11, Jan. 2021, https://doi.org/10.3390/rs13112169.
Y. Du, S. Zhou, X. Jing, Y. Peng, H. Wu, and N. Kwok, “Damage detection techniques for wind turbine blades: A review,” Mech. Syst. Signal Process., vol. 141, p. 106445, Jul. 2020, https://doi.org/10.1016/j.ymssp.2019.106445.
J. Meyers et al., “Wind farm flow control: Prospects and challenges,” Wind Energy Sci., vol. 7, no. 6, pp. 2271–2306, Nov. 2022, https://doi.org/10.5194/wes-7-2271-2022.
F. P. García Márquez and A. M. Peco Chacón, “A review of non-destructive testing on wind turbines blades,” Renew. Energy, vol. 161, pp. 998–1010, Dec. 2020, https://doi.org/10.1016/j.renene.2020.07.145.
I. Paliy, A. Sachenko, V. Koval, and Y. Kurylyak, “Approach to face recognition using neural networks”, in Proc. 2005 IEEE Intell. Data Acquisition Adv. Comput. Syst.: Technol. Appl., Sofia, Bulgaria, Sep. 5–7, 2005. New York, NY, USA: IEEE, Inc., 2005, pp. 112–115, https://doi.org/10.1109/idaacs.2005.282951.
M. Komar, A. Sachenko, V. Golovko and V. Dorosh, “Compression of network traffic parameters for detecting cyber attacks based on deep learning,” Proceedings of the 2018 IEEE 9th International Conference on Dependable Systems, Services and Technologies (DESSERT), Kyiv, Ukraine, 2018, pp. 43-47, https://doi.org/10.1109/DESSERT.2018.8409096.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, Jun. 2017, https://doi.org/10.1109/tpami.2016.2577031.
M. Tan, R. Pang, and Q. V. Le, “EfficientDet: Scalable and efficient object detection”, in Proc. 2020 IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Seattle, WA, USA, Jun. 13–19, 2020. New York, NY, USA: IEEE, Inc., 2020, pp. 10781–10790, https://doi.org/10.1109/cvpr42600.2020.01079.
G. Jocher et al., ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation. (Nov. 22, 2022). Python. Zenodo. https://doi.org/10.5281/zenodo.7347926.
G. Jocher, J. Qiu, and A. Chaurasia, Ultralytics YOLO (Version 8.0.0). (Jan. 2023). Python. GitHub, Inc. Accessed: Aug. 17, 2024. [Online]. Available: https://github.com/ultralytics/ultralytics
M. Memari, P. Shakya, M. Shekaramiz, A. C. Seibi, and M. A. S. Masoum, “Review on the advancements in wind turbine blade inspection: Integrating drone and deep learning technologies for enhanced defect detection”, IEEE Access, vol. 12, pp. 33236–33282, Feb. 2024, https://doi.org/10.1109/access.2024.3371493.
M. Brenner, N. H. Reyes, T. Susnjak, and A. L. C. Barczak, “RGB-D and thermal sensor fusion: A systematic literature review”, IEEE Access, vol. 11, pp. 82410–82442, Aug. 2023, https://doi.org/10.1109/access.2023.3301119.
D. Thomas, M. Gündel, A. Wickers, M. Alpen, and J. Horn, “Multivariate inspection of German steel civil infrastructure using autonomous UAS”, in Life-Cycle of Structures and Infrastructure Systems. London: CRC Press, 2023, pp. 493–500, https://doi.org/10.1201/9781003323020-58.
B. Szeląg, S. Sobura, and R. Stoińska, “Application of multispectral images from unmanned aerial vehicles to analyze operations of a wastewater treatment plant,” Energies, vol. 16, no. 6, Art. no. 6, Jan. 2023, https://doi.org/10.3390/en16062871.
O. Melnychenko and O. Savenko, “A self-organised automated system to control unmanned aerial vehicles for object detection”, in Proc. 4th Int. Workshop Intell. Inf. Technol. & Syst. Inf. Secur. (IntelITSIS’2023), Khmelnytskyi, Ukraine, Mar. 22–24, 2023. Aachen: CEUR-WS.org, 2023, pp. 589–600. Accessed: Aug. 17, 2024. [Online]. Available: https://ceur-ws.org/Vol-3373/paper40.pdf
C. R. Petersen, S. Fæster, J. I. Bech, K. M. Jespersen, N. M. Israelsen, and O. Bang, “Non-destructive and contactless defect detection inside leading edge coatings for wind turbine blades using mid-infrared optical coherence tomography,” Wind Energy, vol. 26, no. 5, pp. 458–468, 2023, https://doi.org/10.1002/we.2810.
P. G. Regodeseves and C. S. Morros, “Unsteady numerical investigation of the full geometry of a horizontal axis wind turbine: Flow through the rotor and wake,” Energy, vol. 202, p. 117674, Jul. 2020, https://doi.org/10.1016/j.energy.2020.117674.
Q. Zhang, T. Xiao, N. Huang, D. Zhang, and J. Han, “Revisiting feature fusion for RGB-T salient object detection,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 5, pp. 1804–1818, May 2021, https://doi.org/10.1109/TCSVT.2020.3014663.
L. Morando, C. T. Recchiuto, J. Calla, P. Scuteri, and A. Sgorbissa, “Thermal and visual tracking of photovoltaic plants for autonomous UAV inspection”, Drones, vol. 6, no. 11, p. 347, Nov. 2022. Accessed: Oct. 29, 2024, https://doi.org/10.3390/drones6110347
W. Zhou, Z. Wang, M. Zhang, and L. Wang, “Wind turbine actual defects detection based on visible and infrared image fusion,” IEEE Trans. Instrum. Meas., vol. 72, pp. 1–8, 2023, https://doi.org/10.1109/TIM.2023.3251413.
J. Zhu, C. Wen, and J. Liu, “Defect identification of wind turbine blade based on multi-feature fusion residual network and transfer learning,” Energy Sci. & Eng., vol. 10, no. 1, pp. 219–229, 2022, https://doi.org/10.1002/ese3.1024.
K.-A. Kwon, M.-Y. Choi, H.-S. Park, J.-H. Park, Y.-H. Huh, and W. J. Choi, “Quantitative defects detection in wind turbine blade using optical infrared thermography,” J. Korean Soc. Nondestruct. Testingvol., vol. 35, no. 1, pp. 25–30, Feb. 2015, https://doi.org/10.7779/JKSNT.2015.35.1.25.
H. Sanati, D. Wood, and Q. Sun, “Condition monitoring of wind turbine blades using active and passive thermography,” Appl. Sci., vol. 8, no. 10, Art. no. 10, Oct. 2018, https://doi.org/10.3390/app8102004.
M. Memari, M. Shekaramiz, M. A. S. Masoum, and A. C. Seibi, “Data fusion and ensemble learning for advanced anomaly detection using multi-spectral rgb and thermal imaging of small wind turbine blades”, Energies, vol. 17, no. 3, p. 673, Jan. 2024. https://doi.org/10.3390/en17030673.
Q. Fang, P. M. Meaney, and K. D. Paulsen, “Singular value analysis of the Jacobian matrix in microwave image reconstruction,” IEEE Trans. Antennas Propag., vol. 54, no. 8, pp. 2371–2380, Aug. 2006, https://doi.org/10.1109/TAP.2006.879192.
B. Chen and E. Jungstedt, “Fast and large-converge-radius inverse compositional Levenberg–Marquardt algorithm for digital image correlation: principle, validation, and open-source toolbox,” Opt. Lasers Eng., vol. 151, p. 106930, Apr. 2022, https://doi.org/10.1016/j.optlaseng.2021.106930.
T. Sánchez and A. Augusto, “Image space and time interpolation for video navigation,” Master thesis, Universitat Politècnica de Catalunya, Barcelona, Spain (Online), 2011. Accessed: Aug. 17, 2024. [Online]. Available: https://upcommons.upc.edu/handle/2099.1/13425
M. Petravic and G. Kuo-Petravic, “An ILUCG algorithm which minimizes in the euclidean norm”, J. Comput. Phys., vol. 32, no. 2, pp. 263–269, Aug. 1979, https://doi.org/10.1016/0021-9991(79)90133-5
C. Mayfield, “Automating the classification of thematic rasters for weighted overlay analysis in GeoPlanner for ArcGIS,” Master of Science, University of Redlands, Redlands, CA, 2015. https://doi.org/10.26716/redlands/master/2015.6.
B. K. P. Horn, “Understanding image intensities”, Artif. Intell., vol. 8, no. 2, pp. 201–231, Apr. 1977, https://doi.org/10.1016/0004-3702(77)90020-0
I. E. Komari, M. Fedorenko, V. Kharchenko, Y. Yehorova, N. G. Bardis, L. Lutai, “The neural modules network with collective relearning for the recognition of diseases: Fault-tolerant structures and reliability assessment,” International Journal of Circuits, Systems and Signal Processing, vol. 14, pp. 792-800, 2020. https://doi.org/10.46300/9106.2020.14.102.
P. Khan et al., “Machine learning and deep learning approaches for brain disease diagnosis: Principles and recent advances,” IEEE Access, vol. 9, pp. 37622-37655, 2021, https://doi.org/10.1109/ACCESS.2021.3062484.
Downloads
Published
How to Cite
Issue
Section
License
International Journal of Computing is an open access journal. Authors who publish with this journal agree to the following terms:• Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
• Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
• Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
