Application of Infrared and Visible Image Fusion

Application of Infrared and Visible Image Fusion: Comparison

Please note this is a comparison between Version 1 by Yongyu Luo and Version 4 by Catherine Yang.

Infrared and visible light image fusion combines infrared and visible light images to provide a more comprehensive image with more features from two photos by extracting the main information from each image and fusing it together.红外和可见光图像融合通过从每张图像中提取主要信息并将其融合在一起，将红外和可见光图像结合起来，从两张照片中提供具有更多特征的更全面的图像。

infrared and visible image fusion
image fusion
multi-scale decomposition
compressed sensing

1. Introduction简介

Image fusion attempts to use various techniques as enhancement strategies to create rich images with many aspects and information. Combining multiple sensors to produce an image is the process of image fusion. The continuous progress of current science and technology has led to the development of image fusion technology because an image with a single piece of information cannot meet the needs of people^[1]. Infrared and visible image fusion^[2], multi-focus image fusion, medical image fusion, and remote sensing image fusion are the main branches of image fusion.

图像融合试图使用各种技术作为增强策略来创建具有许多方面和信息的丰富图像。组合多个传感器产生的图像是图像融合的过程。当前科学技术的不断进步导致了图像融合技术的发展，因为具有单个信息的图像无法满足人们的需求[1]。红外和可见光图像融合[2]、多焦点图像融合、医学图像融合、遥感图像融合等是图像融合的主要分支。

Infrared and visible image fusion are commonly used in the above four image fusion techniques. The visible light band, with its high resolution and unusually detailed texture, is most consistent with the visual field of the human eye, producing images very similar to those people see in their daily lives. However, it will be severely disrupted by shielding, weather, and other factors. The ability to recognize and identify targets in infrared images captures thermal targets even in the most challenging weather conditions, such as heavy rain or smoke. On the other hand, low resolution, fuzziness, and poor contrast are further disadvantages of infrared images.

红外和可见光图像融合是上述四种图像融合中比较常用的技术。可见光波段具有很高的分辨率和异常详细的纹理，与人眼的视野最一致，产生的图像与人们日常生活中看到的图像非常相似。但是，它将受到屏蔽，天气和其他因素的严重干扰。识别和识别红外图像中目标的能力即使在最具挑战性的天气情况下（如大雨或烟雾）也能捕获热目标。另一方面，低分辨率、模糊性和对比度差是红外图像的进一步缺点。

In practical applications, the combination of infrared and visible images can solve various problems. For example, in some cases, operators must simultaneously monitor a large number of visible and infrared images from the same scene, each with its own unique requirements. Humans have found it very challenging to combine information from visible and infrared images just by staring at various visible and infrared images. In some cases with complex backgrounds, infrared images can overcome the constraints of visible images, obtain target information at night or in low-illumination environments, and improve target recognition abilities. By fusing infrared and visible light photos, workflow efficiency and convenience can be greatly improved. At the same time, infrared and visible image fusion is widely used in the fields of night vision, biometric recognition, detection, and tracking [4]. This highlights the importance of infrared and visible image fusion research.

在实际应用中，红外和可见光图像的合并可以解决各种问题。例如，在某些情况下，操作员必须同时监控来自同一场景的大量可见光和红外图像，每个图像都有自己独特的要求。人类发现，仅仅通过凝视各种可见光和红外图像来组合来自可见光和红外图像的信息是非常具有挑战性的。在一些背景复杂的情况下，红外图像可以克服可见光图像的约束，在夜间或低照度环境下获取目标信息，提高目标识别能力。通过融合红外和可见光照片，可以大大提高工作流程的有效性并带来便利。同时，红外和可见光图像融合在夜视、生物特征识别、检测和跟踪等领域存在广泛的应用[4]。这凸显了红外和可见光图像融合研究的重要性。

2. Application of night vision夜视的应用

The thermal radiation information of the target object or scene is usually converted to false color images because the human visual system is more sensitive to color images than grayscale photos. Thanks to the use of color transfer technology, the resulting color images have a realistic daytime color appearance, which makes the scene more intuitive and helps the viewer understand the image. Figure

目标物体或场景的热辐射信息通常转换为假彩色图像，因为人类视觉系统对彩色图像比对灰度照片更敏感。由于采用了色彩转移技术，产生的彩色图像具有逼真的白天色彩外观，这使得场景更加直观，有助于观众理解图像。图1

shows an example of blending visible and infrared images at night to achieve color vision.

7显示了在夜间融合可见光和红外图像以实现色觉的实例。

Figure图 17. Example of infrared and visible image fusion for color vision at night. (

夜间彩色视觉的红外和可见光图像融合示例。（

a) visible light images; (

）可见光图像;（

b) infrared images; (

）红外图像;（

c) reference images; (

）参考图像;（

d) fusion of images.

）融合图像。

Grayscale images are less responsive to human vision than color images. Human eyes are able to distinguish between thousands of colors, but they can only distinguish between about 100 grayscale images. Therefore, it is necessary to color grayscale images, especially because the fusion method of infrared and visible images with color contrast enhancement has been widely adopted in military equipment^[3]. In addition, due to the rapid growth of multi-band infrared and night vision systems, there is now greater interest in the color fusion ergonomics of many image sensor signals.

灰度图像对人类视觉的响应不如彩色图像。人眼能够区分数千种颜色，但它们只能区分大约 100 张灰度图像。因此，必须对灰度图像进行着色，特别是因为具有颜色对比度增强的红外和可见光图像的融合方法已在军事装备中被广泛采用[94]。此外，由于多波段红外和夜视系统的快速增长，现在人们对许多图像传感器信号的色彩融合人体工程学表示更感兴趣。

3. Application in the field of biometrics在生物识别领域的应用

The subject of facial recognition research is progressing rapidly. The face recognition technology for visual images has been developed to a very advanced stage and has achieved great success^[4]. In the case of low light, the face recognition rate using visual technology will be reduced. However, thermal infrared face recognition technology can perform well. Figure 2 shows images of faces captured in infrared and visible light.

面部识别研究的课题进展迅速。视觉图像的人脸识别技术已经发展到非常先进的阶段，并取得了巨大的成功[95]。在弱光情况下，使用视觉技术的人脸识别率会降低;但是，热红外人脸识别技术可以表现良好。图18显示了在红外光和可见光下捕获的人脸图像。

Figure图 218. Examples of infrared and visible face images. (

红外和可见人脸图像的示例。（

a) visible light images; (

）可见光图像;（

b) infrared images.

）红外图像。

Although face recognition technology based on visible images has been well studied, there are still significant problems with its practical implementation. For example, the recognition effect can be significantly affected by changes in lighting, facial expressions, background, and so on in the actual scene. To recognize faces, infrared photos can complement information hidden in visible-light photos. In recent years, the application of infrared and visible image fusion based on biometric optimization algorithms has increased. By increasing the amount of computation, this approach can improve identification accuracy and provide more supplementary data for biometrics. The future application of infrared and visible light fusion technology in the field of biometrics will also become more extensive.

尽管基于可见图像的人脸识别技术已经得到了深入研究，但其实际实施仍然存在重大问题。例如，识别效果会受到实际场景中的照明、面部表情、背景等变化的显著影响。为了识别人脸，红外照片可以补充隐藏在可见光照片中的信息。近年来，基于生物特征优化算法的红外和可见光图像融合的应用有所增加。通过增加计算量，这种方法可以提高识别精度，并为生物特征提供更多补充数据。未来红外和可见光融合技术在生物识别领域的应用也将变得更加广泛。

However, the growing use of facial recognition technology has also raised some ethical and privacy concerns. For example, while surveillance systems in public places contribute to social security, they also raise questions about whether people's facial information could be stolen and improperly used by outside parties. Various countries where governments are using facial recognition technology to monitor citizens' activities have raised concerns about abuses and human rights violations.

然而，面部识别技术的使用越来越多，也带来了一些道德和隐私问题。例如，虽然公共场所的监控系统有助于社会安全，但它们也引发了人们的面部信息是否可能被外部各方窃取和不当使用的问题。各国政府正在利用面部识别技术监控公民活动的各个国家都对虐待和侵犯人权的行为表示担忧。

4. Application in the field of detection and tracking在检测和跟踪领域的应用

In the field of target detection, visible light and infrared images can work together to detect targets. 在目标检测领域，可见光和红外图像可以协同作用来检测目标。Bulanon et al.^[5] combined the thermal image and visual image of the orange tree crown scene, overcoming the limitations of the two imaging techniques and improving the accuracy of fruit detection. First, fruit can be seen in visible light pictures because of the color difference between the fruit and the tree crown, but because visible light images are sensitive to light fluctuations, fruits can be misclassified. After real-world testing, it is clear that the temperature of the fruit at night is significantly greater than the temperature of the treetop, enabling the resulting infrared image to effectively detect the fruit. Finally, the accuracy of fruit detection has been improved. In order to improve the overall effectiveness of the monitoring system, [32]等人将橘子树冠场景的热图像和视觉图像合并，克服了两种成像技术的局限性，提高了果实检测的准确性。首先，由于果实和树冠之间的色差，可以在可见光图片中看到水果，但由于可见光图像对光波动敏感，因此水果可能会被错误分类。经过真实测试，很明显，晚上水果的温度明显大于树顶的温度，使产生的红外图像能够有效地检测出水果。最后，水果检测的精度有所提高。为了提高监测系统的整体有效性，Elguebaly et al.^[6] proposed a target detection method based on the fusion of visible and infrared images.等人[97]提出了一种基于可见光和红外图像融合的目标检测方法。

In subsequent frames of the video, object tracking locates the object specified in the current frame. In order to locate the object item in the time series, the object tracking algorithm must determine the relationship between the frames. Single-mode tracking is the most popular but less reliable. If it is at night or in poor lighting conditions, target tracking performance cannot be guaranteed because the quality of the visible image is highly related to the imaging environment. This is similar to the lack of texture in infrared photos, a poor three-dimensional understanding of the scene, and no guarantee of their performance in a particular situation. Therefore, Liu et al.^[7] proposed a visual tracking method that combines color images and infrared images, namely 在视频的后续帧中，对象跟踪会定位当前帧中指定的对象。为了在时间序列中定位目标项，目标跟踪算法必须确定帧之间的关系。单模跟踪是最受欢迎的，但不太可靠。如果是夜间或照明条件差，则无法保证目标跟踪性能，因为可见图像的质量与成像环境高度相关。这类似于红外照片缺乏纹理，对场景的三维理解不佳，无法保证它们在特定情况下的性能。因此，刘[98]等人提出了一种将彩色图像和红外图像相结合的视觉跟踪方法，即RGBT tracking, which can fuse complementary information in infrared and visible images to make target tracking more robust.跟踪，它可以融合红外和可见光图像中的互补信息，使目标跟踪更加鲁棒。

5. Application in the field of medical diagnosis在医学诊断领域的应用

Medical image fusion aims to improve image quality by maintaining specific functions, expanding the use of images in clinical diagnosis, and evaluating medical problems. With the continuous development of clinical application needs, medical image fusion is increasingly proving to have important advantages. Through the study of medical images, computers or clinicians are able to make most medical diagnoses. Different types of medical images employ a variety of imaging techniques and give different emphasis to the depiction of the human body. 医学图像融合旨在通过保持特定功能、扩大图像在临床诊断中的使用以及评估医疗问题来提高图像质量。随着临床应用需求的不断发展，医学图像融合越来越多地被证明具有重要优势。通过对医学图像的研究，计算机或临床医生能够进行大多数医学诊断。不同类型的医学图像采用各种成像技术，并对人体的描述给予不同的强调。计算机断层扫描[99]（Computed tomography ^[8] (CT), magnetic resonance imaging ^[9] (），磁共振成像[100]（MRI), single photon emission computed tomography^[10](），单光子发射计算机断层扫描[101]（SPECT), positron emission tomography ^[11](），正电子发射断层扫描[102]（PET), and ultrasound ^[12] are examples of common medical techniques. These methods include those that focus on regional metabolic capacity and those that focus on organ structure. If medical images of various modes can be merged, the effectiveness and accuracy of diagnosis will be significantly improved, while removing redundant information and improving image quality. As shown in Figure 3, ）和超声[103]是常见医疗技术的例子。这些方法包括那些专注于区域代谢能力的方法和那些专注于器官结构的方法。如果可以合并各种模式的医学图像，诊断的有效性和准确性将显着提高，同时删除冗余信息并提高图像质量。如图19所示，MRI images can better see more deformed soft tissue, while 图像可以更好地看到更变形的软组织，而CT scans can better see denser and less twisted tissue. Two different types of photos can then be combined using image fusion technology to provide more precise patient information.扫描可以更好地观察更密集和更少扭曲的组织。然后可以使用图像融合技术将两种不同类型的照片组合在一起，以提供更精确的患者信息。

Figure 319.

Image fusion in clinical imaging field. (

) MRI; (

) CT; (

) Fused image.

6. Applications in the Field of Autonomous Vehicles

The field of autonomous vehicles is constantly evolving, and vision, radar, and LiDAR sensors are also widely used in autonomous vehicle perception technology. The first step in autonomous driving and a crucial element in deciding how well a vehicle performs is the ability to precisely and properly recognize the surroundings around it. General vision sensors may have trouble identifying things, particularly in low light, intense sunlight, or severe weather, which is a test for self-driving automobiles. In order to increase the visual effects of automated vehicles and the safety and dependability of autonomous driving in challenging conditions, infrared and visible light image fusion has been introduced into the application of the field of driverless vehicles. For usage in the field of autonomous cars, Li ^[13][104] et al. suggested a novel two-stage network (SOSMaskFuse) that can efficiently cut down on noise, extract critical thermal data from infrared photos, and display more texture features in visible images. In order to efficiently detect and identify objects even in environments with limited visibility, such as day or night, Choi ^[14][105] et al. presented a sensor fusion system that integrates thermal infrared cameras with LiDAR sensors. This system effectively ensures the safety of autonomous vehicles.