RAW Image Denoising: History
View Latest Version
Please note this is an old version of this entry, which may differ significantly from the current revision.
Contributor:
Jing Xu
,
Yifeng Liu
,
Ming Fang

Given the challenges encountered by industrial cameras, such as the randomness of sensor components, scattering, and polarization caused by optical defects, environmental factors, and other variables, the resulting noise hinders image recognition and leads to errors in subsequent image processing. An increasing number of papers have proposed methods for denoising RAW images. 

  • image denoising
  • RAW noise
  • convolutional neural network

1. Introduction

The field of industrial cameras, as a significant application area of image processing technology, has gained widespread attention and application. Through the acquisition and processing of image data, industrial cameras are extensively utilized in automation inspection, quality control, logistics, and other domains. However, the images captured by industrial cameras often contain noise due to various factors, such as the influence of sensors and internal components of the imaging system [1]. The distribution and magnitude of this noise are non-uniform, severely impacting the retrieval of image information. The removal of image noise has become an indispensable step in image processing. Furthermore, while removing image noise, it is essential to ensure the preservation of complete image information. 
In recent years, image denoising has become a prominent research area in the fields of computer vision and image processing. Research methods can be broadly categorized into two groups: traditional denoising methods and learning-based deep learning methods. Among these, a representative method is the non-local means (NLM) proposed by Buades [2], which aims to remove noise by exploiting the similarity between pixels in an image. Subsequently, Dabov, Foi [3], and others introduced the block matching and 3D filtering (BM3D) technique, which identifies blocks similar to the current block through block matching and applies 3D filtering to these blocks, resulting in a denoised image. Mairal, Bach, and et al. [4] proposed dictionary learning-based sparse representation and self-similarity non-local methods, both of which exploit the properties of the image to eliminate noise [5]. Additionally, there are other popular image denoising methods, including but not limited to denoising using Markov random fields [6][7][8], gradient-based denoising [9][10], and total variation denoising [11][12][13].
The aforementioned methods are conventional denoising approaches that, at the time, achieved decent results in image denoising. However, they inevitably suffer from several issues: (1) reliance on manual design and prior knowledge, (2) the need for extensive parameter tuning, and (3) difficulty in handling complex noise. In contrast, deep learning-based image denoising methods exhibit strong learning capabilities, enabling them to fit complex noise distributions and achieve excellent results through parallel computing and GPU utilization, effectively reducing resource consumption. Initially, multilayer perceptrons (MLPs) [14][15] and autoencoders [16][17][18] were employed for denoising tasks, but due to their limited network capacity and inability to effectively capture noise characteristics, they fell short of the performance achieved by traditional methods. However, the introduction of ResNet by Zhang [19] addressed these issues, enabling continuous improvement in the performance of deep learning-based denoising and gradually establishing its dominance in the field. In recent years, the emergence of ViT [20][21][22] brought the application of Transformers to the field of computer vision, yielding remarkable outcomes. Nowadays, Transformers [23][24][25][26] and CNNs [27][28][29] have become mainstream methods in the domain of image denoising.
In the realm of deep learning-based denoising methods, noise can be effectively modeled. However, the current mainstream deep learning approaches primarily focus on denoising RGB images. As shown in Figure 1, many noises are generated during the camera imaging process [30]. Hence, the denoising methods designed for RGB images perform unsatisfactorily when applied to denoise RAW images. In recent years, an increasing number of papers have proposed methods for denoising RAW images. Zhang [1] modeled the noise in RAW images and made assumptions about different types of noise in the images, enabling the network to learn the noise distribution more effectively. Subsequently, Wei [31] presented a model for synthesizing real noise images, forming an extreme low-light RAW noise formation model, and proposed a noise parameter correction scheme. At the same time, in the field of RGB denoising, Brooks [32] introduced a denoising approach that involves converting RGB images to RAW format for denoising, followed by reconversion to RGB format, yielding significant improvements.
Electronics 12 04346 g001
Figure 1. Imaging noise generating processes.

2. RAW Image

Compared to sRGB images processed through the Image Signal Processing (ISP) unit of a camera, direct processing of RAW images is superior. This is because ISP processing includes steps such as white balance, denoising, gamma correction, color channel compression, and demosaicking, which result in information loss in high spatial frequencies and dynamic range. Moreover, the nature of image noise becomes more complex and challenging to handle. Throughout the camera imaging process, from photons to electrons, and then to voltage, before being converted into digital signals, noise is introduced. These noise sources primarily include thermal noise, photon shot noise, readout noise, and quantization error noise. After going through the ISP modules, noise continues to be generated, amplified, or its statistical characteristics altered, leading to an amplified impact on image quality and increasingly uncontrollable noise characteristics. Therefore, it is more feasible and effective to perform denoising on RAW images before ISP processing. In the experiments conducted by Brooks [32] they inverted RGB images to RAW images and introduced the estimated noise into the network for denoising. Their approach achieved promising results in the field of RGB image denoising.

3. RAW Image Denoising

The study of image denoising has always been an essential component of the computer vision field. With the advent of deep learning, utilizing deep neural networks for denoising has become the mainstream approach in image denoising. Early deep learning denoising methods primarily focused on removing additive Gaussian white noise from RGB images. However, for the RAW image denoising benchmark established in 2017 [33], these denoising methods, while outperforming some traditional approaches, exhibited poor performance in denoising the original image data. In the case of RAW images, neighboring pixels belong to different color channels and exhibit weak correlation, lacking the traditional notion of pixel smoothness. Furthermore, since each pixel in RAW image data contains information for only one color channel, denoising algorithms designed for color images are not applicable. In recent years, noise modeling methods, such as those proposed by Wang [34] and Wei [31], have simulated the noise distribution generated during the image signal processing (ISP) pipeline, achieving promising denoising results through network-based learning. Feng [35] introduced a method that decomposes real noise into shot noise and read noise, improving the accuracy of data mapping. Zhang [36] further extended this by proposing two components of noise synthesis, namely signal-independent, and signal-dependent, which were implemented using different methods. Although utilizing noise modeling methods can provide a good understanding of the statistical characteristics and distribution patterns of noise, which also contributes to noise removal, in practical applications, noise often exhibits diversification and is influenced by various factors, such as sensor temperature and environmental lighting, among others. A single noise model cannot fully describe all noise situations, resulting in unsatisfactory denoising performance in specific scenarios. Therefore, the researchers employ a real noise dataset to fit image noise, aiming to achieve denoising effects that accurately simulate and reproduce noise conditions in the real world. This approach enables the algorithm to learn more types and features of noise, enhancing its generalization capability and adaptability. Consequently, the algorithm becomes more versatile, applicable to a wide range of scenarios, and more closely aligned with real-world applications.

This entry is adapted from the peer-reviewed paper 10.3390/electronics12204346

References

  1. Zhang, Y.; Qin, H.; Wang, X.; Li, H. Rethinking noise synthesis and modeling in raw denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 11–17 October 2021; pp. 4593–4601.
  2. Buades, A.; Coll, B.; Morel, J.M. Non-local means denoising. Image Process. Online 2011, 1, 208–212.
  3. Dabov, K.; Foi, A.; Katkovnik, V.; Egiazarian, K. Image denoising with block-matching and 3D filtering. In Image Processing: Algorithms and Systems, Neural Networks, and Machine Learning; SPIE: Bellingham, WA, USA, 2006; Volume 6064, pp. 354–365.
  4. Mairal, J.; Bach, F.; Ponce, J.; Sapiro, G. Online dictionary learning for sparse coding. In Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada, 14–18 June 2009; pp. 689–696.
  5. Gu, S.; Zhang, L.; Zuo, W.; Feng, X. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 2862–2869.
  6. Malfait, M.; Roose, D. Wavelet-based image denoising using a Markov random field a priori model. IEEE Trans. Image Process. 1997, 6, 549–565.
  7. Cao, Y.; Luo, Y.; Yang, S. Image denoising based on hierarchical Markov random field. Pattern Recognit. Lett. 2011, 32, 368–374.
  8. Li, Y.; Li, C.; Li, X.; Wang, K.; Rahaman, M.M.; Sun, C.; Chen, H.; Wu, X.; Zhang, H.; Wang, Q. A comprehensive review of Markov random field and conditional random field approaches in pathology image analysis. Arch. Comput. Methods Eng. 2022, 29, 609–639.
  9. Zanella, R.; Boccacci, P.; Zanni, L.; Bertero, M. Efficient gradient projection methods for edge-preserving removal of Poisson noise. Inverse Probl. 2009, 25, 045010.
  10. Zeng, N.; Zhang, H.; Li, Y.; Liang, J.; Dobaie, A.M. Denoising and deblurring gold immunochromatographic strip images via gradient projection algorithms. Neurocomputing 2017, 247, 165–172.
  11. Beck, A.; Teboulle, M. Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process. 2009, 18, 2419–2434.
  12. Chan, T.F.; Chen, K. An optimization-based multilevel algorithm for total variation image denoising. Multiscale Model. Simul. 2006, 5, 615–645.
  13. Frohn-Schauf, C.; Henn, S.; Witsch, K. Nonlinear multigrid methods for total variation image denoising. Comput. Vis. Sci. 2004, 7, 199–206.
  14. Burger, H.C.; Schuler, C.J.; Harmeling, S. Image denoising: Can plain neural networks compete with BM3D? In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; pp. 2392–2399.
  15. Fan, L.; Zhang, F.; Fan, H.; Zhang, C. Brief review of image denoising techniques. Vis. Comput. Ind. Biomed. Art 2019, 2, 1–12.
  16. Bajaj, K.; Singh, D.K.; Ansari, M.A. Autoencoders based deep learner for image denoising. Procedia Comput. Sci. 2020, 171, 1535–1541.
  17. Gondara, L. Medical image denoising using convolutional denoising autoencoders. In Proceedings of the 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW), Barcelona, Spain, 12–16 December 2016; pp. 241–246.
  18. Cho, K. Boltzmann machines and denoising autoencoders for image denoising. arXiv 2013, arXiv:1301.3468.
  19. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778.
  20. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929.
  21. Chen, X.; Hsieh, C.J.; Gong, B. When vision transformers outperform resnets without pre-training or strong data augmentations. arXiv 2021, arXiv:2106.01548.
  22. Steiner, A.; Kolesnikov, A.; Zhai, X.; Wightman, R.; Uszkoreit, J.; Beyer, L. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv 2021, arXiv:2106.10270.
  23. Liang, J.; Cao, J.; Sun, G.; Zhang, K.; Van Gool, L.; Timofte, R. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 1833–1844.
  24. Wang, Z.; Cun, X.; Bao, J.; Zhou, W.; Liu, J.; Li, H. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 17683–17693.
  25. Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.H. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 5728–5739.
  26. Fan, C.M.; Liu, T.J.; Liu, K.H. SUNet: Swin transformer UNet for image denoising. In Proceedings of the 2022 IEEE International Symposium on Circuits and Systems (ISCAS), Austin, TX, USA, 28 May–1 June 2022; pp. 2333–2337.
  27. Chen, L.; Chu, X.; Zhang, X.; Sun, J. Simple baselines for image restoration. In Proceedings of the Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 23–27 October 2022; Part VII. Springer Nature: Cham, Switzerland, 2022; pp. 17–33.
  28. Chen, L.; Lu, X.; Zhang, J.; Chu, X.; Chen, C. Hinet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 182–192.
  29. Mou, C.; Zhang, J.; Fan, X.; Liu, H.; Wang, R. COLA-Net: Collaborative attention network for image restoration. IEEE Trans. Multimed. 2021, 24, 1366–1377.
  30. Konnik, M.; Welsh, J. High-level numerical simulations of noise in CCD and CMOS photosensors: Review and tutorial. arXiv 2014, arXiv:1412.4031.
  31. Wei, K.; Fu, Y.; Yang, J.; Huang, H. A physics-based noise formation model for extreme low-light raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 2758–2767.
  32. Brooks, T.; Mildenhall, B.; Xue, T.; Chen, J.; Sharlet, D.; Barron, J.T. Unprocessing images for learned raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 11036–11045.
  33. Plotz, T.; Roth, S. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1586–1595.
  34. Wang, Y.; Huang, H.; Xu, Q.; Liu, J.; Liu, Y.; Wang, J. Practical deep raw image denoising on mobile devices. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; Springer International Publishing: Cham, Switzerland, 2020; pp. 1–16.
  35. Feng, H.; Wang, L.; Wang, Y.; Huang, H. Learnability enhancement for low-light raw denoising: Where paired real data meets noise modeling. In Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal, 10–14 October 2022; pp. 1436–1444.
  36. Zhang, F.; Xu, B.; Li, Z.; Liu, X.; Lu, Q.; Gao, C.; Sang, N. Towards General Low-Light Raw Noise Synthesis and Modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2–6 October 2023; pp. 10820–10830.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.
This entry is offline, you can click here to edit this entry!
Video Production Service
Feedback