1. Introduction
With the advent of the Internet of Things, cloud computing, and the big data era, there has been explosive growth in the scale of information. However, the physical separation among perception, computation, and storage in the conventional computing architecture requires frequent data shuttling among the units, thereby causing significant system consumption and speed loss and making it difficult to meet the requirements of information analysis and processing
[1][2][3]. Therefore, developing new electronic components for intelligent processing systems that are closer to the structure and function of biological brains has become a hot research topic in the fields of modern electronic circuits and image processing
[4][5][6].
Image processing technology, which aims to automatically acquire high-level and abstract information from images, after which it simulates how human eyes work with such information, has become increasingly useful in human life and social production. Exploring the basic structure of human brains, simulating their working mechanisms, and establishing neural network models that integrate perception, storage, and computation as a whole have gradually become research hotspots in the fields of image processing and cognitive computing
[7]. The current mainstream neural network models can simulate the reasoning and learning functions of human brains to a certain extent, and they have shown some potential in image processing
[8]. However, they are confined to certain types and structures with limited processing capabilities. Additionally, the existing ones lack the process of information perception, transmission, and storage prior to the processing stage. Furthermore, the hardware for neural networks is essential to truly realize the conversion from theoretical studies of brain cognition to new technologies of brain-computer intelligence. Nevertheless, most of the current research focuses on the theoretical analysis of the network structures and algorithms, and the research on implementation schemes for neural network hardware is still in its infancy
[9][10][11]. Influenced by factors, such as device size, energy consumption, and integrability, conventional implementation schemes for image processing cannot well trade-off the relationship between speed, accuracy, and system consumption
[12][13][14]. We schematically compare the traditional image processing systems and memristor-based image processing system as shown in
Figure 1. From the perspective of the device, leakage currents become a problem when the channel length and the gate dielectric thickness of a transistor get closer to the scaling limit
[3]. With respect to the architecture, the data transfer between processors and memory units significantly reduces both speed and energy efficiency (referred to as the ‘von Neumann bottleneck’). Furthermore, the performance mismatch between the memory and processing units leads to great latency (also called the ‘memory wall’).
Figure 1. The comparation between the traditional image processing systems and memristor-based image processing system.
The successful preparation of memristors provides a fresh perspective on the hardware implementation of artificial neural networks. It was proposed by Leon Chua, a scientist at the University of California, Berkeley
[15] and discovered by Hewlett-Packard (HP) laboratories in 2008 as the fourth fundamental electronic component after the resistor, capacitor, and inductor
[16]. Experiments have shown that the memristor has properties, such as non-volatility, variable resistance, nanoscale size, threshold characteristics, low power consumption, and synapse-like structure
[17][18][19]. In particular, by taking full advantage of being synapse-like, the memristor can be used as an “electronic synapse” or an “artificial synapse” in the hardware design of neural networks
[20]. Further, after choosing a proper memristor model to simulate the weight of the neural network, a more integrated architecture for hardware implementation can be constructed and applied to different image processing tasks
[21][22]. Compared with conventional artificial neural networks, the memristive ones incorporate powerful perception ability, massive storage capacity, and intelligent processing mechanisms to enable deeper analysis and exploration, which are expected to solve slow training speed and insufficient online processing capability in image processing
[23][24][25].
2. Image Processing Based on Memristive Neural Networks
The successful preparation of memristors brings new ideas for simulating the cognitive functions of artificial synapses. By applying memristive synapses to the hardware implementation of neural networks, a new type of neural network with high integration can be built. It possesses powerful image processing capabilities and plays an important role in fields with high computational complexity, such as image recognition, classification, and segmentation.
2.1. Image Recognition Based on Memristive Neural Networks
In the literature
[26], an impulsive neural network based on memristors was constructed in which the memristive synapses used STDP rules to update the weights, and the memristive neurons adopted the “winner-take-all” strategy to complete the task of handwriting recognition. It was found that its recognition accuracy could reach 83%. A study conducted by Yakopcic et al.
[27] presented a memristor-based convolutional neural network to perform convolutional operations using memristor crossbar arrays, and the accuracy of its handwritten digit recognition reached 94%. Furthermore, a transformation method for neural network models was brought forward
[28], as shown in
Figure 2.
Figure 2. Handwriting recognition based on the ReRAM array.
Specifically, the method of sparsity was taken to divide the original neural network into appropriately sized sub-networks. The limited hardware accuracy was solved by quantizing the input data and somewhat improved to approximately 99.8% from the software side. In the study
[29], a 1M structured memristive synapse was introduced to the memristor-based multilayer neural network, and an adaptive backpropagation algorithm was applied to train the neural networks, thereby achieving character recognition. Kang Jinfeng’s team at Peking University
[30] reported a memristor-based binary neural network. It was trained online, its weight update was achieved using the 2T2R structure of the memristive synapse, and its correctness and effectiveness were verified on the MNIST dataset with a recognition accuracy of 97.4%. In addition, Hu et al.
[22] used 2 phase-change memories to construct artificial synapses, based on which a 3-layer perceptron network was built, and they proved its correctness on the MNIST dataset with a recognition accuracy of 82.2%. For Wang et al.
[31], they constructed a memristor-based convolutional neural network, which was significantly improved in terms of array area and energy efficiency compared with previous ones for the handwriting recognition task. In 2020, a research team from Tsinghua University
[32] designed a memristor-based convolutional neural network (see
Figure 3). Meanwhile, a hybrid training method was suggested to enhance the robustness of the network, and the handwriting recognition task realized an accuracy of over 96%.
Figure 3. Five-layer mCNN with memristor convolver.
In addition, memristor-based neural networks have been applied to other image recognition tasks. For instance, Professor Wu Huaqiang and his team members from Tsinghua University constructed a multilayer perceptron neural network based on 1T1R memristive synapses
[33], as shown in
Figure 4. The network achieved grey scale face image recognition from the Yale Face Database through online learning, and the recognition rate could reach 88.08% for 9000 test images with noise added. Other researchers
[34] investigated a hierarchical temporal memory (HTM) network based on memristors, which applied sparse distributed representations to obtain spatial information of input signals, after which they used parallel learning to adjust the network weights and finally verified the correctness and effectiveness of the network through face recognition tasks. As for the memristor-based probabilistic neural network
[35], it carried out product multiplication using memristor crossbar arrays as well as normalization operations on weights to reduce the complexity of the circuit. The network was validated on the Iris Flower dataset with a recognition accuracy of 98%. Furthermore, the multilayer perceptron neural network studied by Yu et al.
[36] showed increased adaptive capability by introducing nonlinear features in the learning process and superior performance on general datasets, such as MNIST, Iris, and Car Evaluation.
Figure 4. Face recognition task is realized in 1T1R array.
2.2. Image Classification Based on Memristive Neural Networks
In 2013, Alibart et al.
[37] successfully prepared a TiO
2−x-based memristor, after which they developed a single-layer perceptron (SLP) neural network based on the TiO
2−x memristor crossbar array to achieve image classification. Its circuit structure is displayed in
Figure 5.
Figure 5. Single-layer perceptron network memristor circuit.
Another (SLP) neural network was made based on 2M memristive synapses
[10], and its circuit structure is presented in
Figure 6. The network, which was trained using delta rules, achieved the classification of 3 × 3-pixel black-and-white images. Professor Strukov’s team at the University of California, Santa Barbara
[38] prepared a 20 × 20 memristor crossbar array, as depicted in
Figure 7. The array adopted TiO
2−x and Al
2O
3 as the functional and stacked layers, respectively, after which it was interconnected with traditional CMOS peripheral circuits, thereby constructing an SLP neural network to achieve the image classification with an accuracy of more than 97%.
Figure 6. Single layer perceptron implemented using 10 × 6 memristor crossbar array.
Figure 7. Three-layer fully connected perceptron network realized though Pt/Al2O3/TiO2−x/Ti/Pt memristor arrays.
Wang et al.
[39] prepared a three-dimensional structured memristor and applied it to image classification, which improved the operational efficiency of the algorithm and opened a new path for the in-depth integration of computer vision and novel nanodevices. Additionally, a memory computing framework based on memristors was proposed by Zhang et al. in 2021
[40], which used a greedy search algorithm to improve the robustness and anti-interference capability of the system, and its accuracy reached 92.3% on the classification tasks involving the CIFAR-10 dataset.
2.3. Image Segmentation Based on Memristive Neural Networks
As early as 2014, Myonglae et al.
[41] proposed a memristor-based visual recognition system, where the system used a programmable gate array to convert image signals into pulse signals and performed weight updates based on STDP learning rules. As a result, the foreground and background segmentation of figure images from “0” to “9” were achieved. One year later, Chiu and his team members
[42] constructed a differential 2R crossbar array, which applied RRAM as a cache to reduce system energy consumption, and they verified its correctness and effectiveness using image segmentation tasks. In the literature
[43], a fully convolutional neural network based on memristors was introduced. It utilized voltage selectors and memristor arrays to construct its max-pooling layers as well as a sliding window approach to enhance operation efficiency. Moreover, the weight updates of memristor arrays were implemented through the ex-situ training method, and the effectiveness of the proposed network was finally verified through image segmentation. The study
[44] designed a memristor-based cell neural network based on the fractional-order calculus theory, as illustrated in
Figure 8.
Figure 8. Memristor-based cell neural network.
In the process of image edge extraction, it took the fractional-order control method to increase the high-frequency information and retain more low-texture information. The simulation results proved that the edge images extracted by this network had more complete and clear contour information and richer texture detail information. Another example is the prepared memristor with NbO
x as its functional layer
[45]. An artificial sensory neuron was constructed, then in combination with an InGaZnO
4 optical sensor (see
Figure 9), which encoded optical information into impulses, image segmentation in complex backgrounds was achieved by such a pulse-coupled neural network. It is believed that this study has paved the way for the integration of neuromorphology and bioelectronics. In 2021, Chen et al.
[46] proposed an efficient memristor-based fully convolutional neural network, which adopted a convolutional kernel-first (CKF) algorithm to achieve effective parameter pruning, thereby significantly reducing circuit power consumption and demonstrating high accuracy and adaptiveness for medical image segmentation tasks.
Figure 9. Impulse coupled neural network based on memristor.
2.4. Others
Tsai et al.
[47] reported a long short-term memory network, which mapped and programmed the network weights into the phase-change memory devices, as demonstrated in
Figure 10. Compared to other methods, this network realized the software-equivalent text prediction as well as a larger improvement in the accuracy of weight mapping and text prediction.
Figure 10. Realization method of long short-term memory network based on phase-change memory device unit.
Another long short-term memory network was built on a 128 × 64 1T1R memristor array
[48], as shown in
Figure 11. Through utilizing the memristor arrays to store synaptic weights for different time steps, the network performed the prediction task of the number of global airline passengers and the recognition task of human gait, and it verified the feasibility of the memristor-based long short-term memory neural network in performing tasks, such as linear regression and pattern recognition.
Figure 11. LSTM network based on memristor synaptic array.
Moreover, Farkhani et al.
[49] designed a neuromorphic computing system based on spintronic memristors, where the read circuit was replaced with a proposed real-time sensing circuit, and the input signals were turned into the switching of magnetic moments, thereby substantially reducing circuit energy consumption, providing system operational efficiency, and achieving the real-time tracking of targets. As for the study
[50], the chaotic trajectories of memristive circuits were included, which combined the homotopy analysis method (HAM) and multi-objective optimization (MO) to tackle the high computational complexity and low computational efficiency of traditional analysis methods.
In this paper, the architectural characteristics of several image processing algorithms based on memristive neural networks are comprehensively summarized, including their input coding patterns, weight representations and the data types of interlayer communication. The specific comparative information is summarized in Table 1.
Table 1. Comparative information of memristive neural network-based image processing.
Reference |
Architectural Characteristics of Image Processing Algorithms Based on Memristive Neural Networks |
Input Coding |
Weight Representation |
Neural Network Communication |
[51] |
Amplitude Encoding/Time Encoding Analogue Signal |
Differential Amplifier |
Multi-precision |
[52] |
Amplitude Encoding Analogue Signal |
Multi-precision |
MSB |
[38] |
Amplitude Encoding Analogue Signal |
Differential Amplifier |
Multi-precision |
[40] |
Amplitude Encoding Analogue Signal |
Differential Amplifier |
Multi-precision |
[53] |
Amplitude Encoding Analogue Signal |
Peripheral Circuit Processing |
MSB |
The above key research questions will provide references for building the next generation of novel memristive neural networks with integrated perception-storage-computation architectures.
2.5. Summary
With the expanded research in nanomaterials science and image processing technology, image processing based on memristive neural networks has become one of the hot issues in the study on neural network hardware implementation schemes. Currently, there are the following problems that must be solved timely.
- (1)
-
The existing memristive synaptic circuits can only simulate the basic functions and behavioural characteristics of biological synapses, and they lack enough theoretical support from computational neuroscience. Therefore, it is crucial to design a fully functional and simple structured memristive synaptic circuit, which can address the problems of insufficient portray, unclear mechanism, and single plasticity of the conventional ones.
- (2)
-
There is the accumulation of computational errors in memristor-based neural network circuits, which is mainly owing to the discrete nature of memristors, and it is difficult to avoid at the device level. Therefore, designing a newly structured memristor crossbar array that offsets the accumulated errors can provide a new perspective for the hardware implementation of neural networks.
- (3)
-
The current research on image processing based on memristive neural networks is still stuck in the simulation of existing artificial neural networks. Therefore, the next research hotspot will involve taking both the advantages of neurocomputing science and image processing studies, exploring brain-inspired neural network training algorithms, and building memristive neural networks with brain-like memory.