Encyclopedia
Self-driving cars are a hot research topic in science and technology, which has a great influence on social and economic development. Deep learning is one of the current key areas in the field of artificial intelligence research. It has been widely applied in image processing, natural language understanding, and so on. In recent years, more and more deep learning-based solutions have been presented in the field of self-driving cars and have achieved outstanding results. This paper presents a review of recent research on theories and applications of deep learning for self-driving cars. This survey provides a detailed explanation of the developments of self-driving cars and summarizes the applications of deep learning methods in the field of self-driving cars. Then the main problems in self-driving cars and their solutions based on deep learning methods are analyzed, such as obstacle detection, scene recognition, lane detection, navigation and path planning. In addition, the details of some representative approaches for self-driving cars using deep learning methods are summarized. Finally, the future challenges in the applications of deep learning for self-driving cars are given out.
- self-driving cars
- deep learning method
- obstacle detection
- scene recognition
- lane detection
The rapid development of the Internet economy and Artificial Intelligence (AI) has promoted the progress of self-driving cars. The market demand and economic value of self-driving cars are increasingly prominent [[1]]. At present, more and more enterprises and scientific research institutions have invested in this field. Google, Tesla, Apple, Nissan, Audi, General Motors, BMW, Ford, Honda, Toyota, Mercedes, Nvidia, and Volkswagen have participated in the research and development of self-driving cars [[2]].
Google is an Internet company, which is one of the leaders in self-driving cars, based on its solid foundation in artificial intelligence [[3]]. In June 2015, two Google self-driving cars were tested on the road (as shown in Figure 1a). So far, Google vehicles have accumulated more than 3.2 million km of tests, becoming the closest to the actual use. Another company that has made great progress in the field of self-driving cars is Tesla. Tesla was the first company to devote self-driving technology to production. Followed by the Tesla models series, its “auto-pilot” technology has made major breakthroughs in recent years. Although the tesla’s autopilot technology is only regarded as Level2 stage by the national highway traffic safety administration (NHTSA), as one of the most successful companies in autopilot system application by far, Tesla shows us that the car has basically realized automatic driving under certain conditions (see Figure 1b) [[4]].
Figure 1. Self-driving cars of Google and Tesla: (a) Google’s self-driving car; (b) Tesla’s self-driving car.
In addition to the companies mentioned above, lots of Internet companies and car companies worldwide are also focusing on the self-driving car field recently. For example, in Sweden, Volvo and Autoliv established a joint company-Zenuity, which is committed to the security of self-driving cars [[5]]. In South Korea, Samsung received approval from the South Korean government to test its driverless cars on public roads in 2017. It should be noted that Samsung applied for the highest number of patents in the world in the field of self-driving cars from 2011 to 2017 [[6]]. In China, Baidu deep learning institute led the research project of self-driving cars in 2013. In 2014, Baidu established the automotive networking business division and successively launched CarLife, My-car, CoDriver and other products [[7]]. In 2016, Baidu held a strategic signing ceremony with Wuzhen Tourism, announcing that the unmanned driving at Level4 will be implemented in this scenic area (see Figure 2a). Other information technology (IT) companies in China also have intensively studied and made great progress in this field based on their technology in the field of artificial intelligence, such as Tencent, Alibaba, Huawei, and so on [[8]]. For example, Tencent has displayed the red flag Level3 self-driving car in cooperation with FAW (First Auto Work) (see Figure 2b).
Figure 2. Self-driving cars of the companies in China: (a) Baidu’s self-driving car; (b) Tencent’s self-driving car.
According to the autonomous driving level, a car in Level0 to Level2 needs a driver to mainly monitor the environment. Advanced Driver Assistance Systems (ADAS) are intelligent systems that reside inside the vehicles categorized from Level0 to Level2 and help the driver in the process of driving [[9][10]]. The risks can be minimized based on the ADAS, such as the electronic stability control system and forward emergency braking system, by reducing driver errors, continuously alerting drivers, and controlling the vehicle if the driver is incompetent [[11]].
In this paper, we focus on self-driving cars which are categorized as level 3 or above. The overall technical framework of self-driving cars that are equipped with a Level3 or higher autonomy system can be divided into four parts, namely the driving environment perception system, the autonomous decision system, the control execution system and the monitor system [[12]]. The architecture of it is shown in Figure 3.
Figure 3. The overall technical framework of self-driving cars with a Level3 or higher autonomy system.
The environment perception system utilizes the prior knowledge of the environment to establish an environmental model including obstacles, road structures, and traffic signs through obtaining surrounding environmental information. The main function of the environment perception system is to realize functions like lane detection, traffic signal detection, and obstacle detection, by using some hardware devices such as cameras and laser radars.
The main function of the autonomous decision system is to make some decisions for the self-driving car, including obstacle avoidance, path planning, navigation, and so on. For example, in the path planning, the autonomous decision system plans a global path according to the current location and the target location firstly, then reasonably plans a local path for the self-driving car by combining the global path and the local environment information provided by the environment perception system.
The control execution system’s function is to execute the commands received from the autonomous decision system, such as braking, steering, and accelerating to complete the speed control and path-following control. The control execution system will perform some actions according to the situations of the environment directly sometimes, without any commands from the autonomous decision system, to deal with some emergencies, such as pedestrian avoidance.
The monitor system is responsible to check whether the car is making actual progress towards its goal and reacts with recovery actions when meeting problems like unexpected obstacles, faults, etc.
The self-driving car is a complex autonomous system, which requires the support of the theories and technologies. At the technical level, it is impossible to achieve such rapid development of the self-driving car without the rapid development of the hardware and software. There are various good hardware and soft platforms capable of rapid data analysis, as well as managing and understanding of self-driving cars [[13]]. For example, the NVIDIA DRIVE PX2 driverless car platform can perform 30 trillion deep learning operations per second and can achieve Level4 autopilot [[14]]. It supports 12-channel camera inputs, laser positioning, radar, and ultrasonic sensors, and includes two new-generation NVIDIA Tegra processors (see Figure 4). When it comes to softwares, Tensorflow is one of the main libraries for deep learning used in the field of self-driving cars [[15]].
Figure 4. The NVIDIA DRIVE PX2 driverless car platform.
At the theoretical level, some methods used in the robotic field can be applied in self-driving cars for the similarity between them, including path planning, environmental sensing, autonomous navigation and control, etc. [[16][17]]. Various artificial intelligence algorithms have been used in self-driving cars, such as fuzzy logic, neural network, and so on. Among these methods, deep learning methods have achieved great success in the self-driving field for their distinct advantages, such as high accuracy, strong robustness, and low cost. This paper will focus on the deep learning methods used in the field of self-driving cars.
Remark 1 (About the autonomous level of self-driving cars).
The NHTSA has adopted the level classification provided by the Society of Automotive Engineers, which ranges from Level 0 to Level 5 [[18]]. In this standard, Level 0 represents a vehicle without any autonomy. Level 1 has basic driving assistance such as adaptive cruise and emergency braking. Level 2 has partial autonomy while the driver needs to supervise the system and perform some tasks. At Level 3, the system has full autonomy under certain conditions, but the human operator is still required to take control if necessary. The vehicle at Level 4 is still a semi-autonomous system, which has higher automation than Level 3. Vehicles in Level 5 are fully autonomous in all conditions.
Remark 2 (About Tensorflow).
Tensorflow is a common deep learning platform that was created by Google. The design of Tensorflow is intended to simplify the construction of deep neural networks and speed up the learning process with a heterogeneous distributed computational environment [[19]]. Tensorflow provides lots of Application Programming Interface (API) libraries to build and train deep learning models, which can support Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and other deep neural network models.
This entry is adapted from the peer-reviewed paper 10.3390/app10082749
References
- Lewis M. Clements; Kara M. Kockelman; Economic Effects of Automated Vehicles. Transportation Research Record: Journal of the Transportation Research Board 2017, 2606, 106-114, 10.3141/2606-14.
- Nathan A. Greenblatt; Self-driving cars and the law. Quantum computers scale up: Constructing a universal quantum computer with a large number of qubits will be hard but not impossible 2016, 53, 46-51, 10.1109/mspec.2016.7419800.
- Birdsall, M.; Google and ITE: The road ahead for self-driving cars. ITE J. (Inst. Transp. Eng.) 2014, 84, 36–39, .
- Dikmen, M.; Burns, C. Trust in autonomous vehicles: The case of Tesla autopilot and summon. In Proceedings of the 2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017, Banff, AB, Canada, 5–8 October 2017; pp. 1093–1098.
- Erik Coelingh; Jonas Nilsson; Jude Buffum; Driving tests for self-driving cars. Quantum computers scale up: Constructing a universal quantum computer with a large number of qubits will be hard but not impossible 2018, 55, 41-45, 10.1109/mspec.2018.8302386.
- Jiyoun Park; Changi Nam; Hye-Jin Kim; Exploring the key services and players in the smart car market. Telecommunications Policy 2019, 43, 101819, 10.1016/j.telpol.2019.04.003.
- Toschi, A.; Sanic, M.; Leng, J.; Chen, Q.; Wang, C.; Guo, M. Characterizing perception module performance and robustness in production-scale autonomous driving system. In Proceedings of the 16th IFIP WG 10.3 International Conference on Network and Parallel Computing, Hohhot, China, 23–24 August 2019; pp. 235–247.
- Li, Y.; Wang, J.; Xing, T.; Liu, T.; Li, C.; Su, K. TAD16K: An enhanced benchmark for autonomous driving. In Proceedings of the 24th IEEE International Conference on Image Processing, Beijing, China, 17–20 September 2017; pp. 2344–2348.
- Lishengsa Yue; Mohamed Abdel-Aty; Yina Wu; Ling Wang; Assessment of the safety benefits of vehicles' advanced driver assistance, connectivity and low level automation systems.. Accident Analysis & Prevention 2018, 117, 55-64, 10.1016/j.aap.2018.04.002.
- Bylykbashi, K.; Qafzezi, E.; Ikeda, M.; Matsuo, K.; Barolli, L. Fuzzy-based driver monitoring system (FDMS): Implementation of two intelligent FDMSs and a testbed for safe driving in VANETs. Future Gener. Comput. Syst. 2020, 105, 665–674. [Google Scholar] [CrossRef]
- Scott Schnelle; Junmin Wang; Richard Jagacinski; Haijun Su; A feedforward and feedback integrated lateral and longitudinal driver model for personalized advanced driver assistance systems. Mechatronics 2018, 50, 177-188, 10.1016/j.mechatronics.2018.02.007.
- Brian Paden; Michal Cap; Sze Zheng Yong; Dmitry Yershov; Emilio Frazzoli; A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles. IEEE Transactions on Intelligent Vehicles 2016, 1, 33-55, 10.1109/tiv.2016.2578706.
- Sparsh Mittal; A Survey on optimized implementation of deep learning models on the NVIDIA Jetson platform. Journal of Systems Architecture 2019, 97, 428-442, 10.1016/j.sysarc.2019.01.011.
- Jones, L.; Driverless when and cars: Where?. Eng. Technol. 2017, 12, 36–40, .
- Jonathan Dekhtiar; Alexandre Durupt; Matthieu Bricogne; Benoît Eynard; Harvey Rowson; Dimitris Kiritsis; Deep learning for big data applications in CAD and PLM – Research review, opportunities and case study. Computers in Industry 2018, 100, 227-243, 10.1016/j.compind.2018.04.005.
- Jianjun Ni; Kang Wang; Qingyun Cao; Zubair Khan; Xinnan Fan; A MEMETIC ALGORITHM WITH VARIABLE LENGTH CHROMOSOME FOR ROBOT PATH PLANNING UNDER DYNAMIC ENVIRONMENTS. International Journal of Robotics and Automation 2017, 32, , 10.2316/journal.206.2017.4.206-4998.
- Jianjun Ni; Liuying Wu; Xinnan Fan; Simon X. Yang; Bioinspired Intelligent Algorithm and Its Applications for Mobile Robot Control: A Survey. Computational Intelligence and Neuroscience 2015, 2016, 1-16, 10.1155/2016/3810903.
- Rod McCall; Fintan McGee; Alexander Mirnig; Alexander Meschtscherjakov; Nicolas Louveton; Thomas Engel; Manfred Tscheligi; A taxonomy of autonomous vehicle handover situations. Transportation Research Part A: Policy and Practice 2019, 124, 507-522, 10.1016/j.tra.2018.05.005.
- Mingliang Liu; Dario Grana; Accelerating geostatistical seismic inversion using TensorFlow: A heterogeneous distributed deep learning framework. Computers & Geosciences 2019, 124, 37-45, 10.1016/j.cageo.2018.12.007.
