Autonomous passenger vehicles are a direct implementation of transportation-related autonomous robotics research. They are also known as self-driving vehicles or driverless vehicles. Shakey the robot (1966–1972) is the first autonomous mobile robot that has been documented [1]. It was developed by Stanford Research Institute’s Artificial Intelligence Centre and was capable of detecting the environment, thinking, planning, and navigation. In basic settings, vision-based lane tracking and obstacle avoidance sparked interest in autonomous vehicles [2]. In the early 1990s, The Royal Armament Research and Development Establishment in the United Kingdom created two vehicles for obstacle-free navigation on and off the road [3]. In the United States, the first operations of autonomous driving in realistic settings dates back to Carnegie Mellon University’s NavLab in the early 1990s [4]. The vehicle developed by NavLab was operated at very low speeds due to the limited computational power available at the time. Early US research projects also included the California PATH project, which developed the automated highway [5]. Vehicle steering was automated with manual longitudinal control in the “No Hands Across America” project [6]. In early 2000, CyberCars, one of several European projects began developing technologies based on automated transport [7]. The announcement of the defence advanced research projects agency (DARPA) grand challenge in 2003 generated research interest in autonomous cars. Following that, in 2006, the DARPA urban challenge was performed in a controlled situation with a variety of autonomous and human-operated vehicles. Since then, many manufactures, including Audi, BMW, Bosch, Ford, GM, Lexus, Mercedes, Nissan, Tesla, Volkswagen, Volvo and Google, have launched self-driving vehicle projects in collaboration with universities [8]. Google’s self-driving car has experimented and travelled 500 thousand kilometres and has begun building prototypes of its own cars [9]. A completely autonomous vehicle would be expected to drive to a chosen location without any expectation of shared control with the driver, including safety-critical tasks.

The performance of lane detection and tracking depends on the well-developed roads and their lane markings, so smart cities are also a prominent factor in autonomous vehicle research. The idea of a smart city is often linked with an eco-city or a sustainable city, both of which seek to enhance the quality of municipal services while lowering their costs. Smart cities’ primary goal is to balance technological innovation with the economic, social, and environmental problems that tomorrow’s cities face. The greater closeness between government and people is required in smart cities that embrace the circular economy’s concepts [10]. The way materials and goods flow around people and their demands will alter, as will the structure of cities. Several car manufacturers such as Tesla and Audi have already launched autonomous vehicle marketing for private use. Soon, society will be influenced by autonomous vehicles’ spread to urban transport systems [11]. The development of smart cities with the introduction of connected and autonomous vehicles could potentially transform cities and guide long-term urban planning [10].

Autonomous vehicles and Advanced Driver Assistance Systems (ADAS) are predicted to provide a higher degree of safety and reduce fuel and energy consumption and road traffic emissions. ADAS is implemented for safe and efficient driving, which has many driver assistance features such as warning drivers about forwarding collision warning or safe lane change [12]. Research shows that most accidents occur because of driver errors, and the ADAS can reduce the accidents and workload of the driver. If there is a likelihood of an accident, ADAS can take the necessary action to avoid it [13]. Lane departure warning (LDW), which utilizes lane detection and tracking algorithms, is an essential feature of the ADAS. The LDW warns the driver when a vehicle crosses white lane lines unintentionally and controls the vehicle by bringing it back into the desired safe path. Three types of approaches for lane detection are usually discussed in the existing literature: learning-based approach, features-based approach, and model-based approach [13,14,15,16,17,18] (detailed analysis ^{[13][14][15][16][17][18]}are presented in Section 3.2). Many challenges and issues have been highlighted in the literature regarding the LDW systems, such as visibility conditions change, variation in images, and lane appearance diversity [17]. Since different countries have used various lane markers, there is a challenge for lane detection and tracking to solve the problems.

2. Lane Detection and Tracking Algorithms

The feature-based approach uses edges and local visual characteristics of interest, such as gradient, colour, brightness, texture, orientation, and variations, which are relatively insensitive to road shapes but sensitive to illumination effects. The model-based approaches apply global road models to fit low levels of features that are more robust against illumination effects, but they are sensitive to road shapes ^[13][14][13,14]. The geometrics parameters are used in the model-based approach for lane detection ^[16][17][18][16,17,18]. The learning-based approach consists of two stages: training and classification. The training process uses previously known errors and system properties to construct a model, e.g., program variables. In addition, the classification phase applies the training model to the user set of properties and outputs that are more likely to be correlated with the error ordered by their probability of fault discloser [19]. It is then followed up by summary tables (Table 12, Table 23, Table 34 and Table 45) that present the key features of these algorithms and strengths, weaknesses, and future prospects.

Table 12. A summary of methods used for lane detection and tracking with general remarks.

and Table 5 are summarized below:

• Frequent calibration is required for accurate decision making in a complex environment.
• Reinforcement learning with the model predictive control could be a better choice to avoid false lane detection.
• Model-based approaches (robust lane detection and tracking) provide better results in different environmental conditions. Camera quality plays an important role in determining lane marking.
• The algorithm’s performance depends on the type of filter used, and the Kalman filter is mostly used for lane tracking.
• In a vision-based system, image smoothing is the initial lane detection and tracking stage, which plays a vital role in increasing systems performance.
• External disturbances like weather conditions, vision quality, shadow and blazing, and internal disturbances such as too narrow, too wide, and unclear lane marking, drop algorithm performance.
• The majority of researchers (>90%) have used custom datasets for research.
• Monocular, stereo and infrared cameras have been used to capture images and videos. The algorithm’s accuracy depends on the type of camera used, and a stereo camera gives better performance than a monocular camera.
• The lane markers can be occluded by a nearby vehicle while doing overtake.
• There is an abrupt change in illumination as the vehicle gets out of a tunnel. Sudden changes in illumination affect the image quality and drop the system performance.
• The results show that the lane detection and tracking efficiency rate under dry and light rain conditions is near 99% in most scenarios. However, the efficiency of lane marking detection is significantly affected by heavy rain conditions.

Methods	Steps	Tool Used	Data Used	Methods Classification	Remarks
Image and sensor-based lane detection and tracking		Image frames are pre-processed Lane detection algorithm is applied

Table 23. A comprehensive summary of lane detection and tracking algorithm.

• It has been seen that the performance of the system drops due to unclear and degraded lane markings.
• IMU (Inertia measurement unit) and GPS are examples that help to improve RADAR and LIDAR’s performance of distance measurement.
• One of the biggest problems with today’s ADAS is that changes in environmental and weather conditions have a major effect on the system’s performance.

Sources	Data		Method Used	Advantages	Drawbacks	Results	Tool Used	Future Prospects	Data	Reason for Drawbacks
	Simulation	Real
[
KITT
----

Table 34. A comprehensive summary of learning-based model predictive controller lane detection and tracking.

Sources	Data		Method	Advantages	Method UsedDrawbacks	Result	Tool Used	Future Prospects	Data	AdvantagesReason for Drawback
	Drawbacks	Result									Tool Used	Future Prospects	Data	Simulation	Real
20
Reason for Drawbacks
	Simulation	Real
]	[	The sensors values are used to track the lanes		24]
[38]	[42]		Camera		Sensors		sensors values	YFeature-based approach	Frequent calibration is required for accurate decision making in a complex environment
Inverse perspective mapping method is applied to convert the image to bird’s eye view.	Minimal error and quick detection of lane.	The algorithm performance drops when driving in tunnel due to the fluctuation in the lighting conditions.	The lane detection error is 5%. The cross-track error is 25% and lane detection time is 11 ms.	Fisheye dashcam, inertial measurement unit and ARM processor-based computer.	Enhancing the algorithm suitable for complex road scenario and with less light conditions.
[	Y	45]	[Gradient cue, color cue and line clustering are used to verify the lane markings.	49The proposed method works better under different weather conditions such as rainy and snowy environments.	The suitability of the algorithm for multi-lane detection of lane curvature is to be studied.	]Except rainy condition during the day, the proposed system provides better results.	C++ and OpenCV on ubuntu operating system. Hardware: duel ARM cortex-A9 processors.	----		Y	Inverse perspective mapping method is applied to convert the image to bird’s eye view.Data obtained by using a model car running at a speed of 100 m/s.	Quick detection of lane.Performance drop in determining the lane, if the vehicle is driving in a tunnel and the road conditions where there is no proper lighting. The complex environment creates unnecessary tilt causing some inaccuracy in lane detection.	The algorithm performance drops due to the fluctuation in the lighting conditions.	The lane detection error is 5%. The cross-track error is 25% lane detection time is 11 ms.	Fisheye dashcam: inertial measurement unit; Arm processor-based computer.48 video clips from USA and Korea	Enhancing the algorithm suitable for complex road scenario and with less light conditions.Since the road environment may not be predictable, leads to false detection.
Predictive controller for lane detection and controller	Machine learning technique (e.g., neural networks,)
[21]	[25	Model predictive controller Reinforcement learning algorithms
-----
Data obtained by using a model car running at a speed of 1 m/s	data obtained from the controller	Learning-based approach	]		Reinforcement learning with model predictive controller could be a better choice to avoid false lane detection.
The complex environment creates unnecessary tilt causing some inaccuracy in lane detection.	Y	[39]	Kinematic motion model to determine the lane with minimal parameters of the vehicle.	[No need for parameterization of the vehicle with variables like cornering stiffness and inertia. Prediction of lane even in absence of camera input for around 3 s.	The algorithm suitable for different environment situation not been considered	43Lateral error of 0.15 m in the absence of camera image.	Mobileye camera, carsim and MATLAB/Simulink, Auto box from dSPACE.	]Trying the fault tolerant model in real vehicle.	Test vehicle	Y		Extraction of lanes from the captured image Random, sample consensus algorithm is used to eradicate error in lane detection.	Multilane detection even during poor lane markings. No prior knowledge about the lane is required.
[46	----	Urban driving scenario quality has to be improved in cardova 2dataset since it perceives the curb of the sidewalk as a lane.	]	[50	The Caltech lane datasets consisting of four types of urban driving scenarios:	Cordova 1; Cordova 2; Washington2; with a total of 1224 frames containing 4172 lane markings.	MATLAB	Real time implementation of the proposed algorithm	]Data from south Korea road and Caltech dataset.	IMU sensors could be incorporated to avoid the false detection of lanes.
Y		Deep learning-based reinforcement learning is used for decision making in the changeover. The reward for decision making is based on the parameters like traffic efficiency	Cooperative decision-making processes involving the reward function comparing delay of a vehicle and traffic.	Validation expected to check the accuracy of the lane changing algorithm for heterogeneous environment	The performance is fine-tuned based on the cooperation for both accident and non-accidental scenario	Custom made simulator	Dynamic selection of cooperation coefficient under different traffic scenario	Newell car following model.	----	Robust lane detection and tracking
[22]	[26]	[40		Capture an image through camera Use Edge detector to data for extract the features of the image Determination of vanishing point		Based on robust lane detection model algorithms	Real-time	Model-based approach	Provides better result in different environmental conditions. Camera quality plays important role in determining lanes marking

Y

Usage of inverse mapping for the creation of bird’s eye view of the environment.

Improved accuracy of lane detection in the range of 86%to 96% for different road types.

Performance under different vehicle speed and inclement weather conditions not considered.

The algorithm requires 0.8 s to process frame. Higher accuracy when more than 59% of lane markers are visible.

]

[44]

Y

YFirewire color camera, MATLAB

Real-time implementation of the work

Highway and streets and around Atlanta

[

----

47]

[51]

Y

Rectangular detection region is formed on the image. Edge points of lane is extracted using threshold algorithm. A modified Brenham line voting space is used to detect lane segment.

Robust lane detection method by using a monocular camera in which the roads are provided with proper lane markings.

Performance drops when road is not flat

In Cardova 2 dataset, the false detection value is higher around 38%. The algorithm shows better performance under different roads geometries such as straight, curve, polyline and complex

Software based performance analysis on Caltech dataset for different urban driving scenario. Hardware implementation on the Tuyou autonomous vehicle.

----

Caltech and custom-made dataset

Due to the difficulty In image capturing false detection happened. More training or inclusion of sensors for live dataset collection will help to mitigate it.

Reinforcement learning-based approach for decision making by using Q-function approximator.

Decision-making process involving reward function comprising yaw rate, yaw acceleration and lane changing time.

Need for more testing to check the efficiency of the approximator function for its suitability under different real-time conditions.

The reward functions are used to learn the lane in a better way.

Custom made simulator

To test the efficiency of the proposed approach under different road geometrics and traffic conditions. Testing the feasibility of the reinforcement learning with fuzzy logic for image input and controller action based on the current situation.

custom

More parameters could be considered for the reward function.

[23]

[27]

Y

Y

Hough transform to extract the line segments, usage of a convolutional neural network-based classifier to determine the confidence of line segment.

Tolerant to noise

In the custom dataset, the performance drops compared to Caltech dataset.

[41]

[45]

Y

Based on voting map, detected vanishing points, usage of distinct property of lane colour to obtain illumination invariant lane marker and finally found main lane by using clustering methods.

Overall method test algorithm within 33 ms per frame.

[

Need to reduce computational complexity by using vanishing point and adaptive ROI for every frame.

48]

[52

Under various

Illumination condition lane detection rate of the algorithm is an average 93%

]

Y

For urban scenario, the proposed algorithm provides accuracy greater than 95%. The accuracy obtained in lane detection in the custom setup is 72% to 86%.

OV10650 camera and I MU is Epson G320.

Performance improvement is future consideration.

Caltech dataset and custom dataset.

The device specification and calibration, it plays important role in capturing the lane.

Software based analysis done.

There are chances, to test algorithm at day time with inclement weather conditions.

Custom data based on Real-time

-----

Probabilistic and prediction for the complex driving scenario.

Usage of deterministic and probabilistic prediction of traffic of other vehicles to improve the robustness

Analysis of the efficiency of the system under real-time noise is challenging.

Robust decision making compared to the deterministic method. Lesser probability of collision.

MATLAB/Simulink and carsim. Used real-time setup as following:

Hyundai-Kia motors K7, mobile eye camera system, micro auto box II, Delphi radars, IBEO laser scanner.

Testing undue different scenario

Custom dataset (collection of data using test vehicle).

The algorithm to be modified for real suitability for real-time monitoring.

[24]

[28]

Y

Feature-line-pairs (FLP) along with Kalman filter for road detection.

[

Faster detection of lanes, suitable for real-time environment.

42

Testing the algorithm suitability under different environmental conditions could be done.

Around 4 ms to detect the edge pixels, 80 ms to detect all the FLPs, 1 ms to determine the extract road model with Kalman filter tracking.

]

C++; camera and a matrox meteor RGB/ PPB digitizer.

Robust tracking and improve the performance in urban dense traffic.

[Test robot.

------

46]

Y

[49]

[53]

Y

Proposed a sharp curve lane from the input image based on hyperbola fitting. The input image is converted to grayscale image and the feature namely left edge, right edge and the extreme points of the lanes are calculated

Better accuracy for sharp curve lanes.

The suitability of the algorithm for different road geometrics yet to study.

The results show that the accuracy of lane detection is around 97% and the average time taken to detect the lane is 20 ms.

Custom made simulator C/C++ and visual studio

-----

Improved performance by using support vector machines and artificial neural networks on the image.Custom data

Usage of pixel hierarchy to the occurrence of lane markings. Detection of the lane markings using a boosting algorithm. Tracking of lanes using a particle filter.

Detection of the lane without prior knowledge on-road model and vehicle speed.-----

Machine with 4-GHz processor capable of working on image approximately 240 × 320 image at 15 frames per second.

[25]

[

Usage of vehicles inertial sensors GPS information and geometry model further improve performance under different environmental conditions

To test the efficiency of the algorithm by using the Kalman filter.

custom data

Calibration of the sensors needs to be maintained.

29]

[

Y

43

Dual thresholding algorithm for pre-processing and the edge is detected by single direction gradient operator. Usage of the noise filter to remove the noise.

]

[47]

Y

The lane detection algorithm insensitive headlight, rear light, cars, road contour signs.

The algorithm detects the straight lanes during the night.

Detection Of straight lanes.

vanishing point detection method for unstructured roadsCamera with RGB channel.

Accurate and robust performance for unstructured roads.

Difficult to obtain robust vanishing point for detection of lane for unstructured scene.

The accuracy of vanishing point range between 80.9% to 93.6% for different scenarios.

Unmanned ground vehicle and mobile robot.-------

Custom dataset

Suitability of the algorithm for different types of roads during night to be studied.

Future scope for structured roads with different scenarios.

Custom data

[26]

[30]

Y

Determination of region of interest and conversion of binary image via adaptive threshold.

Better accuracy

The algorithm needs changes for checking its suitability for the day time lane detection

90% accuracy during night at isolated highways

Firewire S400 camera and MATLAB

Geometrics transformation of image for increasing the accuracy and intensity normalization.

Custom dataset

The constraints and assumption considered do not suit for the day time.

Complex background interference and unclear road marking.

[31]

[44]

[48]

Y

Proposed a lane detection approach using Gaussian distribution random sample consensus (G-RANSAC), usage of rider detector to extract the features of lane points and adaptable neural network for remove noise.

Provides better results during the presence of vehicle shadow and minimal illumination of the environment.

----

The proposed algorithm is tested under different illumination condition ranging from normal, intense, normal and poor and provides lane detection accuracy as 95%, 92%, 91% and 90%.

Software based analysis

Need to test proposed method under various times like day, night.

Test vehicle

----

[27]

[31]

Y

Canny edge detector algorithm is used to detect the edges of the lanes.

Hough transform improves the output of the lane tracker.

------

Performance of the proposed system is better.

Raspberry pi based robust with camera and sensors.

Simulation of the proposed method by using raspberry Pi based robot with a monocular camera and radar-based sensors to determine the distance between neighboring vehicles.

Custom data

------

[28]

[32]

Y

Video processing technique to determine the lanes illumination change on the region of interest.

----

----

Robust performance

vision-based vehicle

Determine the lanes illumination changes on the region of interest for curve line roads

Simulator

----

[29]

[33]

Y

Y

A colour-based lane detection and representative line extraction algorithm is used.

Better accuracy in the day time.

Algorithm needs changes to test in different scenario.

The results show that the lane detection rate is more than 93%.

MATLAB

There is scope to test the algorithm in the night time.

Custom data

Unwanted noise reduces the performance of the algorithm.

[30]

[34]

Y

Proposed hardware architecture for detecting straight lane lines using Hough transform.

Proposed algorithm provides better accuracy for occlusion, poor line paintings.

Computer complexity and high cost of HT (Hough transform)

Algorithm tested under various conditions of roads such as urban street, highway and algorithm provides a detection rate of 92%.

Virtex-5 ML 505 platform

Algorithm need to test with different weather condition.

Custom

[35]

Y

Proposed a lane detection methodology in a circular arc or parabolic based geometric method.

Video sensor improves the performance of the lane marking.

Performance dropped in lane detection when entering the tunnel region

Experiment performed with different road scene and provided better results.

maps, video sensors, GPS.

Proposed method can test with previously available data.

Custom

Due to low illumination

[32]

[36]

Y

Proposed a hierarchical lane detection system to detect the lanes on the structured and unstructured roads.

Quick detection of lanes.

----

The system achieves an accuracy of 97% in lane detection.

MATLAB

Algorithm can test on an isolated highway, urban roads.

----

[33]

[37]

Y

LIDAR sensor-based boundary detection and tracking method for structured and unstructured roads.

Regardless of road types, algorithm detect accurate lane boundaries.

Difficult to track lane boundaries for unstructured roads because of low contract, arbitrary road shape

The road boundary detection accuracy is 95% for structured roads and 92% for unstructured roads.

Test vehicle with LIDAR, GPS and IMU.

Algorithm needs to test with RADAR based and vision-based sensors.

Custom data

Low contract arbitrary road shape

[34]

[38]

Y

Proposed a method to detect the pedestrian lanes under different illumination conditions with no lane markings.

Robust performance for pedestrian lane detection under unstructured environment.

More challenging for indoor and outdoor environment.

The result shows that the lane detection accuracy is 95%.

MATLAB

There is scope for structured roads with different speeds limit

New dataset of 2000 images (custom)

Complex environment

[35]

[39]

Y

Y

The proposed system is implemented using an improved Hough transform, which pre-process different light intensity road images and convert it to the polar angle constraint area.

Robust performance for a campus road, in which the road does not have lane markings.

Performance drops due to low intensity of light

------

Test vehicle and MATLAB

-------

Custom data

Low illumination

[36]

[40]

Y

A lane detection algorithm based on camera and 2D LIDAR input data.

Computational and experimental results show the method significantly increases accuracy.

------

The proposed approach shows better accuracy compared with the traditional methods for distance less than 9 m.

Proposed method need to test with RADAR and vision-based sensors data

software based analysis and MATLAB

Fusion of camera and 2D LIDAR data

-----

[37]

[41]

Y

A deep learning-based approach for detecting lanes, object and free space.

The Nvidia tool comes with SDK (software development kit) with inbuild options for object detection, lane detection and free space.

Monocular camera with advance driver assistance system is costly.

The time taken to determine the lane falls under 6 to 9 ms.

C++ and NVidia’s drive PX2 platform

Complex road scenario with different high intensity of light.

Table 45. A comprehensive summary of robust lane detection and tracking.

Sources	Data

Some of the key observations from Table 3, Table 4