A Review of Description Languages and Systems: Comparison
Please note this is a comparison between Version 1 by Jing Ma and Version 2 by Camila Xu.

Testing and validation of the functionalities and safety of automated vehicles shifted from a distance-based to a scenario-based method in the past decade. A number of domain-specific languages and systems were developed to support scenario-based testing.

  • scenario description
  • SDL
  • domain specific language
  • DSL

1. Introduction

The development of automated vehicles (AV) progressed rapidly in the past few years. Many companies were testing their AVs on roads and highways in the United States and other nations around the world [1][2][1,2]. However, the roll-out of AVs on public roads took longer than expected [3], and one of the main reasons is related to trust. Can the public trust that these AVs are safe? The answer to this question is complicated by the fact that an AV integrates both the mechanics of a vehicle and the ability of the driver into a single entity. Any safety testing regime has to include both the mechanics of the car and its ability to drive, which currently is handled by driver licensing.
One way to ensure the safety of AVs is to have it driven enough. How much distance an AV has to be driven to adequately evaluate its safety was hotly debated. It was argued that to make statistical safety comparisons with the performance of human drivers, hundreds of millions of miles of driving is required [4]. However, doing so would take decades and is obviously impractical. Therefore, the focus of AV testing shifted from a driving distance-based approach to a scenario-based approach [5][6][7][5,6,7]. This later approach also aligns well with the development process described in the 2016 version of the ISO-26262 standard [8] which is concerned with the development of safety-critical electrical and electronic systems.
Scenario-based testing requires well-designed traffic scenarios that can sufficiently test the capabilities of the automated driving systems [7]. A driving scenario is typically made up of static, dynamic, and temporary elements. These include the environment such as road widths, number of lanes, road curvature, road signs, pedestrian sidewalks, a set of vehicles (automated or human-driven) and their initial states (position, orientation, speed, etc.), pedestrians, traffic signals, and the ego automated vehicle (the AV under test) with its initial state and ultimate goal. It can also be viewed as a time sequence of scenes [5][9][5,9]. Such scenarios need to be described and specified, either textually or graphically or both. There are now a variety of scenario description languages and systems (SDLS) which are either simulation platform dependent or independent. Choosing the right SDLS is important to researchers and developers of AVs, as well as regulatory authorities who are concerned with safety testing.

2. Discussion

A number of competing and complimentary scenario description languages and systems were discussed in the previous section. Their characteristics can be summarized in Table 1. For each SDL, this table lists their primary data format for scenario description, and whether they are open-source. Most of them make use of an external standard or system for describing the road network and its conditions. Furthermore, we listed the simulation platforms which support each SDLS. This is important, as scenario descriptions are mostly used for simulation studies. If a researcher, developer, or organization is committed to a particular simulation software, then their choice of SDL is restricted. The simulator CARLA, which is an open-sourced software itself, supports the most SDLS.
Table 1.
Comparison of listed scenario description language.
SDL Data Open Road Simulation
  Format Source Description Platform
stiEF XML Yes OpenDRIVE ViresVTD2
Hesperia Programming Unknown 3D CxxTest
  language     Prescan
OAS HTML Yes 2D OAS
SceML XML Yes OpenDRIVE CARLA
OpenSCENARIO XML Yes OpenDRIVE Prescan
        CARLA
        ViresVTD2
GeoScenario XML Yes Lanlets Unreal Engine
CommonRoad XML Yes Lanlets SUMO
SCENIC Probabilistic Yes OpenDRIVE VERIFAI
  programming   OSM CARLA
  language     Webots
        LGSVL
M-DSL Python-like Yes OpenDRIVE CARLA
  Syntax     SUMO
Each SDLS is designed with different goals in mind. For example, openSCENARIO aims to be an open standard scenario description language that will be accepted by many users and simulation tools. On the other hand, M-SDL makes it easy for many test cases to be generated from a basic scenario. This feature will be useful and convenient for testing and validation purposes. But regardless of the design goals of the SDLS, tools must be available to allow their scenario descriptions to be imported by simulation tools and platforms to allow the AV to be tested in those situations. Although traditionally, simulations are carried out in 2D, several photorealistic 3D simulators are now available, for instance [10][11][12][13][14][15][22,26,27,46,47,49]. Thus, the trend is to enable the description of scenarios in 3D by incorporating photorealistic 3D models. Not only do 3D models look more realistic, the scenarios are more realistic in the sense that not all information are available to the ego AV at any given time. For instance, in our example scenario, the oncoming car V5 on the main road is obscured by the parked cars V7 and V8. It would were much easier for the ego AV to decide on its action if the position and speed of V5 is made available. However, if the sensor on the ego AV cannot pick out V5 until V5 appears in its line-of-sight, then the situation will become much more realistic. In fact, this can become one of the edge or corner cases for testing purposes. Using text or graphics alone may be sufficient describe simple scenarios. However, as pointed out earlier, simple scenarios are unlikely to be sufficient for testing and verification of safety purposes. Therefore, a combination of textual and graphical representations will be needed, especially when real-world maps and weather data are to be incorporated. Currently, scenarios are almost entirely designed by human experts. This approach greatly limits the number of test scenarios that can be generated. Combing data-driven and knowledge-driven approaches is a logical next-step in the process. By making use of artificial intelligence (AI) techniques, edge and corner cases could be identified automatically. These cases are expected to help identify failures or strange behaviors of AVs, and consequently increase the assurance of safety of these vehicles. SDLS should therefore facilitate this. In this sense, M-SDL is currently the leading language in this direction. Different scenarios are typically treated as being independent of each other and are not characterized by their relative level of complexity. Such characterization will become more important for test engineers and regulators to identify edge and corner cases properly. At the same time, an agreed-upon measure of complexity will aid scenario designers to classify the difficulty of their scenarios in the same way the automation level of a vehicle is classified into one of six levels [16][50]. The level of difficulty could be included in each scenario in the SDLS. Many different countries will be developing their own AV regulations and policies, so a common understanding of the terminologies used in different languages is critical to the car manufacturers. Currently, stiEF is the only multilingual SDL that is designed to overcome this problem. More SDLS that takes the issues involved in multilingual documents will be beneficial.