The challenge
The growing complexity of automated driving systems such as Lane Keep Assist, Lane Centering and Adaptive Cruise Control challenge existing verification and validation (V&V) methodologies used in the automotive industry. As these systems become more prevalent, bugs surface and failures occur. A collision example took place earlier this month, on June 2nd. A Tesla Model 3 using an automated driving function collided with a stationary truck on a Taiwanese highway.
Video 1 shows a simulated reproduction of the Tesla accident, based on released footage and using Foretify™ and Carla Simulator
While reproducing failures after they have occurred is useful for verifying a fix, the real need is to eliminate as many failures as possible in advance. The number of possible circumstances and risk dimensions is infinite and many of these are unknowns. The upcoming SOTIF standard (ISO 21448) recognizes the challenge and gravity of the unknowns, as illustrated in Figure 1 below.
While you may be able to enumerate vehicle maneuver and risk dimension categories such as sensor and camera faults or stationary objects such as cones, puddles or even faded road markings, the possible combinations of these are infinite and cannot be predicted up front. As shown above, existing technologies such as residual risk calculation provide a data-driven grade for the knowns but no formula can calculate the risk of unknown and unpredictable scenarios. The verification plan enumerates all the thought-out scenarios, but what about the unexpected and unpredicted?
As demonstrated in the Tesla incident, the result of the unpredictable nature of a scenario’s space is expensive recalls and compromised safety.
This leads to two frequently asked questions that the automotive industry is busy with in terms of V&V:
- How do we confront infinity with finite resources and tight timelines?
- How do we find the unknown unknowns?
The Foretellix solution – Foretify™
Foretellix’s Foretify platform combines use of controlled-random test generation to scale up and search for the unknowns, easy mixing of scenarios and risk dimensions, and powerful data analytics to address this challenge:
- Use of controlled-random test generation to scale up and search for the unknowns – Foretify allows leveraging a generic constraint solver to achieve a massive number of high quality scenarios. This scale is not reached with a systematic walk over of attribute values. In order to achieve unexpected and edge case scenarios, all attributes are randomized by default. Foretify randomly selects locations on the map, including attribute values such as distance and speed, the timing and order of events, the angle, roll and location of stationary objects and so on. Users can apply constraints (simple rules) to ensure the value legality and consistency of the generated test. For example, Foretify can select either a random location or a location constrained to have enough lanes to accommodate a cut-in and cut-out maneuvers.
Video 2 shows a few automatically generated variations of the same Tesla Model 3 incident. Note that the map location has changed along with other attributes.
Per user request, Foretify can generate hundreds of thousands of scenarios in which a truck or a random stationary object resides in random locations, orientations, lanes, color, and so on.
- Mixing scenarios and risk dimensions – Foretify allows mixing numerous vehicle maneuvers and risk dimensions to achieve the next level of thoroughness. The constraint solver can take sub-scenarios and find multiple proper locations, speeds and circumstances in which they can co-exist. Also, since bugs typically come in clusters, Foretify users can create multiple scenarios and variations of an already discovered bugs.
- Test table for requesting cross combinations of values – For adding project specific tests, Foretify uses a simple interface Test tables constitute a productivity tool that allows requesting thousands of executions with all cross-value combinations.
- Powerful Coverage Driven Analytics – Generating a large number of tests requires powerful data analytics and management tools. Foretify provides a dashboard that displays the executed conditions and KPIs in a simple, multi-hierarchical view. The dashboard reflects what was actually executed (given the unpredictable AV responses) and allows analysis of what was tested and verified. The tool also allows removing test duplications by creating a minimal set of tests that achieves maximum V&V coverage. Utilizing functional coverage and metrics to guide the verification efforts is a proven and productive approach to pragmatically explore most of your ODD in a minimal amount of time.
Figure 2 shows an automatically generated metrics report, including both coverage metrics and KPIs.
Foretify introduces an innovative approach with scalable random scenario creations, scenario combinations and mixing, cross combination values, and data-analytics to meet the ADAS and AV industry challenge of identifying the unknown unknowns.
