Unlock the full potential of intelligent large-scale virtual testing
The companies developing autonomous vehicles are under serious financial pressure. Their long-standing reliance on real-world physical testing has resulted in the expenditure of tens of billions of dollars on maintaining expensive test fleets. Even those that have augmented their testing with virtual simulations are far from achieving widespread commercial deployment. These setbacks are now pushing some companies to the brink of failure.
Achieving sufficient safety levels for transitioning from limited pilots to full-scale production is not easy, but it’s doable. Level 3 and Level 4 Automated Driving Systems (ADS) must handle millions of scenario variants, including emergent cases stemming from complex interactions with road users and environmental conditions. This nearly infinite universe of test cases will never be fully addressed by relying only on test drives. Some AV companies have already recognized the need for large-scale simulation, but they use limited simulation techniques and rely primarily on costly physical drives.
However, by combining the strength of state-of-the-art large-scale virtual testing technologies with intelligent utilization of physical test-driving, ADS companies can tame infinity to reach proper test coverage, address the safety challenges, and achieve large-scale commercial deployment at a sustainable cost.
The leading autonomous vehicles company Nuro highlighted the fundamental tension in the development of self-driving solutions between capital efficiency and the speed of building an initial service. The Nuro founders explained the hard choices Nuro had to take and how moving from multiple pilot deployments to data-driven R&D focus will accelerate their development and reduce expenditure. Ury Zhilinsky, Head of Learned Behavior and Simulation at Nuro, said in our recent discussion: ’Data-driven large-scale virtual simulation is a key enabling technology for accelerating the safe deployment of our autonomous delivery solutions while controlling our cost.’
The AV industry must revolutionize the verification and validation process to overcome this existential crisis, mitigate financial risks, and confidently transition from limited pilots to full-scale deployment and commercialization.
Running out of time and money
Autonomous vehicle companies are facing dire financial conditions. The industry’s long-standing reliance on road test drives, which naturally evolved from fast prototyping and early capabilities demonstrations, has grown over the past decade into an estimated average annual testing cost of $800M to $2.7B per program (largely depending on the mix of physical world testing and simulations). The staggering costs of this inefficient and never-ending testing method pose an existential challenge for the entire industry.
This crisis threatens the viability of not just many companies’ ADS development programs but, for some companies, their ultimate survival. From the shutdown of Argo AI and Embark Trucks winding down its operations to Tesla not delivering on their recurring ’by the end of the year‘ promise of Full Self Driving cars, and Waymo and Cruise burning through billions a year while grappling with limited pilots in a handful of cities. Despite close to $200B invested over a decade, AV companies are struggling to achieve the necessary level of safety required for full-scale deployment. This is a grim reality check for our industry.
Data collection by test vehicles and commercial fleets and replaying some of the test drives in simulation – even when combined with brute-force generation of simplistic scenarios – is merely too expensive and not scalable. It yields such low testing coverage that the time-to-market is constantly extended due to the ‘long tail’ of edge cases, preventing AV companies from achieving substantial revenue in a timely manner.
The sheer complexity and unpredictability of real-world scenarios, coupled with the immense variability of road conditions and human behavior, make it simply impossible to exhaustively test and validate AV systems through these conventional means. Studies suggest that more than half the cost of AV programs goes to validation. Companies are confronted with the harsh reality that, without a radical breakthrough in testing methodology, they will simply be running out of funding before they can reach acceptable levels of safety performance.
The infinite universe of test cases
First, let’s recognize that Level-3 and Level-4 autonomous vehicles are among the most complex systems ever designed. The challenge in ensuring ADS safety lies in the need to ensure that vehicles can handle the very broad and highly diverse array of real-world scenarios. This includes finding infrequent but dangerous edge cases that must be handled systemically and safely.
Take, for example, a very rare but extremely safety-critical scenario. How often does a pedestrian jump in front of a car in the real world? Very rarely, but whether the jump is from the left or the right, at a distance of 200 feet or 20 feet, with a car traveling at 15 or 50 mph, in dim lighting or at night, rain, snow, or sunshine, on an empty 2-lane country road or a 4-lane arterial with other vehicles limiting evasion maneuverability – all these and dozens of other variables must be handled as safely as possible. Combined, we are easily talking about tens of thousands of variants of this single “jumping pedestrian” scenario category alone.
Of course, there are hundreds more scenario categories, dealing with other cars’ maneuvers, handling unexpected road geometries, dangerous road conditions, interactions with pedestrians, reacting to emergency vehicles, etc. Each one of these scenario categories spawns tens of thousands of possible variants that need to be tested. Subsequently, tens of millions of test scenario variants need to be validated in a timely and cost-effective manner. And not just once – after the initial development is done – but continuously during development and after each subsequent bug fix or software enhancement. Such periodic regression testing throughout the AV software’s lifetime, including over-the-air updates, must be handled efficiently to ensure continued safety performance… all without incurring significant costs.
It has been obvious for the past several years that relying purely on physical test drives, leveraging large production data-collecting fleets (i.e., Tesla), and limited-scope ODD Robotaxi pilots featuring hundreds of cars (e.g., Cruise and Waymo) won’t yield sufficient testing to achieve the desired test coverage and ensure safety. From a statistical perspective, near-accidents and dangerous situations occur too infrequently in real life to serve as primary testing basis. Therefore, synthetically generated tests have been increasingly deployed to start closing this testing gap. But today’s virtual testing has shortcomings that limit its potential: inefficient generation of the necessary millions of scenarios, inability to automatically uncover unknowns, lack of coverage metrics, and limited simulation accuracy.
New solutions available today are already addressing some of these shortcomings. By adopting a collaborative open approach, the industry must come together and deliver on the mandated full potential of virtual testing.
Taming infinity: the role of automated large-scale virtual testing
To achieve the necessary level of safety, the AV industry must fully embrace a transformative leap forward in testing, utilizing automated large-scale test generation technology capable of covering a wide range of complex driving scenarios and infusing randomly generated emergent scenario variants. Large-scale automated testing should utilize state-of-the-art simulation tools. This includes perception and behavior simulators, among others, which are narrowing the simulation accuracy gap. Furthermore, the transformation requires analyzing massive amounts of test results to generate clear metrics for test coverage, performance, and safety. Physical test drives continue to serve an essential role for capturing emergent complex interactions in a real-world context, correlating simulation results to the physical world, and covering testing aspects where simulation accuracy still falls short.
A similar transformation occurred in the semiconductor chip industry when overreliance on expensive physical testing resulted in a financial crisis. The semiconductor industry had to transform its strategy from physical silicon prototype testing to intelligent virtual verification and validation. Today, large-scale intelligent virtual testing is enabling extremely complex semiconductors with billions of transistors to work flawlessly in everyday applications, from cloud servers and mobile phones to medical devices and software-defined vehicles.
The time has come to transition from limited pilots to cost-effective full-scale commercial deployments using these state-of-the-art methods!
ABOUT THE AUTHOR
Ziv Binyamini is the CEO of Foretellix. He has 30 years of experience in the verification and validation of highly complex systems, including the most advanced microprocessors, system-on-chip, advanced ADAS, autonomous trucks, and AVs. Foretellix provides safety-driven V&V solutions that enable safe, large-scale deployment of Automated Driving Systems.
