Greetings and welcome to a brand new segment on our blog, “Under the Hood,” featuring industry experts. In this post, we’ll discuss how to use ODDs practically in your safety V&V.
We’re thrilled to have Gil Amid, Foretellix’s Chief Regulatory Affairs Officer, VP of Operations & Co-Founder join us. Gil is a member of the ASAM Technical Steering Committee and has been instrumental in developing the ASAM OpenSCENARIO® 2.1 DSL (OSC DSL) standard.
Today’s interview is in a Q&A format, giving you the chance to gain firsthand insights from Gil into a deeper dive into how Operational Design Domain (ODD)-based scenario generation and coverage analysis can streamline V&V processes and ensure the safety and success of autonomous vehicles.
Welcome, Gil, and thank you for being here. Let’s start with the industry’s big question: Is autonomous technology safe enough for large-scale commercial deployment yet?
The short answer is “probably not yet.” In reality, that’s a question for regulators, who must ensure autonomous vehicles are safe. Currently, there are no standardized rating systems, like the five-star NCAP rating for traditional vehicles, to certify AV safety for commercial deployment. The EU’s regulation EU 1426 addresses this by requiring manufacturers to define and meet their safety criteria and demonstrate this to regulators. Essentially, manufacturers must create their own safety assessments to prove the safety of their AVs.
ISO 34503 is a standard for defining ODD attributes. While it proposes a basic set of attributes for ODD specification, it does not suggest how to incorporate it into V&V. Can OSC (DSL) be used to leverage these ISO standard attributes when defining the boundaries of an ODD?
Indeed, ISO 34503 defines a taxonomy for ODD specification. The key question is how to model this taxonomy in OSC DSL. OSC DSL programming language-like capabilities can be used to define a full taxonomy. So, yes, it is feasible and relatively simple to represent ISO 34503 taxonomy using OSC DSL and provides a standard vocabulary for ODD attributes like weather and road types. Once you have the taxonomy defined, you can set operational design limits by applying constraints on these attributes, such as only functioning in dry weather or on specific road types. The combination of a taxonomy and a set of constraints on the attributes – is a full ODD specification, all using native OSC DSL constructs.
Can you elaborate on the difference between ODDs and scenarios in V&V?
Sure. An ODD defines the “playing field” in which an Automated Driving System (ADS) was designed to operate safely, specifying factors like weather and road structures. Scenarios are like plays within that field, simulating various situations the ADS might encounter, such as merging onto a highway or handling unexpected stopped vehicles. OSC DSL provides a language to describe these scenarios, simplifying large-scale testing and coverage analysis.
How can ODDs be modeled using OpenSCENARIO® 2.1 (OSC DSL)?
OSC DSL lets us model ODDs by setting constraints on all taxonomy attributes, such as environmental conditions. As the ODD is modeled using OSC entities like classes, structures, and constraints, it can simply be incorporated (imported ) into the scenario code itself. Once the ODD definition is merged with the scenario, all relevant checks, KPI, and coverage monitors supplied by OSC DLS can be used to produce the relevant outcome of the scenario.
How do coverage monitors help ensure a test suite comprehensively explores the boundaries of an ADS’s Operational Design Domain (ODD)?
Coverage monitors are crucial for thorough testing. They can be reused across scenarios, track what parts of the ODD are tested, and identify gaps in coverage. By collecting and analyzing this data, they ensure the test suite comprehensively explores the ODD boundaries. In fact, Foretellix’ solution enables aggregation of ODD coverage data from all your various V&V/testing platform (see the following figure).
How does this translate into V&V practices?
Here’s the exciting part. Test scenarios can reference the ODD entities defined in OSC DSL. This allows us to create checks and measurements within the scenario to evaluate the ADS’s behavior. Imagine a scenario where the ADS encounters unexpected rain – we can define a check to see if the ADS correctly triggers an ODD exit and transitions to a safe state. Even better, these checks and coverage monitors can be independent and reused across different scenarios, saving significant time and effort in creating comprehensive test suites. Plus, using a standardized language for scenarios and the ODD specification really improves communication across the board.
Safety assurance in ADS development is a critical bottleneck that many AV companies struggle with. How can OpenSCENARIO® 2.1 (DSL) improve V&V processes?
OSC DSL integrates ODDs and scenarios into a unified testing flow, providing consistent metrics and coverage towards the safety assessment. This makes it easier to run large-scale tests and get aggregated results across different platforms. By adopting OSC DSL, we can establish a more standardized and efficient V&V process, ensuring the safety and success of autonomous vehicles.
This benefits manufacturers by streamlining development and gives consumers greater confidence in self-driving cars. Moreover, OSC DSL can extend to other automated systems, paving the way for safer transportation in the future.
Can you elaborate on the benefits of this approach?
Sure. With ODDs and OSC DSL, we can do a lot. We can run large-scale tests by creating different scenario variations, and we get aggregated results across various testing platforms, making analysis easier. This approach ensures comprehensive testing by systematically covering the entire ODD. It also allows us to optimize our testing efforts by zeroing in on areas that need more attention.
There are two ways to assess ODD coverage. Can you explain the limitations of the simpler approach and how Foretellix addresses achieving a more robust ODD coverage strategy?
The simpler approach to ODD coverage involves checking each element of the ODD boundary individually, such as: highway driving, rain, and daylight. While this method ensures that each specific element is covered, it doesn’t account for the complexity and variety of real-world scenarios an ADS might encounter while operating within the ODD. For instance, it might miss combinations of elements, like driving on a highway during heavy rain at night, or it may miss various scenarios that can happen within the ODD, which are crucial for thorough testing.
Foretellix’s approach, known as “real ODD coverage,” takes a more comprehensive view. It includes all possible events and interactions within the ODD, acknowledging the multi-dimensional nature of the operational environment. Foretellix uses multi-platform coverage monitoring to track testing across virtual simulations, test tracks, and real-world driving logs. By analyzing the data, Foretellix identifies coverage gaps and prioritizes future testing efforts, ensuring a robust and systematic ODD coverage strategy. This method ensures that the ADS is tested under a wide range of scenarios, leading to a more reliable and safer autonomous vehicle.
