Self-Driving Cars Test Traditional Procedures

The sci-fi scenario of self-driving cars isn’t looking all that futuristic these days. Yet as big names and startups throttle up development efforts, they are also mapping out new test and validation practices to ensure autonomous vehicles can merge safely into mainstream traffic.

Early prototypes of these vehicles from automotive giants like Ford and Tesla along with tech titans like Google and Uber are already hitting city streets. Ford conducted winter weather testing of autonomous vehicles at Mcity, a full-scale simulated real-world city environment at the University of Michigan, while Google claims to have logged over 1.5 million road miles with its self-driving car project. This summer, Uber’s Ford Fusion self-driving fleet took to city streets in Pittsburgh as part of a pilot to test the technology and gather feedback from customers.

The latest frenzy of development and test activity, coupled with the federal government’s recent release of policies and guidelines around autonomous vehicle safety, are fueling what experts project will become a sizable market in just a few years. By 2025, Boston Consulting Group is estimating a global market of $36 billion for partially autonomous vehicles and $6 billion for fully autonomous units.

Proponents are bullish on autonomous vehicle technology for a variety of reasons, including its ability to increase driver safety and comfort, reduce accidents, prevent traffic congestion and even cut back on emissions. For example, platooning—a type of autonomous operation where a convoy of trucks is electronically linked and controlled by a lead vehicle—would save about 4% in fuel compared to separately running rigs, according to a new report from the North American Council on Freight Efficiency.

Yet for all the potential upsides, the nascent industry faces a long road ahead to get consumers comfortable with the technology, particularly as it relates to safety. A study just released by Kelley Blue Book found that just over half (51%) of respondents prefer to have full control over a vehicle, even if it’s not as safe for other drivers. The unfamiliarity with autonomous vehicle technology and heightened concerns over safety is mounting pressure on industry players to ramp up and innovate more sophisticated test and validation practices beyond what’s typically performed in traditional vehicle development.

“How do you really understand that the system you’re putting on the road will perform over a lifespan of the vehicle and at a high enough level that will ensure you don’t have catastrophes?” asks Lee Barnes, director, connected and autonomous vehicle business at global engineering consultancy Ricardo. “There’s no real answer for that today.”

A Holistic Test Approach

Because autonomous vehicles are considered machines, they’re held to a different standard than human drivers, experts say. Moreover, the current methods of testing—

real-world track tests and existing simulation practices—don’t completely map to this new area of development, raising the possibility of significant safety gaps, Barnes explains. “The standard environmental testing approaches used today for testing vehicle hardware for lifelong durability and functionality are not transferable to an autonomous vehicle platform,” he says. “We can still use those approaches, but we have to go beyond them for testing the system as a whole in an environment that has random situations taking place.”

As part of its consulting practice in this area, Ricardo is leveraging agent-based modeling (ABM) simulation methodologies to support advanced testing and analysis of autonomous vehicle performance. The approach combines agents (vehicles, people or infrastructure) with specific behaviors (selfishness, aggression) and connects them to a defined environment (cities or test tracks) to understand the emergent behaviors during a simulation, Barnes says. The practice is used to recreate real-world driving scenarios in a virtual environment to test complex driving scenarios.

ABM and other novel forms of modeling and simulation practices are required because it’s simply not possible to rack up all of the necessary physical testing miles, nor is it viable to virtually explore all of the variables when autonomous operation is involved.

Rand Corp. just released a study titled “Driving to Safety,” which concluded that autonomous vehicles would have to be driven hundreds of millions of miles (sometimes even billions) to fully demonstrate their reliability in terms of fatalities and injuries. That isn’t feasible considering the industry’s aggressive go-to-market timeframes, according to Nidhi Kalra, senior information scientist and the director of the Center for Decision Making Under Uncertainty at Rand.

“If we want to achieve the test driving miles necessary before making these vehicles available for consumer use, we would be waiting forever,” she explains, advocating the need for alternative methods to supplement real-world testing, including well-designed pilot studies, modeling and simulation, and mathematical modeling and analysis.

ANSYS, whose simulation software is used liberally in traditional automotive testing, is working on a number of technologies to address the challenges related to simulating and testing autonomous vehicles. ANSYS is evolving its existing tools into a platform that would address simulation as a controlled loop spanning the model of the vehicle, the system of sensors sensing what is going on in the environment, the control system that interprets signals and makes decisions, and the actuator that adjusts the braking system or steering system, for example, based on vehicle dynamics, says Sandeep Sovani, ANSYS’ director for the global automotive industry. The key, Sovani says, is creating a simulation platform that spans and integrates all of these aspects. “It’s an integration challenge—how to move data from a scenario model into a sensor model effectively and fast,” he says.

ANSYS Simplorer, a systems-level simulation tool, will play a key role, as will the work ANSYS is doing with reduced order models, an approach designed to allow highly detailed models to run in real time as opposed to taking hours or even days, he explains. ANSYS’ Engineering Knowledge Management tool, which manages simulation data, will also be instrumental for integrating data across multidisciplinary simulations.

Testing the Self-Driving Cars’ Algorithms

Another big challenge involves the still nascent machine learning capabilities, which are critical for allowing autonomous vehicles to properly discern the nature of a line of sight object, but still have critical gaps in knowledge. Despite being trained to recognize a “pedestrian,” the algorithms could be tripped up by someone in a wheelchair or walking with crutches, and that ambiguity can be further exacerbated by environmental conditions or sensor failures, affecting the ability to initiate a safe response. “You have the potential for millions of combinations that the algorithm may not have learned yet, and each represents a gap in the knowledge of the pedestrian detector, a software error, and a potential safety hazard,” explains Michael Wagner, CEO and founder of Edge Case Research, formed by Carnegie Mellon researchers and focused on solving this very problem.

Because exhaustive testing of autonomous systems is near impossible, Edge Case Research is developing an automated software robustness testing tool code-named “free2code” that prioritizes tests that are most likely to find safety hazards. Such scalable testing tools give developers the feedback they need early in development, so that they can get on the road more quickly with safer, more robust vehicles. “What we’re doing at a software level is addressing the stuff that doesn’t happen often to make sure the system doesn’t trip when it sees something weird,” explains Philip Koopman, an associate professor of Electrical and Computer Engineering at Carnegie Mellon, who is working with Edge Case. “This kind of testing provides confidence that all the failure modes that might happen have been explored … and it’s helpful to pair [the practice] with simulation testing.”

Another change to typical simulation and test practices is the requirement for systems-level simulation when dealing with autonomous vehicles, notes Frédéric Merceron, Transportation & Mobility Industry Solution Experience Director at Dassault Systèmes. “Today, each functional area uses different simulation software and integrates everything together to certify that things are working well,” he explains. “With an autonomous vehicle, the braking system will have some impact on other systems in the car, on the sensors or controllers. You have to have a backbone across all the different disciplines and have people working together early on multidisciplinary testing.”

In addition to the 3DEXPERIENCE backbone, Dassault’s Simpack, a general purpose multi-body simulation tool, and Dymola, a modeling and simulation environment based on the open Modelica language, will come into play for systems-level simulation. In addition, the recent acquisition of Computer Simulation Technology AG (CST) provides electromagnetic (EM) and electronics simulation software for simulating radars and sensors—another important component for autonomous vehicle simulation and testing. EXALEAD, Dassault’s Big Data intelligence tool, will help mine data collected from the test beds and integrate it to create higher fidelity simulation models.

Given that it’s early in autonomous vehicle development, many of these tools and techniques are evolving with new methods and solutions still to come. With the safety of these vehicles an imperative to their success, experts say the optimal test strategy remains a multi-pronged approach.

“There is no single way to prove their safety, but a lot of good ways to manage risk,” says Rand’s Kalra.

More Info

• ANSYS
• Dassault Systèmes
• Ford
• Edge Case Research
• fka SV
• Rand Corp.
• Ricardo
• Tesla