Terrain Twins and Software in the Loop

In the June 2021 IEEE Spectrum article “How Software is Eating the Car,” the author Robert Charette noted, “Today, high-end cars like the BMW 7-series with advanced technology like advanced driver-assist systems (ADAS) may contain 150 ECUs (electric control units) or more, while pick-up trucks like Ford’s F-150 top 150 million lines of code. Even low-end vehicles are quickly approaching 100 ECUs and 100 million lines of code …” 

Software’s growing role in modern cars suggests it must now be an integral part of automotive simulation. Simulating the vehicle’s aerodynamics and electromechanical operations in isolation may be enough for the early design phase. But for the later phases, engineers must also ensure that the software-driven functions kick in as needed during normal operations and when failure modes occur. In this article, we examine the practice, as exemplified by Southwest Research Institute (SwRI)’s newly released 3D simulation tool.

Terrain Digital Twins

In its press release announcing the new software, the SwRI writes that its “simulated environment, or a 3D software loop, supports evaluations of an infinite number of scenarios that would be cost-prohibitive to test in the real world. The technology meets U.S. Department of Defense demands for modeling and simulation tools to help advance the development of unmanned ground vehicles (UGVs), the military term for automated or autonomous vehicles.”

SwRI’s simulation environment is built on the Unreal game engine; therefore, it can work with common 3D file formats from CAD programs, according to Dan Beer, assistant manager in SwRI’s Robotic Autonomy Department. However, SwRI’s technology is different from the computational fluid dynamics or finite element analysis programs you might use to verify vehicle aerodynamics and driving dynamics. 

“Our tool is focused on providing a range of terrain models and environmental conditions,” says Beer. “This capability is less about the optimization of the design of the vehicle itself and is all about providing a way to test autonomy software solutions and capabilities for a vehicle across a full range of real-world conditions.”

The software offers a way to deploy digital twins, but not in the way you might think. What it allows you to use is the digital twin of a real-world environment where the AV [autonomous vehicle] is expected to operate. 

“With a digital twin of the environment in which an AV will operate, developers are afforded the opportunity to conduct additional planning and testing that could otherwise be costly to conduct through in-person testing,” says Beer.

The Institute said it had developed a pipeline of technology with custom algorithms, off-the-shelf software, open-source tools and public map data. It comes with a Simulation Scene Adjustment Tool with a video game-style 3D interface. The simulator can spin up a digital twin, a virtual representation of an automated vehicle that looks and behaves like its counterpart in the real world.

SwRI’s Simulation Scene Adjustment Tool evaluates automated vehicles in 3D simulations of off-road environments. The image shows 3D scenes featuring different ground cover, simulating grass, foliage or rocks to virtually test military unmanned ground vehicles (UGVs). Images courtesy of SwRI.

“It allows engineers to test scenarios that may be dangerous or expensive to establish and repeat during in-person testing, such as those involving failure scenarios or pedestrians and wildlife,” Beer says.

Autonomy Software Onboard

Simulation software makers such as Ansys are now incorporating software-in-the-loop, hardware-in-the-loop, and drive-in-the-loop testing functions as part of their offerings. NVIDIA, which develops the DRIVE Sim platform, also allows similar types of testing with NVIDIA Isaac Sim. 

SwRI’s simulator combines a 3D visualization engine, dynamics engine, vehicle modeling tools, vehicle terrain interaction models, and plug-ins to communicate with an autonomy software stack. It renders 3D environments using elevation maps captured from geographic information system (GIS) data. 

According to the Institute, its “machine learning algorithms simulate computer vision and sensing outputs for lidar, radar, cameras, GPS and other systems to perceive scene objects, movements and position when calculating driving responses. A dynamics engine simulates forces caused by gravity and motion as a vehicle model moves through an environment. Simulated vehicles can be programmed with weight, speed, horsepower, center of gravity and other realistic characteristics. A graphics engine simulates trees, grass, terrain objects and visual effects such as sky and clouds.” In the first round of research, SwRI imported the digital elevation models (DEMs) from aerial scans conducted by the San Antonio River Authority and other government agencies. 

“We developed algorithms to perturb DEM and GIS data in user-configurable ways that generate synthetic environments,” says Joe Auchter, lead engineer for SwRI’s Intelligent Systems Division. “This allows for testing of new algorithms and techniques in simulation, building numerous test environments that share certain relevant characteristics with a real geo-specific location where vehicles will eventually operate.”

“For the autonomy stack, we used the Robot Operating System (ROS), which allows for easier communication between different parts of the system. Rather than every system needing to speak the same language, a camera driver written in C++ will have no problems providing data to an image classifier written in Python,” explains Beer. 

ROS (www.ros.org) is an open source software development kit for robotics applications. It has been in use for more than a decade, giving birth to a robust ecosystem and community around it. Its users include students with a single project as well as institutions with ongoing projects. 

“While ROS is nearing end of life and ROS 2 has been seeing more and more active development from industry, academia, and other research institutes, this testing environment and other SwRI development is not restricted to these frameworks. We’re here to bridge the gap from client needs to client results, and always ready to find the solution that best gets them there,” Beer adds.

Currently SwRI’s testing environment is a research project, but Beer says, “We are happy to hear from companies seeking such solutions and discussing how we can help them meet their needs.”