Computing Uncertainty

Can you be certain about uncertainty? This is not a paradox but a real question that engineers wrestle with regularly. 

In simulating various scenarios in which a product might fail, whether a smartphone slipping into a bathtub or an autonomous car sliding on an icy road, engineers employ many approximations, which add numerical uncertainty to the predictions. Now that AI-based approaches, reduced order models (ROMs), and surrogate models are gaining acceptance, the need to account for the uncertainty involved takes on greater importance. 

For this article, we speak to the specialists who work in the domain of uncertainty quantification, or UQ. Uncertainty is an inherent part of simulation, but the ability to quantify it helps engineers understand why they should or shouldn’t be confident in a particular set of predictions.

The Value Proposition

Rand SIM, part of Rand Worldwide, is an Elite Channel Partner of Ansys. It resells Ansys software along with relevant training, support and consulting services. Chris Smith, consulting division manager, Rand SIM, and his colleagues sometimes recommend use of a ROM to their clients because it’s a good value proposition. 

“We know if a client runs a full physics simulation, it might cost $300,000 and a year’s time, and we know they don’t have that kind of time or money. In that case, we’ll propose modeling subsystems of their broader configuration, and then creating ROMs for each subsystem, and then integrating them into a broader model of ROMs to come up with a representation. This costs them less time and money,” he explains. 

As a simple example, Smith points to an ammonia lance—an ammonia-injection device commonly used in the oil and gas industry. “In a massive turbine stack, you have thousands of ammonia lances. You don’t need to model every single ammonia lance to accurately characterize the full system. You can do a model of one ammonia lance, and extend that to the system,” he says. 

Smith has seen customers with different attitudes toward ROMs. He warned against two extreme approaches: “avoiding [use of] ROMs because of the fear of uncertainty, and ignoring the uncertainty because of the eagerness to use them.” The first, he quips, is the root cause of “analysis paralysis.” 

Calculating ROM uncertainty is not always straightforward, Smith cautions. “Let’s say, each ROM of a component has 5% uncertainty. Obviously, you would have aggregate uncertainty in a system, but sometimes in a non-linear fashion, because of how the physics integrate,” he says. Smith usually proposes conducting selective physics simulation as a way to judge the ROM’s reliability. 

“That way, when we do our comparison, we can give the customer a breakdown of the expected uncertainties versus what we see in the system. Most of the time, those may be higher because we’re working on a new system where we don’t have as much experimental data, but we can still give them an idea on what the differential is in terms of the uncertainty of the ROM versus the actual physics model,” he explains.

The Data-Driven Approach

SmartUQ software, a modern machine learning and uncertainty quantification tool, specializes in data-driven ROMs. Classic physics-based simulation employs solvers that faithfully replicate the thermal, fluid and structural phenomena using detailed 3D models. The data-driven approach focuses on the underlying mathematical correlations in design parameters and performance: for example, how the curvature and diameter of a valve affect the water pressure. 

The use of ROM is gaining popularity, largely driven by the ease of use that comes from AI integration, and the time and cost savings it offers. However, “many people don’t have a good understanding of the level of uncertainty or approximations involved. That means, it’s difficult for them to determine the level of trust they should have in the predictions from the models,” notes Gavin Jones, principal application engineer, SmartUQ. 

One effective way to minimize uncertainty is to minimize extrapolation, Jones suggests. “The further you get away from any validated point, the more you’re extrapolating. As a general rule, you want to keep things within the convex hull [in the graph] defined by the input data used to train the ROM,” he says. 

The data-driven approach doesn’t concern itself with the underlying physics, be it fluid, thermal or structural. Therefore, when a user attempts to make predictions outside the scope covered by the training data, there’s a risk that he or she is venturing into a territory where the physics is no longer the same. 

“The ROM will still make predictions, assuming the trends in the training data hold true. However, outside the bounds of the training data, you may be dealing with a different physics regime, or the physics may no longer be linear,” warns Jones.

Some ROM solution providers are exploring physics-informed approaches to data-driven ROMs. SmartUQ is among them. “With physics-informed methods, essentially you are adding a constraint to the hyperparameter optimization, typically in the form of partial differential equations or ordinary differential equations,” explains Jones.

Judging the trustworthiness of a ROM is not as simple as saying, for example, 90% accuracy is good; 60% is not good enough. Jones reasons, “If the consequences of incorrect predictions put lives at risk, can cause injury or prompt costly product recalls, and if the ROM is the only thing you’re relying on, and there is no other validation, like physical test results, then you want a high degree of accuracy.” 

Uncertainty in Multiphysics

In 2021, multiphysics simulation software developer COMSOL added an Uncertainty Quantification Module as a new add-on to Release 6 of its software suite. “The Uncertainty Quantification Module can efficiently test the validity of model assumptions, convincingly simplify models, understand the key input to the quantities of interest, explore the probability distribution of the quantities of interest, and discover the reliability of the quantities of interest,” the company wrote.

The UQ Module can be used on any trained ROMs or surrogate models, regardless of where the simulations were conducted. Therefore, use of the module is not restricted to simulations performed in COMSOL software.

“To ensure that a surrogate model or ROM captures the behavior of a system accurately, the input samples should be space-filling; that is, they should cover the input parameter space uniformly to avoid leaving out important regions,” Fanlong Meng, development manager for COMSOL, advises.

As a general rule of practice he recommends starting with at least 10 to 20 simulations per input parameter. “With N input parameters, that means roughly 20 × N simulations. Based on the resulting model quality, additional simulations can be run to improve the surrogate model further,” he explains. 

The UQ Module comes with a built-in training-error metric specific to each surrogate model type. “A large training error typically indicates that the surrogate model is not sufficiently accurate, suggesting the need for additional training data. For a more independent assessment of model accuracy, performing validation against new data points (not included in the training set) is recommended,” says Meng.

Meng further explains that fidelity of the model also depends on the mathematical complexity. “I’m referring to the input vs. quantity-of-interest relationship itself: for example, linear versus nonlinear. This is not about the underlying physics,” he clarifies. “For example, a turbulent CFD model may yield a simple input-output map depending on which quantity or quantities of interest the user selects. But a seemingly simple model can also produce a nonlinear response in certain cases. For instance, a steel bracket under bending shows a nonlinear relationship between displacement and material thickness, even if the material is linearly elastic.”

The Blackbox Predictions

MathWorks, known for MATLAB and Simulink software, offers tools to develop ROMs from high-fidelity computational fluid dynamics and finite element analysis simulation models. 

“If users are working with linear parametric black-box models (such as transfer functions or state-space models) or with linear and nonlinear grey-box models, the System Identification Toolbox can help compute model uncertainty by estimating the variability in model parameters resulting from random disturbances in the output,” says Kishen Mahadevan, senior product manager, Controls and Identification, MathWorks. “If users are working with neural networks, then Deep Learning Toolbox and Deep Learning Toolbox Verification Library can help verify and test properties of deep learning networks to increase confidence in model behavior.”

Japanese carmaker Subaru is one of MathWorks’ customers. To bypass the need to conduct full physics simulation of its hydraulic control system, Subaru engineers developed an AI surrogate model using MATLAB. The model is shown to be capable of reproducing waveforms with arbitrary current, oil temperature, and source pressure readings. And it achieved a 99% reduction in calculation time compared to the original 1D model, according to MathWorks.

“When estimating 1D/0D dynamic ROMs, the number of simulations is often less important than ensuring adequate coverage of the design space and that system dynamics are properly excited,” Mahadevan notes. “For example, using excitation signals such as Pseudorandom Binary Sequence (PRBS) can enable capturing system dynamics with fewer simulations, provided they sufficiently excite the system.”

Uncertainty analysis is just one way of assessing model quality, Mahadevan adds. Ideally it should be used in conjunction with other methods: “Cross-validation, residual analysis, including the whiteness test and correlation test, visualizing error histograms to assess estimated model fit, and persistence of excitation tests can provide further insights into [the] model’s reliability,” he says.  DE