Catching Your AI’s Hallucination

In traditional FEA and CFD software, setting up a simulation job requires users to explicitly pick and choose the parameters to be used, from the load-bearing surfaces to the flow turbulence types. Often, a menu or dialog box limits the input choices to the sensible, relevant, computable ones. This is an intimidating step for new and inexperienced users, but it also makes troubleshooting simpler, leaving breadcrumbs to follow to identify the faulty parameters in case of problems. 

Now that simulation software is shifting toward the use of AI agents and copilots, the step is expected to become a series of natural language prompts. This makes FEA and CFD much more accessible, especially for the general designers with limited exposure to simulation. But could this make debugging harder when things go awry? Would it open the door to AI hallucination in FEA or CFD? In this article, we explain how to recognize the signs of hallucination, and what you can do to minimize it.

Message Confusion

Just like humans, AI agents can sometimes misinterpret messages while they attempt to carry out instructions, explained Krystian Link, Technical Product Marketing Manager, Rand. “In the backend, the AI agent is trying to identify keywords and analyze how you have structured them,” he said. “As you add more and more keywords, the message can become disjointed.” Rand Simulation, part of Rand Worldwide, is an elite Ansys software reseller. Ansys simulation software with natural language-driven chatbots and copilots. 

Santosh Kini is the Product Manager for Simulation at Keysight, catering to the aerospace, automotive, datacenter, healthcare, and semiconductor markets, among others. He said, “Hallucination may come from the algorithm itself. Typically, it will give a statistically likely result, but not necessarily what’s physically correct. And the algorithm may have no inbuilt validation mechanism. For example, the algorithm doesn’t know your temperatures cannot go below zero, or that your inflow and outflow have to be the same to maintain the system.” 

In the conceptual design phase, AI hallucination can be a source of creativity, leading to unexplored design options. Kini said, “If you’re an automotive designer, you know what a car should look like. But when you use AI, you’re also expecting the AI tool to use its imagination, to come up with novel ideas. In that sense, it can argued hallucination is a feature of the tool thinking outside the box.”

Training vs. Usage

AI training needs data—lots of it. Often new users underestimate the volume of validated FEA runs required to develop a reliable surrogate or reduced order model that can stand in for the full physics model. And inaccurate or corrupt data in the training data set can sow the seed for AI hallucination.

“The data should not be noisy or incomplete. It should not be very sparse on the edge,” said Kini. “And the user should only use the AI-trained model for what it’s trained to predict. If the entire data set is made up of laminar flow simulations, but you’re asking it to predict turbulent flows, you’re asking for trouble.”

Room for Assumptions

Simulation is fundamentally mathematics, a series of equations. Therefore, simulation programs work best with calculable, quantifiable parameters, like temperature, weight, volume, and tensile strength. But with natural language prompts, users could inadvertently sidestep the critical parameters, forcing the AI tool to make assumptions.

“If you give the tool an underdefined problem, and don't give it enough information, then you’re asking it to fill the gaps. That means you’re opening the door for hallucination,” said Kini.

Signs of Hallucination

Recognizing AI hallucination usually requires domain knowledge. It resides with the industry veterans who can look at the shape of a part and intuitively know where its vulnerabilities are, or the CFD or FEA experts who know what the expected results should look from a quick glance at the mesh model or CAD geometry.

“If you come from the analog background [having used pre-AI simulation tools], you know what to expect, what to look for, so it’s easier to catch AI hallucination,” said Link.

“See if the underlying physics is violated. There are standard benchmarks that you use to validate the solvers. If your AI prediction is something completely off, if it doesn’t align with those benchmarks, it's probably hallucinating,” said Kini. “You can catch a lot of hallucination by looking for inconsistencies. If you gave the tool two very similar sets of inputs, but the results are widely different, something is wrong.”

Link recalled that, on a number of occasions, he had had to interrogate the AI tools to identify the source of simulation results that seemed odd. Instead of asking the AI tool more questions about the same simulation model, feeding more examples of the same type of simulation would be more helpful, he suggested.

“If you show it enough example, it might say, based on these setups and flow conditions, here are the factors that should be present, so you can check for them,” he said. “The more feedback you give, the more it will incorporate the new knowledge into its future runs.”

Solver in the Loop, Human in the Loop

Kini warned that, when it comes to safety critical systems, human oversight is nonnegotiable. “With aircrafts, power plants, or something similar, because their failure means a big disaster, you have to have more safeguards and humans should definitely be in the loop all the times,” he said.

He also proposed the solver-in-the-loop workflow, where AI predictions are not accepted at face value but also confirmed with a full simulation run on a time-test solver. “AI proposes, but the traditional solver validates it, or you do spot checks with the solver,” he explained.

Bruno Lecointre, AI innovation lead for CAE at Keysight, said, “The markets we serve are heavily regulated. There's a lot of automation that already exists. So customers need to find out where the value is, where agentic AI may be superior to automation. That's not a given,” he said. “There is also a tradeoff with AI solutions. How much do you need to invest in AI training? I’d say the market is in the exploratory phase right now.”

As natural language prompts become a standard feature in CFD and FEA programs, Link feels the software should also include more guardrails to keep the human-machine exchanges on the right path. “If it’s designed to funnel the conversation toward simulation, to ask for things like industry-standard physics types and flow conditions, then it minimizes the risk of hallucination,” he reasoned.