The Metadata Challenge

A few years ago, the ASSESS Initiative (part of NAFEMS) drafted the Unified Model Characteristics for Engineering Simulation (UMC4ES), an attempt to define a comprehensive set of model characteristics for the complete range of engineering simulation models that could potentially be used across different approaches for implementing metadata structures. 

That effort comes at a time when there is keen interest in better ways of collecting and accessing this simulation metadata. Companies need better ways of exploring, reproducing and sharing the results of their simulation activities, and metadata plays a key role in making that data organized and accessible. With the proper collection and storage of metadata, simulations have improved traceability, and collaboration can be enhanced. 

In addition to the ASSESS effort, other organizations are working on ways to improve or standardize metadata structures for simulation. Incose, for example, has its Model Characterization Pattern (MCP). Fraunhofer has developed VMAP, an open, vendor-neutral standard for computer-aided engineering (CAE) data storage to enhance interoperability between different simulation software packages. The development of artificial intelligence (AI) and machine learning tools that can automate metadata collection and improve search capabilities could potentially revolutionize metadata management.

We spoke to several experts in the market about new activities around simulation metadata standards and automation.

What is the role of simulation metadata? Why is it important?

Ian Symington, Chief Technical Officer, NAFEMS: The most obvious value is being able to find useful work again. In a fast-paced design environment, it can be really easy to lose track of which version of a model you analyzed. When you are doing simulation modeling work, you remove features not necessary for analysis, so what you are working with no longer has a direct, one-to-one relationship with the CAD file. Having traceability of why you ran a simulation and what the results were is incredibly useful. Metadata connects everything together. 

Jacob Surber, Vice President of Products, Rescale: Any time you are running a simulation you are using many gigabytes of input files and getting many gigabytes of data in the results. Often these are large, binary blobs. Metadata represents the real value of that simulation. How do you make it serviceable and actionable? Typically simulation results go into a vault never to be heard from again. What metadata does is offer a higher level view of the outputs and outcomes of simulation, so you can use that information again.

You can think about metadata in a couple of different ways. There is operational metadata—what software were you using? How long did it take? Those are basic, operational descriptors of what took place. There is also metadata around the results of the simulation. If you look across the different solvers, they all have slightly different formats. Because we support all of the solvers, we are looking at how to standardize and make this accessible.

Jason Ghidella, Principal Product Manager, Simulink Platform, MathWorks: Simulation metadata plays the role of documenting and organizing the information that describes models, experiments, and results. It captures details such as the model name, purpose, assumptions, parameters, release information, and confidentiality level.

Its importance comes from two main aspects:

• Traceability and credibility: Metadata links simulation results back to the exact models, configurations, and requirements they address. This allows engineers to verify results, reproduce simulations, and build confidence in their validity.
• Interoperability and reuse: Standards like SSP [System Structure and Parameterization] rely on metadata to ensure that simulation data can be shared across teams, tools, and organizations. Without clear metadata, results may be misunderstood or misapplied.

In short, metadata is what turns a simulation into reliable and reusable evidence rather than an isolated experiment.

What are some of the key challenges of metadata management?

Symington, NAFEMS: It’s typically a manual process. For mature companies, everything is on a hard disk, or if you are in aerospace or other industries, things might be on tape. It might be listed in a spreadsheet or recorded by hand. All of this needs to be digitized. Ideally the state of the art would be that metadata is in a database, with all linkages to find the files easily and all the different items of metadata connected to the thing you are interested in. Ideally, there would be an automated process to create that metadata as you are performing your work, so it’s not just another chore for the engineer.

Surber, Rescale: First, there is a lot of siloed data. There is a lot of legacy data from simulations run years ago. That metadata was never extracted. Customers need to set up pipelines to go through old simulations and extract that metadata. There is also a lack of context or knowledge loss around this legacy data.

There also needs to be a cultural shift. The challenge will be getting engineers on board to adopt new processes. Workflows have to change and expectations need to be managed. Without consistent extraction and capture of metadata, the digital thread doesn’t exist. With automated extraction and the ability to understand unstructured data at scale, that can make the digital thread extremely useful and relatively easy for companies to deploy.

We have seen some efforts around standardization of simulation metadata in the industry (i.e., ASSESS and a few others). What is the goal of standardization in this context? What are some of the obstacles around sharing simulation data that are driving this activity?

Symington, NAFEMS: The Holy Grail would be everyone using the same items for metadata. If everyone agreed that this is the only set of data we need, then you can put that into tools we can use. It would be handy for all of us to be speaking the same language, but I’m not sure it’s realistic. In terms of the ASSESS document, it’s incredibly useful to have this enormous list of every possible item of metadata you could want. It’s a great starting place, and there’s nothing else out there that’s nearly as comprehensive. If I were looking at what simulation metadata a team should collect, the first thing I would do is look at that document and curate what I think is important.

Ghidella, MathWorks: One major goal of the standardization of simulation metadata is to improve traceability and support the product development process by ensuring that simulation models, parameters, assumptions, and results can be consistently documented, interpreted, and reused across teams.

Current obstacles in sharing simulation data include:

• Interface mismatches: Different tools and teams often use incompatible formats, making it difficult to integrate simulation results.
• Inconsistent parameter specifications: Variations in how parameters are defined can lead to misinterpretation of results.
• Version control issues: Without clear tracking, it is hard to know which model or dataset a result came from.
• Ambiguity in requirements: Unclear or incomplete requirement documentation reduces confidence in simulation outcomes.

Some of the standardization work seeks to address these challenges by defining a structured process for conveying simulation information, enabling metadata to be stored and exchanged in a standardized format. This improves consistency, transparency, and reuse of simulation data across teams, tools, and organizations, reducing errors and inefficiencies in the development lifecycle. 

Surber, Rescale: If you are talking about standardization and definition in a broad sense, I think that’s helpful for educating engineers and organizations as to what kinds of metadata might be available or that they might want to store. In terms of implementation, the technology is changing, though.

In the past, you had schema standardization or key value parastructures that would help you write a sQL query to find the right metadata based on the job. It helped you find the thing in a sea of data. 

Bringing in artificial intelligence (AI) reduces the need to have that strict structure or standard. We just launched Rescale Data Intelligence, which has an automation feature that can execute before, during or after a simulation to extract metadata. This opens up implementation flexibility where we can extract a lot more data, even more than is expressly going to be used, and you can have it on hand if you want to come back and look for something else. We also have data connectors around PLM and SPDM solutions so you can enrich simulations with the context from those systems. AI allows you to interrogate these massive databases of information and attach that to a simulation with external metadata.

What role can/will AI/LLMs play in this space?

Ghidella, MathWorks: AI and LLMs can play several transformative roles in managing and using simulation metadata.

Automating Metadata Generation and Enrichment: LLMs can analyze models, simulation logs, and documentation to automatically generate or enrich metadata. For example, an LLM-powered system can infer descriptions, suggest tags, and identify relationships between simulation artifacts. This reduces the burden on engineers and improves metadata consistency across teams and tools.

Semantic Search and Discovery: LLMs enable semantic search, allowing users to ask natural language questions like “Which component in the model uses a PID controller with a gain above 10?” and receive relevant results, even if the metadata uses different terminology.

Intelligent Assistance and Reasoning: AI assistants embedded within simulation environments facilitate users’ comprehension of metadata by providing contextual explanations. For example, they can clarify the origin of a result, identify potential inconsistencies, or recommend relevant annotations that may be missing.

Enabling Interoperability and Standardization: As industry groups like ASSESS push for metadata standardization, LLMs can help bridge gaps between proprietary formats and emerging standards. AI can map internal metadata structures to standardized schemas, easing integration across tools and organizations.

Supporting Compliance and Auditability: In regulated industries, metadata is essential for demonstrating compliance. AI can assist by flagging missing or outdated metadata, generating audit trails, and ensuring that simulation workflows align with documentation and standards.