VVUQ · Module 10 of 10
Credibility Assessment and Decision-Making
All the verification, validation, and uncertainty evidence exists to serve one thing: a decision. This closing module scales the required rigor to what is at stake and turns the evidence into a defensible go or no-go.
Readiness check
This closing module turns evidence into decisions. Tick only what you can do closed-notes.
- Recall the validation uncertainty and prediction uncertainty.
- Multiply two scores to combine factors.
- Compute a margin as a difference divided by an uncertainty.
- Recall that risk combines likelihood and consequence.
- Recall that a model has an intended use.
The core idea
Credibility is VVUQ evidence sufficient for a specific decision. The ASME V&V 40 approach scales the required rigor to the model's influence and the decision's consequence, and adequacy is judged by whether the prediction, with its uncertainty, leaves a sufficient margin.
model risk = influence × consequencehigher risk ⇒ more VVUQ evidencemargin = (limit − prediction) / uThe point of verification, validation, and uncertainty quantification is not to produce numbers but to support a decision, and the amount of evidence required depends on what the decision is worth. The ASME V&V 40 credibility framework, developed for medical devices but general in spirit, makes this risk-informed. Model risk combines two factors: the model's influence on the decision (how much the outcome relies on the simulation rather than on tests or other evidence) and the consequence of the decision being wrong (how severe a failure would be). A high-influence, high-consequence use demands extensive VVUQ evidence, code and solution verification, validation on relevant data, and quantified uncertainty; a low-risk use can be credible with far less. Adequacy for intended use is then the judgement that the evidence matches the risk. In quantitative terms, a prediction is acceptable when it leaves a sufficient margin: the gap between the predicted value and the limit, measured in units of the combined uncertainty, must exceed a required number of standard deviations. Documenting the evidence, its limits, and this margin is what makes a simulation-based decision defensible to a reviewer or regulator.
The skills, taught in order
Five skills turn the accumulated evidence into a risk-informed, documented decision.
10.1 Credibility and intended use
Credibility is not a property of a model in the abstract but of a model for a stated purpose. The same simulation may be credible for a rough concept study and inadequate for a certification, so the intended use must be fixed before credibility is judged.
10.2 Model risk
Model risk combines the model's influence on the decision and the consequence of an incorrect decision. High influence with high consequence is high risk; either being low lowers the risk. This risk sets how much VVUQ evidence is required.
| Model risk | Evidence required |
|---|---|
| Low | basic verification, limited validation |
| Moderate | solution verification, relevant validation, UQ |
| High | full code and solution verification, extensive validation, quantified UQ |
The V&V 40 principle: the rigor of the VVUQ evidence is scaled to the model risk, not applied uniformly.
10.3 Adequacy for intended use
Adequacy is the judgement that the assembled evidence matches the model risk. It asks whether the verification, validation, and uncertainty are sufficient for this decision, not whether they are perfect. Falling short means either more evidence or a lower-stakes use.
10.4 Margins and acceptance
Quantitatively, a prediction is acceptable when the margin, the gap between the prediction and the limit measured in units of the combined uncertainty, exceeds a required number of standard deviations. A margin that ignores the uncertainty is meaningless.
10.5 Documenting the evidence
The final product is a credibility record: the intended use, the VVUQ evidence, the quantified uncertainty, the margin, and the limits of applicability. This documentation is what lets a reviewer, auditor, or regulator trust a simulation-based decision.
Engineering connection: a digital twin or simulation replacing a physical test can only do so when its credibility record shows the evidence matches the decision's risk, the bridge to AI-Enabled Digital Engineering.
Worked example 1: scoring model risk
A simulation has a model influence of 0.7 (on a 0 to 1 scale) and supports a decision whose consequence is scored 0.8. Compute the model risk and judge the VVUQ rigor required.
- ProblemCompute the model risk and required rigor in Figure 1.
- Given / findInfluence = 0.7, consequence = 0.8. Find the risk score and the rigor implied.
- AssumptionsInfluence and consequence are scored independently on a 0 to 1 scale, combined multiplicatively.
- ModelModel risk = influence × consequence; higher risk requires more VVUQ evidence.
- Equationsrisk = influence × consequence
- SolveRisk = 0.7 × 0.8 = 0.56. With both factors high, this falls in the high-risk band, requiring full code and solution verification, extensive validation, and quantified uncertainty.
- CheckLowering either factor would reduce the risk: if influence dropped to 0.3, the risk would be 0.24 and the rigor could be lighter. The multiplicative form captures that either low factor lowers risk.
- ConclusionThe model risk is 0.56, in the high band, so the decision demands the full VVUQ evidence chain. The score sets the standard the evidence must meet.
Worked example 2: the acceptance margin
A design limit is 100 (normalised). The simulation predicts 85 with a combined uncertainty of 6. Find the margin in standard deviations and decide whether it meets a requirement of at least 2.
- ProblemFind the acceptance margin and decide the outcome in Figure 2.
- Given / findLimit = 100, prediction = 85, combined uncertainty u = 6, requirement ≥ 2. Find the margin.
- AssumptionsThe uncertainty is the combined validated prediction uncertainty; the limit is firm.
- ModelMargin = (limit − prediction)/u, compared to the required number of standard deviations.
- Equationsmargin = (limit − prediction)/u
- SolveMargin = (100 − 85)/6 = 15/6 = 2.5σ. Since 2.5 > 2, the design meets the requirement.
- CheckHad the uncertainty been 8 instead of 6, the margin would be 15/8 = 1.9σ, below the requirement, showing how the uncertainty, not just the gap, decides acceptance.
- ConclusionThe margin of 2.5 standard deviations exceeds the required 2, so the simulation supports acceptance. The decision rests on the prediction and its uncertainty together.
Misconceptions and diagnostics
| Mistake | Symptom | Diagnostic question | Correction |
|---|---|---|---|
| Uniform rigor | Same evidence for every model | "What is the model risk?" | Scale VVUQ evidence to influence and consequence. |
| Credibility without intended use | A model called valid in general | "Valid for what decision?" | Judge credibility for a stated purpose. |
| Margin without uncertainty | Gap reported as if exact | "In how many standard deviations?" | Divide the gap by the combined uncertainty. |
| Undocumented evidence | A decision a reviewer cannot audit | "Is the evidence recorded?" | Document use, evidence, uncertainty, and limits. |
Practice ladder
A model has influence 0.5 and consequence 0.4. Find the model risk.
Show answer
Risk = 0.5 × 0.4 = 0.20, a low-to-moderate risk requiring modest evidence.
A limit is 50, the prediction is 40, and the uncertainty is 4. Find the margin in standard deviations.
Show answer
Margin = (50 − 40)/4 = 2.5σ.
For a requirement of 3σ, a prediction of 70 against a limit of 100 needs what maximum uncertainty?
Show answer
Margin = (100 − 70)/u ≥ 3 ⇒ 30/u ≥ 3 ⇒ u ≤ 10. The uncertainty must be 10 or less to meet the 3σ requirement.
You want a simulation to replace a physical qualification test. Describe the credibility case you would build.
What good work looks like
State the intended use, score the model risk from its influence and the consequence of failure, assemble verification, validation, and quantified uncertainty scaled to that risk, show an acceptance margin against the limit, and document the evidence and its applicability limits for the reviewer.
Working with AI, and proving it yourself
Use AI as an examiner, not a solver
Portfolio task
For a real simulation-based decision, score the model risk, assemble the VVUQ evidence to match, compute the acceptance margin, and write a short credibility record.
Retrieval and spaced review
Closed notes. Answer out loud, then reveal.
1. What is credibility?
VVUQ evidence sufficient for a specific intended use.
2. What determines model risk?
The model's influence on the decision and the consequence of being wrong.
3. How does risk set the evidence?
Higher risk requires more verification, validation, and uncertainty evidence.
4. Write the acceptance margin.
Margin = (limit − prediction)/u, in standard deviations.
5. What makes a decision defensible?
A documented credibility record of use, evidence, uncertainty, margin, and limits.
Standards mapping
This module follows the ASME Verification, Validation, and Uncertainty Quantification standards. Use these references to read further.
| Topic in this module | Where to read more |
|---|---|
| Risk-informed credibility assessment | ASME V&V 40, Model Credibility |
| Adequacy for intended use | ASME V&V 40, Model Credibility |
| Machine-learning model credibility | ASME VVUQ 70, Machine Learning |
Standard designations refer to the ASME V&V series; the risk-informed credibility framework is defined in ASME V&V 40 and extended to machine learning in VVUQ 70.