Home / Courses / Verification and Validation / Validation Metrics

VVUQ · Module 5 of 10

Validation Metrics and Validation Uncertainty

A comparison error means nothing without knowing how precisely it is known. This module’s validation-error and uncertainty treatment is based on the ASME V&V 20-2009 framework as summarized in NIST IR 8298. It does not reproduce the standard’s normative text. The numerical, input, and experimental standard uncertainties combine into one validation uncertainty, the yardstick the discrepancy is measured against.

01

Readiness check

Learning objectives

  • Construct the combined validation standard uncertainty uval from compatible standard uncertainties, including covariance where relevant.
  • Calculate the ratio R = |E|/uval and interpret what the comparison resolves.
  • Justify why a small R does not bound model-form error, since contributions can cancel.
  • Evaluate how reducing the experimental uncertainty sharpens the comparison's resolving power.

This module quantifies the validation. Tick only what you can do closed-notes.

  • Recall the comparison error E = S − D.
  • Recall the grid convergence index and that it carries a safety factor.
  • Combine independent standard uncertainties as a root sum of squares.
  • Recall that a measurement carries an uncertainty.
  • Compare a value to an interval.
0 or 1 weak itemsContinue with this module.
2 weak itemsRevisit the GCI in Module 3.
02

The core idea

The comparison discrepancy E cannot be interpreted alone. Numerical, input, and experimental uncertainty all affect it. Expressed on compatible standard-uncertainty bases and treated as independent, they combine into a validation standard uncertainty uval. Comparing |E| with uval tells you what the comparison can resolve; it does not measure model-form error.

E = S − D = δmodel + δnum + δinput − δDuval = √(unum2 + uinput2 + uD2)R = |E| / uval

A comparison discrepancy is only interpretable alongside its uncertainty. Conceptually the discrepancy carries four signed error contributions:

E = δmodel + δnum + δinput − δD
  • δmodel: model-form error, the inadequacy of the mathematical model itself.
  • δnum: numerical error, the difference between the discrete solution and the exact solution of the model equations.
  • δinput: the error induced in the simulation result by errors in its inputs.
  • δD = D − T: the error in the experimental result, where T is the unknown true value of the quantity at this validation point.

The experimental term enters with a minus sign because E is defined as S − D. That sign is what gives the covariance terms involving δD their negative signs. Only E is directly calculated. The signs and magnitudes of the individual error contributions are generally unknown, so what is estimated is their associated uncertainties, not the unknown signed errors themselves.

Three standard uncertainties are estimated: unum for the unresolved numerical-error contribution, uinput for the effect of uncertain simulation inputs on the simulation result, and uD for the experimental result D. When these are expressed on compatible standard-uncertainty bases, share matching units and quantity definitions, are treated as independent or sufficiently uncorrelated, and do not double count a shared source, they combine as uval = √(unum2 + uinput2 + uD2). The ratio R = |E|/uval then expresses the discrepancy in units of that combined standard uncertainty. R is a measure of what the comparison can resolve, not a verdict on the model.

The skill works when: you combine the three uncertainties and compare the comparison error to uval.
The skill breaks down when: quantities on different uncertainty bases are combined as though they matched, a shared source is double counted, or R is read as a pass or fail.
The concept. Three standard uncertainties on compatible bases combine into the validation standard uncertainty. The discrepancy is then expressed relative to it as R = |E|/uval, which reports resolving power rather than a verdict.
03

The skills, taught in order

Five skills build the validation uncertainty and the metric that uses it.

5.1 The three uncertainty sources, on one basis

A validation comparison carries three estimated standard uncertainties: unum for the unresolved numerical-error contribution, uinput for the effect of uncertain simulation inputs on the simulation result, and uD for the experimental result D. Each must share the units and quantity definition of the compared quantity, and each must be on a compatible standard-uncertainty basis, before any combination is meaningful.

Module 3 reported a GCI-based spatial-discretization uncertainty estimate using a safety factor. A raw GCI value is not inserted directly into this standard-uncertainty budget. Relating GCI to a standard uncertainty requires an explicit coverage interpretation and project-specific assumptions. Therefore, the examples in this module supply unum directly on a standard-uncertainty basis.

Under an explicitly adopted GCI coverage interpretation, a standard discretization uncertainty may be written as uh = GCI / kGCI. The appropriate coverage factor is not universal and is not derived in this module. Note also that unum may need to cover spatial and temporal discretization, iterative, and roundoff contributions, not only the mesh GCI.

5.2 Combining on a compatible basis

When the contributing errors are represented by compatible standard uncertainties and their covariance terms are zero, the variance of their sum is the sum of their variances. This produces the root-sum-square expression uval = √(unum2 + uinput2 + uD2). It is the simplified case, not a general law.

Measured experimental quantities may also be used as simulation inputs. Their shared sources can couple uD and uinput; these contributions must be represented consistently to avoid omitted covariance or double counting.

Advanced: when the terms are not independent. Propagating E = δmodel + δnum + δinput − δD in general gives

uval2 = unum2 + uinput2 + uD2 + 2cov(num, input) − 2cov(num, D) − 2cov(input, D)

The two minus signs arise because the experimental error enters E = S − D with the opposite sign to the simulation-side errors. Setting every covariance to zero recovers the root-sum-square form above.

SymbolSourceFrom
unumunresolved numerical errorsolution verification, expressed as a standard uncertainty
uinputuncertain inputspropagation of input uncertainty
uDexperimental errormeasurement uncertainty
uvalcombined validation standard uncertaintyroot sum of squares, when the three are compatible and uncorrelated

Every symbol in this table is a standard uncertainty. Combining them is only meaningful once they share that basis, their units, and their quantity definition.

5.3 What the comparison resolves

Form the course-defined diagnostic comparison ratio R = |E| / uval. R is not a standardized metric and has no universal pass/fail threshold; it is used here to organize the comparison between discrepancy magnitude and combined standard uncertainty. When R is about one or smaller, the non-model uncertainty contributions are not small relative to the discrepancy, and the comparison does not support isolating model-form error. When R is greater than one but not clearly large, no binary conclusion should be drawn. Only when |E| is much larger than uval may E be treated as an approximate estimate of model-form error under this interpretation, and this module defines no universal numerical threshold for much larger: that requires an authoritative source or a project criterion.

Standard uncertainty is not a confidence interval. Where a justified distribution and coverage factor are available, an expanded uncertainty Uval = k·uval may be formed, and then conditionally δmodel ∈ [E − Uval, E + Uval]. This is a conditional coverage interval under the adopted assumptions, and k = 2 is not universally valid. If zero lies within the interval, the comparison does not resolve nonzero model-form error at that coverage level; if zero lies outside it, the adopted uncertainty model supports a nonzero model-form discrepancy at that validation point. Neither outcome is a universal pass or fail verdict on the model.

5.4 Interpreting the result

A comparison that does not resolve a discrepancy is not evidence of zero error, and it does not bound the model-form error: the contributions can partially cancel, so a small E is consistent with a large model-form error offset by other sources. Reducing uval, by a finer mesh, better inputs, or better measurements, increases the resolving power of the comparison and can reveal a discrepancy a looser comparison could not detect.

5.5 Validation metrics

The comparison error with its validation uncertainty is one validation metric. Others compare whole distributions, such as the area between the simulated and measured cumulative distributions, capturing more than a single point. The ASME multivariate metrics extend this to several outputs at once.

Engineering connection: reporting E alongside uval, its basis, and its assumptions is what lets a reviewer judge what the comparison established for a specific quantity and condition.

04

Worked example 1: the validation uncertainty

A validation supplies three standard uncertainties on compatible bases: unum = 0.02, uinput = 0.03, and uD = 0.04, all in the units of the compared quantity. Find the combined validation standard uncertainty.

Figure 1. Three standard uncertainties on compatible bases combine as a root sum of squares, like the legs of a right triangle in higher dimensions. The result, 0.054, is a combined standard uncertainty dominated by the largest source.
  1. ProblemFind the validation uncertainty for the sources in Figure 1.
  2. Given / findunum = 0.02, uinput = 0.03, uD = 0.04, each a standard uncertainty in the units of the compared quantity. Find uval.
  3. AssumptionsThe three inputs are supplied as mutually compatible standard uncertainties, and independence is assumed for this teaching example, so every covariance term is taken as zero. No shared source is double counted.
  4. Modeluval = √(unum2 + uinput2 + uD2).
  5. Equationsuval = √(0.022 + 0.032 + 0.042)
  6. SolveSum of squares = 0.0004 + 0.0009 + 0.0016 = 0.0029. uval = √0.0029 = 0.054.
  7. CheckThe result exceeds the largest single source (0.04) but is far below their linear sum (0.09), the signature of adding variances rather than magnitudes. The experimental term dominates.
  8. ConclusionThe combined validation standard uncertainty is 0.054. A comparison discrepancy is expressed relative to this, not relative to zero. It is a standard uncertainty, not a bound, a tolerance, a confidence interval, or an accuracy level.
Result. uval = 0.054, a combined standard uncertainty.
05

Worked example 2: what does the comparison resolve?

For that validation (uval = 0.054), the comparison discrepancy is E = 0.03. Determine what this comparison resolves.

Figure 2. The discrepancy is smaller than the combined validation standard uncertainty, giving R = 0.56. The comparison does not have the resolution to isolate model-form error at this uncertainty level.
  1. ProblemDetermine what the comparison in Figure 2 resolves.
  2. Given / findE = 0.03, uval = 0.054 as a combined standard uncertainty. Find R = |E|/uval.
  3. AssumptionsThe three contributions were estimated on compatible standard-uncertainty bases and combined with zero covariance. No coverage factor or distribution has been adopted, so no probability statement is available.
  4. ModelR = |E|/uval expresses the discrepancy in units of the combined standard uncertainty. R near or below one means the non-model contributions are not small relative to the discrepancy.
  5. EquationsR = |E| / uval = 0.03 / 0.054 ≈ 0.56
  6. SolveR = 0.03 / 0.054 ≈ 0.56. The comparison discrepancy is not large relative to the combined standard uncertainty.
  7. CheckThis does not bound the model-form error at 0.054, and it is not evidence that the error is zero. The contributions can partially cancel, so a small E is consistent with a large model-form error offset by other sources. A smaller uval, from better data or a finer mesh, would increase the resolving power of the comparison.
  8. ConclusionThe comparison discrepancy is not large relative to the combined standard uncertainty. The comparison therefore does not support isolating model-form error or approximating it by E. This is a statement about the resolution of this validation comparison, not a model-level pass or validation verdict.
Result. E = 0.03, uval = 0.054, R ≈ 0.56: the comparison does not resolve model-form error at this uncertainty level.
06

Misconceptions and diagnostics

MistakeSymptomDiagnostic questionCorrection
Adding uncertainties linearlyuval too large"Are the covariance terms zero?"With compatible standard uncertainties and zero covariance, variances add, giving the root sum of squares.
Omitting a sourceuval too small"Did I include numerical, input, and data?"All three enter the validation uncertainty.
Reading a small R as zero errorZero error assumed"What did this comparison resolve?"It resolved nothing about model-form error at this uncertainty level, and does not bound it.
Judging E against zeroSmall discrepancy called meaningful"Relative to what uncertainty?"Express E relative to uval, not relative to zero.
07

Practice ladder

Level 1 · Direct skill

Combine standard uncertainties unum = 0.03 and uD = 0.04 as a root sum of squares (no input uncertainty, zero covariance).

Show answer

uval = √(0.032 + 0.042) = √0.0025 = 0.05.

Level 2 · Mixed concept

If the comparison discrepancy there is E = 0.08, what does the comparison resolve?

Show answer

R = 0.08 / 0.05 = 1.6. The discrepancy exceeds the combined standard uncertainty, but it is not much larger. Without an explicitly defined coverage criterion or stronger separation, do not interpret 0.08 as the model-form error and do not issue a pass/fail conclusion.

Level 3 · Independent problem

Standard uncertainties are unum = 0.01, uinput = 0.02, uD = 0.02, and E = 0.015. What does the comparison resolve?

Show answer

uval = √(0.0001 + 0.0004 + 0.0004) = √0.0009 = 0.03, so R = 0.015 / 0.03 = 0.5. The discrepancy is not large relative to the combined standard uncertainty, so the comparison does not resolve model-form error at this uncertainty level.

Transfer task | Real engineering

Your comparison gives a small R, but the experimental uncertainty is large. Explain why that is a weak result and how to strengthen it.

What good work looks like

A large uD inflates uval, so R is small only because the comparison is imprecise; a real discrepancy could sit undetected inside it. Reducing the measurement uncertainty tightens uval and increases the resolving power of the comparison.

08

Working with AI, and proving it yourself

Use AI as an examiner, not a solver

"Check that my three uncertainties are on compatible standard-uncertainty bases before I combine them."
"Give me three validation cases; I will compute R and say what each comparison resolves."
"Is my model validated?" Computing uval and judging what the comparison resolves is the skill.
"What is the model error?" Reasoning from E and uval is the point.

Portfolio task

For a real validation, estimate the numerical, input, and experimental standard uncertainties, state the basis of each, combine them into uval, and report E, uval, and R with a statement of what the comparison resolves for that quantity and condition.

Must include: three standard-uncertainty estimates with their bases, a uval, the ratio R, and a statement of what was and was not resolved.
09

Retrieval and spaced review

Closed notes. Answer out loud, then reveal.

1. Name the three uncertainty sources in a validation.

Numerical, input, and experimental.

2. Write the validation uncertainty.

uval = √(unum2 + uinput2 + uD2).

3. What does R = |E|/uval tell you?

How large the discrepancy is relative to the combined standard uncertainty, and therefore what the comparison can resolve. It is not a pass or fail.

4. Does a small R bound the model-form error?

No. Contributions can partially cancel, so a small E is consistent with a large model-form error offset by other sources.

5. How is the comparison made more discriminating?

By reducing uval through a finer mesh, better inputs, or better measurements.

TodayFinish this quiz and Levels 1 and 2 of the ladder.
+1 dayRe-derive uval and the verdict from a blank page.
+3 daysJudge three new validation cases.
+7 daysTurn to the uncertainties themselves, Module 6.
+30 daysReport E, uval and its basis, and what the comparison resolved.