VVUQ · Module 9 of 10
Model Calibration and Predictive Capability
Calibrating a model to data tunes its parameters to fit; it is not the same as validating it. The goal is predictive capability: a scoped prediction at an application point the model was not fitted to, carrying an uncertainty built from every source that matters there, not a fixed pair of terms.
Readiness check
Learning objectives
- Distinguish calibration from validation and justify why calibration agreement is not validation evidence.
- Calculate a through-origin least-squares parameter estimate and its illustrative standard uncertainty.
- Construct an application-point prediction uncertainty budget from compatible output-space or relative standard uncertainties, accounting for dependence.
- Distinguish prediction uncertainty from the signed prediction error, and standard from expanded uncertainty.
This module fits parameters and predicts. Tick only what you can do closed-notes.
- Fit a parameter by least squares.
- Combine compatible standard uncertainties as a root sum of squares.
- Distinguish fitting data from testing against data.
- Recall model-form uncertainty from earlier modules.
- Recall that predicting outside a range is extrapolation.
The core idea
Calibration tunes model parameters to fit data; it improves agreement but does not prove the model. Predictive capability is a scoped prediction at an application point, with an uncertainty budget assembled from the sources active there: numerical, input, parameter, and model-form contributions, combined only when they are compatible standard uncertainties and their dependence is accounted for.
calibration: fit parameters to datavalidation: test on data not used for tuning (Module 5)upred2 = unum2 + uinput2 + uparam2 + umodel2 + covariance termsModels almost always contain parameters not known from first principles: a friction coefficient, a heat-transfer constant, a material property. Calibration estimates these by fitting the model to data, typically by least squares. This carries a trap: a flexible model may fit calibration data well even when its physical representation is inadequate, because parameter adjustment can absorb part of the structural discrepancy. Calibration agreement alone is therefore not validation evidence. Validation uses data not used to estimate or tune the parameters, and it characterises model-form uncertainty rather than handing you the model-form error, as Module 5 set out.
The aim is predictive capability: a prediction at an application point where no tuning data exist, reported with an uncertainty scoped to that point. What validation evidence supports there depends on similarity in physics, model assumptions, influential inputs, and response regime, not only on geometric proximity in one coordinate. A prediction can sit numerically between two tested points and still lie outside the physics that validation exercised.
The uncertainty of a prediction is a budget, not a fixed pair of terms. It may include numerical uncertainty at the application calculation (Module 3), uncertainty in the application inputs propagated through the model (Module 7), the calibrated-parameter uncertainty carried into the output, and the model-form uncertainty from validation. Which terms appear is set by the prediction model and the sources represented, not by a universal list of symbols. The terms combine only when they are compatible standard uncertainties for the same predicted quantity, and only with dependence accounted for: calibration correlates the parameters and can make the parameter and model-form terms dependent, so treating them as independent is an assumption to be stated and examined, not a default.
Report what the number is. A standard prediction uncertainty upred becomes an interval only through a coverage factor kcov and a stated distribution, and the interval is not proof that the unknown signed prediction error lies inside it. Extrapolation beyond the calibration and validation evidence changes the budget and can make some terms unquantifiable, so it demands explicit justification.
The skills, taught in order
Seven skills build calibration, its distinction from validation, what validation supports at a new condition, and the uncertainty budget of a prediction.
9.1 Model calibration
Calibration estimates unknown model parameters by fitting the model to data, usually minimising the sum of squared residuals as in nonlinear least squares. The result is a best-fit parameter value with an associated uncertainty from the scatter of the fit.
Calibration is conditional on the data and the model used to fit. The resulting parameter uncertainty must be propagated through the application model into the units of the predicted output before it can join the budget (Module 7); a relative parameter uncertainty carries straight to a relative output uncertainty only for a proportional model like the one below. Fitted parameters need not be true physical constants, and a good fit does not make them so; several parameters can also compensate for one another and for model inadequacy during the fit, leaving them correlated.
9.2 Calibration is not validation
A flexible model may fit calibration data well even when its physical representation is inadequate, because parameter adjustment can absorb part of the structural discrepancy. Calibration agreement alone is therefore not validation evidence. Validation uses data not used to estimate or tune the parameters. Confusing the two, or reusing the same evidence under two labels, is a common and serious error.
| Activity | Uses data to | Tells you |
|---|---|---|
| Calibration | tune parameters | best-fit parameters, conditional on the data |
| Validation | test the tuned model on data not used for tuning | how large the discrepancy is relative to the combined uncertainty (Module 5), not the model-form error itself |
| Prediction | neither (new condition) | a scoped prediction, with an uncertainty budget |
Calibration and validation must use different evidence. Prediction applies the calibrated model, validated on separate data, to a new condition.
9.3 What validation supports at a new condition
A validation point is a specified tested condition and response quantity at which simulation and experiment are compared. The application point is the condition and response quantity for which the new prediction is required. Evidence at a validation point supports an application point only to the extent that the relevant physics, assumptions, influential inputs, and response regimes remain applicable.
Validation evidence is generated for specified quantities at specified conditions. Its relevance to an application point depends on similarity in physics, model assumptions, influential inputs, response regime, and distance from the evidence, not only geometric proximity in one coordinate. Ask whether the physical mechanisms exercised in validation are still active at the application point: a prediction that crosses a transition, a threshold, yielding, contact, a phase change, or a change of dominant physics may be numerically close to the tested conditions yet physically outside them. A model validated for one quantity is not thereby validated for every quantity, and the model-form uncertainty measured at one condition does not automatically apply at another.
9.4 The prediction uncertainty budget
Write the signed prediction error as a sum of the represented contributions,
δpred = δnum + δinput + δparam + δmodelso that, taking variances,
upred2 = Σi ui2 + 2 Σi<j cov(δi, δj)with unum the numerical uncertainty of the application calculation, uinput the application-input uncertainty propagated through the model, uparam the calibrated-parameter uncertainty in output units, and umodel the model-form uncertainty. This variance expression applies when the represented contributions are additive output-space quantities with compatible probabilistic meanings and finite second moments. The uncertainty sources active in a particular prediction define the budget; the displayed four-term list is a teaching scaffold, not a universal mandatory inventory.
Two conditions must hold before any terms are combined. Each must be a standard uncertainty for the same predicted quantity, on a compatible basis (Module 5): a GCI, an interval half-width, an RMS discrepancy, a maximum error, or a safety factor is not automatically a one-standard-deviation quantity. And their dependence must be accounted for through the covariance terms; where aleatory and epistemic contributions use incompatible representations, propagate them jointly or conditionally where justified, or report them separately rather than forcing them into one root-sum-of-squares result.
A prediction budget may be constructed entirely in output units or entirely on a relative basis. Absolute and relative uncertainty components must not be mixed in the same variance equation. The additive budget above is written in output units; to work relative to the predicted value Sapp, divide each contribution by |Sapp| first, and convert back with upred,abs = |Sapp|·upred,rel when output units are needed.
umodel deserves particular care. It characterises uncertainty about the model's inadequacy; it is not a validation discrepancy. Setting umodel = |E| or umodel = |E| − uval is not defensible, because a discrepancy is a signed comparison that can partly cancel other effects and does not bound the model-form error (Module 5). A defensible umodel needs a declared uncertainty model with a stated statistical meaning; possible approaches include a fitted discrepancy model, a multiple-model assessment, a probabilistic assessment across conditions, or structured expert judgement, but none is automatically a one-standard-deviation quantity.
9.5 Standard and expanded prediction uncertainty
upred is a standard uncertainty. An interval requires a coverage factor and a distributional statement: Upred = kcov·upred, with kcov chosen for a stated coverage, and kcov = 2 is not universally 95%. JCGM conventionally uses k for the coverage factor; this module writes kcov to distinguish it from the calibrated slope k. On a relative basis the expanded uncertainty is Upred,rel = kcov·upred,rel. For a skewed or bounded predicted quantity, empirical or propagated quantiles describe the interval better than a symmetric ± Upred. Say whether the interval is for the output quantity, for an expected value, or for an estimator, because they differ.
9.6 Prediction uncertainty is not prediction error
The signed prediction error δpred = Sapp − Tapp is the gap between the prediction and the unknown truth. It is unknown until truth is observed, and it has a sign. Prediction uncertainty upred is non-negative and quantifies doubt under the adopted model. An uncertainty interval is not a promise that δpred lies inside it, a narrower interval is not automatically more accurate, and uncertainty is easily underestimated by omitting sources or assuming independence. Report upred as doubt, never as the prediction error.
9.7 Extrapolation and its cost
Predicting beyond the calibration and validation evidence is extrapolation. Under extrapolation, prediction-uncertainty contributions may change because sensitivities, influential mechanisms, parameter effects, and discrepancy relevance change. Model-form uncertainty often becomes harder to justify and may need to be enlarged, but neither its magnitude nor its rate of growth is determined by distance alone. Interpolation and extrapolation are multidimensional: a prediction can be interpolative in one coordinate while extrapolative in another or across a regime boundary. Interpolation is not automatically safe and extrapolation is not automatically invalid; regime transitions may matter more than geometric distance, and any expansion of uncertainty with distance needs a supported model rather than an assertion.
Engineering connection: a calibrated fatigue or heat-transfer model is only trustworthy for design once it is validated on data not used for tuning, its prediction carries the budget active at the design condition, and that budget is scoped to the condition rather than assumed uniform across all operating points.
Source. This module's validation-to-prediction framing follows the point-specific V&V concepts summarised in NIST IR 8298. Its discussion of calibrated-parameter uncertainty and explicit model discrepancy is consistent with the framework introduced by Kennedy and O'Hagan. It does not implement a complete Bayesian calibration method or claim full normative compliance with ASME V&V 20.
Worked example 1: calibrating a parameter
A linear model F = k·x is calibrated to the data (1, 2.1), (2, 3.9), (3, 6.2) by least squares. Find the best-fit parameter k and its standard uncertainty.
- ProblemCalibrate the parameter k for the data in Figure 1, and find its standard uncertainty.
- Given / findF = k·x, data (1, 2.1), (2, 3.9), (3, 6.2). Find k and u(k).
- AssumptionsThe proportional model through the origin is the adopted calibration model; the xi are treated as fixed; the errors are in Fi, independent, with a common variance. Three observations give only an illustrative uncertainty estimate.
- ModelLeast squares for a proportional model: k = Σ(xiFi)/Σ(xi2). One parameter is fitted, so the residuals carry n − 1 = 2 degrees of freedom, and u(k) = √(s2/Σxi2) with s2 = SSE/(n − 1).
- EquationsΣ(xF) = 1(2.1) + 2(3.9) + 3(6.2) = 28.5Σ(x2) = 1 + 4 + 9 = 14u(k) = √(s2/Σx2)
- Solvek = 28.5/14 = 2.0357. Residuals 0.0643, −0.1714, 0.0929 give SSE = 0.04214, so s2 = 0.04214/2 = 0.02107 and u(k) = √(0.02107/14) = 0.0388, a relative uncertainty u(k)/k = 0.0191 (about 1.9%).
- CheckThe fit gives F ≈ 2.036x, close to each data point (2.04, 4.07, 6.11 versus 2.1, 3.9, 6.2). For this proportional model the relative slope uncertainty carries directly to the relative output, giving uparam = 0.0191 used in the prediction of Worked Example 2.
- ConclusionThe calibrated parameter is k = 2.036 with a relative standard uncertainty of about 1.9%. This tunes the model to the data; validating it requires data not used for tuning.
Worked example 2: the uncertainty of a prediction
Continue Worked Example 1. Propagating the calibration fit gives a relative parameter uncertainty uparam,rel = 0.0191 in the predicted output. A separate model-discrepancy assessment supplies a relative model-form uncertainty umodel,rel = 0.050 for this application condition; its detailed construction is outside this arithmetic example. Assume the numerical and application-input uncertainties are negligible here, and state that assumption. Find the prediction uncertainty and say what it is and is not.
- ProblemFind the prediction uncertainty for the case in Figure 2.
- Given / finduparam,rel = 0.0191 and umodel,rel = 0.050, both relative standard uncertainties for the same predicted quantity; unum and uinput assumed negligible. Find upred,rel.
- Assumptionsuparam,rel and umodel,rel are compatible relative standard uncertainties for the same predicted quantity, so the budget is built entirely on a relative basis. Although the calibration and validation datasets are separate, the discrepancy assessment uses predictions generated with the calibrated parameters, so the two contributions are not automatically independent; this example adopts zero covariance only as a simplifying assumption, and a real analysis would disclose it and check its effect. unum and uinput are taken as negligible at this condition.
- ModelWith zero covariance, upred,rel = √(uparam,rel2 + umodel,rel2). Convert with upred,abs = |Sapp|·upred,rel if output units are needed.
- Equationsupred,rel = √(0.01912 + 0.0502)
- Solveupred,rel = √(0.000365 + 0.0025) = √0.002865 = 0.0535, which rounds to about 5.4% as an engineering result.
- CheckThe model-form term (0.050) dominates the parameter term (0.0191), so a better model would help far more than a tighter calibration here. This is a standard uncertainty, not an interval: a 95% statement would need a coverage factor and a distribution, which this example does not adopt.
- ConclusionThe prediction carries a relative standard uncertainty of about 5.4%, combining the parameter and model-form terms under the stated zero-covariance assumption. It is not the prediction error, which stays unknown until measured, and it excludes any numerical or input contribution assumed negligible. Both included terms matter; so do the excluded ones whenever they are not.
Misconceptions and diagnostics
| Mistake | Symptom | Diagnostic question | Correction |
|---|---|---|---|
| Calibration counted as validation | Model tested on its fit data | "Was this data used to tune?" | Validate on data not used for tuning. |
| Reducing the budget to two terms | Numerical and input uncertainty dropped | "What sources are active at this point?" | Include unum and uinput when they matter; the budget is set by the problem. |
| Setting umodel to a validation discrepancy | Model-form term is really |E| | "Is this a standard uncertainty or a discrepancy?" | A discrepancy is signed and does not bound model-form error (Module 5). Use a declared uncertainty model. |
| Combining dependent terms in quadrature | Parameter and model-form treated as independent | "Did the discrepancy use the calibrated parameters?" | State the covariance assumption and check its effect; calibration links the terms. |
| Reporting upred as an interval | "± upred at 95%" with no basis | "What coverage factor and distribution?" | upred is a standard uncertainty; an interval needs kcov and a distribution. |
| Over-fitting parameters | Great fit, poor prediction | "Are there too many free parameters?" | Keep the model physically constrained. |
| Extrapolating a calibration | Confident prediction far from data | "Same mechanisms and regime as validation?" | Justify the extrapolation and re-examine every budget term; distance alone does not set them. |
Practice ladder
Calibrate F = k·x to (1, 3), (2, 5), (3, 8) by least squares.
Show answer
k = Σ(xF)/Σ(x2) = (3 + 10 + 24)/(1 + 4 + 9) = 37/14 = 2.64.
Combine a parameter uncertainty of 0.04 with a model-form uncertainty of 0.03 for a prediction.
Show answer
upred = √(0.042 + 0.032) = √0.0025 = 0.05, combined only because both are output-relative standard uncertainties for the same quantity and assumed uncorrelated.
A prediction reports only uparam = 0.03 and claims 0.03 total. What is missing, what is the honest value if umodel = 0.06, and name one further source the budget might still omit.
Show answer
The model-form uncertainty is missing. Honest upred = √(0.032 + 0.062) = √0.0045 = 0.067, more than double the reported value, and this still assumes zero covariance. The budget might also omit numerical uncertainty at the application calculation or application-input uncertainty, either of which would raise it further.
You calibrated a model on all your data and it fits well. Explain why you cannot yet trust its predictions and what to do.
What good work looks like
Fitting all the data leaves none for validation, so the good fit shows nothing about predictive capability. Reserve data not used for tuning (or gather more) to validate, and scope the application point against that evidence rather than just checking it lies within a range. Give umodel a declared uncertainty model, propagate the parameter uncertainty into output units, and report the budget active at the condition, stating any covariance assumption and what the number does not prove.
Working with AI, and proving it yourself
Use AI as an examiner, not a solver
Portfolio task
Calibrate a model on one data set, validate it on a separate set, and report a prediction at a new application point. State the application point and whether it is interpolative or extrapolative and in which variables. Build the uncertainty budget from the sources active there, each as a compatible standard uncertainty, and state your covariance assumption.
Retrieval and spaced review
Closed notes. Answer out loud, then reveal.
1. What does calibration do, and what is it conditional on?
Estimates unknown parameters by fitting the model to data; the result is conditional on that data and model.
2. Why is calibration not validation?
A flexible model can fit its calibration data even when the physics is inadequate, because parameter adjustment absorbs part of the discrepancy; validation needs data not used for tuning and characterises model-form uncertainty rather than delivering the error.
3. What does a prediction uncertainty budget contain?
The sources active at the application point, numerical, input, parameter, and model-form terms, combined only as compatible standard uncertainties with dependence accounted for. Not a fixed pair.
4. Why can't umodel be set to a validation discrepancy?
A discrepancy is signed, can partly cancel, and does not bound the model-form error; umodel needs a declared uncertainty model.
5. Is upred the prediction error?
No. The signed error Sapp − Tapp is unknown until truth is observed; upred is non-negative doubt, and its interval is not proof the error lies inside it.