9.1
Surrogates, expensive simulations, and design of experiments
Why this lesson matters
Optimization and uncertainty studies may require thousands of model evaluations, while one high-fidelity CFD or FEA run may take hours.
Learning objectives
- Define and distinguish Surrogate model and Design of experiments.
- Apply the lesson method to the worked surrogates, expensive simulations, and design of experiments case.
- Evaluate evidence, uncertainty, and AI-assisted output before making a claim.
Readiness check
Before continuing, explain what decision this topic supports and name one upstream source that must be controlled.
Check your response
A sound answer names a specific engineering decision, its configuration, and a controlled requirement, model, dataset, interface, or standard that constrains the work.
Core idea
A surrogate approximates a defined input-output map over a stated domain. Its training design must cover the variables, constraints, nonlinearities, and decision regions that matter. A fast fit to poorly chosen data is not a useful engineering model.
Key concepts
| Surrogate model | A computationally cheaper approximation of a more expensive model, experiment, or process. |
|---|
| Design of experiments | A planned set of input combinations chosen to learn effects and interactions efficiently. |
|---|
| Training domain | The bounded input region represented by the training design. |
|---|
| Quantity of interest | The output used for prediction, constraint, risk, or decision. |
|---|
Step-by-step explanation
- Define intended use, inputs, outputs, domain, constraints, and acceptable error.
- Screen variables using physics and sensitivity, not convenience alone.
- Choose a space-filling, factorial, adaptive, or physics-informed design appropriate to dimension and purpose.
- Run and verify the source simulations before fitting.
- Reserve independent validation cases, especially near boundaries and active constraints.
Worked example
A thermal model predicts maximum component temperature T = 55 + 18 q - 9 v + 2 q² + 1.5 qv, where q is normalized heat load and v is normalized coolant flow. Each high-fidelity run costs 40 minutes.
- 1
Define q and v ranges and the safety constraint T <= 85 °C.
- 2
Select points that cover corners, center, and interior rather than a one-dimensional sweep.
- 3
Keep several boundary and near-85 °C cases out of training for validation.
- 4
Record solver verification and convergence status so failed runs are not learned as physics.
Result. The experiment is designed around the constraint surface and interaction qv, not around maximizing the number of easy simulations.
Independent check. Training and validation points cover the intended domain, active constraints, and important interactions without data leakage.
Common misconceptions
| Misconception | Correction |
|---|
| More training points always fix a surrogate | Poor domain definition, biased design, solver errors, missing variables, or wrong model class can persist with more data. |
|---|
| A tool output closes the question | A result remains a candidate until its inputs, method, configuration, uncertainty, and relevance have been checked. |
|---|
Diagnostic questions
Why reserve boundary cases?
Optimization and feasibility decisions often occur near constraints, where independent error evidence matters most.
What would make this work reproducible?
Controlled inputs, method or code, versions, assumptions, outputs, and a stated interpretation tied to the decision.
Practice ladder
BasicIdentify inputs, output, domain, and constraint in the thermal example.
IntermediatePropose a 12-point design and justify boundary coverage.
AdvancedDesign an adaptive rule that adds points where prediction uncertainty and decision sensitivity are both high.
AI-assisted engineering task
Ask AI to critique a proposed DOE for missing corners, interactions, constraints, and physical symmetries. Require it to reference the supplied domain only.
How to prove the AI output yourself
- Plot the design in every pair of important variables.
- Check source-run convergence and configuration.
- Measure validation coverage near decision boundaries.
Retrieval and spaced review
Answer closed-notes today, then again after 1, 3, 7, and 30 days.
Define Surrogate model.
A computationally cheaper approximation of a more expensive model, experiment, or process.
What role does Design of experiments play here?
A planned set of input combinations chosen to learn effects and interactions efficiently.
What must a reviewer be able to reconstruct?
Training and validation points cover the intended domain, active constraints, and important interactions without data leakage.
End-of-lesson summary
A surrogate approximates a defined input-output map over a stated domain. Its training design must cover the variables, constraints, nonlinearities, and decision regions that matter. A fast fit to poorly chosen data is not a useful engineering model.
Student notes
Write intended use, domain, variables, constraints, source-model status, and validation split before fitting.
Recommended readings
Instructor notes
Use a DOE that has many points but misses one corner. Density and coverage are different concepts.
9.2
Gaussian process regression for uncertainty-aware prediction
Why this lesson matters
A surrogate should express where its prediction is supported and where data are sparse. Gaussian processes provide a probabilistic model, but their uncertainty is conditional on kernel, noise, and modeling assumptions.
Learning objectives
- Define and distinguish Gaussian process and Kernel.
- Apply the lesson method to the worked gaussian process regression for uncertainty-aware prediction case.
- Evaluate evidence, uncertainty, and AI-assisted output before making a claim.
Readiness check
Before continuing, explain what decision this topic supports and name one upstream source that must be controlled.
Check your response
A sound answer names a specific engineering decision, its configuration, and a controlled requirement, model, dataset, interface, or standard that constrains the work.
Core idea
A Gaussian process defines a distribution over functions through a mean and covariance kernel. Conditioning on training observations gives a posterior predictive mean and variance. The kernel encodes assumptions about smoothness, scale, anisotropy, and correlation.
Key concepts
| Gaussian process | A stochastic-process model whose finite collections have a joint Gaussian distribution. |
|---|
| Kernel | A covariance function encoding how outputs co-vary with input separation and structure. |
|---|
| Length scale | A kernel parameter controlling how rapidly the modeled function can vary with an input. |
|---|
| Predictive standard deviation | Conditional spread from the GP model, not automatically total engineering uncertainty. |
|---|
Step-by-step explanation
- Scale inputs and define a mean or normalization consistent with physics.
- Choose a kernel whose smoothness and anisotropy assumptions are defensible.
- Represent observation or simulation noise using alpha or an explicit noise kernel as appropriate.
- Fit hyperparameters while checking bounds, multiple optima, and numerical conditioning.
- Inspect predictive mean and standard deviation with held-out errors and physical constraints.
Worked example
A one-dimensional simulator gives y = x sin(x) at x = [1, 3, 5, 6, 7, 8]. Fit a GP with a constant times RBF kernel plus white noise and predict across [0, 10].
- 1
Normalize or scale x and y consistently.
- 2
Use ConstantKernel * RBF + WhiteKernel and several optimizer restarts.
- 3
Request both mean and predictive standard deviation.
- 4
Expect uncertainty to grow away from observations, especially near 0, 10, and large gaps.
- 5
Compare predictions with held-out simulator evaluations before interpreting uncertainty as calibrated.
Result. The GP returns a smooth posterior mean and location-dependent standard deviation. Both depend materially on kernel and noise assumptions.
Independent check. Training data, scaling, kernel, hyperparameter bounds, optimizer outcome, and held-out coverage are recorded.
Common misconceptions
| Misconception | Correction |
|---|
| GP standard deviation is the true model error | It is conditional on the GP model, data, kernel, noise, and hyperparameter treatment and may be miscalibrated. |
|---|
| A tool output closes the question | A result remains a candidate until its inputs, method, configuration, uncertainty, and relevance have been checked. |
|---|
Diagnostic questions
What does the kernel contribute?
Prior assumptions about similarity, smoothness, amplitude, periodicity, and anisotropy.
What would make this work reproducible?
Controlled inputs, method or code, versions, assumptions, outputs, and a stated interpretation tied to the decision.
Practice ladder
BasicExplain the roles of mean, kernel, length scale, and noise.
IntermediatePredict how an excessively long length scale affects fit and uncertainty.
AdvancedCompare RBF and Matérn kernels for a response with a sharp but continuous transition.
AI-assisted engineering task
Ask AI to explain a fitted kernel in engineering language and flag hyperparameters at bounds, without declaring model adequacy.
How to prove the AI output yourself
- Inspect the fitted kernel and log-marginal likelihood.
- Test multiple initializations and kernels.
- Evaluate held-out residuals and interval coverage.
- Check physical limits.
Retrieval and spaced review
Answer closed-notes today, then again after 1, 3, 7, and 30 days.
Define Gaussian process.
A stochastic-process model whose finite collections have a joint Gaussian distribution.
What role does Kernel play here?
A covariance function encoding how outputs co-vary with input separation and structure.
What must a reviewer be able to reconstruct?
Training data, scaling, kernel, hyperparameter bounds, optimizer outcome, and held-out coverage are recorded.
End-of-lesson summary
A Gaussian process defines a distribution over functions through a mean and covariance kernel. Conditioning on training observations gives a posterior predictive mean and variance. The kernel encodes assumptions about smoothness, scale, anisotropy, and correlation.
Student notes
Always record GP mean, kernel, scaling, noise model, hyperparameter bounds, seed, software version, and validation results.
Recommended readings
Instructor notes
Show two GPs with similar mean error but very different interval calibration. Uncertainty quality is a separate learning target.
9.3
Surrogate validation, extrapolation, and credibility
Why this lesson matters
Optimization can exploit tiny surrogate errors and drive designs into sparse or extrapolated regions that look artificially favorable.
Learning objectives
- Define and distinguish Interpolation and Extrapolation.
- Apply the lesson method to the worked surrogate validation, extrapolation, and credibility case.
- Evaluate evidence, uncertainty, and AI-assisted output before making a claim.
Readiness check
Before continuing, explain what decision this topic supports and name one upstream source that must be controlled.
Check your response
A sound answer names a specific engineering decision, its configuration, and a controlled requirement, model, dataset, interface, or standard that constrains the work.
Core idea
Validate a surrogate for its intended decisions using independent cases, boundary coverage, residual diagnostics, interval calibration, and constraint-classification performance. Distance from training data and physical-domain limits must be monitored explicitly.
Key concepts
| Interpolation | Prediction inside a region supported by training data, though not necessarily close in high dimensions. |
|---|
| Extrapolation | Prediction outside the supported data domain or beyond the learned physical regime. |
|---|
| Interval coverage | Fraction of validation observations contained in stated predictive intervals. |
|---|
| Constraint classification | Correct identification of feasible and infeasible cases near a decision boundary. |
|---|
Step-by-step explanation
- Freeze the surrogate before using the independent validation set.
- Evaluate RMSE, maximum error, bias, and residual structure at representative and boundary cases.
- Check predictive-interval coverage and sharpness, not width alone.
- Measure false-feasible and false-infeasible classifications near constraints.
- Define trust-region, retraining, or high-fidelity confirmation rules for optimization use.
Worked example
A temperature surrogate has validation errors [0.4, -0.8, 1.1, 2.6, -3.0] °C. The last two cases are near the 85 °C constraint and both are falsely predicted feasible.
- 1
Compute RMSE = sqrt((0.16+0.64+1.21+6.76+9.00)/5) = 1.885 °C.
- 2
Global RMSE hides the decision-critical false-feasible errors.
- 3
Report maximum absolute error 3.0 °C and constraint confusion near 85 °C.
- 4
Add training points or a conservative constraint margin and require high-fidelity confirmation before release.
Result. The surrogate is inadequate for unqualified feasibility decisions even if 1.89 °C RMSE seems acceptable for visualization.
Independent check. Metrics are tied to use, validation cases are independent, and constraint errors are evaluated in physical units and consequence.
Common misconceptions
| Misconception | Correction |
|---|
| Low average error means safe optimization | Rare or structured errors near active constraints can dominate decisions. |
|---|
| A tool output closes the question | A result remains a candidate until its inputs, method, configuration, uncertainty, and relevance have been checked. |
|---|
Diagnostic questions
How can extrapolation be detected?
Use domain bounds, scaled distance or density, kernel support, physics checks, and explicit out-of-domain flags.
What would make this work reproducible?
Controlled inputs, method or code, versions, assumptions, outputs, and a stated interpretation tied to the decision.
Practice ladder
BasicRecompute RMSE and maximum error.
IntermediateBuild a feasible-infeasible confusion table.
AdvancedDesign a trust-region and high-fidelity confirmation policy for surrogate-based optimization.
AI-assisted engineering task
Ask AI to draft a validation summary from computed metrics, forcing separate statements for average error, worst case, interval coverage, and constraint performance.
How to prove the AI output yourself
- Recalculate metrics from predictions.
- Check validation independence.
- Plot error versus inputs and distance.
- Confirm candidate optima with the source model.
Retrieval and spaced review
Answer closed-notes today, then again after 1, 3, 7, and 30 days.
Define Interpolation.
Prediction inside a region supported by training data, though not necessarily close in high dimensions.
What role does Extrapolation play here?
Prediction outside the supported data domain or beyond the learned physical regime.
What must a reviewer be able to reconstruct?
Metrics are tied to use, validation cases are independent, and constraint errors are evaluated in physical units and consequence.
End-of-lesson summary
Validate a surrogate for its intended decisions using independent cases, boundary coverage, residual diagnostics, interval calibration, and constraint-classification performance. Distance from training data and physical-domain limits must be monitored explicitly.
Student notes
Report surrogate validity by domain, quantity, decision, and metric. Never write 'accurate surrogate' without qualifiers.
Recommended readings
Instructor notes
Create one surrogate that passes RMSE but fails the decision. This is the central credibility lesson.
9.4
Reduced-order models and when not to approximate
Why this lesson matters
Surrogates and reduced-order models are often grouped together, but they preserve different structure and fail in different ways.
Learning objectives
- Define and distinguish Reduced-order model and Basis.
- Apply the lesson method to the worked reduced-order models and when not to approximate case.
- Evaluate evidence, uncertainty, and AI-assisted output before making a claim.
Readiness check
Before continuing, explain what decision this topic supports and name one upstream source that must be controlled.
Check your response
A sound answer names a specific engineering decision, its configuration, and a controlled requirement, model, dataset, interface, or standard that constrains the work.
Core idea
A reduced-order model projects or compresses a high-dimensional dynamical or field model while retaining selected structure. A response surrogate learns an input-output map. Both require purpose-specific validation, regime awareness, and error control.
Key concepts
| Reduced-order model | A lower-dimensional approximation that retains selected structure of a full-order model. |
|---|
| Basis | A set of modes or vectors used to represent high-dimensional states compactly. |
|---|
| Projection | Mapping governing equations or states into the reduced space. |
|---|
| Regime change | A qualitative shift such as separation, contact, buckling, phase change, or control saturation that can invalidate an approximation. |
|---|
Step-by-step explanation
- Choose the target outputs, transient behavior, and parameter domain.
- Generate representative snapshots or data with verified full-order models.
- Construct and truncate a basis using energy and decision-sensitive criteria.
- Validate state reconstruction and output prediction on unseen trajectories.
- Detect regime changes and fall back to higher fidelity when approximation assumptions fail.
Worked example
A thermal ROM uses modes from steady and slow transients between 20 °C and 60 °C. It is proposed for a boiling onset transient at 110 °C.
- 1
The new case lies outside the temperature and physics regime used to construct the basis.
- 2
Boiling introduces phase-change and heat-transfer behavior absent from the snapshots.
- 3
A small projection residual alone would not prove physical adequacy if the full-order model itself lacks boiling.
- 4
Reject use, expand the governing model and snapshot domain, then rebuild and validate.
Result. Approximation is inappropriate because both training regime and underlying physics change.
Independent check. The applicability detector considers physical regimes, not only numeric input bounds.
Common misconceptions
| Misconception | Correction |
|---|
| A ROM is always physics-based and trustworthy | It inherits full-model assumptions and introduces reduction error; data selection and regime coverage remain critical. |
|---|
| A tool output closes the question | A result remains a candidate until its inputs, method, configuration, uncertainty, and relevance have been checked. |
|---|
Diagnostic questions
When should a surrogate not be used?
When data are insufficient, regimes change, extrapolation dominates, consequence is high without confirmation, or the source model itself is not credible.
What would make this work reproducible?
Controlled inputs, method or code, versions, assumptions, outputs, and a stated interpretation tied to the decision.
Practice ladder
BasicDistinguish a polynomial response surface, GP, and projection ROM.
IntermediateDesign validation cases for a vibration ROM across speed and damping.
AdvancedDefine fallback triggers for contact, buckling, phase change, or controller saturation.
AI-assisted engineering task
Ask AI to list possible regime changes for a stated system, then require physics-owner review and evidence for each trigger.
How to prove the AI output yourself
- Compare with full-order cases.
- Check conservation and stability.
- Test outside nominal trajectories.
- Verify fallback behavior.
Retrieval and spaced review
Answer closed-notes today, then again after 1, 3, 7, and 30 days.
Define Reduced-order model.
A lower-dimensional approximation that retains selected structure of a full-order model.
What role does Basis play here?
A set of modes or vectors used to represent high-dimensional states compactly.
What must a reviewer be able to reconstruct?
The applicability detector considers physical regimes, not only numeric input bounds.
End-of-lesson summary
A reduced-order model projects or compresses a high-dimensional dynamical or field model while retaining selected structure. A response surrogate learns an input-output map. Both require purpose-specific validation, regime awareness, and error control.
Student notes
Add an applicability statement and fallback rule to every approximate-model report.
Recommended readings
Instructor notes
Do not reduce the distinction to 'ROM has equations, surrogate has data'. Hybrid and non-intrusive ROMs exist. Focus on retained structure and validation.
LAB 9
Lab 9: Train and validate a Gaussian process surrogate
Lab objective
Train a GP on a small thermal simulation design, then validate mean error, interval coverage, and constraint classification on held-out cases.
Engineering context
The teaching simulator maps normalized heat load and coolant flow to maximum temperature. The 85 °C threshold defines feasibility.
Input data
- A 5 by 5 candidate design
- A deterministic thermal teaching function
- A seeded training-validation split
Step-by-step task
- Generate source-model data
- Fit ConstantKernel * RBF + WhiteKernel
- Predict mean and standard deviation
- Compute RMSE, 95% interval coverage, and false-feasible count
Python code
import numpy as np
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import ConstantKernel, RBF, WhiteKernel
def thermal_model(X):
q, v = X[:, 0], X[:, 1]
return 55.0 + 18.0*q - 9.0*v + 2.0*q**2 + 1.5*q*v
grid = np.array([(q, v) for q in np.linspace(0, 1, 5)
for v in np.linspace(0, 1, 5)])
rng = np.random.default_rng(21)
order = rng.permutation(len(grid))
train, valid = grid[order[:16]], grid[order[16:]]
y_train, y_valid = thermal_model(train), thermal_model(valid)
kernel = ConstantKernel(1.0, (1e-2, 1e3)) * RBF([0.4, 0.4], (1e-2, 10.0)) + WhiteKernel(1e-5, (1e-8, 1.0))
gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True,
n_restarts_optimizer=5, random_state=21)
gp.fit(train, y_train)
mean, std = gp.predict(valid, return_std=True)
rmse = np.sqrt(np.mean((mean - y_valid)**2))
coverage = np.mean(np.abs(mean - y_valid) <= 1.96*std)
false_feasible = np.sum((mean <= 85.0) & (y_valid > 85.0))
print("kernel:", gp.kernel_)
print(f"RMSE={rmse:.4f} degC coverage={coverage:.3f} false_feasible={false_feasible}")
Explanation of code
Step 1 generate source-model data Step 2 fit ConstantKernel * RBF + WhiteKernel Step 3 predict mean and standard deviation Step 4 compute RMSE, 95% interval coverage, and false-feasible count
Expected output
A fitted kernel, low interpolation RMSE for this smooth teaching function, reported interval coverage, and a false-feasible count. Exact hyperparameters may vary by library version.
Interpretation
The example validates the surrogate only on a tiny deterministic teaching domain. Real use needs source-model VVUQ, richer DOE, noise treatment, and decision-specific thresholds.
Common errors
- Tuning on the validation set
- Assuming GP intervals are automatically calibrated
- Ignoring false-feasible predictions
Extension tasks
- Add noisy simulation output
- Compare Matérn and RBF kernels
- Use adaptive sampling near 85 °C
- Confirm the predicted optimum with the source model
Reflection questions
- What did the held-out set test?
- Why can exact coverage be unstable with nine cases?
- Which metric matters most for a feasibility decision?
PROJECT
Mini-project 3: Credible surrogate for an engineering model
Deliverable
A DOE, verified source dataset, GP or justified alternative, validation report, applicability statement, constraint test, and retraining rule.
Required checks
Independent validation, physical-unit errors, worst-case error, interval assessment, extrapolation detection, and one high-fidelity confirmation.