Home / Courses / Optimization / Least Squares

Optimization · Module 5 of 10

Nonlinear Least Squares

Fitting a model to data is optimization with a special structure. The objective is a sum of squared residuals, and that structure lets Gauss-Newton approximate the Hessian for free, from the Jacobian alone.

Readiness check

This module fits models by minimising residuals. Tick only what you can do closed-notes.

Recall a residual as data minus model.
Solve the normal equations for a straight-line fit.
Form a Jacobian of partial derivatives.
Recall the Newton step from Module 4.
Multiply a matrix transpose by a vector.

0 or 1 weak itemsContinue with this module.

2 weak itemsRevisit least-squares regression in Numerical Methods, Module 6.

3 or more weak itemsRevisit the Newton step in Module 4.

The core idea

A least-squares problem minimises half the sum of squared residuals. Its gradient is J^Tr and its Hessian is well approximated by J^TJ, so Gauss-Newton solves J^TJ Δ = −J^Tr, using only first derivatives of the model.

minimise ½ Σ r_i(x)² = ½ ‖r(x)‖²∇f = J^Tr, ∇²f ≈ J^TJGauss-Newton: J^TJ Δ = −J^Tr

Model fitting is the most common optimization in engineering: choose parameters x so a model best matches data. The residuals r_i(x) are the mismatches, and the objective is f(x) = ½‖r(x)‖², half the sum of their squares. This structure is special. The gradient is ∇f = J^Tr, where J is the Jacobian of the residuals (their partial derivatives with respect to the parameters), and the Hessian is J^TJ plus a term involving second derivatives of the residuals. When the residuals are small or nearly linear, that second term is negligible, so J^TJ is an excellent Hessian approximation, available from first derivatives alone. The Gauss-Newton method exploits this, solving J^TJ Δ = −J^Tr for the parameter step. For a linear model the Jacobian is constant and this is exactly the normal equations, solved in one step. When Gauss-Newton takes too bold a step, Levenberg-Marquardt adds a damping term (J^TJ + λI), blending it toward steepest descent for robustness.

The skill works when: you form the residuals and Jacobian and solve the Gauss-Newton system for the step.

The skill breaks down when: residuals are large and nonlinear, so J^TJ is a poor Hessian and damping is needed.

The concept. Each residual is the vertical gap between a data point and the model. Least squares tunes the parameters to make the sum of those gaps squared as small as possible.

The skills, taught in order

Five skills build the least-squares objective, its Gauss-Newton solution, and the Levenberg-Marquardt safeguard.

5.1 The least-squares objective

Given residuals r_i(x) = model(x) − data, the objective is f(x) = ½Σr_i². Minimising it finds the parameters that best fit the data in the least-squares sense. The factor of one half makes the gradient clean.

5.2 The gradient and the Gauss-Newton Hessian

The gradient is ∇f = J^Tr, where J holds the residual derivatives. The true Hessian is J^TJ plus a residual-curvature term; dropping that term gives the Gauss-Newton approximation ∇²f ≈ J^TJ, positive semidefinite and free from second derivatives.

Quantity	Expression	Role
Objective	½‖r‖²	sum of squared residuals
Gradient	J^Tr	zero at the optimum
Hessian (approx)	J^TJ	Gauss-Newton curvature
Step	−(J^TJ)⁻¹J^Tr	Gauss-Newton update

The special structure of least squares: the Jacobian alone gives both the gradient and an approximate Hessian.

5.3 The Gauss-Newton method

Gauss-Newton solves J^TJ Δ = −J^Tr for the parameter step Δ, then updates x. It is Newton's method with the cheap J^TJ Hessian, converging fast when residuals are small. For a linear model J is constant and the step reaches the exact least-squares solution at once.

5.4 Levenberg-Marquardt damping

When Gauss-Newton overshoots, Levenberg-Marquardt solves (J^TJ + λI) Δ = −J^Tr. A small λ recovers Gauss-Newton's speed; a large λ shrinks the step toward steepest descent for safety. Adapting λ each iteration gives a robust, widely used fitter.

5.5 Model fitting in practice

Nonlinear least squares calibrates material laws, sensor curves, and dynamic models to test data. The residuals encode the fit, the Jacobian the sensitivities, and the converged parameters, with their uncertainties, are the engineering result.

Engineering connection: curve-fitting a stress-strain law or a heat-transfer correlation to measured data is a nonlinear least-squares problem solved exactly this way.

Worked example 1: a linear least-squares fit

Fit the line y = a + bx to the three points (0, 1), (1, 3), (2, 4) by minimising the sum of squared residuals. Find a, b, and the residual sum of squares.

Figure 1. The least-squares line balances the three residuals. For a linear model the normal equations give the parameters directly, no iteration needed.

ProblemFind the least-squares line for the data in Figure 1.
Given / findPoints (0, 1), (1, 3), (2, 4). Find a, b, and the residual sum of squares.
AssumptionsErrors in y only; the model is linear, so the Jacobian is constant.
ModelMinimise ½Σ(a + bx_i − y_i)²; the normal equations give b = (nΣxy − ΣxΣy)/(nΣx² − (Σx)²), a = ȳ − bx̄.
Equationsn = 3, Σx = 3, Σy = 8, Σxy = 11, Σx² = 5
Solveb = (3·11 − 3·8)/(3·5 − 9) = (33 − 24)/6 = 1.5. a = 8/3 − 1.5·1 = 1.167. Residuals: −0.167, 0.333, −0.167; sum of squares = 0.167.
CheckThe gradient J^Tr is zero here: the residuals sum to zero and are orthogonal to the x-values, the defining condition of the least-squares optimum.
ConclusionThe best-fit line is y = 1.167 + 1.5x with residual sum of squares 0.167. A linear model is solved in one step by the normal equations.

Result. a = 1.167, b = 1.5, residual sum of squares 0.167.

Worked example 2: a Gauss-Newton step

Fit the one-parameter model y = a·x to the points (1, 2), (2, 3), (3, 5). Take one Gauss-Newton step from a = 1.5 and confirm it reaches the exact least-squares value.

Figure 2. With a constant Jacobian, the Gauss-Newton step from any start reaches the exact least-squares parameter in one move, mirroring Newton on a quadratic.

ProblemTake one Gauss-Newton step for the model in Figure 2 and compare to the exact fit.
Given / findModel y = a·x, points (1, 2), (2, 3), (3, 5), start a = 1.5. Find the step and the new a.
AssumptionsResiduals r_i = a x_i − y_i; Jacobian J_i = x_i, constant in a.
ModelGauss-Newton: Δa = −J^Tr / J^TJ; a_new = a + Δa.
Equationsr = (1.5·1 − 2, 1.5·2 − 3, 1.5·3 − 5) = (−0.5, 0, −0.5)J^TJ = 1 + 4 + 9 = 14, J^Tr = −0.5 − 1.5 = −2
SolveΔa = −(−2)/14 = 0.143. a_new = 1.5 + 0.143 = 1.643. The exact least-squares value is Σxy/Σx² = 23/14 = 1.643.
CheckThe one Gauss-Newton step lands exactly on the least-squares solution, because the model is linear in a, so J is constant and J^TJ is the exact Hessian.
ConclusionGauss-Newton reaches the exact fit in one step here, the least-squares analogue of Newton solving a quadratic. For a nonlinear model it would take several damped steps.

Result. Δa = 0.143, a = 1.643, the exact least-squares value.

Misconceptions and diagnostics

Mistake	Symptom	Diagnostic question	Correction
Using the full Hessian	Needless second derivatives	"Are the residuals small?"	Gauss-Newton's J^TJ suffices when they are.
Gauss-Newton on large residuals	Steps overshoot or diverge	"Is J^TJ a good Hessian here?"	Add Levenberg-Marquardt damping (J^TJ + λI).
Confusing residual with error	Fit quality misjudged	"Is r data minus model?"	Residual is the model-to-data mismatch, not the parameter error.
Ignoring the Jacobian structure	Treating fitting as generic optimization	"Did I exploit ½‖r‖²?"	Use J^Tr and J^TJ from the least-squares form.

Practice ladder

Level 1 · Direct skill

For residuals r = (0.2, −0.1, 0.3) and Jacobian J = (1, 1, 1), find J^Tr.

Show answer

J^Tr = 0.2 − 0.1 + 0.3 = 0.4.

Level 2 · Mixed concept

With that J, find J^TJ and the Gauss-Newton step for a single parameter.

Show answer

J^TJ = 3. Δ = −J^Tr/J^TJ = −0.4/3 = −0.133.

Level 3 · Independent problem

Fit y = a·x to (1, 3), (2, 5), (3, 8) by the exact least-squares formula.

Show answer

a = Σxy/Σx² = (3 + 10 + 24)/(1 + 4 + 9) = 37/14 = 2.643.

Transfer task | Real engineering

You fit a nonlinear material law to test data and Gauss-Newton diverges. Explain what is likely wrong and how Levenberg-Marquardt helps.

What good work looks like

Large or highly nonlinear residuals make J^TJ a poor Hessian, so the Gauss-Newton step overshoots. Levenberg-Marquardt adds λI to damp the step toward steepest descent, restoring convergence, then relaxes λ as the fit improves.

Working with AI, and proving it yourself

Use AI as an examiner, not a solver

"Check that my residuals are orthogonal to the Jacobian columns at the fit."

"Give me three data sets; I will set up the Gauss-Newton step for each."

"Fit the model for me." Forming J and r and solving the step is the skill.

"Why did it diverge?" Reasoning about residual size and damping is the point.

Portfolio task

Fit a real model to measured data by Gauss-Newton or Levenberg-Marquardt, report the converged parameters, and confirm the gradient J^Tr is near zero.

Must include: the residuals, the Jacobian, the converged parameters, and a gradient check.

Retrieval and spaced review

Closed notes. Answer out loud, then reveal.

1. Write the least-squares objective.

f(x) = ½‖r(x)‖², half the sum of squared residuals.

2. Give its gradient and approximate Hessian.

∇f = J^Tr and ∇²f ≈ J^TJ.

3. Write the Gauss-Newton system.

J^TJ Δ = −J^Tr.

4. What does Levenberg-Marquardt add?

A damping term: (J^TJ + λI) Δ = −J^Tr.

5. Why does a linear model converge in one step?

Its Jacobian is constant, so J^TJ is the exact Hessian.

TodayFinish this quiz and Levels 1 and 2 of the ladder.

+1 dayRe-derive the Gauss-Newton step from a blank page.

+3 daysFit three new models by least squares.

+7 daysAdd constraints to the problem, Module 6.

+30 daysReuse least squares to calibrate a real model to data.

Textbook mapping

This module follows Nocedal and Wright, Numerical Optimization, 2nd edition. Use these references to read further.

Topic in this module	Where to read more
The least-squares objective	Nocedal & Wright, Chapter 10
The Gauss-Newton method	Nocedal & Wright, Chapter 10
Levenberg-Marquardt	Nocedal & Wright, Chapter 10

Chapter numbers refer to the 2nd edition. The least-squares methods are standard, so any recent edition will align closely.