Comparison of Methods & Implementation of Levenberg-Marquardt Algorithm for Nonlinear Leas, Study notes of Mathematical Methods for Numerical Analysis and Optimization

Download Comparison of Methods & Implementation of Levenberg-Marquardt Algorithm for Nonlinear Leas and more Study notes Mathematical Methods for Numerical Analysis and Optimization in PDF only on Docsity!

Gradient-based Methods for Optimization. Part II.^ Nathan L. Gibson^ [email protected]^ Department of MathematicsOregon State University

OSU – AMC Seminar, Nov. 2007 – p. 1

Summary from Last Time^ •^ Unconstrained Optimization^ •^ Nonlinear Least Squares^ •^ Parameter ID ProblemSample Problem:′′^ ′^ u+^ cu+^ ku^ = 0;

′ u(0) = u; u(0) = 0^0

MAssume data {u}is given for some timesj j=^

ton thej^ interval^ [0, T^ ]. Find^ x=[c, k

T^ ]such that the following objective function is minimized:^ M∑^1 f^ (x) =^2 j=

(^2) |u(t; x) − u|.j j OSU – AMC Seminar, Nov. 2007 – p. 2

Summary of Methods^ •^ Newton:N^ m(x) =^ f^ (x) +^ ∇f^ (xk^ kk^

(^1) T T^2 )(x − x) + (x^ −^ x)∇f^ (x)(x^ − k k^ k^2 x)k^

-^ Gauss-Newton:GNT^ m(x) =^ f^ (x) +^ ∇f^ (x)(x^ −k^ k^ k^

1 T^ ′T^ ′ x) + (x^ −^ x)R(x)R(x)(x^ −k k^ k^ k^2 x)k^

-^ Steepest Descent:SDT^ m(x) =^ f^ (x) +^ ∇f^ (x)k^ k^ k^

11 T^ (x − x) + (x^ −^ x)^ I(x^ −^ x)k k^ k^2 λk

-^ Levenberg-Marquardt:LMT^ m(x) =^ f^ (x)+∇f^ (x)(x−xk^ k^ kk^

“^ ” 1 T ′T^ ′)+ (x−x)^ R(x)R(x) +^ νI k k^ k^ k^2 (x−x)k^ 0 =^ ∇m(x) =⇒^ Hs=^ −∇f^ (x)k^ k^ k^ k^

OSU – AMC Seminar, Nov. 2007 – p. 4

Levenberg-Marquardt Idea^ •^ If iterate is not close enough to minimizer so thatGN does not give a descent direction, increase

ν to take more of a SD direction. • As you get closer to minimizer, decrease

ν^ to take more of a GN step.^ •^ For zero-residual problems, GN convergesquadratically (if at all)^ •^ SD converges linearly (guaranteed)

OSU – AMC Seminar, Nov. 2007 – p. 5

Step Length^ Steepest Descent Method^ •^ We define the^ steepest descent direction

to be d=^ −∇f^ (x). This defines a direction but not ak^ k step length. • We define the Steepest Descent update step to beSD s=^ λdfor some^ λkk^ kk^

>^0.

-^ We would like to choose

λso that^ f^ (x)k^ decreases sufficiently. • Could ask simply that^ f^ (x)k+

< f^ (x)k^ OSU – AMC Seminar, Nov. 2007 – p. 7

Predicted Reduction^ Consider a linear model of

f^ (x) T^ m(x) = f (x) + ∇f^ (x)(x^ −^ x).kkkk Then the^ predicted reduction

using the Steepest Descent step^ (x=^ xk+1^ k

−^ λ∇f^ (x))^ is kk pred = m(x) − m(x) =^ λ‖∇f^ (x)‖kkkk+1kk

The^ actual reduction^ in^ f

is ared = f (x)^ −^ f^ (x).kk+1^ OSU – AMC Seminar, Nov. 2007 – p. 8

Armijo Rule^ We can define a strategy for determining the steplength in terms of a sufficient decrease criteria asfollows:mLet^ λ^ =^ β, where^ β^ ∈

(^1) (0, 1) (think ) and^ m^ ≥ 2 0 is the smallest integer such that^ ared > α pred, where^ α^ ∈^ (0,^ 1).

OSU – AMC Seminar, Nov. 2007 – p. 10

Line Search^ •^ The^ Armijo Rule^ is an example of a line search:Search on a ray from

xin direction of locallyk^ decreasing f. • Armijo procedure is to start with^ m^ = 0^ thenincrement m until sufficient decrease is achieved,m (^2) i.e., λ = β= 1, β, β,... • This approach is also called “backtracking” orperforming “pullbacks”. • For each m a new function evaluation is required.^ OSU – AMC Seminar, Nov. 2007 – p. 11

Damped Gauss-Newton Step^ Thus the step for Damped Gauss-Newton isDGN^ s

mGN=^ βd where^ β^ ∈^ (0,^ 1)^ and^ m^

is the smallest non-negative integer to guarantee sufficient decrease.

OSU – AMC Seminar, Nov. 2007 – p. 13

Levenberg-Marquardt-Armijo ′^ •^ If^ R(x)^ does not have full column rank, or if the′T^ ′matrix^ R(x)R(x

)^ may be ill-conditioned, youshould be using Levenberg-Marquardt. • The LM direction is a descent direction. • Line search can be applied. • Can show that if ν=^ O(‖R(x)‖)^ then LMAk^ k converges quadratically for (nice) zero residualproblems.^ OSU – AMC Seminar, Nov. 2007 – p. 14

0.8^1 1.2^ 1.4^ 1.

1.8^2 Search Direction 1.8 1.6 1.4 k 1.2 (^1) 0.8 0.6^ c Gauss−Newton Steepest Descent Levenberg−Marquardt^ OSU – AMC Seminar, Nov. 2007 – p. 16

1 1.1^ 1.2^ 1.3^ 1.4^ 1.^

1.6^ 1.7^ 1. Iteration history Gauss−Newton wAR 1.7 Steepest Descent wAR Levenberg−Marquardt wAR 1.6 1.5 1.4 k 1.3 1.2 1.1 1 c OSU – AMC Seminar, Nov. 2007 – p. 17

Gauss−Newton with Armijo rule 210 010 −2 10 −4 (^10) Gradient Norm−6 10 −8 100 1 2 3 4 5 Iterations

Gauss−Newton with Armijo rule 510 Iterations^ Pullbacks 010 −5 (^10) Function Value−10 10 −15 100 1 2 3 4 5 Iterations Steepest Descent with Armijo rule1.7 10 1.6 10 1.5 (^10) Gradient Norm1.4 10 0 2 4 6 8 10 Iterations

Steepest Descent with Armijo rule 410 Iterations^ Pullbacks (^310210) Function Value 110 0100 2 4 6 8 10 Iterations^ OSU – AMC Seminar, Nov. 2007 – p. 19

Word of Caution for LM^ •^ Note that blindly increasing

ν^ until a sufficient decrease criteria is satisfied is NOT a good idea(nor is it a line search). • Changing^ ν^ changes direction as well as steplength. • Increasing^ ν^ does insure your direction isdescending. • But, increasing^ ν^ too much makes your steplength small.

OSU – AMC Seminar, Nov. 2007 – p. 20