Basic of Unconstrained Optimization - Applied Math Programming | OPR 992, Study notes of Operational Research

Download Basic of Unconstrained Optimization - Applied Math Programming | OPR 992 and more Study notes Operational Research in PDF only on Docsity!

OPR 992 - Applied Mathematical Programming

Week 2

Basics of Unconstrained Optimization

OPR 992

Applied Mathematical Programming

l^ Problem Formulation Optimality Conditions Methods for SolvingUnconstrained NLPs^ OPR 992 - Applied Mathematical Programming

Problem Formulation

min

x

f^ (

x)

We are interested in finding local minima.

l^ Problem Formulation Optimality Conditions l^ Calculus as usual l^ Example 1 l^ Example 2 Methods for SolvingUnconstrained NLPs^ OPR 992 - Applied Mathematical Programming

Calculus as usual^ n

We assume that

f

is twice continuously differentiable.

n^

Let

x

∗^

be a local minimum.

n^

First-order necessary condition:

f^ (

x∗

n^

Second-order necessary condition:

2 f

(x

is positive

semidefinite.

l^ Problem Formulation Optimality Conditions l^ Calculus as usual l^ Example 1 l^ Example 2 Methods for SolvingUnconstrained NLPs^ OPR 992 - Applied Mathematical Programming

Example 1^ Find all local minima for

f^ (

x) = (

x^2

x

x^2

(^2) x 1

l^ Problem Formulation Optimality Conditions Methods for SolvingUnconstrained NLPs l^ Common Characteristics l^ Finding the right steplength l^ Newton’s Method l^ Steepest Descent l^ Quasi-Newton Methods^ OPR 992 - Applied Mathematical Programming

Methods for Solving Unconstrained

NLPs

l^ Problem Formulation Optimality Conditions Methods for SolvingUnconstrained NLPs l^ Common Characteristics l^ Finding the right steplength l^ Newton’s Method l^ Steepest Descent l^ Quasi-Newton Methods^ OPR 992 - Applied Mathematical Programming

Common Characteristics^ 1. Start at

x

0

and

k

2. Compute step direction

xk

using some method.

3. Find a steplength

α

k^

such that

x

k+

x

k^

α

∆k

xk

gives

sufficient descent.

4. Let

x

k+

x

k^

α

∆k

xk

and

k

k

5. If

f^ (

xk

, stop. Otherwise, go to step 2.

l^ Problem Formulation Optimality Conditions Methods for SolvingUnconstrained NLPs l^ Common Characteristics l^ Finding the right steplength l^ Newton’s Method l^ Steepest Descent l^ Quasi-Newton Methods^ OPR 992 - Applied Mathematical Programming

Newton’s Method^ n

Finds the roots of an equation. n To find a stationary point, we need

f^ (

x) = 0

n^

Let

xk

[∇

2 f

(x

)]k −^1

f^ (

xk

n^

If^

2 f

(x

)k is positive semidefinite, it is invertible and

xk

is

a descent direction. n^

Fast convergence near the optimum.

l^ Problem Formulation Optimality Conditions Methods for SolvingUnconstrained NLPs l^ Common Characteristics l^ Finding the right steplength l^ Newton’s Method l^ Steepest Descent l^ Quasi-Newton Methods^ OPR 992 - Applied Mathematical Programming

Steepest Descent^ n

No second derivative computations n No system of equations to solve n Let

xk

f^ (

xk

n^

Simple to implement, but not very good convergenceproperties.