Multidimensional Gradient Methods - Numerical Methods - Lecture Slides, Slides of Mathematical Methods for Numerical Analysis and Optimization

Download Multidimensional Gradient Methods - Numerical Methods - Lecture Slides and more Slides Mathematical Methods for Numerical Analysis and Optimization in PDF only on Docsity!

Multidimensional Gradient Methods in

Optimization

Multidimensional Gradient Methods

-Overview

• Use information from the derivatives of the

optimization function to guide the search

• Finds solutions quicker compared with

direct search methods

• A good initial estimate of the solution is

required

• The objective function needs to be

differentiable

Gradients-Example

Calculate the gradient to determine the direction of the

steepest slope at point (2, 1) for the function

Solution: To calculate the gradient we would need to

calculate

which are used to determine the gradient at point (2,1) as

2 2

f x , y = x y

2 2

xy

x

f

2 2

x y

y

f

∇ f = 4 i + 8 j

Hessians

• The Hessian matrix or just the Hessian is the

Jacobian matrix of second-order partial derivatives

of a function.

• The determinant of the Hessian matrix is also

referred to as the Hessian.

• For a two dimensional function the Hessian matrix

is simply

2

2 2

2

2

2

y

f

y x

f

x y

f

x

f

H

Hessians-Example

Calculate the hessian matrix at point (2, 1) for the function

Solution: To calculate the Hessian matrix; the partial

derivatives must be evaluated as

resulting in the Hessian matrix

2 2

f x , y = x y

2 2 ( 1 ) 4

2 2 2 2

2 = = = ∂

∂ y x

f 2 2 ( 2 ) 8

2 2 2

2 = = = ∂

∂ x y

f (^) 4 4 ( 2 )( 1 ) 8

2 2 = = = ∂ ∂

∂

∂ ∂

∂ xy y x

f

x y

f

 

  



























∂

∂

∂ ∂

∂

∂ ∂

∂

∂

∂

= 8 8

4 8

2

2 2

2

2

2

y

f

y x

f

x y

f

x

f

H

Steepest Ascent/Descent Method

• Starts from an initial point and looks for a

local optimal solution along a gradient.

• The gradient at the initial solution is

calculated.

• A new solution is found at the local

optimum along the gradient

• The subsequent iterations involve using the

local optima along the new gradient as the

initial point.

Solution

Iteration 1: To calculate the gradient; the partial derivatives

must be evaluated as

Now the function can be expressed along the direction

of gradient as

x

x

f = 2 = 2 ( 1 )= 2

∂

∂ y y

f

∇ f = 4 i + 2 j

f ( x , y )

, ( 2 4 , 1 2 ) ( 2 4 ) ( 1 2 ) 2 ( 2 4 ) 4

2 2 (^0 0) = + + = + + + + + + 

 

 



∂

∂

∂

∂

• h f h h h h h y

f h y x

f f x

( ) 20 28 13

2 g h = h + h +

Solution Cont.

Iteration 1 continued:

This is a simple function and it is easy to determine

by taking the first derivative and solving for its roots.

This means that traveling a step size of along the

gradient reaches a minimum value for the function in this

direction. These values are substituted back to calculate a

new value for x and y as follows:

1. 7

h =−

h =− 0. 7

1 2 ( 0. 7 ) 0. 4

2 4 ( 0. 7 ) 0. 8

= + − = −

= + − = −

y

x

Note that f (^2 ,^1 )^ =^13 f^ (−^0.^8 ,−^0.^4 )^ =^3.^2

Solution Cont.

Iteration 3: The new initial point is .We calculate the

gradient at this point as

= 2 + 2 = 2 (− 1 )+ 2 = 0 ∂

∂ x x

f (^) = 2 = 2 ( 0 )= 0 ∂

∂ y y

f

∇ f = 0 i + 0 j

This indicates that the current location is a local optimum

along this gradient and no improvement can be gained by

moving in any direction. The minimum of the function is

at point (-1,0).