Download Numerical Optimization: Newton Method, Random Search, and Golden Section Search and more Study notes Civil Engineering in PDF only on Docsity!
Numerical Methods Lecture 6 - Optimization
NOTE: The unit on differential equations will not be available online. We will use notes on the board only.
Topics: numerical optimization
- Newton again
- Random search
- Golden Section Search
Optimization - motivation
What?
- Locating where some function reaches a maximum or minimum
Find x where or
Why?
- For example: A function represents the cost of a project minimize the cost of the project When?
- When an exact solution is not available or a big pain
Example 1:
Given: , Find x where y is minimum
analytical solution: ==>
Example 2:
Given: , Find x where y is minimum
Depends on the range we’re interested in...
Having lots of local maxima and minima means having lots of zero slope cases. An exact solution would be a big pain...
Single variable - Newton
Recall the Newton method for finding a root of an equation
, where
We can use a similar approach to find a min or max of
The min / max occurs where the slope is zero So if we find the root of the derivative, we find the max / min location
Find roots of using Newton
Example: Newton Method
find the maximum of this function
we’ll need these
for x = [0,3]
OR
we can use central difference to replace the analytical derivatives.
to find where the max occurs...
find the root of the function’s derivative
Illustration of the code
Advantages of random search:
- Simple to implement
- Distinguishing global from local maxima is easy
Example: same equation over a larger range Finding roots of derivative (Newton) leaves us with lots of minima and maxima (if Newton can even find them all) Random Search can simply pick through and I.D. the biggest maximum Since it compares all the f(x) values anyway.
Disadvantages of Random Search:
- Brute force method - uses no special information about the function or its derivatives.
- Random number pattern may miss narrow peaks
- Becomes very slow for multi-dimensional problems if needs 1000 numbers to get a good look...
needs numbers to get a good look
Note that we’ve zoomed in
first iteration second iteration
local
local local
global? (^) global?
Single Variable - Golden Section Search Optimization Method
Similar to the bisection method
- Define an interval with a single answer (unique maximum) inside the range sign of the curvature does not change in the given range
- Divide interval into 3 sections by adding two internal points between ends
- Evaluate the function at the two internal points x1 and x if f(x1) > f(x2) the maximum is between xmn and x redefine range xmn = xmn, xmx = x if f(x1) < f(x2) the maximum is between x1 and xmx redefine range xmn = x1, xmx = xmx
- Divide new smaller interval into 3 sections and start over
Q1: Where should we place the two internal points? Let’s see if we can pick some ‘efficient’ points
Set the following conditions: (1) L = L1 + L (2) R = L/L2 = L2/L
substitute (1) into (2) ==> 1 + R = 1/R
solve for R (R is called the Golden Ratio, named by ancient Greeks)
R = (sqrt(5)-1)/2 =.
So if the range is [0 , 1]
X1 = 1 - R =.
xmn x1^ x2^ xmx xmn x1^ x2 xmx look inside red range for second iteration
L
L
L
xmn x1 x2 xmx
So if we set the tolerance to. the final x guess is within .001 of the x location of the actual maximum A few iterations
xmn (^) x1 x2 xmx xmn
xmx
x1 x
xmn x2 x
xmx
xmx
x
xmn
x
A working algorithm
GoldenSearch ( f , a, b,tol ) grat
( (^5) − 1 )
d ← grat ⋅( b − a)
x2 ← a + d
x1 ← b − d
err ← 100
numit ← 0
numit ← numit + 1
xopt ← x
err ( 1 − grat )
( b − a)
xopt
b ← x
x2 ← x
d ← grat ⋅( b − a)
x1 ← b − d
if err > tol
if f ( x1) > f ( x2)
xopt ← x
err ( 1 − grat )
( b − a)
xopt
a ← x
x1 ← x
d ← grat ⋅( b − a)
x2 ← a + d
if err > tol
otherwise
while err > tol
xopt
f ( xopt )
