Download Optimization Techniques, Lecture Notes- Physics - Prof IB Leader 2 and more Study notes Physics in PDF only on Docsity!
5. Search Methods
Search methods include line and multidimensional searches. The basic idea of the methods is as follows:
- Start with a reasonable estimate for the minimum.
- Determine a search direction.
- Determine a step size.
- Get a new estimate for the minimum.
- Iterate until convergence, tested by, for example: , and/or.
With the exception of random (or pseudo-random) search methods, all search methods con- verge on a local minimum. If the function to be optimized is not unimodal, then several opti- mizations from different starting points are required if the global minimum is to be located with any certainty.
5.1 Line Searches (1-D)
The simplest and most time consuming method is to select a trial solution and keep incre- menting it by a small amount as follows:
x 0 d k αk x k + 1 = x k + αk d k f ( x k + 1 ) −f ( x k) <ε 1 x k + 1 − x k < ε 2 ∇f ( x k) <ε 3
x 0 δx
Select x 0 and δ x
Evaluate f ( x 0 )
Evaluate f ( xk +δ x)
Is f ( xk +δ x) < f ( xk)? Set xk + 1 =xk +δ x
Is δx small enough?
Set δ x = δ x ⁄ 2
yes
no yes STOP no
Figure 5.1 : An Example of Simple 1-D Line Search.
5.2 Interval Reduction
A more efficient method is by interval reduction. The aim is to keep reducing the interval I within which the minimum is known to lie until the interval is acceptably small.
Figure 5.2 : An Example of Interval Reduction Line Search.
x
f ( )x
x 0 x 1 x 2 x 3
δx
x
f ( )x
A B C D
I 1
I 2
5.2.2 Golden Section Interval Reduction
By being a little more subtle, we can increase the rate of convergence of the interval reduction method. This method makes use of the fact that we have already evaluated the function at one point in the new interval.
We position B and C such that
, i.e. , (5.5)
where d and ∆x are defined as shown in Figure 5.3. This means that one of the interior points remains an interior point for the new interval.
Figure 5.3 : An Example of Golden Section Interval Reduction Line Search.
The order of convergence is now
This order of convergence p is called the Golden Section ratio.
∆x d −∆x
d −∆x = (^) d ∆x d = 0.
x
f ( )x
A B C D
d = I 1 I 2
∆x ∆x
p
I 1
I 2
1 (^1) d = = (^) d − ∆x = 1.
5.2.3 Example: Brake with Internal Expanding Shoes
Problem: Fixing all other variables, find the brake span θ which minimises the self-actuation from the leading shoe.
Let be the moment of the frictional forces on the leading shoe and be the moment of the normal forces on the leading shoe:
Thus, we need to maximize or minimize where, substituting values:
F F
R
r
a
α 1
α 2
θ
Rotation
a = 123 mm r = 115 mm R = 150 mm pa = 1 MPa (pressure limit) μ = 0. b = 32 mm (face width)
α 1 ≈ α 2
M (^) f Mn
M (^) f= μ pa bR (^) R − Rcos θ−a 2 sin 2 θ
Mn = pa bRa θ 2 −sin 42 θ
( Mn −Mf) f( θ) =−( Mn −Mf)
f( θ ) =− 295.2 θ+ 147.8 sin 2 θ− 324.0cos θ− 132.8sin^2 θ+324.
5.3 Polynomial Fitting
Interval reduction methods are known as zero-order methods since they do not make use of the gradient of the function. To increase the search efficiency, we shall now consider fi rst- order methods which do make use of the gradient (or an approximation of it).
Any continuous function can be approximated by a polynomial, and then the minimum value can be determined analytically.
It is often sufficient to fit a quadratic (second-order polynomial). For this, we need to know the value of the objective function at 3 points.
Let the quadratic be:
, (5.7)
where , and are unknown constants. At the three known points, we get:
Hence,
(5.9)
xtrue x^ xu x
f ( )x
xl xi
f ( )x
q x( ) = a 0 + a 1 x +a 2 x^2
q x( ) = a 0 + a 1 x +a 2 x^2 a 0 a 1 a 2
f ( xl) =a 0 + a 1 x (^) l +a 2 xl^2 f ( xi) =a 0 + a 1 x (^) i +a 2 xi^2 f ( xu) =a 0 + a 1 xu +a 2 xu^2
a (^2) x^1 u −xl
f ( xu) −f ( xl) xu −x (^) l
f ( xi) −f ( xl) = − x (^) i −xl
The minimum of the quadratic equation, , is given by:
, (5.12)
subject to
. (5.13)
5.4 Newton’s Method
A further increase in efficiency can be obtained by making use of the first and second deriva- tives of the function. Such methods are called second-order methods, of which the most popu- lar ones are Newton’s method and its variants.
The underlying idea of Newton’s method is the following. From the Taylor series of :
, (5.14)
approximate as a quadratic, i.e. let.
Differentiate with respect to x to get
. (5.15)
If x is the location of the minimum of , then , and therefore
. (5.16)
Let be a candidate for the minimum ( being the previous guess), then , and hence, identifying with x and with :
, (5.17)
provided.
This can then be used as the basis for an iterative procedure, making the new value of. This iteration will refine the estimate of the minimum — unless the function is a quadratic, in which case the minimum is found immediately.
There are several variations of Newton’s method, but as they apply to functions of any dimen- sion, they will be discussed in detail in the next section on multidimensional search methods.
a 1
f ( xi) −f ( xl) = xi − xl −a 2 ( xu −xi)
a 0 =f ( xl) −a 1 xl −a 2 xl^2 x* dq dx x ( ) = 0 ⇒ x a^1 = − 2 a 2
d 2 q dx^2
( x*) > 0 ⇒ a 2 > 0
f ( )x
f ( )x = f ( )xˆ +(x −xˆ) f ′ ( )xˆ + 12 (x −xˆ)^2 f ″ ( )xˆ +R
f ( )x R = 0
f ′ ( )x = f ′ ( )xˆ +( x −xˆ) f ″ ( )xˆ f ( )x f ′ ( )x = 0
x xˆ f^ ′^ ( )xˆ f ″ ( )xˆ
xk + 1 xk f ′ ( xk + 1 ) = 0 xk + 1 xk xˆ
xk + 1 xk f ′ ( xk) f ″ ( xk)
f ″ ( xk) ≠ 0 xk + 1 xk
