Download Optimization Techniques, Lecture Notes- Physics - Prof IB Leader 1 and more Study notes Physics in PDF only on Docsity!
4M13 LECTURE NOTES — OPTIMIZATION TECHNIQUES
Books
- P.E. Gill, W. Murray and M.H. Wright “Practical Optimization”, Academic Press ER 115
- R.C. Johnson “Optimum Design of Mechanical Elements”, Wiley Interscience AP 190
- D.G. Luenberger “Linear and Nonlinear Programming”, Addison-Wesley ER 142
- J.N. Siddall “Optimal Engineering Design”, Marcel Dekker AP 229
Introduction
“‘What’s new?’ is an interesting and broadening eternal question, but one which, if pursued exclusively, results only in an endless parade of trivia and fashion, the silt of tomorrow. I would like, instead, to be concerned with the question ‘What is best?’, a question which cuts deeply rather than broadly, a question whose answers tend to move the silt downstream.” Robert M. Pirsig “Zen and the Art of Motorcycle Maintenance” (1974)
Mathematical optimization is the formal title given to the branch of computational science that seeks to answer the question ‘What is best?’ for problems in which the quality of any answer can be expressed as a numerical value. Such problems arise in all areas of mathematics, the physical, chemical and biological sciences, engineering, architecture, economics, and manage- ment, and the range of techniques available to solve them is nearly as wide.
The primary objective of this course is to provide a broad overview of standard optimization techniques and their application to practical problems.
Aims
When using optimization techniques, one must:
- understand clearly where optimization fits into the problem;
- be able to formulate a criterion for optimization;
- know how to simplify a problem to the point at which formal optimization is a practical proposition;
- have sufficient understanding of the theory of optimization to select an appropriate optimi- zation strategy, and to evaluate the results which it returns.
1. Definitions
The goal of an optimization problem can be stated as follows: find the combination of param- eters (independent variables) which optimize a given quantity, possibly subject to some restrictions on the allowed parameter ranges. The quantity to be optimized (maximized or minimized) is termed the objective function ; the parameters which may be changed in the quest for the optimum are called control or decision variables ; the restrictions on allowed parameter values are known as constraints.
A maximum of a function f is a minimum of − f. Thus, the general optimization problem may be stated mathematically as:
minimize , subject to , (1.1) ,.
where is the objective function, x is the column vector of the n independent control vari- ables, and is the set of constraint functions. Constraint equations of the form are termed equality constraints , and those of the form are inequality constraints. Taken together, and are known as the problem functions. If ine- quality constraints are simply a restriction on the allowed values of a control variable, e.g. minimum and maximum possible dimensions: , (1.2)
these are known as bounds.
2. Classifications
There are many optimization algorithms available to the engineer. Many methods are appro- priate only for certain types of problems. Thus, it is important to be able to recognize the char- acteristics of a problem in order to identify an appropriate solution technique. Within each class of problems there are different minimization methods, varying in computational require- ments, convergence properties, and so on. Optimization problems are classified according to the mathematical characteristics of the objective function, the constraints and the control vari- ables.
Probably the most important characteristic is the nature of the objective function. If the rela- tionship between and the control variables is of a particular form, such as linear , e.g.
, (2.1)
where b is a constant-valued vector and c is a constant, or quadratic , e.g.
, (2.2)
f ( x ) x = ( x 1 , x 2 ,... , xn ) T ci ( x ) = 0 i =1 2,..., , m ′ ci ( x ) ≥ 0 i = m ′ +1,... , m f ( x ) { ci ( x )} ci ( x ) = 0 ci ( x ) ≥ 0 f ( x ) { ci ( x )}
xi min ≤ xi ≤ xi max
f ( x )
f ( x ) = b T x + c
f ( x ) = x T A x + b T x + c
3.1 Create A Basic Configuration
To create a Basic Configuration it is usually necessary to produce a simplifying model of the problem to highlight the Decision Variables.
3.2 Identify The Decision Variables
These should be independent parameters. They can be continuous (e.g. linear dimensions, vol- ume), discrete (e.g. bolt sizes: M10, M12, etc.), or integer (e.g. number of cylinders in a piston engine).
Discrete and integer variables can often be replaced by continuous variables and the necessary adjustments made at the end of the optimization process (see Johnson for further details). We shall therefore concentrate mainly on continuous variables.
3.3 Establish The Objective Function
The most common objective function is some measure of cost and the aim is to minimize this. Other objective functions, e.g. weight, may be used, but these can usually be related to cost.
The objective function must be established in terms of the decision variables, i.e. given values for the decision variables it must be possible to calculate the objective function.
3.4 Identify Any Constraints
Not all decision (control) variables can assume values over an infinite range. Constraints are therefore applied. They can be equality constraints (e.g. fixed volume) or inequality con- straints (e.g. minimum radius, maximum height).
Equality constraints can often be incorporated directly into the objective function (by elimi- nating a variable).
Inequality constraints are more difficult to handle, but they can sometimes be left out initially and then considered when interpreting the results from candidate optima. Other approaches are discussed later (e.g. penalty functions).
3.5 Select And Apply An Optimization Method
There are two basic classes of optimization methods:
3.5.1 Optimality Criteria
Analytical methods. Once the conditions for an optimal solution are established, then either:
- a candidate solution is tested to see if it meets the conditions, or
- the equations derived from the optimality criteria are solved analytically to determine the optimal solution.
3.5.2 Search Methods
Numerical methods. An initial trial solution is selected, either using common sense or at ran- dom, and the objective function is evaluated. A move is made to a new point (2nd trial solu- tion) and the objective function is evaluated again. If it is smaller than the value for the first trial solution, it is retained and another move is made. The process is repeated until the mini- mum is found.
Search methods are used when:
- the number of variables and constraints is large;
- the problem functions (objective and constraint) are highly nonlinear; or
- the problem functions (objective and constraint) are implicit in terms of the decision/control variables — making the evaluation of derivative information difficult.
The most appropriate method will depend on the type (classification) of problem to be solved.
3.6 Example: Beer Container
Task: Design a beer container to hold 330 ml of beer.
Step 1: Create a basic configuration
Step 2: Decision Variables
4. Optimality Conditions
4.1 Definitions And Background Theory
The goal of the optimization process is to:
minimize subject to , (4.1)
where S is the set of feasible values (the feasible region ) for x defined by the constraint equa- tions (if any), i.e. the values of x for which all constraints are satisfied. Obviously, for an unconstrained problem S is infinitely large.
The gradient vector of
(4.2)
denotes the direction in which the function will increase most per unit distance travelled.
The Hessian of is an symmetric matrix giving the spatial variation of the gradient:
d is a feasible direction at x if an arbitrarily small move from x in the direction d remains in the feasible region, i.e. if there exists an such that
. (4.4)
The definition of the global minimum of is that
. (4.5)
If we can replace with then this is a strict or strong global minimum.
f ( x ) x ∈ S
f ( x )
g x ( ) ∇ f ( x ) ∂ f ∂ x 1
∂ f ∂ x 2 … ∂ f ∂ xn
T = =
f ( x ) n × n
H ( x ) ∇( ∇ f ( x ))
∂^2 f ∂ x 12
(^2) f ∂ x 1 ∂ xn
∂^2 f ∂ xn ∂ x 1
(^2) f ∂ xn^2
α 0 x + α d ∈ S ∀ 0 < α <α 0
infeasible space
constraint boundary
feasible directions
x *^ f ( x )
f ( x *) ≤ f ( y ) ∀ y ∈ S , y ≠ x * ≤ <
A relative or weak local minimum exists at if an arbitrarily small move from in any fea- sible direction results in either staying constant or increasing, i.e.
. (4.6)
If is a smooth function with continuous first and second derivatives for all feasible x , then a point is a stationary point of if
. (4.7)
Figure 4.1 illustrates the different types of stationary points for unconstrained univariate func- tions.
Figure 4.1: Types of Minima for Unconstrained Optimization Problems.
As shown in Figure 4.2, the situation is slightly more complex for constrained optimization problems. The presence of a constraint boundary, in Figure 4.2 in the form of a simple bound on the permitted values of the control variable, can cause the global minimum to be an extreme value, an extremum (i.e. an endpoint), rather than a true stationary point.
x *^ x * f ( x )
f ( x *) ≤ f ( y ) ∀ y = x *+ ε d ∈ S , y ≠ x * f ( x ) x *^ f ( x )
g x ( *) = ∇ f ( x *) = 0
x
f ( ) x weak local minimum
strong local minima
global minimum
Figure 4.4 : An Example of a Strongly Unimodal Function.
4.2 Conditions For A Minimum
A necessary condition for to be a local minimum of over S is that
(4.8)
for all possible feasible directions d at , since any move from the minimum must cause the function to increase or stay the same.
if is an interior point , a point not on the boundary of the feasible region. Clearly, this condition also applies to a maximum and to a saddle point, since the gradient at both these points is zero. Thus this condition is necessary but not sufficient for a local mini- mum.
A second-order condition is needed.
4.2.1 Sufficient Condition for a Minimum
Any continuous function can be approximated in the neighbourhood of any point by a Taylor series. This series can thus be used to establish necessary and sufficient criteria for a mini- mum.
For the single variable case:
, (4.9)
where R is the remainder of the expansion, and is small compared with the first terms of the
series, and.
Let and.
x 1
x 2 global minimum
x *^ f ( x )
∇ f ( x *)⋅ d ≥ 0 x *
∇ f ( x *)⋅ d = 0 x *
f ( ) x = f ( x *) +( x − x *) f ′ ( x *) + 12 ( x − x *)^2 f ″ ( x *) + R
f ′ = (^) dd xf f ″ d^
(^2) f d x^2
x = x *^ + d f ( ) x = f ( x *) +∆ f
If is a minimum and an interior point, then and.
Thus, from the Taylor series (neglecting R ):
.
Hence
(4.10)
is a sufficient second-order^ condition for a minimum. (If^ then it is a^ strict^ or^ strong mini- mum .)
For a function of several variables , the Taylor series has the form
. (4.11)
Let be an interior minimum, and. Then,
and. Thus, from the Taylor series (neglecting R ):
Thus, the conditions for a local minimum are:
(4.8)
(4.12)
Equation (4.12) implies that the function is convex at , and will be satisfied if the Hessian is a positive definite matrix.
4.2.2 Convex Functions
A function is said to be convex at if for all arbitrarily small the value of the function is greater than or equal to the first two terms of the Taylor expansion. If the function is convex everywhere, i.e. if for all x and y
, (4.13)
then f is called a convex function and is a global minimum.
An example of a convex function is shown in Figure 4.5.
x *^ f ′ ( x *) = 0 ∆ f ≥ 0
∆ f =^12 d^2 f ″ ( x *)
f ″ ( x *) ≥ 0
f ( x ) = f ( x *) + ∇ f ( x *) T^ ( x − x *) + 12 ( x − x *) T^ H ( x *) ( x − x *) + R
x *^ x = x *^ + d f ( x ) = f ( x *) +∆ f f ( x ) ≥ f ( x *) ∇ f ( x *)= 0
f ( x *) + ∆ f = f ( x *) +^12 d T^ H ( x *) d ∴ d T^ H ( x *) d ≥ 0
∇ f ( x *)⋅ d ≥ 0
d T^ H ( x *) d ≥ 0 x * H ( x *)
x *^ δ x = x − x *
f ( y ) ≥ f ( x ) +∇ f ( x ) T^ ( y − x )
x *
4.3 Optimality Conditions Example: Beer Container
