Download Langranges multiplier method, Lecture Notes - Mathematics - and more Study notes Mathematical Methods in PDF only on Docsity!
C12.1B: CONTINUOUS OPTIMISATION
LECTURE 11: THE METHOD OF LAGRANGE MULTIPLIERS
RAPHAEL HAUSER MATHEMATICAL INSTITUTE, UNIVERSITY OF OXFORD
- Examples. In this lecture we will take a closer look at some examples and illustrate how optimality conditions are applied in the Method of Lagrange multipliers.
Example 1.1. Use the method of Lagrange multipliers to solve the problem
min x∈R^2
‖x‖ (1.1)
s.t.
∥x − [^01 ]∥∥ (^) ≥ 1 , ∥ ∥x − [^02 ]∥∥ (^) ≤ 1. (1.2)
x 1
x 2 3
2 1
xˆ x¯ x ˘
F
Fig. 1.1. The feasible domain F is shaded.
Clearly, the problem is equivalent to
min x∈R^2
f (x) = x^21 + x^22
s.t. g 1 (x) = x^21 + (x 2 − 1)^2 − 1 ≥ 0 , g 2 (x) = −x^21 − (x 2 − 2)^2 + 1 ≥ 0.
We have
∇f (x) =
[ (^2) x 1 2 x 2
]
∇g 1 (x) =
[ (^2) x 1 2(x 2 −1)
]
∇g 2 (x) =
[ (^) − 2 x 1 −2(x 2 −2)
]
L(x, λ) = x^21 + x^22 − λ 1
x^21 + (x 2 − 1)^2 − 1
− λ 2
−x^21 − (x 2 − 2)^2 + 1
∇xL(x, λ) =
[
2 x 1 (1 − λ 1 + λ 2 ) 2 x 2 − 2 λ 1 (x 2 − 1) + 2λ 2 (x 2 − 2)
]
The KKT conditions are the following:
2 x 1 (1 − λ 1 + λ 2 ) = 0 (1.3) 2 x 2 − 2 λ 1 (x 2 − 1) + 2λ 2 (x 2 − 2) = 0 (1.4) x^21 + (x 2 − 1)^2 − 1 ≥ 0 (1.5) −x^21 − (x 2 − 2)^2 + 1 ≥ 0 (1.6) λ 1
x^21 + (x 2 − 1) − 1
λ 2
−x^21 − (x 2 − 2)^2 + 1
λ 1 ≥ 0 (1.9) λ 2 ≥ 0. (1.10)
Let us find all the KKT points. We need to distinguish four cases: If A(x) = ∅ then (1.3),(1.4) imply x = 0, which violates (1.6). Thus, there are no KKT points that correspond to A(x) = ∅. If A = { 2 } then λ 1 = 0. (1.3),(1.4) and (1.6) imply
2 x 1 (1 + λ 2 ) = 0 (1.11) 2 x 2 + 2λ 2 (x 2 − 2) = 0 (1.12) x^21 + (x 2 − 2)^2 = 1. (1.13)
(1.11) implies that either x 1 = 0 or λ 2 = −1. The second case contradicts (1.10), so we may assume that the first case holds. But then (1.13) implies x 2 ∈ { 1 , 3 }. If x 2 = 1 then A(x) = { 1 , 2 }, which contradicts our earlier assumption. Thus, we must have x 2 = 3. But then (1.12) implies λ 2 = −3 which contradicts (1.10). Thus, there are no KKT points corresponding to A = { 2 }. If A(x) = { 1 } then λ 2 = 0. (1.3)– (1.5) become
2 x 1 (1 − λ 1 ) = 0, (1.14) 2 x 2 − 2 λ 1 (x 2 − 1) = 0, (1.15) x^21 + (x 2 − 1)^2 = 1. (1.16)
The unique solution of these equations is
xˆ =
[ 0
2
]
, λˆ =
[ 2
0
]
It is easily checked that (ˆx, λˆ) satisfies (1.3)–(1.10) and hence is a KKT point. More- over, the LICQ holds at ˆx because ∇g 1 (ˆx) =
[ 0
2
]
If A(x) = { 1 , 2 }, then (1.5) and (1.6) must hold at equality, that is,
x^21 + (x 2 − 1)^2 − 1 = 0, x^21 + (x 2 − 2)^2 − 1 = 0.
This system of equations implies x 2 = 3/2, x 1 = ±
3 /2. Let us analyse the case
x˘ =
[ 3 / 2
√ 3 / 2
]
only, as the two cases are similar. (1.3),(1.4) imply √ 3(λ 2 + 1 − λ 1 ) = 0, 3 − λ 1 − λ 2 = 0,
Thus, the set of feasible exit directions that satisfy Condition (1.7) from Lecture 10 is the empty set. This shows that the sufficient optimality conditions are satisfied at ˘x, and that this must be a strict local minimiser. Likewise, one finds that the sufficient optimality conditions hold at ¯x.
Example 1.2. Consider the minimisation problem
min − 0 .1(x 1 − 4)^2 + x^22 s.t. x^21 + x^22 − 1 ≥ 0.
(i) Does this problem have a global minimiser? (ii) Set up the KKT conditions for this problem. (iii) Find x∗^ and a vector λ∗^ of Lagrange multipliers so that (x∗, λ∗) satisfy the KKT conditions. (iv) Is the LICQ satisfied at x∗? (v) Characterise the set of feasible exit directions from x∗. (vi) Check that the sufficient optimality conditions hold at x∗^ to show that x∗^ is a local minimiser.
(i) The objective function is unbounded along the line x 2 = 0, x 1 → ∞. Thus, no global solution exists, but we can find a local minimum with the method of Lagrange multipliers.
(ii) We get
∇xL(x, λ) =
− 0 .2(x 1 −4)− 2 λx 1 2 x 2 − 2 λx 2
, ∇xxL(x, λ) =
( (^) − 0. 2 − 2 λ 0 0 2 − 2 λ
The KKT conditions are
− 0 .2(x 1 − 4) − 2 λx 1 = 0, 2 x 2 − 2 λx 2 = 0, x^21 + x^22 − 1 ≥ 0 , λ(x^21 + x^22 − 1) = 0, λ ≥ 0.
(iii) For A(x) = ∅ have λ = 0 and hence, x 1 = 4, x 2 = 0, which implies A = { 1 }, contrary to our assumption. Thus, there are no KKT points corresponding to this case. If A = { 1 } then the unique solution is x∗^ =
[ 1
0
]
, λ∗ 1 = 0.3. (iv) The LICQ holds at x∗^ because ∇g 1 (x∗) =
[ 2
0
]
(v) The set of feasible exit directions from x∗^ for which Condition (1.7) from Lecture 10 holds is
{d ∈ R^2 : d 1 = 0, d 2 6 = 0}.
(vi) For any d from that set we have
dT∇xxL(x∗, λ∗)d = ( 0 d 2 )
0 1. 4
d 2
= 1. 4 d^22 > 0.
Therefore, the sufficient optimality conditions are satisfied and x∗^ is a strict local minimiser.
Example 1.3. Consider the half space defined by H = {x ∈ Rn^ : aTx + b ≥ 0 } where a ∈ Rn^ and b ∈ R are given. Formulate and solve the optimisation problem of finding the point x in H that has the smallest Euclidean norm.
The problem is of course trivial to solve directly, but we want to see how the Lagrange multiplier approach solves the problem “blindly”. The problem is equivalent to solving
min xTx s.t. g(x) = aTx + b ≥ 0.
We may assume that a 6 = 0; otherwise the problem is trivial. The Lagrangian of this problem is
L(x, λ) = xTx − λ(aTx + b).
The gradient ∇g(x) = a is nonzero everywhere, and hence the LICQ holds at all feasible points. The KKT conditions are
x − λa = 0, (1.17) λ(aTx + b) = 0, (1.18) aTx + b ≥ 0 , (1.19) λ ≥ 0. (1.20)
If λ = 0 then x = 0, and then (1.19) implies b ≥ 0. Either this is true, and then (x, λ) = (0, 0) satisfies the KKT conditions, or else b < 0 and then λ = 0 is not a viable choice. If λ > 0, then x = λa 6 = 0, aTx+b = λ‖a‖^2 +b = 0, and then b < 0, which is either true, in which case (x, λ) =
(−b/‖a‖^2 )a, −b/‖a‖^2
satisfies the KKT conditions, or else λ > 0 is not a viable choice. Thus, we have found that both in the case b ≥ 0 and b < 0 there is exactly one point satisfying the KKT conditions, and since the KKT conditions must hold at the minimum of our optimisation problem, the resulting points must be local min- imisers. Since the problem is convex these are also the global minimisers in both cases.
Our last example illustrates that even in the case with only equality constraints the method of Lagrange multipliers is the right tool to solve constrained optimisation problems: eliminating some of the variables can lead to the wrong result.
Example 1.4. Solve the problem
min x∈R^2 x 1 + x 2
s.t. x^21 + x^22 = 1
by eliminating the variable x 2. Show that the choice of sign for a square root opera- tion during the elimination process is critical; the wrong choice leads to an incorrect
