Download Projection Methods for Linear Equality Constrained Problems | IOE 519 and more Study notes Systems Engineering in PDF only on Docsity!
7 Projection methods for linear equality constrained problems
7.1 Optimization over linear equality constraints
Suppose we want to solve
(P) min f (x)
s.t. Ax = b.
Assume that the problem is feasible. Then, without loss of generality matrix A has full row
rank.
The KKT conditions are necessary for this problem and are as follows:
A¯x = b
∇f (¯x) +A
T ¯π = 0.
We therefore wish to find such a KKT point (¯x, ¯π).
Suppose we are at an iterate x where Ax = b, i.e., x is a feasible point. Just like in the steepest
descent algorithm, we wish to find a direction d which is a direction of steepest descent of the
objective function, but in order to stay in the feasible region, we also need to have Ad = 0.
Therefore, the direction-finding problem takes form
min ∇f (x)
T d
s.t. d
T Id ≤ 1
Ad = 0.
The first constraint of the problem requires that the search direction d has length 1 in the Euclidean
norm. We can, however, adapt a more generalized approach and replace the Euclidean norm
‖d‖ =
d
T Id =
d
T d with a more general norm ‖d‖ Q
d
T Qd, where Q is an arbitrary
symmetric p.d. matrix. Using this general norm in the direction-finding problem, we can state the
projected steepest descent algorithm:
Step 0 Given x
0 , set k ← 0
Step 1 Solve the direction-finding problem defined at point x
k :
(DFP
x
k )^ d
k = argmin ∇f (x
k )
T d
s.t. d
T Qd ≤ 1
Ad = 0.
If ∇f (x
k )
T d
k = 0, stop. x
k is a KKT point.
Step 2 Choose stepsize α
k by performing an exact (or inexact) line search.
Step 3 Set x
k+ ← x
k
k d
k , k ← k + 1. Go to Step 1.
Notice that if Q = I and the equality constraints are absent, the above is just the steepest descent
algorithm. The choice of name “projected” steepest descent may not be apparent at this point,
but will be clarified later.
7.2 Analysis of (DFP)
Notice that the search direction at each iteration is the solution of a nonlinearly constrained opti-
mization problem (DFP x
k ) above. Note that (DFP x
k ) is a convex program, and d = 0 is a Slater
point. Therefore, the KKT conditions are necessary and sufficient for optimality. These conditions
are (we omit the superscript k for simplicity):
Ad = 0
d
T Qd ≤ 1
∇f (x) + A
T π + 2βQd = 0
β ≥ 0
β(1 − d
T Qd) = 0.
Let d solve these equations with multipliers β and π.
Proposition 7.1 x is a KKT point of (P) if and only if ∇f (x)
T d = 0, where d is the KKT point
of (DFPx).
Proof: First, suppose x is a KKT point of (P). Then there exists v such that
Ax = b, ∇f (x) + A
T v = 0.
Let d be any KKT point of (DFPx) together with multipliers π and β. Then, in particular, Ad = 0.
Therefore,
∇f (x)
T
d = −(A
T
v)
T
d = v
T
Ad = 0.
To prove the converse, suppose that d (together with multipliers π and β) is a KKT point of
(DFPx), and ∇f (x)
T d = 0. Then
0 = (∇f (x) + A
T π + 2βQd)
T d = ∇f (x)
T d + π
T Ad + 2βd
T Qd = 2β,
and so β = 0. Therefore,
Ax = b and ∇f (x) + A
T π = 0,
i.e., the point x (together with multiplier vector π) is a KKT point of (P).
Proposition 7.2 x is a KKT point of (P) if and only if β = 0, where β is the appropriate KKT
multiplier of (DFP x
Proposition 7.3 If x is not a KKT point of (P), then d is a descent direction, where d is the
KKT point of (DFP x
Proposition 7.4 The projected steepest descent algorithm has the same convergence properties and
the same linear convergence as the steepest descent algorithm. Under the same conditions as in the
steepest descent algorithm, the iterates converge to a KKT point of (P), and the convergence rate
is linear, with a convergence constant that is bounded in terms of eigenvalues identically as in the
steepest descent algorithm.
7.3 Solving (DFP
x
Approach 1 to solving DFP: solving linear equations
If ∇f (x
k )
T d
k = 0, stop. x
k is a KKT point.
Step 2 Choose stepsize α
k by performing an exact (or inexact) line search.
Step 3 Set x
k+ ← x
k
k d
k , k ← k + 1. Go to Step 1.
All properties of the projected steepest descent algorithm still hold here.
Some strategies for choosing Q at each iteration are:
• Q = I
- Q is a given matrix held constant over all iterations
• Q = ∇
2 f (x
k ) where ∇
2 f (x) is the Hessian of f (x). It is easy to show that in this case,
the variable metric algorithm is equivalent to Newton’s method with a line-search, see the
proposition below.
• Q = ∇
2 f (x
k ) + δI, where δ is chosen to be large for early iterations, but δ is chosen to
be small for later iterations. One can think of this strategy as approximating the projected
steepest descent algorithm in early iterations, followed by approximating Newton’s method
in later iterations.
Proposition 7.8 Suppose that Q = ∇
2 f (x
k ) in the variable metric algorithm. Then the direction
d in the variable metric method is a positive scalar times the Newton direction.
Proof: If Q = ∇
2 f (x
k ), then the vector
d of the variable metric method is the optimal solution of
(DFP
x
k ):
(DFP
x
k )
d = argmin ∇f (x
k )
T d
s.t. Ad = 0
d
T ∇
2 f (x
k )d ≤ 1.
The Newton direction
d for P at the point ¯x is the solution of the following problem:
(NDP
x
k ) :^
d = argmin ∇f (x
k )
T d +
1
2
d
T ∇
2 f (x
k )d
s.t. Ad = 0.
If you write down the KKT conditions for each of these two problems, you then can easily verify
that
d = γ
d for some scalar γ > 0.
