Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Prof. Devilaal Chandra delivered this lecture for Convex Optimization course at Alagappa University. Its main points are: Equality, Constrained, Minimization, Newton, Method, Implementation, Quadratic, KKT, Matrix
Typology: Slides
1 / 19
This page cannot be seen from the preview
Don't miss anything!
Convex Optimization — Boyd & Vandenberghe
equality constrained minimization
eliminating equality constraints
Newton’s method with equality constraints
infeasible start Newton method
implementation
11–
minimize
f
x
subject to
Ax
b
f
convex, twice continuously differentiable
p
×
n
with
rank
p
we assume
p
⋆
is finite and attained
optimality conditions:
x
⋆
is optimal iff there exists a
ν
⋆
such that
f
x
⋆
T
ν
⋆
Ax
⋆
b
Equality constrained minimization
11–
represent solution of
x
Ax
b
as
x
Ax
b
F z
x
z
n
−
p
x
is (any) particular solution
range of
n
×
(
n
−
p
)
is nullspace of
rank
n
p
and
reduced or eliminated problem
minimize
f
F z
x
an unconstrained problem with variable
z
n
−
p
from solution
z
⋆
, obtain
x
⋆
and
ν
⋆
as
x
⋆
F z
⋆
x,
ν
⋆
T
−
1
f
x
⋆
Equality constrained minimization
11–
docsity.com
example:
optimal allocation with resource constraint
minimize
f
1
x
1
f
2
x
2
f
n
x
n
subject to
x
1
x
2
x
n
b
eliminate
x
n
b
x
1
x
n
−
1
i.e.
, choose
x
be
n
T
n
×
(
n
−
reduced problem:
minimize
f
1
x
1
f
n
−
1
x
n
−
1
f
n
b
x
1
x
n
−
1
(variables
x
1
x
n
−
1
Equality constrained minimization
11–
λ
x
x
T nt
2
f
x
x
nt
1
/
2
f
x
T
x
nt
1
/
2
properties
gives an estimate of
f
x
p
⋆
using quadratic approximation
f
f
x
inf
Ay
=
b
f
y
λ
x
2
directional derivative in Newton direction:
d
dt
f
x
t
x
nt
t
=
λ
x
2
in general,
λ
x
f
x
T
2
f
x
−
1
f
x
1
/
2
Equality constrained minimization
11–
given
starting point
x
∈
dom
f
with
Ax
=
b
, tolerance
ǫ >
0
.
repeat
∆
x
nt
,
λ
(
x
)
.
Stopping criterion.
quit
if
λ
2
/
2
≤
ǫ
.
Line search.
Choose step size
t
by backtracking line search.
Update.
x
:=
x
t
∆
x
nt
.
a feasible descent method:
x
(
k
)
feasible and
f
x
(
k
+1)
< f
x
(
k
)
affine invariant
Equality constrained minimization
11–
2nd interpretation of page 11–6 extends to infeasible
x
i.e.
Ax
b
linearizing optimality conditions at infeasible
x
(with
x
dom
f
) gives
2
f
x
T
x
nt
w
f
x
Ax
b
primal-dual interpretation
write optimality condition as
r
y
, where
y
x, ν
r
y
f
x
T
ν, Ax
b
linearizing
r
y
gives
r
y
y
r
y
Dr
y
y
2
f
x
T
x
nt
ν
nt
f
x
T
ν
Ax
b
same as (1) with
w
ν
ν
nt
Equality constrained minimization
11–
given
starting point
x
∈
dom
f
,
ν
, tolerance
ǫ >
0
,
α
∈
(
,
1
/
,
β
∈
(
,
.
repeat
∆
x
nt
,
∆
ν
nt
.
Backtracking line search on
‖
r
‖
2
.
t
:= 1
.
while
‖
r
(
x
t
∆
x
nt
, ν
t
∆
ν
nt
)
‖
2
(
−
αt
)
‖
r
(
x, ν
)
‖
2
,
t
:=
βt
.
Update.
x
:=
x
t
∆
x
nt
,
ν
:=
ν
t
∆
ν
nt
.
until
Ax
=
b
and
‖
r
(
x, ν
)
‖
2
≤
ǫ
.
not a descent method:
f
x
(
k
+1)
f
x
(
k
)
is possible
directional derivative of
r
y
2
in direction
y
x
nt
ν
nt
is
d
dt
r
y
t
y
2
t
=
r
y
2
Equality constrained minimization
11–
docsity.com
primal problem:
minimize
n i
=
log
x
i
subject to
Ax
b
dual problem:
maximize
b
T
ν
n i
=
log(
T
ν
i
n
three methods for an example with
100
×
500
, different starting points
x
(0)
Ax
(0)
b
k
x( f
)k(
p − )
⋆
0
5
10
15
20
10
−
10
10
−
5
10
0
10
5
Equality constrained minimization
11–
docsity.com
T
ν
(0)
k
p
⋆
ν(g −
)k(
)
0
2
4
6
8
10
10
−
10
10
−
5
10
0
10
5
x
(0)
k
x(r ‖
)k(
, ν
)k(
‖)
2
0
5
10
15
20
25
10
−
15
10
−
10
10
−
5
10
0
10
5
10
10
Equality constrained minimization
11–
minimize
n i
=
φ
i
x
i
subject to
Ax
b
directed graph with
n
arcs,
p
nodes
x
i
: flow through arc
i
φ
i
: cost flow function for arc
i
(with
φ
′′ i
x
node-incidence matrix
(
p
+1)
×
n
defined as
ij
arc
j
leaves node
i
arc
j
enters node
i
otherwise
reduced node-incidence matrix
p
×
n
is
with last row removed
b
p
is (reduced) source vector
rank
p
if graph is connected
Equality constrained minimization
11–
docsity.com
KKT system
T
v
w
g h
diag
φ
′′ 1
x
1
,... , φ
′′ n
x
n
, positive diagonal
solve via elimination:
−
1
T
w
h
−
1
g,
Hv
g
T
w
sparsity pattern of coefficient matrix is given by graph connectivity
−
1
T
ij
T
ij
nodes
i
and
j
are connected by an arc
Equality constrained minimization
11–
solution by block elimination
eliminate
from first equation:
p j
=
w
j
j
substitute
in second equation
p
j
=
tr
i
j
w
j
b
i
i
,... , p
a dense positive definite set of linear equations with variable
w
p
flop count (dominant terms) using Cholesky factorization
T
form
p
products
T
j
pn
3
form
p
p
inner products
tr
T
i
T
j
p
2
n
2
solve (2) via Cholesky factorization:
p
3
Equality constrained minimization
11–
