Download Questions for Assignment 1 - Numerical Optimization | MATH 516 and more Assignments Mathematics in PDF only on Docsity!
AMATH/MATH 516
FIRST HOMEWORK SET
Due by class time Thursday 04/05/07. The purpose of this problem set is to have you brush up and further develop your multi-variable calculus and linear algebra skills. The problem set will be very difficult for some and straightforward for others. If you are having any difficulty, please feel free to discuss the problems with me at any time. Don’t delay in starting work on these problems!
- Let Q be an n × n symmetric positive definite matrix.
(a) Show that the eigenvalues of Q^2 are the square of the eigenvalues of Q. (b) If λ 1 ≥ λ 2 ≥... ≥ λn are the eigen values of Q, show that
λn‖u‖^22 ≤ uT^ Qu ≤ λ 1 ‖u‖^22 ∀ u ∈ IRn.
(c) If 0 < λ < λ¯ are such that λ‖u‖^22 ≤ uT^ Qu ≤ ¯λ‖u‖^22 ∀ u ∈ IRn, then all of the eigenvalues of Q must lie in the interval [λ, ¯λ]. (d) Let λ and λ¯ be as in Part (c) above. Show that
λ‖u‖ 2 ≤ ‖Qu‖ 2 ≤ λ¯‖u‖ 2 ∀ u ∈ IRn^.
Hint: ‖Qu‖^22 = uT^ Q^2 u.
- Consider the quadratic function f : IRn^7 → IR given by
f (x) :=
xT^ Qx − aT^ x + α ,
where Q ∈ IRn×n, a ∈ IRn, and α ∈ IR.
(a) Write expressions for both ∇f (x) and ∇^2 f (x). Since it is not assumed that f is symmetric, be careful in how you express ∇^2 f (x). (b) If it is further assumed that Q is symmetric, what is ∇^2 f? (c) State first– and second–order necessary conditions for optimality in the problem min{f (x) : x ∈ IRn}. (d) State a sufficient condition on the matrix Q under which the problem min f has a unique global solution and then display this solution in terms of the data Q and a.
- Consider the linear equation Ax = b, where A ∈ IRm×n^ and b ∈ IRm. When n < m it is often the case that this equation is over-determined in the sense that no solution x exists. In such cases one often attempts to locate a ‘best’ solution in a least squares sense. That is one solves the linear least squares problem
(lls) : minimize
‖Ax − b‖^22
for x. Define f : IRn^7 → IR by f (x) :=
‖Ax − b‖^22.
(a) Show that f can be written as a quadratic function, that is, it can be written in the same form as the function of the preceding exercise. (b) What are ∇f (x) and ∇^2 f (x)?
(c) Show that ∇^2 f (x) is positive semi-definite. (d) Show that a solution to (lls) must always exist. (e) Provide a necessary and sufficient condition on the matrix A (not on the matrix AT^ A) under which (lls) has a unique solution and then display this solution in terms of the data A and b.
- A mapping 〈·, ·〉 : IRn^7 → IRn^ is said to be an inner product on IRn^ is for all x, y, z ∈ IRn
(i) 〈x, x〉 ≥ 0 Non-Negative (ii) 〈x, x〉 = 0 ⇔ x = 0 Positive (iii) 〈x + y, z〉 = 〈x, z〉 + 〈y, z〉 Additive (iv) 〈αx, y〉 = α 〈x, y〉 ∀ α ∈ IR Homogeneous (v) 〈x, y〉 = 〈y, x〉 Symmetric
Two vectors x, y ∈ IRn^ are said to be orthogonal in the inner product 〈·, ·〉 if 〈x, y〉 = 0 Unless otherwise specified, we use the notation 〈x, y〉 to designate the usual Euclidean inner product:
〈x, y〉 =
∑^ n
i=
xiyi.
(a) Let 〈x, y〉 be the Euclidean inner product on IRn. Given A ∈ IRn×n, show that A = 0 if and only if
〈x, Ay〉 = 0 ∀ x, y ∈ IRn^.
(b) Let H ∈ IRn×n^ be symmetric and positive definite (i.e. H = HT^ and xT^ Hx > 0 ∀ x ∈ IRn^ \ { 0 }). Show that the bi-linear form given by 〈x, y〉H = xT^ Hy ∀ x, y ∈ IRn defines an inner product on IRn. (c) Every inner product defines a transformation on the space of linear operators called the adjoint. For the Euclidean inner product on IRn, this is just the usual transpose. Given a linear transformation M : IRn^7 → IRn, the adjoint is defined by the relation
〈y, M x〉 = 〈M ∗y, x〉 , for all x, y, ∈ IRn.
The inner product given above, 〈·, ·〉H , also defines an adjoint mapping which we can denote by M TH^. Show that M TH^ = H−^1 M T^ H. (d) The matrix P ∈ IRn×n^ is said to a projection if P 2 = P. Clearly, if P is a projection, then so is I − P. The subspace P IRn^ = Ran (P) is called the subspace that P projects onto. A projection is said to be orthogonal with respect to a given inner product 〈·, ·〉 on IRn^ if and only if
〈(I − P )x, P y〉 = 0 ∀ x, y ∈ IRn^ ,
that is, the subspaces Ran (P) and Ran (I − P) are orthogonal in the inner product 〈·, ·〉. Show that the projection P is orthogonal with respect to the inner product 〈·, ·〉H (defined above), where H ∈ IRn×n^ is symmetric and positive definite, if and only if
P = H−^1 P T^ H.
- Consider the minimization problem
P : minimize f (x) subject to Ax = b ,
where f : IRn^7 → IR is assumed to be twice continuously differentiable, A ∈ IRm×n^ has full rank with m ≤ n, and b ∈ IRm. Set P := I − AT^ (AAT^ )−^1 A.
(a) Suppose H = LLT^ for some non–singular matrix L ∈ IRn×n, e.g. L = H^1 /^2. If Q is the orthogonal projector onto the null–space of AL−T^ in the usual (or Euclidean) inner product, show that the operator P given by P = L−T^ QLT is the orthogonal projector onto the null–space of A with respect to the inner product 〈·, ·〉H.
(b) Show that x¯ = x 0 − δ‖P H−^1 c‖− H^1 P H−^1 c solves P where P is as given in part (b) above. Hint: It may be helpful to first reduce the problem to one of the form
min ˆcT^ w subject to Awˆ = 0 ‖w‖^22 ≤ δ^2.
It is also helpful to apply results relating least–squares to orthogonal projection.
