Download 1 Quadratic Forms and more Study notes Law in PDF only on Docsity!
Reading [SB], Ch. 16.1-16.3, p. 375-
1 Quadratic Forms
A quadratic function f : R → R has the form f (x) = a · x^2. Generalization of this notion to two variables is the quadratic form
Q(x 1 , x 2 ) = a 11 x^21 + a 12 x 1 x 2 + a 21 x 2 x 1 + a 22 x^22.
Here each term has degree 2 (the sum of exponents is 2 for all summands). A quadratic form of three variables looks as
f (x 1 , x 2 , x 3 ) = a 11 x^21 + a 12 x 1 x 2 + a 13 x 1 x 3 + a 21 x 2 x 1 + a 22 x^22 + a 23 x 2 x 3 + a 31 x 1 x 3 + a 32 x 3 x 2 + a 33 x^23.
A general quadratic form of n variables is a real-valued function Q : Rn^ → R of the form
Q(x 1 , x 2 , ..., xn) = a 11 x^21 + a 12 x 1 x 2 + ... + a 1 nx 1 xn+ a 21 x 2 x 1 + a 22 x^22 + ... + a 2 nx 2 xn+ ... ... ... ... an 1 xnx 1 + an 2 xnx 2 + ... + annx^2 n
In short Q(x 1 , x 2 , ..., xn) =
∑n i,j aij^ xixj^. As we see a quadratic form is determined by the matrix
A =
a 11 ... a 1 n .............. an 1 ... ann
.
1.1 Matrix Representation of Quadratic Forms
Let Q(x 1 , x 2 , ..., xn) =
∑n i,j aij^ xixj^ be a quadratic form with matrix^ A. Easy to see that
Q(x 1 , ..., xn) = (x 1 , ..., xn) ·
a 11 ... a 1 n .............. an 1 ... ann
·
x 1 .. xn
.
Equivalently Q(x) = xT^ · A · x. Example. The quadratic form Q(x 1 , x 2 , x 3 ) = 5x^21 − 10 x 1 x 2 + x^22 whose
symmetric matrix is A =
( 5 − 5 − 5 1
) is the product of three matrices
(x 1 , x 2 , x 3 ) ·
( 5 − 5 − 5 1
) ·
x 1 x 2 x 3
.
1.1.1 Symmetrization of matrix
The quadratic form Q(x 1 , x 2 , x 3 ) = 5x^21 − 10 x 1 x 2 + x^22 can be represented, for example, by the following 2 × 2 matrices
( 5 − 2 − 8 1
) ,
( 5 − 3 − 7 1
) ,
( 5 − 5 − 5 1
)
the last one is symmetric: aij = aji.
Theorem 1 Any quadratic form can be represented by symmetric matrix.
Indeed, if aij 6 = aji we replace them by new a′ ij = a′ ji = aij^ + 2 aji, this does not change the corresponding quadratic form. Generally, one can find symmetrization A′^ of a matrix A by A′^ = A+A T
1.2 Definiteness of Quadratic Forms
A quadratic form of one variable is just a quadratic function Q(x) = a · x^2. If a > 0 then Q(x) > 0 for each nonzero x. If a < 0 then Q(x) < 0 for each nonzero x. So the sign of the coefficient a determines the sign of one variable quadratic form. The notion of definiteness described bellow generalizes this phenomenon for multivariable quadratic forms.
1.2.1 Generic Examples
The quadratic form Q(x, y) = x^2 + y^2 is positive for all nonzero (that is (x, y) 6 = (0, 0)) arguments (x, y). Such forms are called positive definite.
The quadratic form Q(x, y) = −x^2 − y^2 is negative for all nonzero argu- ments (x, y). Such forms are called negative definite.
The quadratic form Q(x, y) = (x − y)^2 is nonnegative. This means that Q(x, y) = (x − y)^2 is either positive or zero for nonzero arguments. Such forms are called positive semidefinite.
- If D 1 > 0 and D 2 > 0 then the form is of x^2 + y^2 type, so it is positive definite;
- If D 1 < 0 and D 2 > 0 then the form is of −x^2 − y^2 type, so it is negative definite;
- If D 1 > 0 and D 2 < 0 then the form is of x^2 − y^2 type, so it is indefinite;
If D 1 < 0 and D 2 < 0 then the form is of −x^2 + y^2 type, so it is also indefinite;
Thus if D 2 < 0 then the form is indefinite.
Semidefiniteness depends not only on leading principal minors D 1 , D 2 but also on all principal minors, in this case on D 1 ′ = c too.
- If D 1 ≥ 0 , D′ 1 ≥ 0 and D 2 ≥ 0 then the form is positive semidefinite. Note that only D 1 ≥ 0 and D 2 ≥ 0 is not enough, the additional condition D′ 1 ≥ 0 here is absolutely necessary: consider the form Q(x 1 , x 2 ) = −x^22 with a = 0, b = 0, c = −1, here D 1 = a ≥ 0 , D 2 = ac − b^2 ≥ 0, nevertheless the form is not positive semidiefinite.
- If D 1 ≤ 0 , D′ 1 ≤ 0 and D 2 ≥ 0 then the form is negative semidefinite. Note that only D 1 ≤ 0 and D 2 ≥ 0 is not enough,the additional condition D′ 1 ≤ 0 again is absolutely necessary: consider the form Q(x 1 , x 2 ) = x^22 with a = 0, b = 0, c = 1, here D 1 = a ≤ 0 , D 2 = ac − b^2 ≥ 0, nevertheless the form is not negative semidiefinite.
1.2.5 Definiteness of 3 Variable Quadratic Form
Let us start with the following Example. Q(x 1 , x 2 , x 3 ) = x^21 + 2x^22 − 7 x^23 − 4 x 1 x 2 + 8x 1 x 3. The symmetric matrix of this quadratic form is
.
The leading principal minors of this matrix are
|D 1 | =
∣∣ ∣ 1
∣∣ ∣ = 1, |D 2 | =
∣∣ ∣∣ ∣
∣∣ ∣∣ ∣ =^ −^2 ,^ |D^3 |^ =
∣∣ ∣∣ ∣∣ ∣
∣∣ ∣∣ ∣∣ ∣
Now look:
Q(x 1 , x 2 , x 3 ) = x^21 + 2x^22 − 7 x^23 − 4 x 1 x 2 + 8x 1 x 3 = x^21 − 4 x 1 x 2 + 8x 1 x 3 + 2x^22 − 7 x^23 = x^21 − 4 x 1 (x 2 − 2 x 3 ) + 2x^22 − 7 x^23 = [x^21 − 4 x 1 (x 2 − 2 x 3 ) + 4(x 2 − 2 x 3 ) − 4(x 2 − 2 x 3 )] + 2x^22 − 7 x^23 = [x 1 − 2 x 2 + 4x 3 ]^2 − 2 x^22 − 16 x 2 x 3 − 23 x^23 = [x 1 − 2 x 2 + 4x 3 ]^2 − 2(x^22 − 8 x 2 x 3 ) − 23 x^23 = [x 1 − 2 x 2 + 4x 3 ]^2 − 2[x^22 − 8 x 2 x 3 + 16x^23 − 16 x^23 ] − 23 x^23 = [x 1 − 2 x 2 + 4x 3 ]^2 − 2[x 2 − 4 x 3 ]^2 − 16 x^23 ) − 23 x^23 = [x 1 − 2 x 2 + 4x 3 ]^2 − 2[x 2 − 4 x 3 ]^2 + 32x^23 − 23 x^23 = [x 1 − 2 x 2 + 4x 3 ]^2 − 2[x 2 − 4 x 3 ]^2 + 9x^23 = |D 1 |l^21 + D D^21 l^2 + D D^32 l 32 ,
where l 1 = x 1 − 2 x 2 +4x 3 , l 2 = x 2 − 4 x 3 , l 3 = x 3.
That is (l 1 , l 2 , l 3 ) are linear combinations of (x 1 , x 2 , x 3 ). More precisely
l 1 l 2 l 3
=
·
x 1 x 2 x 3
where
P =
is a nonsingular matrix (changing variables).
Now turn to general 3 variable quadratic form
Q(x 1 , x 2 , x 3 ) = (x 1 , x 2 , x 3 ) ·
a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33
·
x 1 x 2 x 3
.
The following three determinants
|D 1 | =
∣∣ ∣ a 11
∣∣ ∣ , |D 2 | =
∣∣ ∣∣ ∣
a 11 a 12 a 21 a 22
∣∣ ∣∣ ∣ ,^ |D^3 |^ =
∣∣ ∣∣ ∣∣ ∣
a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33
∣∣ ∣∣ ∣∣ ∣
are leading principal minors.
It is possible to show that, as in 2 variable case, if |D 1 | 6 = 0, |D 2 | 6 = 0, then
Q(x 1 , x 2 , x 3 ) = |D 1 |l^21 +
|D 2 |
|D 1 |
l 22 +
|D 3 |
|D 2 |
l^23
where
P =
p 11 ... p 1 n
... pn 1 ... pnn
is a nonsingular matrix (changing variables).
Theorem 2 1. A quadratic form is positive definite if and only if
|D 1 | > 0 , |D 2 | > 0 , ... , |Dn| > 0 ,
that is all principal minors are positive;
- A quadratic form is negative definite if and only if
|D 1 | < 0 , |D 2 | > 0 , |D 3 | < 0 , |D 4 | > 0 , ... ,
that is principal minors alternate in sign starting with negative one.
- If some kth order leading principal minor is nonzero but does not fit either of the above two sign patterns, then the form is indefinite.
The situation with semidefiniteness is more complicated, here are involved not only leading principal minors, but all principal minors.
Theorem 3 1. A quadratic form is positive semidefinite if and only if all principal minors are ≥ 0 ;
- A quadratic form is negative semidefinite if and only if all principal minors of odd degree are ≤ 0 , and all principal minors of even degree are ≥ 0.
1.3 Definiteness and Eigenvalues
As we know a symmetric n × n matrix has n real eigenvalues (maybe some multiple).
Theorem 4 Given a quadratic form Q(x) = xT^ Ax and let λ 1 , ... , λn be eigenvalues of A. Then Q(x) is
- positive definite iff λi > 0 , i = 1, ... , n;
- negative definite iff λi < 0 , i = 1, ... , n;
- positive semidefinite iff λi ≥ 0 , i = 1, ... , n;
- negative semidefinite iff λi ≤ 0 , i = 1, ... , n;
Proof. Just xT^ Ax > 0 ⇒ ∀ λi > 0. Let v be the normalized eigenvector of λi, that is Av = λiv. Then
0 < vT Av = λivT v = λi.
1.4 Linear Constraints and Bordered Matrices
1.4.1 Two Variable Case
The quadratic form Q(x 1 , x 2 ) = x^21 − x^22 is indefinite (why?).
But if we restrict Q to the subset (subspace) of R^2 determined by the constraint x 2 = 0 we obtain a one variable quadratic form q(x) = Q(x, 0) = x^2 which is definitely positive definite. Thus the restriction of Q on x 1 axis is positive definite.
Similarly, the constraint x 1 = 0 gives q(x) = Q(0, x) = −x^2 which is negative definite. Thus the restriction of Q on x 2 axis is positive definite.
Now let us consider the constraint x 1 − 2 x 2 = 0. Solving x 1 from this constraint we obtain x 1 = 2x 2. Substituting in Q we obtain one variable quadratic form q(x) = Q(2x, x) = 4x^2 − x^2 = 3x^2 which is positive definite. Thus the restriction of Q on the line x 1 + 2x 2 = 0 is positive definite.
Let us repeat the last calculations for a general 2 variable quadratic form
Q(x 1 , x 2 ) = ax^21 + 2bx 1 x 2 + cx^22 = (x 1 , x 2 )
( a b b c
) ( x 1 x 2
)
subject to the linear constraint
Ax 1 + Bx 2 = 0.
Let us, as above, solve x 1 from the constraint
x 1 = −
B
A
x 2
and substitute in Q:
Q(x 1 , x 2 ) = Q(−BA x 2 , x 2 ) = a 11 (−BA x 2 )^2 + 2a 12 (−BA x 2 )x 2 + a 22 x^22 =
= aB
(^2) − 2 bAB+cA 2 A^2 x
2
So the definiteness of Q(x 1 , x 2 ) on the constraint set Ax 1 + Bx 2 = 0 depends on the sign of coefficient aB
(^2) − 2 bAB+cA 2 A^2 , more precisely on the nominator
aB^2 − 2 bAB + cA^2 ,
which is nothing else than It is easy to see that
−det
0 A B
A a b B b c
.
The first m matrices M 1 , ... , Mm are zero matrices.
Next m − 1 matrices Mm+1, ... , M 2 m− 1 have zero determinant.
The determinant of the next minor M 2 m is ±det B′^2 where B′^ is the left m × m minor of B, it does not carry any information about A.
And only the determinants of last n−m matrices M 2 m+1, ... , Mm+n carry information about the matrix A, i.e. about the quadratic form Q. Exactly these minors are essential for constraint definiteness.
Theorem 6 (i) If the determinant of H = Mm+n has the sign (−1)n^ and the signs of determinants of last m + n leading principal minors
M 2 m+1, ... , Mm+n
alternate in sign, then Q is negative definite on the constraint set Bx = 0, so x = 0 is a strict global max of Q on the constraint set Bx = 0.
(ii) If the determinants of all last m + n leading principal minors
M 2 m+1, ... , Mm+n
have the same sign (−1)m, then Q is positive definite on the constraint set Bx = 0, so x = 0 is a strict global min of Q on the constraint set Bx = 0.
(iii) If both conditions (i) and (ii) are violated by some nonzero minors from last m + n leading principal minors
M 2 m+1, ... , Mm+n
then Q is indefinite on the constraint set Bx = 0, so x = 0 is neither max nor min of Q on the constraint set Bx = 0.
This table describes the above sign patterns:
Mm+m+1 Mm+m+2 ... Mm+n− 1 Mm+n negative (−1)m+1^ (−1)m+2^ ... (−1)n−^1 (−1)n positive (−1)m^ (−1)m^ ... (−1)m^ (−1)m
Example
Determine the definiteness of the following constrained quadratics
Q(x 1 , x 2 , x 3 ) = x^21 + x^22 − x^23 + 4x 1 x 3 − 2 x 1 x 2 , subject to x 1 + x 2 + x 3 = 0.
Solution. Here n = 3, m = 1. The bordered matrix here is
The leading principal minors are
|M 1 | = 0, |M 2 | =
∣∣ ∣∣ ∣
∣∣ ∣∣ ∣ =^ −^1 ,
|M 3 | =
∣∣ ∣∣ ∣∣ ∣
∣∣ ∣∣ ∣∣ ∣
= − 4 < 0 , |M 4 | =
∣∣ ∣∣ ∣∣ ∣∣ ∣
∣∣ ∣∣ ∣∣ ∣∣ ∣
in this case n = 3, m = 1, so the essential minors are M 3 and M 4. For negative definiteness the sign pattern must be
(−1)n−^1 = (−1)^2 = ” + ”, (−1)n^ = (−1)^3 = ” − ”
And for positive definiteness the sign pattern must be
(−1)m^ = (−1)^1 = ” − ”, (−1)m^ = (−1)^1 = ” − ”,
since we have − 4 < 0 , 16 > 0, which differs from both patterns, our con- strained quadratic is indefinite.
1.5 Change of variables*
Let Q = xT^ Ax be a quadratic form of variable x ∈ Rn. Let us step to new variable y ∈ Rn^ which is connected to x by x = P y where P is some nonsingular matrix. Note that in this case xT^ = (P y)T^ = yT^ P T^. Then
Q = xT^ Ax = yT^ P T^ AP y = yT^ (P T^ AP )y,
so the matrix of new quadratic form of variable y is B = P T^ AP.
If A is symmetric, B = P T^ AP is symmetric too (prove it using the definition of symmetric matrix A = AT^ ).
Jacobi’s Theorem states that any symmetric matrix A can be trans- formed to a diagonal matrix Λ = P T^ AP by an orthogonal matrix P. The elements of Λ are uniquely determined up to permutation. If we allow P to be a nonsingular matrix, then A can be transformed to a diagonal matrix where each diagonal element is 1, -1 or 0.
The point x 0 is min if this difference f (x 0 + h) − f (x 0 ) is positive for all small enough values of h. For small enough h the cubical term f^
′′′(x 0 ) 3! ·^ h
(^3) , the term of degree 4 f (4)(x 0 ) 4! ·^ h
(^4) etc are smaller that the quadratic term f^ ′′(x 0 ) 2 ·^ h
(^2) , so the positivity
of the difference f (x 0 + h) − f (x 0 ) depends on positivity of the coefficient f ′′(x 0 ) 2 of that quadratic form^
f ′′(x 0 ) 2 ·^ h
(^2) , that is on the positivity of the second
derivative at x 0. So, at h = 0 this form has minimum (it is positive definite) if its coefficient f ′′(x 0 ) is positive (our good old second order condition).
Now consider a function of two variables F (x 1 , x 2 ). Again there exists Taylor for two variables
F (x 1 + h 1 , x 2 + h 2 ) = F (x 1 , x 2 ) + (^) ∂x∂F 1 (x 1 , x 2 ) · h 1 + (^) ∂x∂F 2 (x 1 , x 2 ) · h 2 + 1 2
∂^2 F ∂x^21 (x^1 , x^2 )^ ·^ h
2 1 +^
∂^2 F ∂x 1 ∂x 2 (x^1 , x^2 )^ ·^ h^1 h^2 +^
1 2
∂^2 F ∂x^22 (x^1 , x^2 )^ ·^ h
2 2 + 1 3
∂^3 F ∂x^31 (x^1 , x^2 )^ ·^ h
3 1 +^ ...^.
As in one-variable case, an optimum (min or max) can be expected at a critical point, where the linear form (^) ∂x∂F 1 (x 1 , x 2 ) · h 1 + (^) ∂x∂F 2 (x 1 , x 2 ) · h 2 vanishes. And the minimality or maximality depends on the quadratic form
Q(h 1 , h 2 ) =
∂^2 F
∂x^21
(x 1 , x 2 ) · h^21 +
∂^2 F
∂x 1 ∂x 2
(x 1 , x 2 ) · h 1 h 2 +
∂^2 F
∂x^22
(x 1 , x 2 ) · h^22.
Namely, if this form is positive definite, that is if Q(h 1 , h 2 ) > 0 for all (h 1 , h 2 ) 6 = (0, 0), then (x 1 , x 2 ) is a point of minimum, and if the form is negative definite, that is Q(h 1 , h 2 ) < 0 for all (h 1 , h 2 ) 6 = (0, 0), then (x 1 , x 2 ) is a point of maximum.
Exercises
- Write the following quadratic forms in matrix form (a) Q(x 1 , x 2 ) = x^21 − 2 x 1 x 2 + x^22. (b) Q(x 1 , x 2 ) = 5x^21 − 10 x 1 x 2 + x^22. (c) Q(x 1 , x 2 , x 3 ) = x^21 + 2x^22 + 3x^23 + 4x 1 x 2 − 6 x 1 x 3 + 8x 2 x 3.
- By direct matrix multiplication express each of the following matrix products as a quadratic form
(a)
( x 1 x 2
) ·
( 4 2 2 3
) ·
( x 1 x 2
) , (b)
( u v
) ·
( 5 2 4 0
) ·
( u v
) .
- Write the quadratic form corresponding to the matrix A and then find a symmetric matrix which determines the same quadratic form for
(a) A =
( 2 − 1 − 2 1
) , (b) A =
(c)^ A^ =
- Write an example of 2 variable quadratic form Q(x 1 , x 2 ) which is (a) positive definite. (b) negative definite. (c) positive semidefinite. (d) negative semidefinite. (e) indefinite.
- Write an example of 3 variable quadratic form Q(x 1 , x 2 , x 3 ) which is (a) positive definite. (b) negative definite. (c) positive semidefinite. (d) negative semidefinite. (e) indefinite.
- Determine definiteness of the following symmetric matrices (quadratic forms)
(a)
( 2 − 1 − 1 1
) , (b)
( − 3 4 4 − 5
) , (c)
( − 3 4 4 − 6
) , (d)
( 2 4 4 8
) ,
(e)
(f )
(g)
.
- Formulate criteria for positive and negative definiteness of the quadratic form given by a diagonal matrix
Short Summary Quadratic Forms
Q(x 1 , x 2 , ..., xn) =
∑^ n
i,j
aij xixj = (x 1 , ..., xn)·
a 11 ... a 1 n .............. an 1 ... ann
·
x 1 .. xn
=^ xT^ ·A·x
A is symmetric. If not, take its symmetrization A′^ = A+A
T
Definiteness of Q(x): (a) positive definite if Q(x) > 0 for all x 6 = 0 ∈ Rn; (b) positive semidefinite if Q(x) ≥ 0 for all x 6 = 0 ∈ Rn; (c) negative definite if Q(x) < 0 for all x 6 = 0 ∈ Rn; (d) negative semidefinite if Q(x) ≤ 0 for all x 6 = 0 ∈ Rn; (e) indefinite if Q(x) > 0 for some x and Q(x) < 0 for some other x.
Definiteness and Optimality If Q is positive definite then x = 0 is global maximum; If Q is negative definite then x = 0 is global minimum.
Leading principal minors
|D 1 | =
∣∣ ∣ a 11
∣∣ ∣ , |D 2 | =
∣∣ ∣∣ ∣
a 11 a 12 a 21 a 22
∣∣ ∣∣ ∣ , ... ,^ |Dn|^ =^ |A|.
A quadratic form Q(x) is: Positive definite iff |D 1 | > 0 , |D 2 | > 0 , ... , |Dn| > 0. Negative definite iff |D 1 | < 0 , |D 2 | > 0 , |D 3 | < 0 , |D 4 | > 0 , .... Indefinite iff some nonzero Dk violates above sign patterns.
Positive semidefinite iff all principal minors Mk ≥ 0. Negative semidefinite iff all M 2 k+1 ≤ 0 and M 2 k ≥ 0.
Definiteness of Q(x) = xT^ · A · x on the constrained set B · x = 0:
Mm+m+1 Mm+m+2 ... Mm+n− 1 Mm+n negative (−1)m+1^ (−1)m+2^ ... (−1)n−^1 (−1)n positive (−1)m^ (−1)m^ ... (−1)m^ (−1)m
where M 2 m+1, ..., Mm+n are last n − m minors of the bordered matrix
0 ... 0 | B 11 ... B 1 n ... | ... 0 ... 0 | Bm 1 ... Bmn − − − − − − B 11 ... Bm 1 | a 11 ... a 1 n ... | ... B 1 n ... Bmn | an 1 ... ann
