Download Linear Algebra, Lecture Notes - Mathematics - 8 and more Study notes Mathematics in PDF only on Docsity!
IB Paper 7: Linear Algebra Handout 8
9 Singular Value Decomposition – the next best thing to eigenvalues
Provided a square n × n matrix is diagonalisable (which means n distinct eigenvectors), it can be
decomposed into
A X = X A = X ΛΛΛΛ X
− 1
where X is a matrix whose columns are the eigenvectors of A , ΛΛΛΛ is a diagonal matrix whose
elements are the eigenvalues of A. Further, if A , is a symmetric matrix, it will always be
diagonalisable and the eigenvectors can be chosen to be orthogonal and unit (i.e. orthonormal)
A Q = Q A = Q ΛΛΛΛ Q
− 1 = Q ΛΛΛΛ Q
t
and the matrix is well and truly tamed. The columns of Q are an ideal basis of
n
and if we use this
as a co-ordinate system then the action of A is simply to apply a stretch of λ in that direction and
we can tell immediately the major effects of A by concentrating on the largest eigenvalues.
Non-square matrices do not have eigenvalues and eigenvectors – A x does not even have the same
number of co-ordinates as x. There is a related concept called singular values , which is the next
best thing. By concentrating on the largest of these, we can extract the most important part of A
and make the same types of simplifications that we would do for the square case.
For definiteness, in what follows we will consider an m × n matrix A , of rank r ( = no of
independent rows = no. of independent columns).
The “big picture”, derived in handout 4, is
Column Space
Left Null Space
Row Space
Null Space
dimension n − r
dimension r dimension r
dimension m − r
A^0
A
n
m
Every
vector here
every
vector here
y here
y ⊥ every
column of A
Can’t get here
with A x
BUT Column Space of A is the Row Space of
t
A
and Left Null Space of A is the Null Space of
t
A
9.1 The eigenvalue problem for A A
t
We saw in section 7.1 that the matrix AA
t
is square. It gives us a round trip and it makes
sense to talk about its eigenvalues and eigenvectors (which are in of
n
).
If A is m × n , then A
t is n × m and AA
t
is n × n.
i.e. AA
t
is a square matrix. It is also symmetric :
t t
t t t t
A A = A A = A A
A A
t
is, therefore, diagonalisable and the
eigenvectors of AA
t
can be chosen to be orthogonal,
unit vectors and we have
t t t
A A Q = Q A A = Q Q
where Q is the n × n orthogonal matrix whose columns are the eigenvectors of AA
t
and is the
n × n diagonal matrix of the corresponding eigenvalues.
Now the eigenvalues of A
t A can not be negative.
A
t Aq = λ q q
t A
t Aq = λ q
t q
Column Space
Left Null Space
Row Space
Null Space
dimension n − r
dimension r dimension r
dimension m − r
A
A
n
m
A
t
A
t
n
m
A
x
A
t
Ax A
t
Ax
t (^) t
Aq Aq = λ q q
2
2
2
( and 1 )
Aq
q
q
Now for the pay-off.
(i) 1
q ˆ^ , 2
q ˆ^ , ... , r
q ˆ are orthogonal, unit vectors
j i
i j
i j
Aq Aq
q .q.
t (^) t t
i j i j
i j i j
Aq Aq q A A q
t 2
i j j j
i j
i j i
q q
q .q which is 1 if i = j and 0 otherwise
(ii) 1
q ˆ^ , 2
q ˆ^ , ... , r
ˆ q are in the column space of A. Remember
1
1
1
Aq
q ˆ^ = , etc.
(iii) 1
q ˆ^ , 2
q ˆ^ , ... , r
ˆ q are eigenvectors for AA
t with
the same eigenvalues
2
i
( ) (^ )^
2
q = q = σ q
t t
A A A A A A A
We are almost there!
We have 1
q ˆ^ , 2
q ˆ^ , ... , r
q ˆ which are orthogonal, unit eigenvectors of AA
t , we can complete this set
by adding r + 1
ˆ q , ... , m
ˆ q , which are eigenvectors corresponding to eigenvalue zero for AA
t .
These are automatically orthogonal to 1
q ˆ^ , 2
q ˆ^ , ... , r
ˆ q , (i.e. and so must be in the left-null space of
A )and can be chosen to be orthogonal, unit vectors. We then have
t t ˆ ˆ ˆ AA = Q Q
where
Q is the orthogonal matrix whose columns are the eigenvectors of
t
AA and
is the
diagonal matrix ( m × m ) of the corresponding eigenvalues.
1 2 1 2
r r + r + m
Q q q q q q q
1
r
q ... q are an orthogonal basis for Column Space
1
r + m
q ... q are an orthogonal basis for Left-Null Space
n
m
A
q Aq
A
t
A
t
Aq = σ
2 q
A
2 Aq
non-zero σ's eigenvalue 0
These q ’s are in
the left- null space
of A
^
These q ’s are in the
column space of A
^
10.4 Summary - The Singular Value Decomposition of A
We have arranged that
1 1 1
Aq = σ q ˆ
2 2 2
Aq = σ q ˆ
r r r
Aq = σ ˆ q and Aq = = Aq = 0
r + n
1
Writing this in matrix notation
A Q =
Q
1
1 1 1 1
r
r r n r r m
A q q q q q q q q
q ˆ^ ... q ˆ 0 ... 0 r r
σ σ 1 1
Because Q is an orthogonal matrix, its inverse is its transpose, so that
AQ = Q
t ˆ A = Q Q
This is the singular value decomposition of A.
^
Column Space
Left Null Space
Row Space
Null Space
dimension n − r
dimension r dimension r
dimension m − r
n
m
q 1
q 2
q r
q r +
q n
q 1
q r
q r +
q m
...
^
^
^
For 1
i i i
i i i
i r
t
Aq q
A q q
2
i i i
i i i
i i
t t
q q
A q A q
m × n n × n m^ ×^ m^ m × n
T T T
1 2 1 1 2 2
r r r
A = σ q q + σ q q + ... + σ q q
Example II
SVD works just as easily for defective matrices A 's, such as the one we saw in section 8:
A
Begin by computing
A
t A =
Now find the eigenvalues and eigenvectors for A
t A,
det( A
t A − λ I ) = ( 1 −λ ) ( 2 −λ) − 1 = 3 1
2
λ − λ+ = 0
1 , 2
2
1,
i.e. σ 1 = 1.6180 and σ 2 = 0.6180.
Eigenvectors given by 1 2 1
x x x
1
2
x
x
giving 1
q and 2
q
i.e. Q =
Now compute
1
1
1
1 1 1 .5257^.
Aq
q
2
2
2
1 1 1 .8507^.
−^ −
Aq
q
Q =
Now extend
Q with eigenvectors corresponding to eigenvalue 0 for AA
t (again no need since it is
a 2 × 2 matrix and has two non-zero eigenvalues – those calculated above for of A
t A ).
Check
t
Q Q =
10.5 An example of SVD Image Processing
The largest singular value, along with the corresponding 1st columns of Q and
Q , forms the most
significant part of the matrix. As the singular values get smaller and smaller, they, and their
corresponding columns of Q and
Q , make up less and less significant information.
The way the SVD splits a matrix into what is important, and what is not, can be seen by its effect
on a matrix representing an image.
The next pages show four views of the same image. The image is supplied with MATLAB , and
shows many of the key figures in the development of linear algebra at a conference in 1964:
Wilkinson, Givens, Forsythe, Householder, Henrici, and Bauer.
The MATLAB commands to produce each of the images are given in courier font.
load gatlin
This loads the image into memory. It is stored in a matrix X , where every entry is a number
between 1 and 255, representing the brightness of a pixel.
colormap(gray)
image(X)
The colormap command tells MATLAB to expect a grey-scale picture. The image command tells
MATLAB to display the image, which is shown next.
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
The full image
A better image can be obtained by incorporating more information. The closest rank 10 matrix to X
is given by multiplying together the first 10 columns of
Q and Q , with the first 10 singular values
X10 = Qhat(:,1:10)Si(1:10,1:10)Q(:,1:10)';
We can see the effect this has:
image(X10)
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
The rank 10 image
The image is now obviously of a group of men, although most of the detail is missing.
It was generated using only 10(480) + 10 + 10(640) = 11210 numbers, and only 3.6% of
the original space of the matrix is required.
The closest rank 50 matrix to X is given by multiplying together the first 50 columns of
Q and Q ,
with the first 50 singular values
X50 = Qhat(:,l:50)Si(1:50,l:50)Q(:,l:50)';
We can see the effect this has:
image(X50)
1 1
1 10
10 10
^ ^
q
X10 q q
q
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
The rank 50 image
This image shows only a little degradation from the original. It was generated using 50(480) + 50
- 50(640) = 56050 numbers, which is about 18% of the original space of the matrix.
Summary The following is taken verbatim from the Maths Databook
Singular Value Decomposition
t
2
1
A = Q Q (orthogonal × diagonal × orthogonal)
Q ( m × m ) are the eigenvectors of
t
A A
Q ( n × n ) are the eigenvectors of
t
A A
- The r singular values, arranged in descending order on the diagonal of ( m × n ) are the
square roots of the non-zero eigenvalues of both
t
A A and
t
A A. r is the rank of the matrix.
Basis of column space: first r columns of 1
Q
Basis of left nullspace: last m − r columns of 1
Q
Basis of row space: first r columns of 2
Q
Basis of nullspace: last n − r columns of 2
Q.
