Download Singular Value Decomposition - Lecture Notes | GEOS 567 and more Study notes Geology in PDF only on Docsity!
CHAPTER 6: SINGULAR-VALUE DECOMPOSITION (SVD)
6.1 Introduction
Having finished with the eigenvalue problem for Ax = b , where A is square, we now turn our attention to the general N × M case Gm = d. First, the eigenvalue problem, per se, does not exist for Gm = d unless N = M. This is because G maps (transforms) a vector m from M -space into a vectordifferent dimensional spaces. d in N -space. The concept of ìparallelî breaks down when the vectors lie in
Since the eigenvalue problem is not defined for G , we will try to construct a square matrix that includesThis eigenvalue problem G (and, as it will turn out, will lead us to singular-value G T^ ) for which the eigenvalue problem is defined. decomposition ( SVD ), a way to decompose G into the product of three matrices (two eigenvector matrices V and U , associated with model and data spaces, respectively, and a singular-value matrix very similar to Λ from the eigenvalue problem for A ). Finally, it will lead us to the generalized inverse operator, defined in a way that is analogous to the inverse matrix to A found using eigenvalue/eigenvector analysis. The end result of SVD is
N M N P P P P M
P P P^ T × × × ×
G = U Λ V (6.1)
where U P are the P N -dimensional eigenvectors of GG T^ , V P are the P M -dimensional eigenvectors of G T G , and ΛΛΛΛ P is the P × P diagonal matrix with P singular values (positive square roots of the nonzero eigenvalues shared by GG T^ and G T G ) on the diagonal.
6.2 Formation of a New Matrix B
6.2.1 Formulating the Eigenvalue Problem With G
The way to construct an eigenvalue problem that includes G is to form a square ( N + M ) × ( N + M ) matrix B partitioned as follows:
B =
N × N
G T
M × N
G
N × M
M × M
N^ ⇑
M ⇑
|⇐ N ⇒||⇐ M ⇒|
B is Hermitian because B T^ = B (6.3) Note, for example, B 1, N +3 = G 13 (6.4)
and B (^) N +3,1 = ( G T^ ) 31 = G 13 , etc. (6.5)
6.2.2 The Role of G T^ as an Operator
Analogous to Equation (1.13), we can define an equation for G T^ as follows: G T^ y = c (6.6) M × N N × 1 M × 1 We do not have to have a particular y and c in mind when we do this. We are simply interested in the mapping of an N -dimensional vector into an M -dimensional vector by G T^. We can combine Gm = d and G T y = c , using B , as
________T __ __
c
d m
y 0
G
G
or B z = b (6.8) ( N + M ) × ( N + M ) ( N + M ) × 1 ( N + M ) × 1 where we have
6.3.2 Partitioning W
Each eigenvector w i is ( N + M ) × 1. Consider partitioning w i such that
__
M
N
i
i i v
u w = (^) (6.15)
That is, we ìstackî an N -dimensional vector u i and an M -dimensional vector v i into a single ( N + M )-dimensional vector. Then the eigenvalue problem for B from Equation (6.11) becomes
__
__
________
i
i i i
i v
u v
u 0
G
G
T^ η^
This can be written as
G v i = η i u i i =1, 2,... , N + M
N × M M × 1 N × 1 (6.17)
and
G T^ u i = η i v i i =1, 2,... , N + M M × N N × 1 M × 1 (6.18) Equations (6.17) and (6.18) together are called the shifted eigenvalue problem for G. It is not an eigenvalue problem for G , since G is not square and eigenvalue problems are only defined for square matrices. Still, it is analogous to an eigenvalue problem. Note thaton an M -dimensional vector and returns an N -dimensional vector. G T (^) operates G operates on an N -dimensional vector and returns an M -dimensional vector. Furthermore, the vectors are shared by G and G T^.
6.4 Solving the Shifted Eigenvalue Problem
Equations (6.17) and (6.18) can be solved by combining them into two related eigenvalue problems involving G T G and GG T^ , respectively.
6.4.1 The Eigenvalue Problem for G T G
Eigenvalue problems are only defined for square matrices. Note, then, that G T G is M × M , and hence has an eigenvalue problem. The procedure is as follows: Starting with Equation (6.18)
G T u i = η i v i (6.18)
Multiply both sides by η i
η i G T u i = η^2 i v i (6.19)
or
G T^ ( η i u i ) = η^2 i v i (6.20)
But, by Equation (6.17), we have
η i u i = Gv i (6.17)
Thus
G T Gv i =η i^2 v i i =1, 2,... , M (6.21)
This is just the eigenvalue problem for G T G! We were able to manipulate the shifted eigenvalue problem into an eigenvalue problem that, presumably, we can solve. We make the following notes:
- G T G is Hermitian.
kj
N kj k ki
N
ij =^ ∑ k = 1 G^ ikG =∑= 1 g g
( G T^ G ) ( T) (6.22)
ki ij
N ki k kj
N ( ) ji (^) k ( G ) jkG g g ( ) T 1 1
G T^ G = ∑ T =∑ = G G
= =
Thus
GG T u i =η i^2 u i i = 1, 2,... , N (6.29)
We make the following notes for this eigenvalue problem:
- GG T^ is Hermitian.
- GG T^ is positive semidefinite.
- Combining the N equations in Equation (6.29), we have GG T U = UN (6.30) where
M M M
L
M M M
U u 1 u 2 u N (6.31) N × N and
2
22
12
0 0
η N
L
M O
M
L
N (6.32)
N × N
- U is an orthogonal matrix U T U = UU T^ = I N (6.33)
6.5 How Many η i Are There, Anyway??
for B , G^ A careful look at Equations (6.11), (6.21), and (6.29) shows that the eigenvalue problemsT G , and GG T^ are defined for ( N + M ), M , and N values of i , respectively. Just how
many η i are there?
6.5.1 Introducing P , the Number of Nonzero Pairs (+ ηηηη i , ñ ηηηη i )
Equation (6.11)
Bw i = η i w i (6.11)
can be used to determine ( N + M ) real η i. Equation (6.21),
G T Gv i = η^2 i v i (6.21)
can be used to determine M real η^2 i^ since G T G is M × M. Equation (6.29)
GG T u i = η i^2 u i (6.29)
can be used to determine N real η^2 i^ since GG T^ is N × N.
This section will convince you, I hope, that the following are true:
1. There are P pairs of nonzero η i , where each pair consists of (+ η i , ñ η i ).
2. If + η i is an eigenvalue of
Bw i = η i w i (6.11)
and the associated eigenvector w i is given by w (^) i = (^) uv ii (6.34)
then the eigenvector associated with ñ η i is given by
w ′ i =− vu ii (6.35)
3. There are ( N + M ) ñ 2 P zero η i.
4. You can know everything you need to know about the shifted eigenvalue problem byretaining only the information associated with the positive η
i.
- P is less than or equal to the minimum of N and M. P ≤ min( N , M ) (6.36)
2
1
2
1
L
O
O
M M
O
L
P
P
D
( N + M ) × ( N + M )
wherearbitrary, but must be internally consistent. η 1 ≥ η 2 ≥ ⋅⋅⋅ ≥ η P. Note that the ordering of matrices in eigenvalue problems is Then the eigenvalue problem for B from Equation
(6.11) becomes BW = WD (6.41) where now the ( N + M ) × ( N + M ) dimensional matrix W is given by
= ________________________________________________
M M M M M M M M
L L L
M M M M M M M M
M M M M M M M M
L L L
M M M M M M M M
P P P N M
P P P NM
v v v v v v v v
u u u u u u u u W 1 2 1 2 2 1
1 2 ñ 1 ñ 2 ñ 2 1 (6.42)
|⇐ P ⇒ | |⇐ P ⇒| |⇐( N + M ) ñ 2 P ⇒| The second P eigenvectors certainly contain independent information about the eigenvectors w i in ( N + M )-space. They contain no new information, however, about u i or v i , in N - and M -space, respectively, since ñ u i contains no information not already contained in + u i.
6.5.4 What About the Zero Eigenvalues ηη ηη i í s , i = (2 P + 1),... , N + M?
For the zero eigenvalues, the shifted eigenvalue problem becomes
Gv i = η i u i = 0 u i = 0 i = (2 P + 1),... , ( N + M ) (6.43)
N × 1
and
G T u i = η i v i = 0 v i = 0 i = (2 P + 1),... , ( N + M ) (6.44)
M × 1
where 0 is a vector of zeros of the appropriate dimension. If you premultiply Equation (6.43) by G T^ and Equation (6.44) by G , you obtain G T Gv i = G T 0 = 0 (6.45) ( M × 1) and GG T u i = G0 = 0 (6.46) Therefore, we conclude that the u ( N^ ×^ 1)
GG T^ and G T G associated with zero eigenvalues for i ,^ v i^ associated with zero GG T^ and^ η i^ for G T^ BG^ , respectively!are simply the eigenvectors of
6.5.5 How Big is P?
Now that we have seen that the eigenvalues come in P pairs of nonzero values, how can we determine the size of P? We will see that you can determine P from either G T G or GG T^ , and thatrespectively. The steps are as follows. P is bounded by the smaller of N and M , the number of observations and model parameters,
Step 1. Let the number of nonzero eigenvaluesonly M η η^2 i^ of G T G be P. Since G T G is M × M , there are
(^2) i (^) all together. Thus, P is less than or equal to M.
Step 2. If η^2 i^ ≠ 0 is an eigenvalue of G T G , then it is also an eigenvalue of GG T^ since
G T Gv i = η^2 i v i (6.21)
and
GG T u i = η i^2 u i (6.29)
Thus the nonzero η i ís are shared by G T G and GG T^.
Step 3. P is less than or equal to N since GG T^ is N × N. Therefore, since P ≤ M and P ≤ N , P ≤ min( N , M ) GG T^ ( N Thus, to determine × N ). It makes sense to choose the smaller of the two matrices.^ P , you can do the eigenvalue problem for either That is, one chooses^ G T G^ ( M^ ×^ M ) or G T G if M < N , or GG T^ if N < M.
and where we have chosen the P positive η i from
G T Gv i = η^2 i v i (6.21)
GG T u i = η i^2 u i (6.29)
Note that it is customary to order the u i , v i such that
η 1 ≥ η 2 ≥ ⋅⋅⋅ ≥ η P (6.51)
6.6.2 Definition of the Singular Value
the eigenvalue^ We define a singular value η^ λ i^ from Equation (6.21) or (6.29) as the positive square root of
(^2) i (^) for G T G or GG T (^). That is,
λ i = + η i^2 (6.52)
Singular values are not eigenvalues problem is not defined for G or G T ,. N λ≠ i is not an eigenvalue for M. They are, of course, eigenvalues for G or G T^ , since the eigenvalue B in Equation
(6.11), but we will never explicitly deal withformulate the shifted eigenvalue problem, but in practice, it is never formed. B. The matrix B is a construct that allowed us to Nevertheless, you
will often read, or hear, λ i referred to as an eigenvalue.
6.6.3 Definition of ΛΛΛΛ , the Singular-Value Matrix
the form^ We can form an^ N^ ×^ M^ matrix with the singular values on the diagonal. If^ M^ >^ N , it has
N
2
1
×Λ^ =
N M N
N M P^^0
L
O
M M
O
L
If N > M , it has the form
____________________ _
×
2
1
N M
M
N M
P
L
O
M M
O
L
|⇐ M ⇒|
Then the shifted eigenvalue problem
Gv i = η i u i (6.17)
and
G T u i = η i v i (6.18)
can be written as
Gv i = λ i u i (6.55)
and
G T u i = λ i v i (6.56)
wherepositive η i η has been replaced by λ i since all information about U , V can be obtained from the
i. Equations (6.55) and (6.56) can be written in matrix notation as G V = U Λ N × M M × M N × N N × M (6.57) and
G T^ U = V ΛT M × N N × N M × M M × N (6.58)
and
V v 1 v v 1 v = [ V | V 0 ]
M × M^ = P P +^ M P
M M M M
L L
M M M M
and
×
2
1
L
O
M M
O
L
N M^ λ^ P
6.8 Singular-Value Decomposition (SVD)
6.8.1 Derivation of Singular-Value Decomposition
We will see below that G can be decomposed without any knowledge of the parts of U or
V associated with zero singular values λ i , i > P. We start with the fundamental decomposition
theorem G = U Λ V T^ (6.60) Let us introduce a P × P singular-value matrix Λ P that is a subset of Λ:
P
P
2
1
L
M O
M
L
We now write out Equation (6.60) in terms of the partitioned matrices as
= _________________ ________
M P
P
N P
P
N
P P P 0 T
T 0 V
V
G U U (6.66)
|⇐ P ⇒| |⇐ N ñ P ⇒| |⇐ P ⇒||⇐ M ñ P ⇒||⇐ M ⇒|
= N U P P 0 VV 0 P TT (6.67)
|⇐ P ⇒||⇐ M ñ P ⇒| = U P Λ P V P^ T^ (6.68)
That is, we can write G as
G = U P Λ P V P^ T N × M N × P P × P P × M (6.69)
Equation (6.69) is known as the Singular-Value Decomposition Theorem for G. The matrices in Equation (6.69) are
- G = an arbitrary N × M matrix.
- The eigenvector matrix U P
M M M
L
M M M
U (^) P u 1 u 2 u P (6.70)
where u i are the P N -dimensional eigenvectors of
GG T u i = η i^2 u i (6.29)
associated with nonzero singular values λ i.
- The eigenvector matrix V P
4. G T^ U 0 = 0 (6.75)
M × N N × ( N ñ P ) M × ( N ñ P ) Note that the eigenvectors in V are a set of M orthogonal vectors which span model space , while the eigenvectors in V U are a set of N orthogonal vectors which span data space. The P vectors in P span a^ P -dimensional subset of model space, while the^ P^ vectors in^ U P span a^ P -dimensional subset of data space. V 0 and U 0 are called null , or zero , spaces. They are ( M ñ P ) and ( N ñ P ) dimensional subsets of model and data spaces, respectively.
6.8.3 Summarizing SVD
In summary, we started with Equations (1.13) and (6.5) Gm = d (1.13) and G T y = c (6.6) We constructed
×
×
×
= __________× ____
T
N M
M
N
M M
N M
M N
N N
G
G
B
We then considered the eigenvalue problem for B
Bw i = η i w i i = 1, 2,... , ( N + M ) (6.11)
This led us to the shifted eigenvalue problem
Gv i = η i u i i = 1, 2,... , ( N + M ) (6.17)
and
G T u i = η i v i i = 1, 2,... , ( N + M ) (6.18)
We found that the shifted eigenvalue problem leads us to eigenvalue problems for G T G and GG T^ :
G T Gv i = η^2 i v i i = 1, 2,... , M (6.21)
and
GG T u i = η i^2 u i i = 1 , 2,... , N (6.29)
We then introduced the singular valuefrom Equations (6.20) and (6.28) λ i , given by the positive square root of the eigenvalues
λ i = + η i^2 (6.52)
Equations (6.16), (6.17), (6.20) and (6.28) give us a way to findeventually, to U , V , and Λ. They also lead,
N × G^ M =^ N U ×^ N N Λ× M M V × T M^ (6.60) We then considered partitioning the matrices based onThis led us to singular-value decomposition P , the number of nonzero singular values.
N ×^ G^ M =^ N U × PP^ P Λ× PP^ P V × PM T^ (6.76) Before considering an inverse operator based on singular-value decomposition, it is probably useful to cover the mechanics of singular-value decomposition.
6.9 Mechanics of Singular-Value Decomposition
The steps involved in singular-value decomposition are as follows: Step 1. Begin with Gm = d. Formmore observations than model parameters; thus, G T G ( M × M ) or GG T^ ( N × N ), whichever is smaller. (N.B. Typically, there are N > M , and G T G is the more common choice.) Step 2. Solve the eigenvalue problem for Hermitian G T G (or GG T^ )
G T Gv i = η^2 i v i (6.21)
1. Find the P nonzero η^2 i^ and associated v i.
2. Let λ i = +( η^2 i^ )1/^.
- Form
