Download Eigenvalue Problems - Computer Sciences - Lecture Slides and more Slides Operating Systems in PDF only on Docsity!
EIGENVALE PROBLEMS AND THE SVD. [5.1 TO 5.3 & 7.4]
Eigenvalue Problems. Introduction
Let A an n × n real nonsymmetric matrix. The eigenvalue problem:
Au = λu λ ∈ C : eigenvalue u ∈ Cn^ : eigenvector Example:
A =
[
]
ä λ 1 = 1 with eigenvector u 1 =
0 1
ä λ 2 = 2 with eigenvector u 2 =
1 2
ä The set of eigenvalues of A is called the spectrum of A O-
Eigenvalue Problems. Their origins
- Structural Engineering [Ku = λM u]
- Stability analysis [e.g., electrical networks, mechanical system,..]
- Quantum chemistry and Electronic structure calculations [Schr¨odinger equation..]
- Application of new era: page ranking on the world-wide web.
Basic definitions and properties A scalar λ is called an eigenvalue of a square matrix A if there exists a nonzero vector u such that Au = λu. The vector u is called an eigenvector of A associated with λ.
ä The set of all eigenvalues of A is the ‘spectrum’ of A. Notation: Λ(A). ä λ is an eigenvalue iff the columns of A − λI are linearly dependent. ä λ is an eigenvalue iff det(A − λI) = 0
- Compute the eigenvalues of the matrix:
- Eigenvectors?
A =
Basic definitions and properties (cont.)
ä An eigenvalue is a root of the Characteristic polynomial:
pA(λ) = det(A − λI)
ä So there are n eigenvalues (counted with their multi- plicities).
ä The multiplicity of these eigenvalues as roots of pA are called algebraic multiplicities.
O-
A =
What are the eigenvalues of A? How many eigenvectors can you find?
- Same questions if a 33 is replaced by one.
- Same questions if a 12 is replaced by zero.
- What are all the eigenvalues of a diagonal matrix?
O-
ä Two matrices A and B are similar if there exists an invertible matrix X such that A = XBX−^1
- Show: A and B represent the same mapping in 2 different bases
Definition: A is diagonalizable if it is similar to a diagonal matrix
ä Note : not all matrices are diagonalizable
ä Theorem 1: A matrix is diagonalizable iff it has n linearly independent eigenvectors
Example: Which of these matrices is/are diagonalizable
A =
B^ =
C^ =
ä Theorem 2: The eigenvectors associated with distinct eigenvalues are linearly independent
- Prove the result for 2 distinct eigenvalues ä Consequence: if all eigenvalues of a matrix A are simple then A is diagonalizable. ä Theorem 3: A symmetric matrix has real eigenvalues and is diagonalizable. In addition A admits a set of orthonormal eigenvectors.
Right and Left Singular vectors:
Avi = σiui AT^ uj = σjvj
ä Consequence AT^ Avi = σ^2 i vi and AAT^ ui = σ^2 i ui
ä Right singular vectors (vi’s) are eigenvectors of AT^ A
ä Left singular vectors (ui’s) are eigenvectors of AAT
ä Possible to get the SVD from eigenvectors of AAT^ and AT^ A – but: difficulties due to non-uniqueness of the SVD
O-
Define the r × r matrix Σ 1 = diag(σ 1 ,... , σr)
ä Let A ∈ Rm×n^ and consider AT^ A (∈ Rn×n):
AT^ A = V ΣT^ ΣV T^ → AT^ A = V
[
Σ^21
]
n×n
V T
ä This gives the spectral decomposition of AT^ A.
O-
ä Similarly, U gives the eigenvectors of AAT^.
AAT^ = U
[
Σ^21
]
m×m
U T
Important:
AT^ A = V D 1 V T^ and AAT^ = U D 2 U T^ give the SVD factors U, V up to signs!
A few applications of the SVD Many methods require to approximate the original data (matrix) by a low rank matrix before attempting to solve the original problem
ä Regularization methods require the solution of a least- squares linear system Ax = b approximately in the ‘dom- inant singular’ space of A ä The Latent Semantic Indexing (LSI) method in infor- mation retrieval, performs the “query” in the dominant singular space of A ä Methods utilizing Principal Component Analysis, e.g. Face Recognition.
Information Retrieval: Vector Space Model
ä Given: a collection of documents (columns of a matrix A) and a query vector q.
ä Collection represented by an m × n term by document matrix with aij = LijGiNj
ä Queries (‘pseudo-documents’) q are represented similarly to a column
O-
Vector Space Model - continued
ä Problem: find a column of A that best matches q ä Similarity metric: angle between column c and query q
cos θ(c, q) =
|cT^ q| ‖c‖‖q‖
ä To rank all documents we need to compute s = AT^ q
ä s = similarity vector. ä Literal matching – not very effective. ä Problems with literal matching: polysemy, synonymy,...
O-
Use of the SVD
ä Solution: Extract intrinsic information – or underlying “semantic” information –
ä LSI: replace matrix A by a low rank approximation using the Singular Value Decomposition (SVD) A = U ΣV T^ → Ak = UkΣkV (^) kT
ä Uk : term space, Vk: document space.
ä Refer to this as Truncated SVD (TSVD) approach
ä Amounts to replacing small sing. values of A by zeros
New similarity vector: sk = ATk q = VkΣkU (^) kT q
LSI : an example
%% D1 : INFANT & TODLER first aid %% D2 : BABIES & CHILDREN’s room for your HOME %% D3 : CHILD SAFETY at HOME %% D4 : Your BABY’s HEALTH and SAFETY %% : From INFANT to TODDLER %% D5 : BABY PROOFING basics %% D6 : Your GUIDE to easy rust PROOFING %% D7 : Beanie BABIES collector’s GUIDE %% D8 : SAFETY GUIDE for CHILD PROOFING your HOME %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% TERMS: 1:BABY 2:CHILD 3:GUIDE 4:HEALTH 5:HOME %% 6:INFANT 7:PROOFING 8:SAFETY 9:TODDLER %% Source: Berry and Browne, SIAM., ’
ä Number of documents: 8 ä Number of terms: 9
