Download Orthogonality - Computer Sciences - Lecture Slides and more Slides Operating Systems in PDF only on Docsity!
THE GRAM-SCHMIDT ALGORITHM AND QR [6.4]
Orthogonality – The Gram-Schmidt algorithm
- Two vectors u and v are orthogonal if u.v = 0.
- They are orthonormal if in addition ‖u‖ = ‖v‖ = 1
- In this case the matrix Q = [u, v] is such
Q
T Q = I
ä We say that the system {u, v} is orthonormal ..
ä .. and that the matrix Q has orthonormal columns
ä .. or that it is orthogonal [Text reserves this term to
n × n case]
2
Example: An orthonormal system {u, v}
u =
v =
Generalization:
ä A system of vectors {v 1 ,... , vn} is orthogonal if vi.vj =
0 for i 6 = j; and orthonormal if in addition ‖vi‖ = 1 for
i = 1, · · · , n
ä A matrix is orthogonal if its columns are orthonormal
ä Then: V = [v 1 ,... , vn] has orthonormal columns
[Note: The term ’orthonormal matrix’ is not used. ‘orthog-
onal’ is often used for square matrices only (textbook)]
The Gram-Schmidt algorithm
Problem: Given a set {u 1 , u 2 } how can we generate
another set {q 1 , q 2 } from linear combinations of u 1 , u 2 so
that {q 1 , q 2 } is ortonormal?
Step 1 Define first vector: q 1 = u 1 /‖u 1
Step 2: Orthogonalize u 2 against q 1 :
q ˆ = u 2 − (u 2 .q 1 ) q 1
Step 3 Normalize to get second vector:
q 2 = ˆq/‖ qˆ‖
ä Result: {q 1 , q 2 } is an orthonormal set of vectors which
spans the same space as {u 1 , u 2 }.
5
The operations in step 2 can be written as
q ˆ := ORT H(u 2 , q 1 )
ORTH (u 2 , q 1 ): “orthonormalize u 2 against q 1 ”
ä ORT H(x, q) denotes the operation of orthogonalizing
a vector x against a unit vector q.
q
z x
(x,q)
Result of z = ORT H(x, q)
6
Example: u 1 =
u 2 =
Step 1: q 1 =
1 2
Step 2: First compute
u 2 .q 1 = ... = 2. Then:
q ˆ =
− 2 ×
Step 3:
Normalize
q 2 =
1 2
Generalization: 3 vectors
ä How to generalize to 3 or more vectors?
ä For 3 vectors : [u 1 , u 2 , u 3 ]
- First 2 steps are the same → q 1 , q 2
- Then orthogonalize u 3 against q 1 and q 2 :
q ˆ = u 3 − (u 3 .q 1 )q 1 − (u 3 .q 2 )q 2
q 3 = ˆq/‖ qˆ‖
General problem: Given U = [u 1 ,... , un], compute Q =
[q 1 ,... , qn] which is orthonormal and s.t. Col(Q) =
Col(U ).
qˆ 2 =
1 2 1 2 −
1 2 −
1 2
; q 2 =
1 2 1 2 −
1 2 −
1 2
q ˆ 3 = u 3 − (u 3 , q 1 )q 1 − (u 3 , q 2 )q 2
− 2 ×
1 2 1 2 1 2 1 2
− (−1) ×
1 2 1 2 −
1 2 −
1 2
1 2 −
1 2 − 2. 5
‖ qˆ 3 ‖ =
13 → q 3 =
qˆ 3
‖ qˆ 3 ‖
1 2 −
1 2 − 2. 5
- 5
13
- For this example: 1) what is Q? what is R?
- Verify (matlab) that U = QR
- Compute Q T Q. [Result should be the identity matrix]
Recall: For a matrix Q with orthonormal columns, we
have
Q T Q = I
Consequence: For an n × n orthogonal matrix
Q
− 1 = Q
T
. (Q is sometimes said to be unitary)
14
Solving Least-Squares systems via the QR factorization
ä In practice: not a good idea to solve the system A T Ax =
A
T b. Use the QR factorization instead. How?
ä Answer in the form of an exercise
Problem: Ax ≈ b in least-squares sense
A is an m × n (full-rank) matrix.
Consider the QR factorization of A
A = QR
- Approach 1: Write the normal equations – then ‘sim-
plify’
- Approach 2: Write the condition b − Ax ⊥ Col(A) and
recall that A and Q have the same column space.
