Download Linear Algebra, Lecture Notes - Mathematics - 5 and more Study notes Mathematics in PDF only on Docsity!
IB Paper 7: Linear Algebra Handout 5
5. Properties of the Fundamental Subspaces of A
5.1 Bases for the Four Spaces for AI and AII
In order to discuss the properties of the subspaces further it is useful to complete the process of finding bases for the two examples examined in section 3.7 and 4.
CASE I A =
CASE II A =
The story so far is
LU =
LU =
Column Space (the columns of L used, i.e. corresponding to non-zero rows of U )
1 0 0
2 , 1 and 0
1 2 1
2 and 1
1 2
Null Space (= Null space of U ) (Set the free variables to 1 in turn and solve U x = 0 )
3 7 / 2 1 1
and 0 1 1 0
Row Space All non-zero rows of U
1 0 0 2 2 0 , and 1 1 1 3 6 1
^ −
and 1 1 3 6
Left Null Space
see later
5.2 A few useful properties of the Fundamental Subspaces
(a) The dimension of row-space is thus equal to the number of non-zero rows of U which is equal to the number of basic variables (those with pivots). So
Dimension of row space = r = rank( A) = dimension of column space
(b) The dimension of the null space of A is equal to the number of free variables (i.e. the number of variables without pivots). Thus
Dimension of null space = n − r.
(c) Suppose n is in the null space of A , this mean that
0 = A n =
1 2 ... ... ...
m
a a
a
←^ →
←^ →
n
1 2 ...
m
a n a n
a n
This means that a vector n is in null-space if, and only if, it is orthogonal to e very row of A and hence orthogonal to every vector in row space. i.e. Null space and Row Space are orthogonal
(d) Everything we have said about A , applies to AT. Thus
the dimension of left null-space = m − r.
Every vector in Column Space is orthogonal to every vector in Left Null Space.
Row space and Null Space are said to be orthogonal complements in n , and Column Space and
Left Null space are orthogonal complements in m. Which means that they are orthogonal and between them they have a total number of dimensions which is equal n and m respectively. (i.e.
there is “nowhere else to go” in n^ and m.
5.3 Orthogonal Complementary Subspaces
Let us step back to 3D for a moment and imagine we have a vector sub-space consisting of a plane through the origin spanned by the vectors u and v. i.e. the plane is
Sub-space 1 x = α u +β v
This is a two-dimensional subspace of ^3. The line through the origin that is perpendicular to this plane is a
one-dimensional subspace of ^3
Sub-space 2 x = λ t
These sub-spaces are orthogonal complements.
line through 0
v
u
plane t
5.4 The Big Picture
Row space and Null space represent all of n^ , in the sense that any x can be written
x = x row + xnull
and xrow. xnull = 0
Column space and Left Null space similarly carve up
m
b = b col + bleft
bcol. bleft = 0
Notes
- If a vector b has a non-zero bleft then A x = b has no solution. The various compatibility
conditions necessary for there to be a of solution of A x = b can also be stated as b being orthogonal to each of the vectors in a basis of Left Null Space.
- If we restrict the vectors that A operates on to Row Space, then it is clear that A maps an r - dimensional space onto Column Space (which is also r -dimensional). The mapping is, therefore, reversible.
Row Space
Null Space n dimension n − r
dimension r
x
xrow
xnull orthogonal
Column space
Left Null Space
m
dimension m − r
dimension r
b
bcol
bleft
orthogonal
5.5 Finding Left-Null space
The fact that Left-Null Space is perpendicular to Column Space is the clue to the easiest way to find it. It should be clear for CASE I ( m = 3, n = 4), for example, that because r = 3 the dimension of left null space must be 3 − 3 = 0 (i.e. null space is just 0 ).
For CASE II,
AII =
Any vector in the left null space is orthogonal to every vector in column space and the basis of column space is the first two columns of L. So if n is in null-space
T 1 RED 2
l b l
^ ← →
L b
2
l b l b
1
2
3
b
b
b
−^ ^
^
^
b 3 is a free variable. If we take it equal to 1, then
b 3 = 1 , b 2 = − 2 and b 1 = b 3 − 2 b 2 = 5, giving a basis of the left null space
If there is more than one “free value”, treat them just as we did the free variables in the general solution of A x = b , i.e. take then as unity in turn with the rest 0.
6. Final Bits and Pieces
6.1 Algorithmic Complexity of LU
We saw in section 3.5 that LU decomposition with partial pivoting is a robust algorithm. The other quality we would want from a good general purpose algorithm is that it should be quick for large values of m and n.
How many operations does it take to:
Complete the LU factorisation?
Complete the solution of A x = b?
Different computers take different amounts of time to perform different operations (for example division used to be very slow relative to addition or multiplication) and there is some ambiguity as to what constitutes an "operation". We will estimate the number of "operations" by considering an n × n (i.e. a square) case. We will ignore assignment statements and count each addition, subtraction, multiplication and division as one operation. The results will be rough, but will give an order of magnitude estimate of how much cpu time is involved.
N.B. Error on handout
For large n , the LU factorisation requires ≈ 3 3
n operations.
Once the LU factorisation has been completed, it requires ≈ 2 n^2 operations to complete the solution of A x = b These numbers are remarkably small.
For comparison let us compare simply multiplying two n × n matrices together AX = B. To form each element of B, we take
b ij =
1 2 1 2
j j i i in
nj
x x a a a
x
^
^
^
^
^
^
^
... (^) ... = ai (^) 1 1 x (^) j + ai (^) 2 x 2 (^) j + ... + ain xnj
n 2 elements, each element is a dot product, dot product = n multiplications + n −1 additions
By this way of reckoning operations, then, it requires n^2 ( n + n − 1 )≈ 2 n^3 to multiply two n × n
matrices together and ≈ 2 n^2 to multiply an n × n matrix and a vector.
6.2 Some other methods involving manipulations of rows of matrices.
We showed in handout 2, that the product
A = B C
where A is an m × n matrix, B is an m × k matrix, C is an k × n matrix, is equivalent to
[ ] [ ]
[ ]
11 12 1 21 22 2 1 2 1 2
1 2
n n n
k k kn k
c c c c c c a a a b b
c c c b
=^ +^ +
Let us assume that, as is the case of U when we are doing LU decomposition, that B is square and C is the same shape as A. i.e. k = m
[ ] [ ]
[ ]
11 12 1 21 22 2 1 2 1 2
1 2
n n n
m m mn m
c c c c c c a a a b b
c c c b
=^ +^ +
Suppose we now add an extra column at the end of the row C to get a larger matrix.
Cext =
11 12 1 1 21 22 2 2
1 2
n n
m m mn m
c c c d c c c d
c c c d
where the extra column is represented by d. This extends the matrix A by a column, which we denote as e.
Aext =
11 12 1 1 21 22 2 2
1 2
n n
m m mn m
a a a e a a a e
a a a e
[ ] [ ]
[ ]
11 12 1 1 21 22 2 2 1 2 1 2
1 2
n n n
m m mn m m
c c c d c c c d a a a e b b
c c c d b
=^ +
The original columns of the matrix A are unaltered by this and e is given by
[ 1 ] [ 2 ] [ ]
m m
d d d e b b b
=^ +^ +^ +
and we can see that the column vector d is simply multiplied by the matrix B to get e. i.e.
[ A e ] = B [ C d ] = [ BC B d ]
This will in fact work with matrices D and E provided they have the same number of rows as A , so that the combined matrix makes sense.
[ A E ] = B [ C D ] = [ BC BD ]
This technique is sometimes used by programs using LU decomposition to solve A x = b to shortcut the solution of L c = b , we stick b in as an extra final column to A and then do our LU factorisation as normal to get
N.B. Error on handout
Subtract 1 × middle row from the top and the third
Multiply the third row by a factor
Add
of the bottom row to the middle and
of the bottom row to the top
−^ +^ +
= [ I D ]
It should be clear that D = A −^1. [ A I ] = L [ I D ] = [ L LD ]
If we had kept track of the matrix L , it would have become A.
Key Points from Lecture 5
The four fundamental subspaces associated with A , partition m^ and n^ as shown.
Column Space
Left Null Space
Row Space
Null Space dimension n − r
dimension r^ dimension^ r
dimension m − r
A^0
A
n^ m
Can’t get here with A x
y here y ⊥ every column of A
Every vector here ⊥ every vector here
Summary ( This is taken verbatim from the Maths Databook)
Rank The rank, r , of a matrix is the number of independent rows, or columns.
Fundamental Subspaces of an m ×××× n matrix A The column space is the space spanned by the columns. It has dimension equal to the rank, r , and is a subspace of R m. The nullspace is the space spanned by the solutions x of the equation Ax = 0. The nullspace has dimension n − r and is a subspace of R n. The row space is the space spanned by the rows of A. It has dimension equal to r and is a subspace of R n. The left-nullspace is the space spanned by the solutions y of the equation y t^ A = 0. It has dimension m − r , and is a subspace of R m.
The nullspace is the orthogonal complement of the row space in R n.
The left-nullspace is the orthogonal complement of the column space in R m.
For A x = b to have a solution, b must lie in the column space, i.e. y t^ b = 0 for any y such that A t^ y = 0
