Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Material Type: Notes; Professor: Blomgren; Class: NUMERICAL MATRIX ANALYSIS; Subject: Mathematics; University: San Diego State University; Term: Spring 2010;
Typology: Study notes
1 / 25
This page cannot be seen from the preview
Don't miss anything!
Linear Algebra
Lecture Notes #2 — Introduction / Review
Peter Blomgren,
Department of Mathematics and Statistics Dynamical Systems Group Computational Sciences Research Center San Diego State University San Diego, CA 92182-
http://terminus.sdsu.edu/
Spring 2010
Linear Algebra
Outline
1 Linear Algebra
Intro, Review / Crash Course
Vectors: Transpose, Addition & Subtraction
We express a row vector using the transpose, i.e.
˜x ∈ R
n ⇒ ˜x
x 1 x 2... xn
Vector addition and subtraction
˜x, ˜y ∈ R
n ⇒ ˜x ± ˜y =
x 1
x 2
. . .
xn
y 1
y 2
. . .
yn
x 1 ± y 1
x 2 ± y 2
. . .
xn ± yn
or ˜z = ˜x + ˜y where
zi = xi + yi , i = 1, 2 ,... , n.
Matrices Matrix-Vector Product
An m × n matrix (m rows, n columns) A with real or complex
entries is represented
a 11 a 12 a 13... a 1 n
a 21 a 22 a 23... a 2 n
. . .
am 1 am 2 am 3... amn
aij ∈ R, or
aij ∈ C
Sometimes we write A ∈ R
m×n (or A ∈ C
m×n .)
If A ∈ R
m×n and ˜x ∈ R
n , then the matrix-vector product,
˜b = A˜x, is well defined, and ˜b ∈ Rm, where
bi =
n ∑
j=
aij xj , i = 1, 2 ,... , m.
Matrix-Vector Product... ...as a Linear Combination
b 1
b 2
. . .
bm
=
a 11 a 12 a 13... a 1 n
a 21 a 22 a 23... a 2 n
. . .
. . .
. . .
.. .
. . .
am 1 am 2 am 3... amn
x 1
x 2
x 3
. . .
xn
= x 1
a 11
a 21 . . .
am 1
a 12
a 22 . . .
am 2
a 13
a 23 . . .
am 3
a 1 n
a 2 n . . .
amn
= x 1 ˜a 1 + x 2 ˜a 2 + x 3 ˜a 3 + · · · + xn˜an
Matrix-Vector Product: Linearity
The map ˜x → A˜x (from R
n to R
m , or from C
n to C
m ) is linear,
i.e. ∀˜x, ˜y ∈ R
n (C
n ), and α, β ∈ R (C)
A(˜x + ˜y) = A˜x + A˜y
A(α˜x) = αA˜x
A(α˜x + β˜y) = αA˜x + βA˜y
Note: Every linear map from R
n to R
m can be expressed as mul-
tiplication by an m × n-matrix.
The Vandermonde Matrix... ...Linear Least Squares
Evaluating polynomials using matrix notation may seem cute and
useless?!?
But, wait a minute — this notation looks vaguely familiar from the
discussion of linear least squares (LLSQ) problems from
Math 541.
In case you forgot (or never studied) LLSQ: The goal is to find the
best model in a class (i.e. low-dimensional polynomials) to
measured data (observations yi , made at the points xi ).
The discrepancy (error) between the model and the observations is
measured in the sum-of-squares norm.
Linear Least Squares: Explicit Example 1 of 4
Find the best straight line p(x) = c 0 + c 1 x fitting the observations
(x, y ) ∈ {(0, 1), (1, 2), (2, 2 .5), (3, 4), (4, 7)}.
We have the 5 × 2 Vandermonde matrix A =
˜1 ˜x
, the 2-vector ˜c
(of polynomial coefficients) and the 5-vector ˜y (of measurements):
, ˜c =
c 0
c 1
, ˜y =
The Linear Least Squares Problem: Find the ˜c which minimizes
the least squares error ‖A˜c − ˜y‖
2
Linear Least Squares: Explicit Example 3 of 4
Given a model ˜c, we can evaluate to corresponding linear
polynomial p(x) = c 0 + c 1 x at the points xi : ˜p = A˜c. The
pointwise error in the model is ˜e = ˜p − ˜y:
˜p =
c 0 + 0c 1
c 0 + 1c 1
c 0 + 2c 1
c 0 + 3c 1
c 0 + 4c 1
, ˜e =
c 0 + 0c 1 − 1
c 0 + 1c 1 − 2
c 0 + 2c 1 − 2. 5
c 0 + 3c 1 − 4
c 0 + 4c 1 − 7. 5
The least squares error is given by
rLSQ = ‖˜e‖
2 2 =
i=
e
2 i =^ ‖A˜c^ −^ ˜y‖
2 2
Linear Least Squares: Explicit Example 4 of 4
In order to identify the optimal choice of ˜c, we compute the partial
derivatives with respect to the model parameters, and set those
expressions to be zero (in order to identify the optimum)
∂rLSQ
∂c 0
∂rLSQ
∂c 1
After some work (which is not central to this discussion), we get
the Normal Equations
T A˜c = A
T ˜y ⇔ A
T (A˜c − ˜y) = 0
Even though the matrix A is (usually) tall and skinny (here 5 × 2),
the matrix A
T A is square (here 2 × 2). The (formal) solution
˜c = [A
T A]
− 1 A
T ˜y, to this linear system gives us the coefficients for
the optimal polynomial (the red line on slide 12).
Matrix-Matrix Product
The matrix-matrix product B = AC is well defined if the matrix C
has as many rows as the matrix A has columns
Bk×n = Ak×m · Cm×n
The elements of B are defined by
bij =
∑^ m
k=
aik ckj
Sometimes it is useful to think of the columns of B, ˜bj as linear
combinations of the columns of A:
b^ ˜ j =^ A˜cj =
m ∑
k=
ckj ˜ak
The Transpose of a Matrix (A
T )
The transpose of a matrix A = {aij } is the matrix A
T = {aji }, e.g.:
a 11 a 12 a 13 a 14
a 21 a 22 a 23 a 24
a 31 a 32 a 33 a 34
a 41 a 42 a 43 a 44
a 11 a 21 a 31 a 41
a 12 a 22 a 32 a 42
a 13 a 23 a 33 a 43
a 14 a 24 a 34 a 44
The operation is intellectually simple — just mirror across the
diagonal — but can be quite (memory-access) expensive,
especially for large matrices.
For complex matrices C = {cij }, the complex (Hermitian)
transpose is given by C
H = {c
∗ ji }, where^ c
∗ is the complex
conjugate of c:
c = a + bi, c
∗ = a − bi.
The Rank of a Matrix Am×n
The column rank of a matrix is the dimension of range(A), its
“column space.” The row rank of a matrix is the dimension of its
“row space,” or range(A
T ).
The column rank is always equal to the row rank (we will see the
proof of this in a few lectures), hence we only refer to the Rank of
a matrix
rank(A)
An m × n matrix is of full rank if it has the maximal possible rank
min(m, n).
An m × n (m ≥ n) matrix A with full rank must have n linearly
independent columns.
Recall: The Normal Equations
T A˜c = A
T ˜y ⇔ A
T (A˜c − ˜y) = 0
Due to the “tall-and-skinniness” of A, the equation A˜c − ˜y = 0
does not have a solution.
Given a vector ˜c we can define the residual, ˜r(˜c) = A˜c − ˜y, which
measures how far from solving the system we are.
We notice that the solution to the normal equations requires that
the residual is in the nullspace of A
T .
The solution is in range(A) such that the residual is orthogonal
(perpendicular) to range
T
