Math 311 Lecture 22
Least squares approximation
NOTE. For column vectors u,v: the dot product vu = the
matrix product vTu.
Let A be a matrix.
Let W = the column space of A = the space spanned by
the columns of A.
THEOREM. AX = b has a solution iff b is in the column
space of A.
PROOF. Suppose v1, v2, ..., vn are the columns of A and
suppose X = [x1, x2, ..., xn]T. WA = (v1 | v2 | ... | vn) and
AX = (v1 | v2 | ... | vn)[x1, x2, ..., xn]T = x1v1+ x2v2 + ... + xnvn.
Hence b = AX
iff b = x1v1+ x2v2 + ... + xnvn
iff b is a linear combination of the columns of A
iff b is in the column space of A.
Suppose b is not in the column space W of A. Thus AX = b
has no solution.
How close can one come to a solution? I.e., what X gives
a value AX which is the closest possible to b?
Since the vectors AX are exactly the vectors of the
column space W of A, this is the same as asking which
vector in W is the closest to b. The answer is projWb,
the projection of b onto W.
Since projWb LW, AX = projWb does have a solution X.
This X is the least-squares solution, it is the best
approximate solution of AX = b.
We could find the least-squares solution by calculating
projWb and then solving AX = projWb. But there is an
easier way.
THEOREM. If A is an m[n matrix of rank n, the
least-squares solution for AX = b, is the exact solution
to the exact equation (ATA)X = (ATb).
PROOF. Suppose A = (v1 | v2 | ... | vn) and suppose X is the
least-squares solution for AX = b.
X the least-squares solution for AX = b
î AX = projWb(by definition of “least-squares”)
î bAX = bprojWb is 7 to the column space of A.
î bAX is perpendicular to each column vi of A.
î vi(bAX) = 0 for each column vi.
î viT(bAX) = 0 for each column vi.
î .
v1
T
v2
T
...
vn
T
(b−AX)=O
î AT(bAX) = O.
î ATb ATAX = O.
î ATAX = ATb. áexact equation E
To solve (ATA)X = (ATb), first find (ATb) and (ATA).
CSuppose b = and A = and X = .
1
1
1
10
01
00
x
y
Find the best approximate solution of AX = b.
Solution: ATb = [1,1]T. (ATA) = I2. Hence
(ATA)X = (ATb) becomes I2 X = [1,1]T. Hence X = [1,1]T.
The error vector of an approximate solution to AX = b is
the difference e = bAX between the desired value b
and approximate value AX found. The least-squares
solution has the smallest error, i.e., ||e|| is minimum.
Approximating functions
Suppose we know the values f(t1), f(t2), f(t3) of an
otherwise unknown function f(t). Suppose we wish to
approximate it as a linear combination
af1(t)+bf2(t)+cf3(t) of three known functions f1, f2, f3.
Thus we wish to find the X = [a, b, c]T such that af1(t)+
bf2(t) + cf3(t) gives the best approximation to f(t) for
the n known values. Thus X=[a, b, c]T is the best
solution to
af1(t1) + bf2(t1) + cf3(t1) = f(t1)
af1(t2) + bf2(t2) + cf3(t2) = f(t2)
af1(t3) + bf2(t3) + cf3(t3) = f(t3)
...
af1(tn) + bfn(t) + cf3(tn) = f(tn)
W
.
f1(t1)f2(t1)f3(t1)
f1(t2)f2(t2)f3(t2)
f1(t3)f2(t3)f3(t3)
...
f1(tn)
...
f2(tn)
...
f3(tn)
a
b
c
=
f(t1)
f(t2)
...
f(tn)
Let A be the first matrix, X = [a, b, c]T and B the last
vector. Hence we are trying to find the best
approximate answer to AX = B. This is the exact
solution to ATAX = ATB.
Once we have X = [a, b, c]T, the approximating function is
af1(t) + bf2(t) + cf3(t) and the error vector e =
[f(t1)
(af1(t1)+bf2(t1)+cf3(t1)),...,f(tn)
(af1(tn)+bf2(tn)+cf3(tn))]
= the differences between f(ti) and af1(ti)+ bf2(ti)+ cf3(ti).
CFind the quadratic function which best fits {(-2,6),
(-1,2), (0,1), (1,2), (2,5)}. Also find the error vector.
Quadratic means . W f1(t) = t2, f2(t) = t, f3(t) = 1.
at2+bt +c
For (-2,6): f1(-2) = 4, f2(-2) = -2, f3(-2) = 1, f(-2) = 6. ...
AX = B and the exact equation ATAX = ATB are
, .
4−21
1−11
001
111
421
a
b
c
=
6
2
1
2
5
34 0 10
0100
10 0 5
a
b
c
=
48
−2
16
Least-squares solution: [a, b, c]T = [8/7, -1/5, 32/35]T.
Answer: .
e = [.11, -.26, .086, .14, -.086]T
8
7t2−1
5t+32
35 ||e|| = .34