Download Approximation Theory - Lecture Notes | MATH 541 and more Study notes Mathematics in PDF only on Docsity!
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials Numerical Analysis and Computing^ Lecture Notes #11— Approximation Theory —
Least Squares Approximation & Orthogonal Polynomials
Peter Blomgren, 〈[email protected]
Department of Mathematics and Statistics
Dynamical Systems GroupComputational Sciences Research Center San Diego State UniversitySan Diego, CA 92182-7720 http://terminus.sdsu.edu/^ Fall 2009 Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (1/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials Outline^1 Discrete Least Squares Approximation
Quick Review Example 2 Continuous Least Squares Approximation Introduction... Normal Equations Matrix Properties 3 Orthogonal Polynomials Linear Independence... Weight Functions... Inner Products Least Squares, Redux Orthogonal Functions Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (2/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Quick ReviewExample
Picking Up Where We Left Off...
Discrete Least Squares, I
The Idea:
Given the data set (
˜˜x,f), where
˜x^ =^ {x^0
,^ x,... ,^1
T x}n
˜ and f^ = {f,^ f,... ,^01
T^ f}we want to fitn
a simple
model^ (usually a low degree polynomial,
p(x)) tom
this data. We seek the polynomial, of degree
m, which minimizes the residual: r^ (˜x) =
n∑^ [p(xm i=
)^ −^ f^ (xi i
2 )].
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (3/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Quick ReviewExample
Picking Up Where We Left Off...
Discrete Least Squares, II
We find the polynomial by differentiating the sum with respect tothe^ coefficients
of^ p(xm
). — If we are fitting a fourth degree
polynomial
p(x) =^4
a+^ ax^0
(^2) + ax 2 (^3) + ax+ 3 (^4) ax, we must 4
compute the partial derivatives wrt.
a,^ a,^ a^01
,^ a,^ a. 234
In order to achieve a minimum, we must set all these partialderivatives to zero. — In this case we get 5 equations, for the 5unknowns; the system is known as the
normal equations
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (4/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Quick ReviewExample
The Normal Equations — Second Derivation^ Last time we showed that the normal equations can be found with purelya Linear Algebra argument. Given the data points, and the model (here^ p(x)), we write down the over-determined system:^4
^ a+^ a^0
x+^ ax 10 2 23 +^ ax^30
(^4) + ax 40 =^ f^0
a+^ ax^0
(^2) + ax+ 21 (^3) ax+^ a 31 (^4) x=^41 f^1
a+^ ax^0
(^2) + ax+ 22 (^3) ax+^ a 32 (^4) x=^42 f^2 ...
a+^ ax^0 1 n
(^2) + ax+ 2 n^ (^3) ax+^ a 3 n^ (^4) x=^4 n^ f.n
We can write this as a matrix-vector problem:
˜ X ˜a = f,
where the
Vandermonde matrix
X^ is tall and skinny. By multiplying
both the left- and right-hand-sides by
T^ X (the transpose of
X^ ), we get a
“square” system — we recover the
normal equations:T T^ ˜ X X ˜a^ =^ X^ f.
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (5/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Quick ReviewExample
Discrete Least Squares: A Simple, Powerful Method.^ Given the data set (
˜˜x,f), where
˜x^ =^ {x^0
,^ x,... ,^1 x}^ andn
˜f^ =^ {f,^0
f,... ,^ f^1 n
}, we can quickly find the best polynomial fit for any^ specified polynomial degree! Notation:
j^ Let ˜xbe the vector
jj {x,^ x,... , 01
j x}.n
E.g.^ to compute the best fitting polynomial of degree 4, p(x) =^4
a+^ ax^0
(^2) + ax 2 (^3) + ax+ 3 (^4) ax, define: 4
X =
|^ |^ |^
|^ |
|^ |^ |^
|^ |
˜ 1 ˜x^ ˜x 2 3 ˜x˜x
4 |^ |^ |^
|^ |
|^ |^ |^
,^ and compute
˜a^ = (X
T^ −^1 X^ )(
T˜X f) ︸^ ︷︷
. ︸ Not like this!See math 543!
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (6/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Quick ReviewExample
Example: Fitting
p(x),^ i^ i^
= 0,^1 ,^2
,^3 ,^ 4 Models.
Figure:^
We revisit the ex- ample from last time;
and fit polynomials up to degreefour to the given data.
The figure shows the best
p(x),^0 p(x), and^1
p(x) fits.^2 Below:^
the errors give us clues when to stop.
0 1
2 3
4 5
8 6 4 2 0
Underlying function f(x) = 1 + x + x^2/25 Measured Data Average Linear Best Fit Quadratic Best Fit
Model^
Sum-of-squares-error p(x) (^0)
p(x)^1
p(x)^2
p(x)^3
p(x)^4
Table:^ Clearly in this example there is verylittle to gain in terms of the least-squares-error by going beyond 1st or 2nd degreemodels.
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (7/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Introduction... Normal EquationsMatrix Properties
Introduction: Defining the Problem.^ Up until now:
Discrete Least Squares Approximation
applied
to a collection of data. Now:^ Least Squares Approximation of Functions. We consider problems of this type: —
Suppose f
∈^ C^ [a
,^ b]^ and we have the class
Pn
which is the set of all polynomials of degree at mostn. Find the p
(x)^ ∈ Pn
which minimizes (^) ∫ b [p(x) −^ f^ (x)]a 2 dx.
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (8/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Introduction... Normal EquationsMatrix Properties
The Normal Equations: Inner Product Notation, II^ Discrete Normal Equations in
∑^ Notation: [ nn∑∑^ ak k=0^ i=
] j+k x^ =i n∑j^ xf,^ i^ i^ i=
j^ = 0,^1 ,... ,
n.
Discrete Normal Equations, in Inner Product Notation:
n∑^ [j^ a˜x,k k=
[] (^) k (^) ˜x=^ ˜x ]j ˜, f^ ,^ j = 0,^1 ,... ,
n.
Continuous Normal Equations in Inner Product Notation:
n∑j^ a〈x,^ k^ k=
k^ x〉^ =^ 〈x j^ ,^ f^ (x)〉,^
j^ = 0,^1 ,... ,
n.
Hey! It’s really the same problem!!!
The only thing that
changed is the inner product — we went from summation tointegration!^ Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (13/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Introduction... Normal EquationsMatrix Properties
Normal Equations for the Continuous Problem: Matrices.^ The bottom line is that the polynomial
p(x) that minimizes ∫^ β^ [p(x) α
(^2) − f (x)] dx
is given by the solution of the linear system
~~ Xa = b , where
X=^ 〈xi,j^ i^ j^ ,^ x〉,^ b
i^ = 〈x,^ fi (x)〉.
We can compute
i^ j^ 〈x,^ x〉^
i+j+1β= i+j+1−^ α i +^ j^ + 1^
explicitly.
A matrix with these entries is known as a
Hilbert Matrix
classical examples for demonstrating
how numerical solutions
run into difficulties due to propagation of roundoff errors. —^ We need some new language, and tools!^ Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (14/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Introduction... Normal EquationsMatrix Properties
The Condition Number of a Matrix^ The^ condition number
of a matrix is the ratio of the largest eigenvalue and the smallest eigenvalue:If^ A^ is an
n^ ×^ n^ matrix, and its eigenvalues are |λ| ≤ |λ^1
λ|, then then
condition number
is
cond(A)
|λ|n = |λ|^1
The condition number is one important factor determining thegrowth of the numerical (roundoff) error in a computation.We can interpret the condition number as a
separation of scales
If we compute with sixteen digits of precision
ǫ≈^ mach^
−^1610 , the
best we can expect from our computations (even if we doeverything right), is accuracy
∼^ cond(A)
·^ ǫ.mach
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (15/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Introduction... Normal EquationsMatrix Properties
The Condition Number for Our Example^0 0.
1 1.
2 2.
3 3.
4
(^810710610510410310210110010)
Polynomial Degree Condition Number
0 1
2 3
4 5
8 6 4 2 0
Underlying function f(x) = 1 + x + x^2/25 Measured Data Average Linear Best Fit Quadratic Best Fit
Figure:^ Ponder, yet again, the example of fitting polynomials to the data (Right
). The plot on the left shows the condition numbers for 0th, through 4th degree polynomial problems. Note that for the 5-by-5system (Hilbert matrix) corresponding to the 4th degree problem thecondition number is already
(^7) ∼ 10.
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (16/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
Linearly Independent Functions.^ Definition (Linearly Independent Functions)^ The set of functions
{Φ(x),^0
Φ(x),... ,^1
Φ(x)}^ n
is said to be
linearly independent
on [a,^ b
] if, whenever n∑^ cΦ(xi^ i^ i=
) = 0,^ ∀
x^ ∈^ [a,^ b]
,
then^ c= 0i^
,^ ∀i^ = 0
,^1 ,... ,^ n
. Otherwise the set is said to be
linearly dependent. Theorem If^ Φ(x)^ j^
is a polynomial of degree j, then the set {Φ(x),^0 Φ(x),... ,^1
Φ(x)}^ n
is linearly independent on any interval
[a,^ b].^ Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (17/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
Linearly Independent Functions: Polynomials.^ Theorem^ If^ Φj^ (x)^ is a polynomial of degree j, then the set {Φ(x),^ Φ(x 01
),... ,^ Φn
(x)}^ is linearly independent on any interval [a,^ b]. Proof. Suppose
c∈^ R,^ i^
i^ = 0,^1 ,... ,
n, and^ P
∑(x) = ncΦ(i^ i^ i=^
x) = 0
∀x^ ∈^ [a, b]. Since
P(x) vanishes on [
a,^ b] it must be the
zero-polynomial,
i.e.^ the coefficients of all the powers of
x^ must be
zero. In particular, the coefficient of
n^ xis zero.
⇒^ c= 0, hencen^
P(x) =^ ∑n−^1 ci^ i=^ Φ(x). By repeating the same argument, we findi^ c= 0,^ ii^
= 0,^1 ,... ,
n.^ ⇒ {Φ
(x),^ Φ( 01 x),... ,^ Φ
(x)}^ is linearlyn
independent.^ Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (18/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
More Definitions and Notation... Weight Function^ Theorem^ If^ {Φ
(x),^ Φ 01 (x),... , Φ(x)}^ n
is a collection of linearly independent
polynomials in
P, then any pn
(x)^ ∈ Pn
can be written uniquely as
a linear combination of
{Φ(x),^0
Φ(x),... ,^1
Φ(x)}.n
Definition (Weight Function) An integrable function
w^ is called a weight function on the interval [a,^ b] if^ w
(x)^ ≥^0
∀x^ ∈^ [a, b], but^ w^ (x)^6 ≡^ 0 on any subinterval of
[a,^ b].^ Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (19/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
Weight Function... Inner Product^ A weight function will allow us to assign different degrees ofimportance to different parts of the interval.
E.g.^ with
w^ (x) = 1
√/^1 −^ x 2 on [−^1
,^ 1] we are assigning more weight away
from the center of the interval. Inner Product, with a weight function:
〈f^ (x),^ g (x)〉w^ (x)
∫^ b =^ f^ a
∗(x)g (x) w^ (x)dx
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (20/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
Building Orthogonal Sets of Functions — The Gram-Schmidt Process^ Theorem (Gram-Schmidt Orthogonalization)^ The set of polynomials
{Φ(x)^0 ,^ Φ(x),... ,^1
Φ(x)}^ n
defined in the
following way is orthogonal on
[a,^ b]^ with respect to w
(x):
Φ(x) = 1^0
,^ Φ(x) = (^1
x^ −^ b)Φ^1
, 0
where
〈xΦ b= (^1) (x),^ Φ( 00 x)〉w^ (x) 〈Φ(x),^ Φ^0
(x)〉 0 w^ (x) ,
for k^ ≥^
2 ,^ Φ
(x) = (xk −^ b)Φk^ k
(x)^ −^ c− 1 Φ(x)k k−^2 ,
where^ b=^ k^
〈xΦ(xk−^1
),^ Φ(xk−^1
)〉w^ (x) 〈Φ(x)k−^1
,^ Φ(xk−^1
,^ )〉w (x)
〈xΦc= (^) k (x),^ Φk− 1
(x)〉k− 2 w (x) 〈Φ(x)k−^2
,^ Φ(xk−^2
.)〉w (x)
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (25/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
Example: Legendre Polynomials
1 of 2
The set of Legendre Polynomials
{P(x)}n
is orthogonal on [
−^1 ,^ 1]
with respect to the weight function
w^ (x) = 1. P(x) = 1^0
,^ P(x^1
) = (x^ −
b)^ ◦^11
where
∫^1 − b= 1 x dx 1 = 0 ∫ 1 dx^ − 1
i.e.^ P(x^1
) =^ x.^ ∫^ b=^2
13 xdx −^1 ∫ 12 xdx −^1
= 0,^
∫^1 −c= 2 (^2) xdx 1 = 1 ∫ 11 dx^ − 1
/^3 ,
i.e.^ P(x^2
(^2) ) = x−
1 /^3.
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (26/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
Example: Legendre Polynomials
2 of 2
The first six Legendre Polynomials are
P(x)^0
=^1
P(x)^1 =^ x P(x)^2
(^2) = x−
1 /^3
P(x)^3
(^3) = x− 3 x/^5 P(x)^4
(^4) = x− (^2 6) x/7 + 3
/^35
P(x)^5
(^5) = x− (^3 10) x/9 + 5 x/^21.
We encountered the Legendre polynomials in the context ofnumerical integration. It turns out that the
roots^ of the Legendre
polynomials are used as the nodes in Gaussian quadrature.Now we have the machinery to manufacture Legendre polynomialsof any degree.^ Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (27/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
Example: Laguerre Polynomials^ The set of Laguerre Polynomials
{L(x)}n
is orthogonal on (
,^ ∞)
with respect to the weight function
w^ (x) =
−x^ e.
L(x) = 1,^0
〈x, b= (^1) 1 〉−xe^ = 1 〈 1 , 1 〉−xe
L(x) =^1 x^ −^1 , 〈x(x^ −^ 1) b= (^2)
,^ x^ −^1 〉e
−x 〈x^ −^1 ,^ x
−^1 〉−xe
= 3,^ c
〈x(x= (^2)
−^ 1),^1 〉
−xe 〈^1 ,^1 〉−xe
L(x) = (^2
x^ −^3 )(x
−^1 )^ −^
(^2 1) = x− 4x^ +^2.
Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (28/29)
Discrete Least Squares ApproximationContinuous Least Squares Approximation
Orthogonal Polynomials
Linear Independence... Weight Functions... Inner ProductsLeast Squares, ReduxOrthogonal Functions
Homework #8 — Due Friday 12/4/2009, 12:34pm^ (Part-1)^ BF-8.1.
— Is this a good prediction model? Why/Why not? BF-8.1.12 (Part-2) BF-8.2.1^ Peter Blomgren,
〈[email protected]
〉^ Least Squares & Orthogonal Polynomials
— (29/29)
