Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Material Type: Notes; Class: INTRO NUM ANALYS & COMPUT; Subject: Mathematics; University: San Diego State University; Term: Fall 2002;
Typology: Study notes
1 / 54
This page cannot be seen from the preview
Don't miss anything!
Lecture Notes # — Approximation Theory — Discrete Least Squares Approximation
Peter Blomgren, 〈[email protected]〉
Department of Mathematics and Statistics Dynamical Systems Group Computational Sciences Research Center San Diego State University San Diego, CA 92182- http://terminus.sdsu.edu/
Fall 2009
Outline
(^1) Approximation Theory: Discrete Least Squares Introduction Discrete Least Squares
(^2) Discrete Least Squares A Simple, Powerful Approach
3 Discrete Least Squares Application: Cricket Thermometer
Discrete Least Squares Discrete Least Squares
Introduction: Matching a Few Parameters to a Lot of Data.
Sometimes we get a lot of data, many observations, and want to fit it to a simple model.
0 1 2 3 4 5
0
2
4
6
8
Measured Data Average
Discrete Least Squares Discrete Least Squares
Introduction: Matching a Few Parameters to a Lot of Data.
Sometimes we get a lot of data, many observations, and want to fit it to a simple model.
0 1 2 3 4 5
0
2
4
6
8
Measured Data Linear Best Fit
Discrete Least Squares Discrete Least Squares
Introduction: Matching a Few Parameters to a Lot of Data.
Sometimes we get a lot of data, many observations, and want to fit it to a simple model.
0 1 2 3 4 5
0
2
4
6
8
Underlying function f(x) = 1 + x + x^2/ Measured Data
Discrete Least Squares Discrete Least Squares
Introduction: Matching a Few Parameters to a Lot of Data.
Sometimes we get a lot of data, many observations, and want to fit it to a simple model.
0 1 2 3 4 5
0
2
4
6
8
Underlying function f(x) = 1 + x + x^2/ Measured Data Average Linear Best Fit Quadratic Best Fit PDF-link: code.
Discrete Least Squares Discrete Least Squares
Interpolation: A Bad Idea?
We can probably agree that trying to interpolate this data set:
0 1 2 3 4 5
0
2
4
6
8
Measured Data with a 50th degree polynomial is not the best idea in the world... Even fitting a cubic spline to this data gives wild oscillations! [I tried, and it was not pretty!]
Discrete Least Squares Discrete Least Squares
Defining “Best Fit” — the Residual.
We are going to relax the requirement that the approximating function must pass through all the data points. Now we need a measurement of how well our approximation fits the data. — A definition of “best fit.”
Discrete Least Squares Discrete Least Squares
What is the Size of the Residual?
There are many possible choices, e.g.
E 1 =
∑^ n
i=
|ri | ⇔ E 1 = ‖˜r‖ 1
E 2 =
√√ √√ ∑n
i=
|ri |^2 ⇔ E 2 = ‖˜r‖ 2
E∞ = max 0 ≤i≤n |ri | ⇔ E∞ = ‖˜r‖∞
In most cases, the sum-of-the-squares version is the easiest to work with. (From now on we will focus on this choice...)
Discrete Least Squares Discrete Least Squares
Discrete Least Squares Approximation
We have chosen the sum-of-squares measurement for errors. Lets find the constant that best fits the data, minimize
E (C ) =
∑^ n
i=
(fi − C )^2.
If C ∗^ is a minimizer, then E ′(C ∗) = 0 [derivative at a max/min is zero]
Discrete Least Squares Discrete Least Squares
Discrete Least Squares Approximation
We have chosen the sum-of-squares measurement for errors. Lets find the constant that best fits the data, minimize
E (C ) =
∑^ n
i=
(fi − C )^2.
If C ∗^ is a minimizer, then E ′(C ∗) = 0 [derivative at a max/min is zero]
E ′(C ) = −
∑^ n
i=
2(fi − C ) = − 2
∑^ n
i=
fi + 2(n + 1)C , ︸ ︷︷ ︸ Set =0, and solve for C
E ′′(C ) = 2(n + 1) ︸ ︷︷ ︸ Positive
hence
C∗^ = 1 n + 1
∑^ n
i= 0
fi, it is a min since E ′′(C ∗) = 2(n + 1) > 0.
is the constant that best the fits the data. (Note: C ∗^ is the average.)
Discrete Least Squares Discrete Least Squares
Discrete Least Squares: Linear Approximation.
The form of Least Squares you are most likely to see: Find the Linear Function, p 1 (x) = a 0 + a 1 x, that best fits the data.
Discrete Least Squares Discrete Least Squares
Discrete Least Squares: Linear Approximation.
The form of Least Squares you are most likely to see: Find the Linear Function, p 1 (x) = a 0 + a 1 x, that best fits the data. The error E (a 0 , a 1 ) we need to minimize is:
E (a 0 , a 1 ) =
∑^ n
i=
[(a 0 + a 1 xi ) − fi ]^2.
The first partial derivatives with respect to a 0 and a 1 better be zero at the minimum:
∂ ∂a 0
E (a 0 , a 1 ) = 2
∑^ n
i=
[(a 0 + a 1 xi ) − fi ] = 0
∂ ∂a 1
E (a 0 , a 1 ) = 2
∑^ n
i=
xi [(a 0 + a 1 xi ) − fi ] = 0.
We “massage” these expressions to get the Normal Equations...
Discrete Least Squares Discrete Least Squares
Linear Approximation: The Normal Equations p 1 (x)
∑^ n
i=
[(a 0 + a 1 xi ) − fi ] = 0
∑^ n
i=
xi [(a 0 + a 1 xi ) − fi ] = 0.
