Least Squares Method: Linear Regression - Fitting Data to Parametric Models, Study notes of Mathematics

Download Least Squares Method: Linear Regression - Fitting Data to Parametric Models and more Study notes Mathematics in PDF only on Docsity!

Computational Methods CMSC/AMSC/MAPL 460Least squares method: linear regression^ Ramani Duraiswami,Dept. of Computer Science

Fitting data to a model • Practical science involves lots of fitting of data to models• Difference between fitting and interpolation? – Interpolation, the fit function passes through the point– Fitting, the fit function satisfies some error criterion • Tasks arise commonly in science – Fit straight lines and curves to data– More generally fit data to a parametric model • Parametric: Model contains parameters – Job of fitting is to estimate the parameters that “best” make themodel fit the data– “best”
define best • Simplest example of model fitting problem – Linear regression

Relationships among Variables • In much science we seek relations between variables One variable is used to “explain” another variable X VariableIndependent VariableExplaining VariableExogenous VariablePredictor Variable

Y^ VariableDependent VariableResponse VariableEndogenous VariableCriterion Variable

slope: b 1

:spredictionerrorlesshadweIf bXaY += intercept: a

X

Simple Least-Squares Regression Y

X

Estimated Regression Line Y ˆ yye^ −=^ iii^ y i ˆ y i x^ i

:LineRegressiontheofEquationˆ bXaY +=

X

Estimated Regression Line Y

errors/residuals

How do we find a and b? NN^22 ( ) ^ ^ +−=∑∑ == 1 i 1 i

idualserrors/ressquaredof

sumtheminimizeto,

In Least-Squares Regression:Find ba^ abxye^ iii

Least Squares for more complex models • Number of equations and unknowns may not match• Look for solution by minimizing some cost function• Simplest and most intuitive cost function: ||

Ax - b ||^2

-^ Define for each data point

x a residual^ ri^ i

-^ Minimize^ ∑ rr with respect to i^ i^ i^

xl

-^ ∑ rr =∑ (Ax-b).^ ∑ ii^ i^ ijji k^ j^

(Ax-b)ikki

Normal equationst t • The system AA x = Ab is called the Normal equations• Can solve least squares problems using these• For A size m × n and x of size n and b of size m what arethe dimensions of the normal equations? – n × n • Solve via LU decomposition• Is this a good idea?t^ – Somewhat expensive as we have to form AA which involvesmatrix multiplication and then solution– More importantly it is poorly conditionedt^2 – cond(AA) = (cond(A))