Least Squares Method: Linear Regression - Fitting Data to Models - Prof. Ramani Duraiswami, Study notes of Computer Science

Download Least Squares Method: Linear Regression - Fitting Data to Models - Prof. Ramani Duraiswami and more Study notes Computer Science in PDF only on Docsity!

Computational Methods CMSC/AMSC/MAPL 460

Least squares method: linear regression

Ramani Duraiswami, Dept. of Computer Science

Fitting data to a model

•^

Practical science involves lots of fitting of data to models

-^

Difference between fitting and interpolation?– Interpolation, the fit function passes through the point– Fitting, the fit function satisfies some norm based criterion

-^

Tasks arise commonly in science– Fit straight lines and curves to data– More generally fit data to a model

-^

Model contains parameters– Job of fitting is to estimate the parameters that “best” make the

model fit the data

• “best”

Æ

define best

•^

Simplest example of model fitting problem– Linear regression

Relationships among Variables

•^

In much science we seek relations between variables

One variable is used to “explain” another variable

X

Variable

Independent VariableExplaining VariableExogenous Variable

Predictor Variable

Y

Variable

Dependent VariableResponse VariableEndogenous VariableCriterion Variable

Scatter Plots

X

Y

X

Y

We will end up being reasonably confidentthat the true regression line is somewherein the indicated region.

X

Y

Estimated Regression Line

errors/residuals

X

Y

Estimated Regression Line

i

i

i^

y

y

e^

ˆ −

=

y^ i ˆ yi

x^ i

bX a Y^

= ˆ

:

Line

Regression the of

Equation

X

Y

i

i

i^

y

y

e^

ˆ −

=

y^ i i

ˆ y

x^ i

(^

)^

have

also

now

will

We

1

1

∑

∑

=

=

N i

i

i

N i

i^

y

y

e

(^

)^

0

:

Remember

1

=

−

N ∑= i

i^

y

y

In Least-Squares Regression:

(^

(^

)^

X b Y a X X

Y

Y

X

X

b

N i

i

N i

i

i

= =^

1

2

1

=^

=

=^

=

=

⎞ ⎟ ⎠

⎛ ⎜ ⎝ −

⎞ ⎟ ⎠

⎛⎞ ⎜⎟ ⎝⎠

⎛ ⎜ ⎝ −

=

N i

N i

i

i

N i

N i

i

N i

i

i i

X

X

N

Y X Y X N b

1

2

1

2

1

1

1

ComputationalFormula

Models

•^

Have a certain model structure– E.g., “ straight line” “parabolic” “trigonometric” “Gaussian”

-^

Models have specifiable parameters

-^

e.g.

-^

Model

Structure

Data

Parameters

Straight line:

a x + by +c =

(x

,yi

) i

(a,b,c)

Polynomial:

y=c

+c 0

1

x+ …+c

n^

n x

(x

,yi

) i

(^ c

,c 0

1

, …,c

n^

)

Trig.:

y=c

+c 0

1

sin x+ …+c

n^

sin nx

(x

,yi

) i

(^ c

,c 0

1

, …,c

n^

)

Gaussian

y=c

0

exp(-(x-

/^ σ

(x

,yi

) i

(c

“Kernel/RBF”

y=

ai

k(x,xi^

)i

(x

,yi

) i

(a

)i

and

(x

)i

•^

These models are all separable

Linear Systems

A

x

b

= =

Square system:

• unique solution• Gaussian elimination

Rectangular system ??

• underconstrained:

infinity of solutions Minimize |Ax-b|

2

• overconstrained:

no solution

A

x

b

Least Squares for more complex models

-^

Number of equations and unknowns may not match

-^

Data may have noise

-^

Look for solution by minimizing some cost function

-^

Simplest and most intuitive cost function: ||

Ax - b

||^2

•^

Define for each data point

x

a residual i

r

i

•^

Minimize

ri

ri with respect to i^

x

l

•^

r^ i

r

= i

(Aj

xij

-bj

).i

k^

(A

ik

xk

-b

)i

Normal equations

•^

The system A

t^ A x = A

tb

is called the Normal equations

-^

Can solve least squares problems using these

-^

For A size m

×

n and x of size n and b of size m what are

the dimensions of the normal equations?– n

×

n

•^

Solve via LU decomposition

-^

Is this a good idea?– Somewhat expensive as we have to form A

t^ A which involves

matrix multiplication and then solution

• More importantly it is poorly conditioned– cond(A

t A) = (cond(A))

2

•^

Example