Generalized Additive Models - Class Notes | ENEE 698A, Study notes of Electrical and Electronics Engineering

Download Generalized Additive Models - Class Notes | ENEE 698A and more Study notes Electrical and Electronics Engineering in PDF only on Docsity!

Generalized Additive Models

T. Hastie, R. Tibshirani and J. Friedman

The Elements of Statistical Learning, Springer 2001

Presented by:

Amit Juneja

Agenda

• A review of linear smoothers (linear non-parametric regression

methods)

• Additive models
• Backfitting algorithm
• Example
• Conclusion

1.1 Examples

1. Running mean smoother

A fit at xi is produced by averaging the data points in a

neighborhood Ni around xi.

2. Cubic smoothing spline

A function g is found that minimizes

n ∑

i

(yi − g(xi))

2

• λ

−∞

[g

′′ (z)]

2 dz (3)

The solution - cubic spline - is a linear smoother of the form

yˆ = Sy

Examples (contd)

2 Additive models

• Suppose the data consists of n realizations of random variable

Y at p design values

{(y 1 , x 11 , ..., xip), ..., (yn, xn 1 , ..., xnp)} (4)

• The additive models take the form

E(Yi|xi 1 , ..., xip) =

p ∑

j=

fj (xij ). (5)

2.1 Motivation for additive models

There are problems related to multi-dimensional smoothers

1. The curse of dimensionality
2. Different dimensions may have different units and may be

highly correlated so the metric assumption may be hard to

justify

3. The multivariate smoothers are computationally expensive

Application of orthogonality principle

• The optimization problem is to minimize E(Y − g(X))

2 over

g(X) =

∑p

j= fj (Xj ) ∈ H

add

• By principle of orthogonality Y − g(X) ⊥ H

add

• Equivalently Y − g(X) ⊥ Hi, ∀i = 1, ..., p
• This gives

fi(Xi) = Pi(Y −

j 6 =i

fj (Xj )) = E(Y −

j 6 =i

fj (Xj )|Xi) (6)

Normal equations

• We get the normal equations

I P 1 P 1 ... P 1

P 2 I P 2 ... P 2

Pp Pp Pp ... I

f 1 (X 1 )

f 2 (X 2 )

fp(Xp)

P 1 Y

P 2 Y

PpYp

or

Pf = QY (8)

3 Algorithm

• We assume that a solution exists
• The backfitting or Gauss-Seidel algorithm

Initialize : f = f

0 i , i^ = 1,^2 , ..., p

Cycle :j = 1, 2 , ..., p, 1 , 2 , ..., p, ...,

fj ← Sj (y −

k 6 =j

fk) (11)

U ntil :The individual functions do not change (12)

3.1 Existence of the solution

The following results hold

1. For symmetric smoothers with eigenvalues in [0, 1], the normal

equations Pfˆ = Qyˆ always have at least one solution

2. The solution is unique unless there exists a g 6 = 0 such that

Pgˆ = 0, a phenomena called concurvity

3. For this same class of smoothers concurvity can only occur if

there is a linear dependence among the eigenspaces of the S

′ j s

corresponding to the eigenvalue +

4. For this same class of smoothers the Gauss-Siedel procedure

always converge to some solution of Pfˆ = Qyˆ

Results

Predicted class

True Class email spam

email 58.5% 2.5%

spam 2.7% 36.2%

Inference from individual estimates

6 References

• A. Buja, T. Hastie, and R. Tibshirani, ”Linear smoothers and

additive models”, The Annals of Statistics, Vol 17., No. 2

(Jun., 1989), 453-

• T. Hastie, and R. Tibshirani, ”Generalized additive models”,

Statistical Science, Vol 1, pp 297-