Committes and Boosing 2 , Lecture Notes - Computer Science, Study notes of Digital Image Processing

Download Committes and Boosing 2 , Lecture Notes - Computer Science and more Study notes Digital Image Processing in PDF only on Docsity!

CS181 Lecture 4: Committees

and Boosting

Prof. David C. Parkes

1

The NetFlix Competition

• $1M for improving RMSE on test data
• Boosting and ensemble method
• 400+ hypotheses through different methods; a committee with linear weighted combination of ¼ 20 wins!

) Train models sequentially on “residual error” to obtain diversity : boosting.

2

AdaBoost

(Schapire et al.)

AdaBoost + C4. UCI character recognition problem 13.8% to 3.1%!

test error

test error of C4.

combined tree has 2million+ nodes! 4

5

7

Bias-Variance Tradeoff

• Low bias tends to give high variance Examples_._ - Decision stumps have lower variance but higher bias than full decision trees. - 1 st^ order curves have lower variance but higher bias than 9th^ order curves
• Provides an explanation for why complex models don‟t generalize (variance)
• Caution: “bias”  “inductive bias” [although high inductive bias tends to lead to high statistical bias] (^19)

Outwitting the Tradeoff

• Can we reduce the variance of an algorithm without increasing its bias?
• Can we reduce the bias of an algorithm without increasing its variance?

21

Outwitting the Tradeoff

• Can we reduce the variance of an algorithm without increasing its bias?
• Can we reduce the bias of an algorithm without increasing its variance?
• Often, yes… Use a committee!

22

100 hypotheses, 24 parameter model

green: true model

red: c‟tee model

less regularization

(Bishop) 24

Committee

• Learn a set (ensemble) of hypotheses
• To classify, let the hypotheses vote
• Majority vote:
  • Choose the class chosen by most classifiers.
• Weighted majority :
  • Each hypothesis has a weight.
  • Choose class with highest total weight.

25

A Space of Nine Points

a b c

p

q

r

X 2

X 1

What do decision-stump hypotheses look like here?

27

Hypothesis 1

a b c

p

q

r

X 2

X 1

predict positive target

28

Hypothesis 3

a b c

p

q

r

X 2

X 1

predict positive target

30

The Committee Hypothesis

a b c

p

q

r

X 2

X^311