Bootstrap and Bagging: Estimating Generalization Error and Combining Classifiers, Slides of Mechanical Engineering

Download Bootstrap and Bagging: Estimating Generalization Error and Combining Classifiers and more Slides Mechanical Engineering in PDF only on Docsity!

-^ We are discussing methods to estimate the test error(or true risk) of a model essentially using the trainingdata.

PR NPTEL course – p.1/

-^ We are discussing methods to estimate the test error(or true risk) of a model essentially using the trainingdata. •^ We discussed cross-validation which is a very popularmethod for this.

PR NPTEL course – p.2/

-^ The final cross-validation estimate is the average oferrors of each learnt model on data points not usedfor training that model.

PR NPTEL course – p.4/

-^ The final cross-validation estimate is the average oferrors of each learnt model on data points not usedfor training that model.

n∑ 1 e= cv n i=

−ρ(i)ˆ L( f (X)i

, y)i^ PR NPTEL course – p.5/

Bootstrap Methods^ •^ Bootstrap estimates provide another general methodfor estimating the generalization error.

PR NPTEL course – p.7/

Bootstrap Methods^ •^ Bootstrap estimates provide another general methodfor estimating the generalization error.^ •^ The idea in bootstrap is to generate many training setsby sampling with replacement from the given data.

PR NPTEL course – p.8/

Bootstrap Methods^ •^ Bootstrap estimates provide another general methodfor estimating the generalization error.^ •^ The idea in bootstrap is to generate many training setsby sampling with replacement from the given data.^ •^ Given the original data of

n^ points, we generate

B

number of training sets, each of size

n, by randomly

sampling from the given data set. • Then we learn a model on each of the

B^ training sets.^ PR NPTEL course – p.10/

Bootstrap Methods^ •^ Bootstrap estimates provide another general methodfor estimating the generalization error.^ •^ The idea in bootstrap is to generate many training setsby sampling with replacement from the given data.^ •^ Given the original data of

n^ points, we generate

B

number of training sets, each of size

n, by randomly

sampling from the given data set. • Then we learn a model on each of the

B^ training sets.

-^ The final error estimate could be the average of errorsof all the models.

PR NPTEL course – p.11/

-^ Unlike in cross-validation, here we can have as manytraining sets as we want (all with size

n).

-^ However, they may all be very similar.

PR NPTEL course – p.13/

-^ Unlike in cross-validation, here we can have as manytraining sets as we want (all with size

n).

-^ However, they may all be very similar.b^ ˆ •^ Let^ f^ denote the model learnt using the

th^ bbootstrap

sample,^ b^ = 1

,^ · · ·^ , B.

PR NPTEL course – p.14/

-^ Here we are using the original data set as test datawhile for each

b^ ˆ b, the f is learnt using some of the

same data.

PR NPTEL course – p.16/

-^ Here we are using the original data set as test datawhile for each

b^ ˆ b, the f is learnt using some of the

same data. • Hence this bootstrap error estimate would not be verygood.

PR NPTEL course – p.17/

-^ Here we are using the original data set as test datawhile for each

b^ ˆ b, the f is learnt using some of the

same data. • Hence this bootstrap error estimate would not be verygood. • As an example, consider a problem where the classlabel is independent of the feature vector. • Then the true error rate is

0.^5 (under 0–1 loss).

PR NPTEL course – p.19/

-^ Here we are using the original data set as test datawhile for each

b^ ˆ b, the f is learnt using some of the

same data. • Hence this bootstrap error estimate would not be verygood. • As an example, consider a problem where the classlabel is independent of the feature vector. • Then the true error rate is

0.^5 (under 0–1 loss).

-^ Suppose we use the 1-nearest neighbour as ourclassifier.

PR NPTEL course – p.20/