Learning and Generalization - Introduction to Pattern Recognition - Lecture Slides, Slides of Advanced Computer Architecture

Download Learning and Generalization - Introduction to Pattern Recognition - Lecture Slides and more Slides Advanced Computer Architecture in PDF only on Docsity!

Learning and Generalization^ •^ We have been discussing the issue of generalizationabilities of learnt classifiers.

Learning and Generalization^ •^ We have been discussing the issue of generalizationabilities of learnt classifiers.^ •^ We learn a classifier using the training examples.

Learning and Generalization^ •^ We have been discussing the issue of generalizationabilities of learnt classifiers.^ •^ We learn a classifier using the training examples.^ •^ Given training examples,

{(X, y)}, we want to learnii

a general rule (a classifier or a function) that wouldpredict the ‘target’ or ‘output’

y^ given the ‘instance’ or

‘input’^ X. • The question is how do we formally define the goal oflearning?

Learning and Generalization^ •^ We have been discussing the issue of generalizationabilities of learnt classifiers.^ •^ We learn a classifier using the training examples.^ •^ Given training examples,

{(X, y)}, we want to learnii

a general rule (a classifier or a function) that wouldpredict the ‘target’ or ‘output’

y^ given the ‘instance’ or

‘input’^ X. • The question is how do we formally define the goal oflearning? • Can we say whether a learning algorithm is learningcorrectly?

-^ Any learning algorithm takes training data as the inputand outputs a specific classifier/function. •^ For this, it searches over some chosen family offunctions to find one that optimizes a chosen criterionfunction.

-^ Any learning algorithm takes training data as the inputand outputs a specific classifier/function. •^ For this, it searches over some chosen family offunctions to find one that optimizes a chosen criterionfunction.^ {(X, y)} →ii

Learning Algorithm (searching over

→^ f^ ∈ F F) PR NPTEL course – p.8/

-^ As discussed in the previous lecture, a simpleexample problem is ‘curve-fitting’.

-^ As discussed in the previous lecture, a simpleexample problem is ‘curve-fitting’. •^ Given training data

{(X, y), i^ = 1ii

,^ · · ·^ , n}, we want

to learn^ f^ so that

y^ ≈^ f^ (x).

-^ As discussed in the previous lecture, a simpleexample problem is ‘curve-fitting’. •^ Given training data

{(X, y), i^ = 1ii

,^ · · ·^ , n}, we want

to learn^ f^ so that

y^ ≈^ f^ (x).

-^ As we saw, the ‘data error’ is not a good measure forrating possible

f^.

-^ There are many different ways of formalizing this.

-^ For example, a generic approach is what is called^ Minimum Description Length

principle.

-^ For example, a generic approach is what is called^ Minimum Description Length

principle.

-^ Suppose we want to send the data over acommunication channel. •^ we can send the

2 n^ numbers,^ X

, yusing someii^

number of bits.

-^ For example, a generic approach is what is called^ Minimum Description Length

principle.

-^ Suppose we want to send the data over acommunication channel. •^ we can send the

2 n^ numbers,^ X

, yusing someii^

number of bits. • Or we can send

X, the functioni

f^ and the errors

y−^ f^ (X).i^ i

-^ If the fit is good, the errors

y−^ f^ (X)^ would havei^ i

small range and we may be able to send them usingsmaller number of bits compared sending

y.i

-^ However, we also need to send

f^.^ PR NPTEL course – p.19/

-^ If the fit is good, the errors

y−^ f^ (X)^ would havei^ i

small range and we may be able to send them usingsmaller number of bits compared sending

y.i

-^ However, we also need to send

f^.

-^ If^ f^ is very complex, then what we save in bits bysending errors instead of

ymay be more than offseti^

by the bits needed to send description of

f^.^ PR NPTEL course – p.20/