Lecture 23: Classification and Inductive Bias in Machine Learning by David C. Parkes, Study notes of Computer Numerical Control

Download Lecture 23: Classification and Inductive Bias in Machine Learning by David C. Parkes and more Study notes Computer Numerical Control in PDF only on Docsity!

CS181 Lecture 23:

Wrap-up

David C. Parkes

Today

• Review
• Applications

Supervised Learners

• An algorithm that generates a classifier

from labeled training data.

Classifier

( x 1 ,…, xm )

y

Learner

( x 1 , y 1 )

…

( x n, y n)

Inductive Bias

• Inductive bias: the assumptions made by

a learning algorithm that allow it to learn

• “a basis for choosing one

generalization over another”

Example

X 1 X 2 X 3 Y

F F F F

T F F F

T T F T

T T T T

How should we classify

x = (T,F,T)?

Consistent hypotheses include:

specific x 1 Æ x 2 : F

general x 2 Ç x 3 : T

Only training examples are unambiguously classified.

) need an inductive bias!

Two Kinds of Inductive Bias

• Restriction bias: H ½ F
  • e.g., perceptrons vs. general neural networks;

naïve Bayes models

• saw monotone formula PAC-learnable in

O(m) but general DNF not* PAC-learnable

• Preference bias:
  • prefer simpler h over complex h
  • intuition: find a good h 2 H in a small class H

then less likely to be good just by chance

Achieving Preference Bias

• Preference bias may be achieved by:
  • Regularization
    • Score( h ) = f 1 (Complexity( h )) + f 2 (Error( h ))
    • e.g., penalize ½ w^2 in neural networks
  • Search bias
    • prefer simple hypotheses
    • e.g., ID3 in decision trees
  • Bayesian method
    • adopt a prior that prefers simpler probabilistic models over more complex models
    • P(M,μ | D) / P(D| M,μ)P(M,μ) = P(D | M,μ)P(μ|M)P(M)

Cross-validation

• Split data into training set and validation

set (e.g., 60:40)

• Evaluate effect of different parameters ¸

on validation set.

• Can also do cross-fold (e.g. 10x)

A B min¸ Error(B | h¸(A))

Example: #Hidden Nodes in

Neural Network (Duda)

validation

error

25

Reporting Error: Test Set!

A B min¸ Error(B | h¸(A))

C

Report: Error(C | h¸*(A))

train validation test

Perceptrons, Logistic

Regression and Neural

Networks

(Bishop) 35

perceptron learning rule

convergent for separable data

Perceptron Fails for XOR

X 1

X 2

-1 1

1

Linear decision boundaries

cannot classify

36

Strong restriction bias!