Bayesian Learning & Classification with AI: Maximum A Posteriori Hypothesis & Naive Bayes, Study notes of Computer Science

Download Bayesian Learning & Classification with AI: Maximum A Posteriori Hypothesis & Naive Bayes and more Study notes Computer Science in PDF only on Docsity!

Artificial Intelligence

Programming Bayesian Learning

Chris Brooks Department of Computer Science University of San Francisco

Learning and Classification

An important sort of learning problem is the classification

problem.

This involves placing examples into one of two or moreclasses.

Should/shouldn’t play tennis Spam/not spam. Wait/don’t wait at a restaurant

Classification is a

supervised

learning task.

Requires access to a set of labeled training examples

From this we induce a hypothesis that describes how todetermine what class an example should be in.

Department of Computer Science — University of San Francisco – p.1/

Bayes’ Theorem

Recall the definition of Bayes’ Theorem P

b

a

P ( a |b ) P ( b ) P ( a )

Let’s rewrite this a bit. Let

D

be the data we’ve seen so far.

Let

h

be a possible hypothesis

P

h

D

P ( D |h ) P ( h ) P ( D ) Department of Computer Science — University of San Francisco – p.3/

MAP Hypothesis

Often, we’re not so interested in the particularprobabilities for each hypothesis. Instead, we want to know: Which hypothesis is mostlikely, given the data?

Which classification is the most probable? Is

P layT ennis

or

P layT ennis

more likely?

We call this the

maximum a posteriori hypothesis

(MAP

hypothesis). In this case, we can ignore the denominator

P

D

in

Bayes’ Theorem, since it will be the same for all

h

h M AP

argmax h ∈ H

P

D

h

P

h

Department of Computer Science — University of San Francisco – p.4/

ML Hypothesis

In some cases, we can simplify things even further. What are the priors

P

h

for each hypothesis?

Without any other information, we’ll often assume thatthey’re equally possible.

Each has probability

(^1) H

In this case, we can just consider the conditionalprobability

P

D

h

We call the hypothesis that maximizes this conditionalprobability the

maximum likelihood

hypothesis.

h M L

argmax h ∈ H

P

D

h

Department of Computer Science — University of San Francisco – p.6/

Example

Imagine that we have a large bag of candy. We want toknow the ratio of cherry to lime in the bag. We start with 5 hypotheses:1.

h 1

: 100% cherry

h 2

75% cherry, 25% lime.

h 3

50% cherry, 50% lime

h 4

25% cherry, 75% lime

h 5

100% lime

Our agent repeatedly draws pieces of candy. We want it to correctly pick the type of the next piece ofcandy.

Department of Computer Science — University of San Francisco – p.7/

Example

How do the hypotheses change as data is observed? Initially, we start with the priors:

Then we draw a lime.

P

h 1

lime

αP

lime

h 1

P

h 1

P

h 2

lime

αP

lime

h 2

P

h 2

α 1 4

α

P

h 3

lime

αP

lime

h 3

P

h 3

α 1 2

α

P

h 4

lime

αP

lime

h 4

P

h 4

α 3 4

α

P

h 5

lime

αP

lime

h 5

P

h 5

α

α

α

Department of Computer Science — University of San Francisco – p.9/

Example

Then we draw a second lime.

P

h 1

lime, lime

αP

lime, lime

h 1

P

h 1

P

h 2

lime, lime

αP

lime, lime

h 2

P

h 2

α 1 4 1 4

α

P

h 3

lime, lime

αP

lime, lime

h 3

P

h 3

α 1 2 1 2

α

P

h 4

lime, lime

αP

lime, lime

h 4

P

h 4

α 3 4 3 4

α

P

h 5

lime

αP

lime

h 5

P

h 5

α

α

α

Strictly speaking, we don’t really care what

α

is.

We can just select the MAP hypothesis, since we justwant to know the most likely hypothesis.

Department of Computer Science — University of San Francisco – p.10/

Learning bias

What sort of bias does Bayesian Learning use? Typically, simpler hypotheses will have larger priors. More complex hypotheses will fit data more exactly (butthere’s many more of them).

Under these assumptions,

h M AP

will be the simplest

hypothesis that fits the data. This is Occam’s razor, again. Think about the deterministic case, where

P

h i

D

is

either 1 or 0.

Department of Computer Science — University of San Francisco – p.12/

Bayesian Concept Learning

Bayesian Learning involves estimating the likelihood ofeach hypothesis. In a more complex world where observations are notindependent, this could be difficult. Our first cut at doing this might be a brute forceapproach:1. For each

h

in

H

, calculate

P

h

D

P ( D |h ) P ( h ) P ( D )

2. From this, output the hypothesis

h M AP

with the

highest posterior probability.

This is what we did in the example.

Challenge - Bayes’ Theorem can be computationallyexpensive to use when observations are not i.i.d. P

h

o 1 , o 2

P ( o 1 |h,o 2 ) P ( h | o 2 ) P ( o 1 |o 2 ) Department of Computer Science — University of San Francisco – p.13/

Bayesian Optimal Classifiers

Suppose we have three hypotheses and posteriors: h

1

, h 2

, h 3

We get a new piece of data -

h 1

says it’s positive,

h 2

and

h 3

negative.

h 1

is the MAP hypothesis, yet there’s a 0.6 chance that

the data is negative. By combining weighted hypotheses, we improve ourperformance.

Department of Computer Science — University of San Francisco – p.15/

Bayesian Optimal Classifiers

By combining the predictions of each hypothesis, we geta Bayesian optimal classifier. More formally, let’s say our unseen data belongs to oneof

v

classes.

The probability

P

v j

D

that our new instance belongs to

class

v j

is:

h i ∈ H

P

v j

h i

P

h i

D

Intuitively, each hypothesis gives its prediction, weightedby the likelihood that that hypothesis is the correct one. This classification method is provably optimal - onaverage, no other algorithm can perform better.

Department of Computer Science — University of San Francisco – p.16/

Naive Bayes classifier

The Naive Bayes classifer makes a strong assumptionthat makes the algorithm practical:

Each attribute of an example is independent of theothers. P

a

b

P

a

P

b

for all a and b.

This makes it straightforward to compute posteriors.

Department of Computer Science — University of San Francisco – p.18/

The Bayesian Learning Problem

Given: a set of labeled, multivalued examples. Find a function

F

x

that correctly classifies an unseen

example with attributes

a 1 , a 2 , ..., a n

Call the most probable category

v map

v map

argmax v i ∈ V

P

v i

a 1 , a 2 , ..., a n

We rewrite this with Bayes’ Theorem as: v

map

argmax v i ∈ V

P

a 1 , a 2 , ..., a n

v i

P

v i

Estimating

P

v i

is straightforward with a large training

set; count the fraction of the set that are of class

v i

However, estimating

P

a 1 , a 2 , ..., a n

v i

is difficult unless

our training set is

very

large. We need to see every

possible attribute combination many times.

Department of Computer Science — University of San Francisco – p.19/