Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
An overview of decision tree learning, including the concepts of entropy and information gain. It also includes slides for textbook content on decision trees for various scenarios, such as predicting c-section risk and play tennis based on weather conditions. Topics like xor, disjunctive hypotheses, and top-down induction of decision trees.
Typology: Study notes
1 / 29
This page cannot be seen from the preview
Don't miss anything!
read Chapter recommended exercises
Decision tree representation ID learning algorithm Entropy Information gain Overtting
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Outlook
Overcast
Humidity
High Normal
No Yes
Wind
Strong Weak
No Yes
Yes
Sunny Rain
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Decision tree representation
Each internal no de tests an attribute Each branch corresp onds to attribute value Each leaf no de assigns a classicati on
How would we represent
XOR A B C D E M of N
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Instances describable by attribute val ue pairs Target function is discrete valued Disjunctive hyp othesis may b e required Possibly noisy training data
Examples
Equipment or medical diagnosis Credit risk analysis Mo deling calendar scheduling preferences
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Entropy(S)
0.0 0.5 1. p+
S is a sample of training examples p is the prop ortion of p ositive examples in S p is the prop ortion of negative examples in S Entropy measures the impurity of S E ntr opy S p log (^) p p log (^) p
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
E ntr opy S exp ected numb er of bits needed to enco de class or of randomly drawn memb er of S under the optimal shortest length co de
Why
Information theory optimal length co de assigns log (^) p bits to message having probability p
So exp ected numb er of bits to enco de or of random memb er of S
p log (^) p p log (^) p
E ntr opy S p log (^) p p log (^) p
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Day Outlo ok Temp erature Humidity Wind PlayTennis D Sunny Hot High Weak No D Sunny Hot High Strong No D Overcast Hot High Weak Yes D Rain Mild High Weak Yes D Rain Co ol Normal Weak Yes D Rain Co ol Normal Strong No D Overcast Co ol Normal Strong Yes D Sunny Mild High Weak No D Sunny Co ol Normal Weak Yes D Rain Mild Normal Weak Yes D Sunny Mild Normal Strong Yes D Overcast Mild High Strong Yes D Overcast Hot Normal Weak Yes D Rain Mild High Strong No
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Which attribute is the best classifier?
High Normal
Humidity
[3+,4-] [6+,1-]
Wind Weak Strong
[6+,2-] [3+,3-]
= .940 - (7/14).985 - (7/14).592= .151 = .940 - (8/14).811 - (6/14)1.0=.
Gain (S, Humidity ) Gain (S, Wind)
E =0.940 E =0.
E =0.985 E =0.592 E =0.811 E =1.
S: [9+,5-] S: [9+,5-]
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
...
A
A
A
...
A
A
A
... (^) ...
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Hyp othesis space is complete Target function surely in there Outputs a single hyp othesis which one Cant play questions No back tracking Lo cal minima Statisical l y based search choices Robust to noisy data Inductive bias approx prefer shortest tree
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Why prefer short hyp otheses
Argument in favor
Fewer short hyps than long hyps a short hyp that ts data unlikely to b e coincidence a long hyp that ts data might b e coincidence
Argument opp osed
There are many ways to dene small sets of hyps eg all trees with a prime numb er of no des that use attributes b eginning with Z Whats so sp ecial ab out small sets based on size of hyp othesis
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
Consider adding noisy training example
S unny H ot N or mal S tr ong P l ay T ennis N o
What e ect on earlier tree
Outlook
Overcast
Humidity
High Normal No Yes
Wind
Strong Weak No Yes
Yes
Sunny Rain
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
0 10 20 30 40 50 60 70 80 90 100
Accuracy
Size of tree (number of nodes)
On training dataOn test data
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
How can we avoid overtting
stop growing when data split not statistical l y signicant grow full tree then p ost prune
How to select b est tree
Measure p erformance over training data Measure p erformance over separate validati on data set MDL minimize siz e tr ee siz e miscl assif ications tr ee
lecture slides for textb o ok Machine Learning c Tom M Mitchell McGraw Hill
