Decision Tree Classifiers, Lecture notes of Machine Learning

Download Decision Tree Classifiers and more Lecture notes Machine Learning in PDF only on Docsity!

Decision tree learning

Sunita Sarawagi

IIT Bombay

http://www.it.iitb.ac.in/~sunita

Decision tree classifiers

• Widely used learning method
• Easy to interpret: can be re-represented as if-then-else

rules

• Approximates function by piece wise constant regions
• Does not require any prior knowledge of data distribution,

works well on noisy data.

• Has been applied to:
  • classify medical patients based on the disease,
  • equipment malfunction by cause,
  • loan applicant by likelihood of payment.
  • lots and lots of other applications..

• Tree where internal nodes are simple decision rules on

one or more attributes and leaf nodes are predicted

class labels.

Decision trees

Salary < 1 M

Prof = teaching

Good

Age < 30

Bad Bad

Good

Training Dataset

age income student credit_rating buys_computer

<=30 high no fair no

<=30 high no excellent no

30…40 high no fair yes

40 medium no fair yes

40 low yes fair yes

40 low yes excellent no

31…40 low yes excellent yes

<=30 medium no fair no

<=30 low yes fair yes

40 medium yes fair yes

<=30 medium yes excellent yes

31…40 medium no excellent yes

31…40 high yes fair yes

40 medium no excellent no

This

follows

an

example

from

Quinlan’s

ID

Weather Data: Play or not Play?

Outlook Temperature Humidity Windy Play?

sunny hot high false No

sunny hot high true No

overcast hot high false Yes

rain mild high false Yes

rain cool normal false Yes

rain cool normal true No

overcast cool normal true Yes

sunny mild high false No

sunny cool normal false Yes

rain mild normal false Yes

sunny mild normal true Yes

overcast mild high true Yes

overcast hot normal false Yes

rain mild high true No

Note:

Outlook is the

Forecast,

no relation to

Microsoft

email program

overcast

high normal false

true

sunny

rain

No Yes No Yes

Yes

Example Tree for “Play?”

Outlook

Humidity

Windy

Tree learning algorithms

• ID3 (Quinlan 1986)
• Successor C4.5 (Quinlan 1993)
• CART
• SLIQ (Mehta et al)
• SPRINT (Shafer et al)

Basic algorithm for tree building

• Greedy top-down construction.

Gen_Tree (Node, data)

make node a leaf?

Yes

Stop

Find best attribute and best split on attribute

Partition data on split condition

For each child j of node Gen_Tree (node_j, data_j)

Selection

criteria

Measures of impurity

• Entropy
• Gini



k

i

i i

Entropy S p p

1

( ) log



k

i

i

Gini S p

1

2

Information gain

• Information gain on partitioning S into r subsets
• Impurity (S) - sum of weighted impurity of each subset

0

p

1

1

Entropy





r

j

j

j

r

Entropy S

S

S

Gain S S S Entropy S

1

1

1

0

Gini

1

Information gain: example

S

K= 2, |S| = 100, p

1

= 0.6, p

2

= 0.

E(S) = -0.6 log(0.6) - 0.4 log (0.4)=0.

S

1

S

2

| S

1

| = 70, p

1

= 0.8, p

2

= 0.

E(S

1

) = -0.8log0.8 - 0.2log0.2 = 0.

| S

2

| = 30, p

1

= 0.13, p

2

= 0.

E(S

2

) = -0.13log0.13 - 0.87 log 0.87=.

Information gain: E(S) - (0.7 E(S

1

) + 0.3 E(S

2

) ) =0.

Weather Data: Play or not Play?

Outlook Temperature Humidity Windy Play?

sunny hot high false No

sunny hot high true No

overcast hot high false Yes

rain mild high false Yes

rain cool normal false Yes

rain cool normal true No

overcast cool normal true Yes

sunny mild high false No

sunny cool normal false Yes

rain mild normal false Yes

sunny mild normal true Yes

overcast mild high true Yes

overcast hot normal false Yes

rain mild high true No

Example: attribute “Outlook”

• “Outlook” = “Sunny”:
• “Outlook” = “Overcast”:
• “Outlook” = “Rainy”:
• Expected information for attribute:

info([2,3]) entropy(2/5,3/5)  2 / 5 log( 2 / 5 ) 3 / 5 log( 3 / 5 )  0. 971 bits

info([4,0]) entropy(1,0)  1 log( 1 ) 0 log( 0 )  0 bits

info([3,2]) entropy(3/5,2/5)  3 / 5 log( 3 / 5 ) 2 / 5 log( 2 / 5 )  0. 971 bits

Note: log(0) is

not defined, but

we evaluate

0*log(0) as zero

info([3,2], [4,0],[3,2]) ( 5 / 14 ) 0. 971 ( 4 / 14 ) 0 ( 5 / 14 ) 0. 971

 0. 693 bits

witten&eibe

Computing the information gain

• Information gain:

(information before split) – (information after split)

• Information gain for attributes from weather

data:

gain(" Outlook")  info([9,5])-info([2,3],[4,0],[3,2])  0.940- 0.

 0. 247 bits

gain("Outlook" )  0. 247 bits

gain("Temperatur e") 0. 029 bits

gain(" Humidity") 0. 152 bits

gain(" Windy")  0. 048 bits

witten&eibe