Understanding Decision Trees and Information Gain in Machine Learning, Slides of Database Management Systems (DBMS)

Download Understanding Decision Trees and Information Gain in Machine Learning and more Slides Database Management Systems (DBMS) in PDF only on Docsity!

• Iterative Dichotomiser

Decision Trees

• A Decision tree is a tree with branching nodes with a choice between 2 or more choices.
• Decision Node: A node that a choice is made
• Leaf Node: The result from that point of the tree

OutLook

Sunny PartiallyCloudy Cloudy

NO (^) Yes Humidty

NO Yes

ID

• Invented by J. Ross Quinlan

• Employs a top-down greedy search through the

space of possible decision trees.

• Select attribute that is most useful for

classifying examples (attribute that has the

highest Information Gain).

Entropy

• Entropy tells us how well an attribute will

separate the given example according to the

target classification class.

• Entropy(S) = -Ppos log 2 Ppos – Pneg log 2 Pneg

• Ppos = Proportion of positive examples

• Pneg = proportion of negative example

Information Gain

• Measures the expected reduction in entropy. The higher the Information Gain, more is the expected reduction in entropy.
• The Equation for Information gain is.

Information Gain

• A is an attribute of collection S

• Sv = subset of S for which attribute A has

value v

• |Sv| = number of elements in Sv

• |S| = number of elements in S

Example (cont)

• Entropy(S) = -Ppos log 2 Ppos – Pneg log 2 Pneg

Entropy(4Y,2N): -(4/6)log 2 (4/6) – (2/6)log 2 (2/6)

Now that we know the Entropy where going to

use that answer to find the Information Gain

Example

Video Contains Car

Contains Violence

Rally Cars

Races

GTA 4 Yes Yes No No

Doom No Yes No No

GTA3 Yes Yes No No

Halo 3 Yes Yes No No

Need for Speed

Yes No No Yes

Rally Sport

Yes No Yes No

Example

Video Contains Car

Contains Violence

Rally Cars

Races

GTA 4 Yes Yes No No

Doom No Yes No No

GTA3 Yes Yes No No

Halo 3 Yes Yes No No

Need for Speed

Yes No No Yes

Rally Sport

Yes No Yes No

Example (Cont)

• For Attributes (Contains Rally Cars)

S = [4Y,2N]

SYes = [0Y,1N] E(SYes ) = 0

SNo = [4Y,1N] E(SNo ) = 0.

Gain (S, Contains Rally Cars) = 0.91829 – [(1/6)0 + (5/6)0.7219] = 0.

Example (Cont)

• For Attributes (Races)

S = [4Y,2N]

SYes = [0Y,1N] E(SYes ) = 0

SNo = [4Y,1N] E(SNo ) = 0.

Gain (S, Races) = 0.91829 – [(1/6)0 + (5/6)0.7219] =

Example (Cont)

• Gain (S, Contains Cars) =
• Gain (S, Contains Rally Cars) = 0.
• Gain (S, Races) = 0.

Contains Rally Cars

No Yes

Not Races Violent

No Yes

Not Violent Violent