Download Analyzing Child's Outdoor Play Based on Weather, Temperature, Humidity, and Wind and more Assignments Machine Learning in PDF only on Docsity!
Decision Trees Example Problem
Consider the following data, where the Y label is whether or not the child goes out to play.
Day Weather Temperature Humidity Wind Play?
1 Sunny Hot High Weak No
2 Cloudy Hot High Weak Yes
3 Sunny Mild Normal Strong Yes
4 Cloudy Mild High Strong Yes
5 Rainy Mild High Strong No
6 Rainy Cool Normal Strong No
7 Rainy Mild High Weak Yes
8 Sunny Hot High Strong No
9 Cloudy Hot Normal Weak Yes
10 Rainy Mild High Strong No
Step 1: Calculate the IG (information gain) for each attribute (feature)
Initial entropy = ๐ป(๐) = โ โ ๐(๐ = ๐ฆ) log 2
๐ฆ
log
2
log
2
= โ( 0. 5 ) log
2
( 0. 5 ) โ ( 0. 5 ) log
2
Temperature:
Total entropy of this division is:
๐ป(๐ | ๐ก๐๐๐) = โ โ ๐(๐ก๐๐๐ = ๐ฅ) โ ๐(๐ = ๐ฆ | ๐ก๐๐๐ = ๐ฅ) log
2
๐ฅ ๐ฆ
= โ(๐(๐ก๐๐๐ = ๐ป) โ ๐(๐ = ๐ฆ |๐ก๐๐๐ = ๐ป) log
2
๐ฆ
๐(๐ก๐๐๐ = ๐) โ ๐(๐ = ๐ฆ |๐ก๐๐๐ = ๐) log
2
๐ฆ
log
2
๐ฆ
1
2
) log
2
1
2
1
2
) log
2
1
2
3
5
) log
2
3
5
2
5
) log
2
2
5
log
2
log
2
IG(Y, temp) = 1 โ 0.7884 = 0.
Temperature
HOT
N, Y, N, Y
MILD
Y, Y, N, Y, N
COLD
N
Humidity:
Total entropy of this division is:
๐ป(๐ | โ๐ข๐) = โ โ ๐(โ๐ข๐ = ๐ฅ) โ ๐(๐ = ๐ฆ | โ๐ข๐ = ๐ฅ) log
2
๐ฅ ๐ฆ
= โ(๐(โ๐ข๐ = ๐ป) โ ๐(๐ = ๐ฆ |โ๐ข๐ = ๐ป) log
2
๐ฆ
๐(โ๐ข๐ = ๐) โ ๐(๐ = ๐ฆ |โ๐ข๐ = ๐) log
2
๐ฆ
3
7
) log
2
3
7
4
7
) log
2
4
7
2
3
) log
2
2
3
1
3
) log
2
1
3
IG(Y, hum) = 1 โ 0.8651 = 0.
Humidity
STRONG
Y, Y, Y, N, N, N, N
WEAK
Y, N, Y
Wind:
Total entropy of this division is:
๐ป(๐ | ๐ค๐๐๐) = โ โ ๐(๐ค๐๐๐ = ๐ฅ) โ ๐(๐ = ๐ฆ | ๐ค๐๐๐ = ๐ฅ) log
2
๐ฅ ๐ฆ
= โ(๐(๐ค๐๐๐ = ๐) โ ๐(๐ = ๐ฆ |๐ค๐๐๐ = ๐) log
2
๐ฆ
๐(๐ค๐๐๐ = ๐) โ ๐(๐ = ๐ฆ |๐ค๐๐๐ = ๐) log
2
๐ฆ
2
6
) log
2
2
6
4
6
) log
2
4
6
1
4
) log
2
1
4
3
4
) log
2
3
4
IG(Y, wind) = 1 โ 0.8755 = 0.
Step 2: Choose which feature to split with!
IG(Y, wind) = 0.
IG(Y, hum) = 0.
IG(Y, weather) = 0.
IG(Y, temp) = 0.
Wind
STRONG
Y, Y, N, N, N, N
WEAK
N, Y, Y, Y
Humidity
Entropy of โSunnyโ node = โ((
1
3
) log
2
1
3
2
3
) log
2
2
3
Entropy of its children = 0
IG = 0.
Entropy of โRainyโ node = โ((
1
4
) log
2
1
4
3
4
) log
2
3
4
Entropy of children = โ(
3
4
1
3
) log
2
1
3
2
3
) log
2
2
3
IG = 0.
SUNNY
N, Y, N
Humidity
HIGH
N, N
NORMAL
Y
CLOUDY
Y, Y, Y
RAINY
N, N, Y, N
Humidity
HIGH
N, Y, N
NORMAL
N
Wind
Entropy of โSunnyโ node = โ((
1
3
) log
2
1
3
2
3
) log
2
2
3
Entropy of its children = โ(
2
3
1
2
) log
2
1
2
1
2
) log
2
1
2
IG = 0.
Entropy of โRainyโ node = โ((
1
4
) log
2
1
4
3
4
) log
2
3
4
Entropy of children = 0
IG = 0.
SUNNY
N, Y, N
Wind
STRONG
N, Y
WEAK
N
CLOUDY
Y, Y, Y
RAINY
N, N, Y, N
Wind
STRONG
N, N, N
WEAK
Y
Boosting
(https://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/boosting.pdf)
