Tree Pruning & Classification: Impurity, Class Assignment, & Pruning, Study notes of Systems Engineering

Download Tree Pruning & Classification: Impurity, Class Assignment, & Pruning and more Study notes Systems Engineering in PDF only on Docsity!

Tree‐Based

Models Kwok‐Leung^ Tsui

Industrial^ &^ Systems

Engineering

Georgia^ Institute

of^ Technology

Regression Models

1. Classical linear model 2. Generalized linear model 3. Additive model

S^ : smooth nonparametric functionj^

4. Generalized additive models

Weather^ Data:

Play^ or^ not

Play?

Outlook^ Temperature^

Humidity^ Windy^ Play? sunny^ hot^

high^ false^ No sunny^ hot^

high^ true^ No overcast^ hot^

high^ false^ Yes rain^ mild^

high^ false^ Yes rain^ cool^

normal^ false^ Yes rain^ cool^

normal^ true^ No overcast^ cool^

normal^ true^ Yes sunny^ mild^

high^ false^ No sunny^ cool^

normal^ false^ Yes rain^ mild^

normal^ false^ Yes sunny^ mild^

normal^ true^ Yes overcast^ mild^

high^ true^ Yes overcast^ hot^

normal^ false^ Yes rain^ mild^

high^ true^ No

overcast

high^ normal

falsetrue

sunny^

rain

No^

No

Yes^

Yes

Example^ Tree^ for^ “Play?”^ Outlook^ Yes Humidity^ Windy

- Models•^ C^ — the regression model prediction value corresponding to the region Rm

m

- Two types•^ Regression trees^ •^ Classification trees • Fundamentals Issues in Tree-based Models•^ How to decide the splitting point? (Tree growing)^ •^ How to control the size of the tree? (Tree pruning)

(^52) , (^15) (^42) , (^14)

(^32) , (^13) (^22) , (^12) 5 2 ,^1112 ,^11

RxxIc

RxxIc

RxxIc

RxxIcR

RxxIcxf mmm xxIc

Tree-Based Models ) ∈= ∑= =

Regression^

Trees

• Partition the space into

M^ regions: R^ , R^ , …, R^1

.M M ∈= RxIc ∑ mm (^) = m

• For each of N observations, input is x^ = (x^ xi^ i1 ,^ xf^ )()(^1 • The best partition: to minimize the sum of squared error:^ N^2 − xfy ))((^ ∑ ii^ = i^1

…^ x^ ), output is continuous yi2 ,^ ,^ ip^

i,

)|( mii

∧ Rxyaveragec^ ∈= m^

Classification and Regression Trees^ (Fundamentals)

Notation

• x – input vector •^ X^ – input space •^ J^ – number of classes •^ C^ – set of classes, e.g.

C ={1,2,L, J

Accuracy Estimation • Question – How good is a classifier in prediction, i.e.,how accurate is a classifier? • True Misclassification Rate – Given the learning sample

L, (x^ , j), n=1,n^ n^

L,N, x^ ∊^ X , j∊^ C , construct d(x). Let (x, j), xn^ n^

∊^ X , j ∊^ C^ be a new sample from the same populationas^ L. The true misclassification rate of d(x),*^ R^ (d), is defined as^ R

*^ (d) = P(d(x)^ ≠ j).

How to Estimate R

*^ (d)

• Three Types of Estimates of R

* (d)

–^ Training Error R(d) –^ Test Error R

ts(d)

–^ Cross-Validation Error R

cv(d)

How to Estimate R

*^ (d) (Cond.)

•^ Cross-Validation Error R

cv(d)

– Randomly split

L^ into^ V^ subsets of as nearly equalsize as possible,^ L,^ L,^ L. For v = 1,^1 V

L,^ V,

vconstruct d(x) using

L^ -^ L. The testing error of dv^

v^ (x)

is ts^ v^ R^ (d) =^ ∑

vI(d(xn)^ ≠(x,j) L^ nn∊ v

j) / Nn^ v

where Nis the number of cases inv^

L.v^

– The^ V-fold cross-validation error is defined as^ R

cv^ Vts^ (d) =^ ∑R^ (dv=^

v^ ) /^ V

Classification Trees • Are tree structured classifiers. • Construction : beginning with the inputspace, repeatedly split into two descendantsubsets. Splits are formed by conditions onthe input vector x = {x

, x^ ,^ L}, e.g., a split 12

could be of the form

{x^ ∊^ X : x^ =^4

Tree Construction • Three Issues – How to grow, i.e., how to split? – Which class should a terminal node beassigned to? – When to stop?

Split Selection