Download CART Algorithm - Mathematics and Statistics - Study Notes and more Study notes Mathematical Statistics in PDF only on Docsity!
CART Algorithm
This document describes the tree growing process of the CART algorithm. The algorithm is
based on Classification and Regression Trees by Breiman et al (1984). A CART tree is a
binary decision tree that is constructed by splitting a node into two child nodes repeatedly,
beginning with the root node that contains the whole learning sample.
Notations
Y The dependent variable, or target variable. It can be ordinal categorical,
nominal categorical or continuous.
If Y is categorical with J classes, its class takes values in C = {1, …, J }.
X (^) m , m = 1, …, M The set of all predictor variables. A predictor can be ordinal categorical,
nominal categorical or continuous.
N n nn y 1
=
3 = x
The whole learning sample.
3 ( t ) The learning samples that fall in node^ t.
w n The case weight associated with case n.
f (^) n The frequency weight associated with case n. Non-integral positive value is
rounded to its nearest integer.
π ( ) j , j = 1, …, J Prior probability of Y = j , j = 1, …, J.
p ( j , t ), j = 1, …, J The probability of a case in class^ j^ and node^ t.
p ( t ) The probability of a case in node^ t.
p ( j | t ), j = 1, …, J The probability of a case in class^ j^ given that it falls into node^ t.
C ( i | j ) The cost of miss-classifying a class^ j^ case as a class^ i^ case. Clearly
C ( j | j )= 0.
Tree Growing Process
The basic idea of tree growing is to choose a split among all the possible splits at each node
so that the resulting child nodes are the “purest”. In this algorithm, only univariate splits are
considered. That is, each split depends on the value of only one predictor variable. All
possible splits consist of possible splits of each predictor. If X is a nominal categorical
variable of I categories, there are 2 1
1 −
I − possible splits for this predictor. If X is an ordinal
categorical or continuous variable with K different values, there are K - 1 different splits on
X. A tree is grown starting from the root node by repeatedly using the following steps on each
node.
- Find each predictor’s best split.
For each continuous and ordinal predictor, sort its values from the smallest to the largest.
For the sorted predictor, go through each value from top to examine each candidate split
point (call it v, if x ≤ v, the case goes to the left child node, otherwise, goes to the right.)
to determine the best. The best split point is the one that maximize the splitting criterion
the most when the node is split according to it. The definition of splitting criterion is in
later section.
For each nominal predictor, examine each possible subset of categories (call it A, if
x ∈ A , the case goes to the left child node, otherwise, goes to the right.) to find the best
split.
- Find the node’s best split.
Among the best splits found in step 1, choose the one that maximizes the splitting
criterion.
- Split the node using its best split found in step 2 if the stopping rules are not satisfied.
Splitting criteria and impurity measures
At node t , the best split s is chosen to maximize a splitting criterion ∆ i ( s , t ). When the
impurity measure for a node can be defined, the splitting criterion corresponds to a decrease
in impurity. In SPSS products, ∆ I ( s , t )= p ( t )∆ i ( s , t )is referred to as the improvement.
Categorical dependent variable
If Y is categorical, there are three splitting criteria available: Gini, Twoing, and ordered
Twoing criteria.
At node t , let probabilities p ( j , t ), p ( t )and p ( j | t )be estimated by
w j
wj
N
j N t p jt
,
j
p ( t ) p ( j , t ) ,
j
p jt
p jt
pt
p jt p j t ( ,)
where
∈
n!
N (^) w , j wnfnI ( yn j )
2
1
j ∈ C
i st it pL itL pRitR pLpR p j tL p j tR.
- Find the super-class C
* 1 of^ C 1 which maximizes^ (^ ( 1 ),)
∆ i s C t.
Continuous dependent variable
When Y is continuous, the splitting criterion ∆ i ( s , t )= i ( t )− pL i ( tL )− pRi ( tR ) is used
with the Least Squares Deviation (LSD) impurity measures
∈
∈
( )
()
2 ( ())
n t
n n
n t
n n n
w f
w f y yt
it
!
! ,
where
p L = Nw ( tL )/ Nw ( t ) , p R = Nw ( tR )/ Nw ( t ) , ∑
∈
()
n t
N (^) w t wnfn
!
,
()
N t
w f y
yt
w
n t
∑ n n n
∈
! .
Stopping Rules
Stopping rules control if the tree growing process should be stopped or not. The following
stopping rules are used:
- If a node becomes pure; that is, all cases in a node have identical values of the dependent
variable, the node will not be split.
- If all cases in a node have identical values for each predictor, the node will not be split.
- If the current tree depth reaches the user-specified maximum tree depth limit value, the
tree growing process will stop.
- If the size of a node is less than the user-specified minimum node size value, the node
will not be split.
- If the split of a node results in a child node whose node size is less than the user-
specified minimum child node size value, the node will not be split.
s of node t , the improvement ( , ) () ( , )
∆ I s t = pt ∆ is t is smaller
than the user-specified minimum improvement, the node will not be split.
Surrogate Split
Given a split
X ≤ s , its surrogate split is a split using another predictor variable X ,
X ≤ s X (or X > sX ), such that this split is most similar to it and is with positive predictive
measure of association. There may be multiple surrogate splits. The bigger the predictive
measure of association is, the better the surrogate split is.
Predictive measure of association
Let X *∩ X
3 (resp. 3 X X
) be the set of learning cases (resp. learning cases in node t ) that
has non-missing values of both X
and X. Let ( | )
p s ≈ sX t be the probability of sending a
case in 3 X X
to the same child by both
s and s (^) X , and
~ s (^) X be the split with maximized
probability | ) max( ( | ))
p s s t ps sX t s
X X
The predictive measure of association λ(
~ | )
s ≈ s (^) Xt between s
and
~ s (^) X at node t is
min( , )
min( , ) ( 1 ( | )
L R
L R X X p p
p p ps s t s s t
where pL (resp. p (^) R ) is the relative probability that the best split s
at node t sends a case
with non-missing value of X
to the left (resp. right) child node, p (^) L = p ( tL ) p ( t ) and
p (^) R = p ( tR ) p ( t )respectively. And where
if is continuous ( )
if iscategorical ( )
,
,
Y
N X X
N s s t
Y
N X X
j N s s t
ps s t
w
w X
j (^) wj
wj X
X
π
,
with
∩
∈
X X
n
Nw X X wnfn
!
∩
∈
()
n t
w n n
X X
N X X t w f
!
∩
∈
()
n t
w X n n X
X X
N s s t w f I n s s
!
