Decision Trees in Machine Learning: Classification with an Example, Study notes of Artificial Intelligence

Download Decision Trees in Machine Learning: Classification with an Example and more Study notes Artificial Intelligence in PDF only on Docsity!

CS181 Lecture 2:

Decision Trees

Prof. David C. Parkes

Classifiers

• A classifier is a function h : X  Y
  • Features: X = (X 1 , …, X m )
    • e.g., X 1 = {0,1,2}, X 2 = {blue, red}, X 3 = R¸ 0
  • Target class: Y
    • e.g., Y={true, false}, Y = {1,.. 5}

( x 1 ,…, xm) Classifier y’

Example: Medical Diagnosis

• Features:
  • symptoms and test results
    • e.g. Sore Throat, Thermometer Reading
• Classes:
  • True/False
    • e.g. does patient have flu?
  • One of a range of possible illnesses
    • e.g. cold, flu or nothing

Example: Plant Eating Robot

• Features:
  • gray-scale pixels in image
  • location in world
• Classes:
  • {nutritious, poisonous}
  • {eat, don’t eat}

Decision Trees

X 1

F T

X 2

F T

N P

X 4

F T

P X 2

N P

F T

Decision trees are a particular representation of a classifier. Node: feature to branch on Edge: value of a feature Leaf: prediction

Example Xj: binary feature Y = { N utritious, P oisonous} (^7)

Classification With Decision Trees X 1 F T X 2 F T N P

X 4

F T

P X 2

N P

F T

To classify x =(T,F,T,F)

Classification With Decision Trees

Begin at root.

Check value of x 1.

To classify x =(T,F,T,F) X 1

F T X 2 F T N P

X 4

F T

P X 2

N P

F T

Classification With Decision Trees

Begin at root.

Check value of x 1.

Follow T branch.

To classify x =(T,F,T,F) X 1

F T X 2 F T N P

X 4

F T

P X 2

N P

F T

Classification With Decision Trees

Begin at root.

Check value of x 1.

Follow T branch.

Check value of x 4.

Follow F branch.

To classify x =(T,F,T,F) X 1

F T X 2 F T N P

X 4

F T

P X 2

N P

F T

Classification With Decision Trees

Begin at root.

Check value of x 1.

Follow T branch.

Check value of x 4.

Follow F branch.

Return class (P)oisonous.

To classify x =(T,F,T,F) X 1

F T X 2 F T N P

X 4

F T

P X 2

N P

F T

Question for Today

• How to train a good decision tree given

labeled data?

Designing a Learning Algorithm:

Things to consider

• Feature construction
• Feature selection
• Selecting a representation
• Selecting a model (= class of hypotheses)
• Selecting an algorithm
• Tuning parameters of the algorithm

2. Feature Selection

• Sometimes the data contains many

features, e.g. 1000 or more

• Many learning algorithms break down with

too many features

• Feature selection is the process of

selecting the most relevant features

• can be done automatically

3. Selecting a Representation

• Learning algorithms represent the

classifiers they learn in different ways

• e.g., curve fitting uses polynomials , decision tree algorithms use …
• Representation constrains the set of

hypotheses