Recap
•We have bee discussing the issue of consistency of
empirical risk minimization (ERM).
PR NPTEL course – p.1/98
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Empirical Risk Minimization - Pattern Recognition - Lecture Slides, Slides of Engineering Dynamics
The key points are: Empirical Risk Minimization, Family of Classifiers, Uniform Convergence, Set of Class Labels, Hypothesis Space, Action Space, Loss Function, Accuracy of Two Estimates, Triangular Inequality, Hoeffiding Bound
Typology: Slides
2012/2013
1 / 98
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Empirical Risk Minimization - Pattern Recognition - Lecture Slides and more Slides Engineering Dynamics in PDF only on Docsity!
Recap
We have bee discussing the issue of consistency ofempirical risk minimization (ERM).
PR NPTEL course – p.1/
Recap
We have bee discussing the issue of consistency ofempirical risk minimization (ERM).
We have seen that ERM is consistent if theconvergence as per law of large numbers is uniformover
H
, the family of classifiers over which we are
minimizing empirical risk.
PR NPTEL course – p.2/
Recap
We have bee discussing the issue of consistency ofempirical risk minimization (ERM).
We have seen that ERM is consistent if theconvergence as per law of large numbers is uniformover
H
, the family of classifiers over which we are
minimizing empirical risk.
In this class we continue our discussion oncharacterizing families
H
where such uniform
convergence holds.
We first briefly recall the notation and the results weproved last class.
PR NPTEL course – p.4/
We are given
PR NPTEL course – p.5/
We are given
X
- input space; (
Feature space
)
Y
- output space (
Set of class labels
)
PR NPTEL course – p.7/
We are given
X
- input space; (
Feature space
)
Y
- output space (
Set of class labels
)
H
- hypothesis space (
family of classifiers
) PR NPTEL course – p.8/
We are given
X
- input space; (
Feature space
)
Y
- output space (
Set of class labels
)
H
- hypothesis space (
family of classifiers
)
Each
h
∈ H
is a function,
h
X → A
,
where
A
is called
action space
.
Training data:
X
i
, y
i
, i
, n
drawn
iid
according to some distribution
P
xy
on
X × Y
.
PR NPTEL course – p.10/
Loss function:
L
Y × A → ℜ
.
PR NPTEL course – p.11/
Loss function:
L
Y × A → ℜ
.
The risk function,
R
H → ℜ
, is given by
R
h
E
[
L
y, h
X
))] =
L
y, h
X
dP
xy
We assume that
L
is bounded so that the expectation
always exists.
Let
h
∗
= arg min
h
∈H
R
h
PR NPTEL course – p.13/
Loss function:
L
Y × A → ℜ
.
The risk function,
R
H → ℜ
, is given by
R
h
E
[
L
y, h
X
))] =
L
y, h
X
dP
xy
We assume that
L
is bounded so that the expectation
always exists.
Let
h
∗
= arg min
h
∈H
R
h
We define the goal of learning as finding
h
∗
, the global
minimizer of risk.
PR NPTEL course – p.14/
However, we can not directly minimize
R
.
The
empirical risk function
,
R
n
H → ℜ
, is defined by
R
n
h
(^1) n
n
i
=
L
y
i
, h
X
i
PR NPTEL course – p.16/
However, we can not directly minimize
R
.
The
empirical risk function
,
R
n
H → ℜ
, is defined by
R
n
h
(^1) n
n
i
=
L
y
i
, h
X
i
Let
ˆh
∗ n
= arg min
h
∈H
R
n
h
PR NPTEL course – p.17/
Consistency of Empirical Risk Minimization
We would like the algorithm to satisfy:
ǫ, δ >
,
N <
, such that
Prob
[
R
ˆh
∗ n
R
h
∗
ǫ
]
δ,
n
N
PR NPTEL course – p.19/
Consistency of Empirical Risk Minimization
We would like the algorithm to satisfy:
ǫ, δ >
,
N <
, such that
Prob
[
R
ˆh
∗ n
R
h
∗
ǫ
]
δ,
n
N
We would also like to (approximately) know the truerisk of the learnt classifier and hence like to have
Prob
[
R
n
ˆh
∗ n
R
h
∗
ǫ
]
δ,
n
N
PR NPTEL course – p.20/