Nur auf Docsity: Lade Statistical foundation of learning und mehr Übungen als PDF für Rechtsinformatik herunter!
Statistical foundations of learning - CIT4230004 (Summer Semester 2024)
Assignment 4
Due: 27.06.2024, 23:
Points: 14
The solutions have to be handed in via Moodle. We do not accept late submissions.
We would recommend using LaTeX for writing your submission but also accept handwrit- ten solutions, but please note that if we can not read or understand it, we cannot grade it.
To get full points, always provide the steps in your derivation/proofs and make clear when/how you use known results, for example, from the lecture (e.g. already proven concentration inequalities).
Statistical foundations of learning - CIT4230004 (Summer Semester 2024)
Exercise 4.1: Validation
This exercise provides an example where leave-one-out error is a poor estimate of the generalisation error. Consider the 0-1 loss and assume that the distribution D is such that P(y = 1|x) = P(y = 0|x) =
for every x ∈ X
Given a training sample S = {(x 1 , y 1 ),... , (xm, ym)} ∼ Dm, we consider the following classification rule
hS (x) :=
0 if
Pm i=
yi is odd,
1 if
Pm i=
yi is even
for every x
- Compare the expected generalisation error for hS , ES [LD(hS )], and the expected leave-one-out error, ES [Lloo(hS )].
- Now compute ES
|LD(hS ) − Lloo(hS )|
, and comment on your result.
- Give reasons behind the similarities / dissimilarities in these findings. (2+3+1=6 points)
Exercise 4.2: Rademacher Complexity
For any p ≥ 1 let Bp denote the set
Bp =
n x ∈ Rd^ : ∥x∥p ≤ 1
o
Prove that the Rademacher complexity of Bp satisfies
R(Bp) = d−^1 /p
(3 points)
Exercise 4.3: Universality of the Gaussian kernel
Let X = {x ∈ Rp^ : ∥x∥ 2 ≤ 1 }. In the lecture, we saw that the exponential kernel k(x, y) = exp(⟨x, y⟩) is universal on X. Conclude that the Gaussian kernel k(x, y) = exp(− 12 ∥x−y∥^2 ) is also universal on X.
Hint: Start with approximating a function f using the exponential kernel. From there, work your way to the Gaussian kernel.
(5 points)
