EE364b Prof. S. Boyd
EE364b Homework 4
1. Support vector machine training via stochastic subgradient. We suppose that feature-
label pairs, (x, y )∈Rn× {−1,1}, are generated from some distribution. We seek a
linear classifier or predictor, of the form ˆy=sign(wTx), where w∈Rnis the weight
vector. (We can add an entry to xthat is always 1 to get an affine classifier.) Our
classifier is correct when ywTx > 0; since this expression is homogeneous in w, we can
write this as ywTx≥1. Thus, our goal is to choose wso that 1 −ywTx≤0 with high
probability.
Asupport vector machine (SVM) chooses wsvm as the minimizer of
f(w) = E1−ywTx++ (ρ/2)kwk2
2,
where ρ > 0 is a parameter. The first term is the average loss, and the second term is a
quadratic regularizer. Finding wsvm involves solving a stochastic optimization problem.
Explain how to (approximately) solve this stochastic optimization problem using the
stochastic subgradient method, with one sample per subgradient step. In this context,
the samples from the distribution are called data or examples, and the collection of
these is called the training data. Since this method only processes one data sample in
each step, it is called a streaming algorithm (since it does not have to store more than
one data sample in each step).
Implement the stochastic subgradient method for a problem with n= 20, and (x, y )
samples generated using
randn(’state’,0)
w_true = randn(n,1); % ’true’ weight vector
% to get each data sample use snippet below
x = randn(n,1);
y = sign(w_true’*x+0.1*randn(1));
Experiment with the choice of ρ, the step size rule, and the number of iterations to
run (but don’t be afraid to run the algorithm for 10000 steps).
To view the convergence, you can plot two quantities at each step: the optimality gap
f(w)−f⋆and the classifier error probability Prob ywTx≤0. To (approximately)
compute these quantities, use a Monte Carlo method, using, say, 10000 samples. (You’ll
want to compute these 10000 samples, and evaluate the Monte Carlo estimates of the
two quantities above, without using Matlab for loops. Also note that evaluation of
these two quantities will be far more costly than each step of the stochastic subgradient
method.) You can use CVX to estimate f⋆.
1
docsity.com
