COMP 652 - ECSE 608: Machine Learning - Assignment 1
Posted Monday, September 10, 2018
Due Friday, September 28, 2018
You should submit an archive of your code, as well as a pdf file with your answers (either typed or
scanned), uploaded to MyCourses. If you cannot access MyCourses, email the assignment directly to both
Riashat and Audrey by the deadline (11:59pm EST on the day the assignment is due).
1. [55 points] Regression, Overfitting and Regularization
For this exercise, you will experiment with regression, regularization and cross-validation. You are
provided with a data set in files hw1-q1x.csv (inputs) and hw1-q1y.csv (targets). You are
allowed to use any programming language of your choice (Python, Matlab, R, etc). You can also
use use any toolbox/package (e.g. scickit-learn) as long as you understand what is going on
behind the scene (for example does the fit linear model() method you use fit a bias term or
does it assume that the data is centered?).
(a) [5 points] Load the data into memory. Make an appropriate Xmatrix and yvector. Split the
data at random into one set (Xtrain,ytrain )containing 80% of the instances, which will be
used for training + validation, and a testing set (Xtest,ytest)(containing remaining instances).
Describe here any preprocessing of the data that you performed for this exercise (did you
handle the bias term in linear regression directly or did you add a 1to each input? Did you
center the data? Why and how?, etc.).
(b) [5 points] Run linear regression on the data using L2regularization, varying the regularization
parameter λ∈ {0,0.1,1,10,· · · ,105}. Plot on one graph the root-mean-square error (RMSE)
for the training data and the testing data, as a function of λ(you should use a log scale for λ).
Plot on another graph the L2norm of the weight vector you obtain. Plot on the third graph the
actual values of the weights obtained (one curve per weight). Explain briefly what you see.
(c) [5 points] Perform five-fold cross-validation on the training data to determine the best value
of the regularization parameter λ(report the training and validation errors for each fold and
briefly explain how you chose the best value of λ). Compare the best value of λobtained to
your results in question 1b and briefly comment.
(d) [5 points] Suppose that the training data was sorted in increasing value of the target variable y,
and you simply partitioned it by splitting it in kfolds (without shuffling the data first). Explain
what would happen if you tried to perform cross-validation with these folds.
(e) [5 points] Re-format the data in the following way: take each of the input variables, and feed
it through a set of Gaussian basis functions, defined as follows. For each variable, use 5
univariate basis functions with means evenly spaced between −1and 1and variance σ2. You
will experiment with σ2values of 0.1,0.5,1and 5.
(f) [5 points] Using no regularization and doing regression with this new set of basis functions,
plot the training and testing error as a function of σ2(when using only basis functions of a given
σ2). Add constant lines showing the training and testing error you had obtained in question 1b.
Explain how σ2influences overfitting and the bias-variance trade-off.
1
