EECS 553: Machine Learning - Lecture 15: Modern Regularization and Optimization Methods, Slides of Biostatistics

Download EECS 553: Machine Learning - Lecture 15: Modern Regularization and Optimization Methods and more Slides Biostatistics in PDF only on Docsity!

EECS 553: Machine Learning

Lecture 15: Modern Regularization and Optimization Methods

First: Wrap up ConvNets

What happens when we continue stacking deeper layers on a “plain” convolutional neural network?

Case Study: ResNet

Q: What’s weird about these training and test curves? 56 - layer model performs worse on both training and test error

• > The deeper model performs worse, but it’s not caused by overfitting!

Training ResNets in practice

• SGD + Momentum (0.9)
• Learning rate: 0.1, divided by 10 when validation error plateaus
• Mini-batch size 256
• Weight decay of 1e- 5
• No dropout used
• Batch Normalization after every CONV layer
• Xavier/2 initialization from He et al.

Time to Revisit:

Optimization and Regularization

Regularization

Regularization

• Regularized empirical risk minimization ! !

!"# $ ' ( !

!

• Ridge:. " = " % %
• Lasso:. " = "
  Penalty function

Implication: Model weights are small Implication: Model weights are small and mostly equal to zero Loss function !(#)

Regularization

Regularized empirical risk minimization ! !

• Gradient updates: " &'#

&

!

& = " &

&

&