Neural Network Exam Flashcards
1. Why does a NN with no activation become linear?
Because stacked linear layers collapse into one linear transform.
2. Does ReLU make a single perceptron nonlinear?
Yes—ReLU introduces a non-linear kink.
3. Why does sigmoid cause vanishing gradients?
Derivative near 0 for large |x|.
4. Why doesn’t ReLU’s non-differentiability break backprop?
Use a subgradient; single point irrelevant.
5. Why use Leaky ReLU?
Prevents dead neurons.
6. Why is MSE bad for classification?
Slow learning; bad probabilities.
7. Why use cross-entropy with softmax?
Measures mismatch of distributions.
8. Why is NN loss non-convex?
Nonlinear activations create complex landscapes.
9. When does GD fail to converge?
Large LR, exploding/vanishing gradients.
10. Why use LR schedules?
Large steps early, fine-tuning later.
11. How does chain rule power backprop?
Layerwise gradient multiplication.
12. What causes vanishing/exploding gradients?
Multiplying many Jacobians.
13. What happens if weights initialize identically?
Neurons stay identical—no symmetry breaking.
14. Why deeper nets have gradient issues?
More multiplicative steps.
15. Why can nets overfit big data?
High capacity can memorize noise.
16. How does dropout help?
Randomly removes units to force robust features.
17. Effects of very small vs large batch size?
Small = noisy grads; large = smooth grads.
