Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
MOdel traning ang guessing on the basis of regression and using linear classification model..
Typology: Lab Reports
1 / 7
This page cannot be seen from the preview
Don't miss anything!
Optimization algorithms are at the core of machine learning, enabling models to learn from data. They minimize a loss function by iteratively adjusting model parameters. The choice of optimizer is critical, influencing training speed, convergence, and the final performance of the model. This document provides a detailed overview of a wide range of optimization algorithms, from foundational first-order methods to advanced second-order techniques.
First-order methods use the gradient (the first derivative) of the loss function to guide the search for the minimum. They are computationally efficient and widely used in deep learning. 2.1. Gradient Descent (GD) Gradient Descent is the most fundamental optimization algorithm. It works by calculating the gradient of the loss function with respect to the model parameters and moving in the opposite direction of the gradient. Formula: θ_{t+1} = θ_t - η · ∇J(θ_t) Where:
Cons: Can sometimes fail to converge to the optimal solution (generalization issues). Weight decay (L2 regularization) is not handled optimally in the original formulation. 2.6. AdamW (Adam with Decoupled Weight Decay) AdamW is a modification of the Adam optimizer that addresses a specific flaw: the way weight decay (L2 regularization) is implemented. In standard Adam, L2 regularization is added to the loss function, which means it gets mixed with the adaptive learning rate logic. AdamW decouples the weight decay from the gradient update step. Why AdamW? By decoupling weight decay, AdamW ensures that the regularization effect is consistent regardless of the adaptive learning rate. This leads to significantly better generalization performance, making it the preferred choice for training modern deep learning models like Transformers. Pros and Cons: Pros: Better generalization than standard Adam. More stable training for large-scale models. Cons: Slightly more complex to implement (though available in all major frameworks).
These methods use the second derivative (Hessian matrix) of the loss function to find the minimum. They can converge much faster but are computationally expensive. 3.1. Conjugate Gradient (CG) An iterative algorithm for solving systems of linear equations, which is equivalent to minimizing a quadratic function. It's more efficient than GD as it avoids redundant search directions. Pros and Cons: Pros: Faster than GD, doesn't require storing a Hessian matrix. Cons: Primarily designed for quadratic problems, less common in deep learning. 3.2. BFGS (Broyden–Fletcher–Goldfarb–Shanno)
A powerful quasi-Newton method that approximates the inverse Hessian matrix, avoiding the need to compute it directly. It's very effective for optimization problems. Pros and Cons: Pros: Very fast convergence, highly accurate. Cons: Requires storing the dense inverse Hessian approximation, which is memory-intensive for large models. 3.3. L-BFGS (Limited-memory BFGS) A memory-efficient version of BFGS. Instead of storing the full inverse Hessian approximation, it stores only a few vectors that represent the approximation implicitly. This makes it suitable for large-scale problems. Pros and Cons: Pros: Memory-efficient, suitable for large models. Cons: Can be slower than the full BFGS method.
