Download Linear Programming and Optimization Problems and more Exams Nursing in PDF only on Docsity!
Convex Optimization Problems
and Applications
Convex Optimization (University of
Manchester)
1. ILO: Linear Programming
(a) Formulating a Linear Programming Problem
Let $x_{ij}$ denote the amount of cargo $i \in {1, 2}$ in compartment $j \in {1, 2}$. The objective function is the total profit: $f(x) = 300 \cdot (x_{11} + x_{12}) + 350 \cdot (x_{21} + x_{22})$ or equivalently $f(x) = 6 \cdot (x_{11} + x_{12}) + 7 \cdot (x_{21} + x_{22})$.
The constraints are given by the weight limitations of the compartments and the amount of cargo available: $x_{11} + x_{12} \leq 25$ (total amount of cargo 1) $x_{21} + x_{22} \leq 30$ (total amount of cargo 2) $x_{11} + x_{21} \leq 25$ (weight constraint on compartment 1) $x_{12} + x_{22} \leq 20$ (weight constraint on compartment 2) $-x_{11} - x_{21} \leq -25$ (weight achieved on compartment 1) $-x_{12} - x_{22} \leq -20$ (weight achieved on compartment 2) $x_{ij} \geq 0$ (cargo can't have negative weight)
Observing that $x_{11} + x_{21} = 25$ and $x_{12} + x_{22} = 20$, we can let $x_{11} = 25 - x_{21}$ and $x_{12} = 20 - x_{22}$. Relabelling $x = x_{21}$ and $y = x_{22}$, the problem becomes: $f(x) = x + y$ with the feasible set: $-x - y \leq -20$ (total amount of cargo 1) $x + y \leq 30$ (total amount of cargo 2) $x \leq 25$ (cargo can't have negative weight) $y \leq 20$ (cargo can't have negative weight) $x \geq 0$ (cargo can't have negative weight) $y \geq 0$ (cargo can't have negative weight)
The feasible set and the objective can be visualized. There are 5 vertices among the 15 minors of the matrix $A$, and the optimal solutions are $x_{1,4}$ and $x_{2,4}$, or any point in between along the hyperplane $x + y = 30$.
In terms of the original system, we take all of Cargo 2, in whichever compartment is convenient, and complete the load with Cargo 1.
(b) Optimality Conditions and Central Path
(i) The optimality conditions are: $Ax - b = 0$ $A^T y + s - c = 0$ $XSe = 0$ $x \geq 0, s \geq 0$
(ii) The solution is obvious: the feasible set is the line segment connecting $(0, 1)^T$ and $(1, 0)^T$, and the optimal value is $x^* = (0, 1)^T$.
(iii) Solving the optimality conditions, we get the solution $x = (0, 1)^T$, $s = (1, 0)^T$, $y = 0$, which is the optimal solution.
(iv) The central path is the set $(x, y, s)$ that arises when we change the conditions $x_i s_i = 0$ to $x_i s_i = \tau$, for $\tau > 0$. The unconstrained problem is solved iteratively, while decreasing $\tau$ until it is suitably small (and ensuring $x, s > 0$ along the way).
(c) Converting a Convex Optimization Problem to Linear Programming
The problem is: $\minimize |Ax - b|_\infty$ subject to $|x|_1 \leq 1$
This can be written equivalently as: $\minimize t_0$ subject to $-t_0 \leq a_i^T x \leq t_0, 1 \leq i \leq m$ $-t_i \leq x_i \leq t_i, i = 1, \dots, n$ $ \sum_{i=1}^n t_i \leq 1$
2. ILO: Application to Supervised Learning, and Convexity
(a) Supervised Learning as an Optimization Problem
(i) Three examples of supervised learning problems are text classification, image recognition, and linear regression.
(ii) When applying gradient descent to a supervised learning problem, the computational cost can be high due to the need to compute the gradient of each summand (which equals the number of data samples). A way to overcome this is to use a randomized method such as stochastic gradient descent, where in each step, one chooses one index at random and computes the gradient of that summand.
(b) Convexity of Least-Squares Regression
(i) The matrix $XX^T \in \mathbb{R}^{n \times n}$ is positive semidefinite. A necessary condition for it to be positive definite is $m \geq n$.
(ii) The function $f(\beta) = |X^T \beta - y|^2$ is convex, as the Hessian $ \nabla^2 f(\beta) = XX^T$ is positive semidefinite.
(iii) Given data $x_1, \dots, x_N \in \mathbb{R}^{n-1}$ and labels $y_1, \dots, y_N \in \mathbb{R}$, we can construct the loss function $f(\beta) = \frac{1}{N} |X^T \beta - y|^2$, which is convex. Solving the optimization problem $\minimize_{\beta \in \mathbb{R}^n} f(\beta)$ gives us a linear regressor $h(x; \beta) = \beta^T x$ that can be used to predict the output for a new input vector $x$.
(c) Convexity of a Specific Function
(i) The function $\phi(z) = z \log z$ is convex, as $\phi'(z) = \log z + 1$ and $ \phi''(z) = 1/z > 0$.
minimize $x^T x$ subject to $Ax = b$
is:
The Lagrangian is $L(x, \mu) = x^T x + \mu^T (Ax - b)$. The dual function is $\phi(\mu) = -\frac{1}{2} \mu^T AA^T \mu - b^T \mu$, which is concave. The optimal dual variable is $\mu^ = -2(AA^T)^{-1}b$, and the optimal primal variable is $x^ = A^T(AA^T)^{-1}b$.
Constrained Optimization Problem
Consider the following problem:
minimize $x_1^2 + 2x_2^2$ subject to $(x_1 - 1)^2 + x_2 \leq 1$ $x_2 = 1/2$
The feasible set is a ball of radius 1 centered at $(1, 0)$ and the line $x_2 = 1/2$. The KKT conditions for this problem are:
$(x_1 - 1)^2 + x_2 \leq 1$ $x_2 = 1/2$ $\lambda \geq 0$ $\lambda((x_1 - 1)^2 + x_2 - 1) = 0$ $2x_1 + 2\lambda(x_1 - 1) = 0$ $4x_2 + 2\lambda x_
Solving these conditions, we find the optimal point $x^ = (2 - \sqrt{3}, 1/2)$ with the corresponding optimal value $f(x^ ) = 9 - \sqrt{3}$.
KKT Conditions
The KKT conditions for the general constrained optimization problem:
minimize $f(x)$ subject to $g(x) \leq 0$ $Ax = b$
are:
$g(x) \leq 0$ $Ax = b$ $\lambda \geq 0$ $\lambda_i g_i(x) = 0, \forall i$ $ \nabla f(x) + \sum_{i=1}^m \lambda_i \nabla g_i(x) + A^T \mu = 0$
For the special case of a linear programming problem:
minimize $\langle c, x \rangle$ subject to $Ax = b$ $x \geq 0$
the KKT conditions have the form:
$-x \leq 0$ $Ax = b$ $\lambda \geq 0$ $x_i \lambda_i = 0, \forall i$ $c - \lambda + A^T \mu = 0$
Farkas Lemma
There exists a vector $x \neq 0$ satisfying $Ax = 0$ and $x \geq 0$ if and only if there is no vector $y$ such that $A^T y > 0$.
The proof of this statement is provided in the original text.
