Download Linear programming tutorial and more Study notes Linear Programming in PDF only on Docsity!
Linear programming tutorial
Ivan Savov
August 17, 2020
git commit 10da63a
Contents
1 Linear programming 1 1.1 Simplex algorithm.............................. 3 1.2 Using SymPy to solve linear programming problems............ 12 1.3 Using a linear program solver........................ 16 1.4 Duality.................................... 17 1.5 Practice problems............................... 20 1.6 Links...................................... 21
1 Linear programming
In the early days of computing, computers were primarily used to solve optimization problems so the term “programming” is often used to describe optimization problems. Linear programming is the study of linear optimization problems that involve linear constraints. Optimization problems play an important role in many business applica- tions: the whole point of a corporation is to constantly optimize profits, subject to time, energy, and legal constraints. Suppose you want to maximize the quantity g(x, y) subject to some constraints on the values x and y. To maximize g(x, y) means to find the values of x and y that make g(x, y) as large as possible. Let’s assume the objective function g(x, y) represents your company’s revenue, and the variables x and y correspond to monthly production rates of “Xapper” machines and “Yapper” machines. You want to choose the production rates x and y that maximize revenue. If the revenue from each Xapper machine is $3000 and the revenue from each Yapper machine is $2000, the monthly revenue is described by the function g(x, y) = 3000x + 2000y. Due to the limitations of the current production facilities, the rates (x, y) are subject to various constraints. We’ll assume each constraint can be written in the form a 1 x + a 2 y ≤ b. The maximum number of Xapper machines that can be produced in a month is three, written x ≤ 3. Similarly, the company can produce at most four Yapper machines, denoted y ≤ 4. Suppose it takes two employees to produce each Xapper machine and one employee to produce each Yapper machine. If the company has a total of seven employees, the human resources limits impose the constraint 2x + y ≤ 7 on the production rates. Finally, logistic constraints allow for at most five machines to be shipped each month, which we write as x + y ≤ 5.
1
1 LINEAR PROGRAMMING 2
This production rate optimization problem can be expressed as the following linear program:
max x,y g(x, y) = 3000 x + 2000y,
subject to the constraints
x ≤ 3 , y ≤ 4 , 2 x + y ≤ 7 , x + y ≤ 5 , x ≥ 0 , y ≥ 0.
Each of the inequalities represents one of the real-world production constraints. We also included the non-negativity constraints x ≥ 0 and y ≥ 0 to show it’s impossible to produce a negative number of machines—we’re not doing an accounting scam here, this is a legit Xapper–Yapper business. On first hand the problem looks deceptively simple. We want to find the coordinates (x, y) that maximize the objective function g(x, y), so we can simply find the direction of maximum growth of g(x, y) and go as far as possible in that direction. Rather than attempt to plot the function g(x, y) in three dimensions, we can visualize the growth of g(x, y) by drawing level curves of the function, which are analogous to the lines shown on topographic maps. Each line represents some constant height: g(x, y) = cn, for n ∈ { 0 , 1 , 2 , 3 ,.. .}. Figure 1 shows the level curves of the objective function g(x, y) = 3000x + 2000y at intervals of 3000.
- 5 1 1. 5 2 2. 5 3 3. 5 4 4. 5 5 5. 5 6 x
y
- 5
1
- 5
2
- 5
3
5
0
5
5
- 5 Level curves of
(^) g( x, y )
Figure 1: The objective function g(x, y) = 3000x + 2000y grows with x and y. The direction of maximum growth is (3, 2)—if you think of g(x, y) as describing the height of a terrain, then the vector (3, 2) points uphill. The dashed lines in the graph represent the following level curves: g(x, y) = 3000, g(x, y) = 6000, g(x, y) = 9000, g(x, y) = 12000, and so on.
Linear programming problems are mainly interesting because of the constraints im- posed on the feasible region. Each inequality corresponds to a restriction on the possible production rates (x, y). A coordinate pair (x, y) that satisfies all constraints is called a
1.1 Simplex algorithm 4
- Place constraint equations and slack variables in the first rows.
- Place −g(~x) in the last row of the tableau.
INITIALIZATION: Start the algorithm from the coordinates ~x 0 that correspond to a vertex of the constraint region.
MAIN LOOP: Repeat while negative numbers in the last row of the tableau exist:
Choose pivot variable: The pivot variable is the most negative number in the last row of the tableau. Choose pivot row: The pivot row corresponds to the first constraint that will become active when the pivot variable increases. Move: The algorithm “moves” from the vertex with coordinates ~x to the vertex with coordinates ~x′, which is defined as a vertex where the pivot row is active. Change of variables: Perform the change of variables from the coordinates system B with origin at ~x to a coordinates system B′^ with origin at ~x′. This involves row operations on the tableau.
OUTPUT: When no more negative numbers in the last row exist, we have reached an optimal solution ~x∗^ ≡ argmax~x g(~x).
Though not exactly identical to the Gauss–Jordan elimination procedure, the simplex algorithm is similar to it because it depends on the use of row operations. For this reason, linear programming and the simplex algorithm are often forced upon students taking a linear algebra course, especially business students. I’m not going to lie to you and tell you the simplex algorithm is simple, but it is very powerful so you should know it exists, and develop a general intuition about how it works. And, as with all things corporate-related, it’s worth learning about them so you’ll know the techniques of the enemy. In the remainder of this section, we’ll go through the steps of the simplex algorithm needed to find the optimal solution to the Xapper-and-Yapper production problem. We’ll analyze the problem from three different angles: graphically by drawing shifted coordinate systems, analytically by writing systems of equations, and computationally by using a new matrix-like structure called a tableau.
Definitions
We now introduce some useful terminology used in linear programming:
- An inequality a 1 x + a 2 y ≤ b is loose (or slack ) for a given (x, y) if there exists a positive constant s > 0 such that a 1 x + a 2 y + s = b.
- An inequality a 1 x + a 2 y ≤ b is tight for a given (x, y) if the equality conditions holds a 1 x + a 2 y = b.
- s: slack variable. Slack variables can be added to any inequality a 1 x + a 2 y ≤ b to transform it into an equality a 1 x + a 2 y + s = b. Note slack variables are always nonnegative s ≥ 0.
- A vertex of an n-dimensional constraint region is a point where n inequalities are tight.
- An edge of an n-dimensional constraint region is place where n − 1 inequalities are tight.
1.1 Simplex algorithm 5
- A pivot variable is a variable whose increase leads to an increase in the objective function.
- A pivot row is a row in the tableau that corresponds to the currently active edge during a “move” operation of the simplex algorithm.
- ~v: a vector that represents the current state of the linear program.
- A tableau is a matrix that represent the constraints on the feasible region and the current value of the objective function. Tableaus allow us to solve linear programming problems using row operations.
Introducing slack variables
In each step of the simplex algorithm, we’ll keep track of which constraints are active (tight) and which are inactive (loose). To help with this bookkeeping task, we introduce nonnegative a slack variable si ≥ 0 for each of the inequality constraints. If si > 0, inequality i is loose (not active), and if si = 0, inequality i is tight (active). If we want to remain within the feasible region, no si can become negative, since si < 0 implies inequality i is not satisfied. Introducing the slack variables, the linear program becomes:
max x,y g(x, y) = 3000x + 2000y,
subject to the equality constraints:
x + s 1 = 3 , y + s 2 = 4 , 2 x + y + s 3 = 7 , x + y + s 4 = 5 , x, y, s 1 , s 2 , s 3 , s 4 ≥ 0.
This is starting to look a lot like a system of linear equations, right? Recall that what matters the most in systems of linear equations are the coefficients, and not the variable names. Previously, when faced with overwhelming complexity in the form of linear equations with many variables, we used an augmented matrix to help us focus only on what matters. We’ll use a similar approach again.
Introducing the tableau
A tableau is a compact representation of a linear programming problem in the form of an array of numbers, analogous to the augmented matrix used to solve systems of linear equations.
x + s 1 = 3 y + s 2 = 4 2 x + y + s 3 = 7 x + y + s 4 = 5 − 3000 x − 2000 y =?
⇔
1 0 1 0 0 0 3 0 1 0 1 0 0 4 2 1 0 0 1 0 7 1 1 0 0 0 1 5 − 3000 − 2000 0 0 0 0?
.
The first six columns of the tableau correspond to the variables x, y, s 1 , s 2 , s 3 , s 4 , and the last column contains the constants from the constraint equations. In the last row, we use the coefficients of the negative objective function −g(x, y). This strange initialization of the last row of the tableau is a trick we use to calculate the current
1.1 Simplex algorithm 7
I know what you’re thinking. Surely it would have been simpler to keep track of the position (x, y) and the slackness separately (s 1 , s 2 , s 3 , s 4 ), rather than to invent a binary state vector ~v ∈ { 0 , 1 }^6 , and then depend on matrix multiplication to find the coordinates. True that. But the energy we invested to represent the constraints as a tableau will allow us to perform complex geometrical operations by performing row operations on the tableau. The key benefit of the combined representation is that it treats problem variables and slack variables on the same footing.
5 1 1. 5 2 2. 5 3 3. 5 x
5
1
- 5
2
- 5
3
5
0
x
y
Figure 3: The (x, y) coordinates are measured with respect to (0, 0).
It’s now time to start making progress. Let’s make a move. Let’s cut some slack!
Choose the pivot variable
The simplex algorithm continues while there exist negative numbers in the last row of the tableau. Recall we initialized the last row of the tableau with the coefficients of −g(x, y) and the initial value to g(0, 0) = 0.
Both the x and y columns contain negative numbers. This means the objective function would increase if were to increase either the x or the y variables. The coefficient −3000 in the last row represents our incentives towards increasing the variable x: each increase of 1 step in the Xapper production will result in an increase of 3000 in the value of g(x, y). Similarly, the coefficient −2000 indicates that a unit-step in the y direction will increase g(x, y) by 2000. It’s a bit complicated why we put the negatives of the coefficients in the row for the objective function, but you’ll have a chance to convince yourself it all works out nicely when you see the row operations.
1.1 Simplex algorithm 8
Given the options x and y, we choose to increase the x variable since it leads to a biggest gain in profits. Remember, we’re playing the role of the greedy corporate capitalist whose only motivation is to maximize profits.
Choose the pivot row
We’ve decided to increase the x variable. The next step is to check how big we can make the x variable before hitting one of the constraints. Which constraint do we hit first when we move in the x direction? In the first equation x + s 1 = 3, we could increase x up to x = 3. In the second equation y + s 2 = 0 + s 2 = 4, the x-variable doesn’t appear so it “allows” an arbitrary increase in x. The third constraint 2x+y +s 3 = 2x+0+s 3 = 7 allows an increase in x up to x ≤ 7 /2 = 3.5. Finally, the fourth equation x + 0 + s 4 = 5 imposes the constraint x ≤ 5. The equation x + s 1 = 3 allows the smallest increase in x, so this equation will become our pivot row. Specifically, pivoting through the equation x + s 1 = 3 means we’re decreasing s 1 from 3 to 0 and correspondingly increasing x from 0 to 3. We have thue de-slacked the constraint x + s 1 = 3. Performing the pivot operation corresponds to the following change in the state vector: ~v = (0, 0 , 1 , 1 , 1 , 1) → ~v′^ = (1, 0 , 0 , 1 , 1 , 1).
To understand the effect of this change of state, let’s look at the constraint equations that result from the new state vector:
1 0 1 0 0 0 0 1 0 1 0 0 2 1 0 0 1 0 1 1 0 0 0 1
1 0 0 1 1 1
=
3 4 7 5 0
⇒
x + = 3 s 2 = 4 s 3 = 7 s 4 = 5.
The effect of the new state ~v′^ is to increase x from 0 to 3, which leads to corresponding decrease of s 1 from 3 to 0. The value of the six variables are now:
x = 3, y = 0, s 1 = 0, s 2 = 4, s 3 = 7, s 4 = 5.
Note the change of state vector ~v → ~v′^ we performed did not change the system of equations that describe the constraints of the linear program:
x + + s 1 = 3 y + s 2 = 4 2 x + y + s 3 = 7 x + y + s 4 = 5.
This system of equations corresponds to the tableau:
Except for the increase in x and the corresponding decrease in s 1 , we haven’t done anything to the tableau. The next step of the simplex algorithm involves performing row operations on the tableau.
1.1 Simplex algorithm 10
Note s′ 1 = 0 in the system of equations so we could completely ignore this variable and the associated column of the tableau. Nevertheless, we choose to keep the s′ 1 column around as a bookkeeping device, because it’s useful to keep track of how many times we have subtracted the first row from the other rows. We have now completed the first step in of the simplex algorithm, and we continue the procedure by looking for the next pivot.
Choose the pivot
Observe the second column of the row that corresponds to the objective function con- tains a negative number, which means the objective function will increase if we increase the variable y. Having decided to increase y′^ we must now decide by what amount can we increase the variable y′^ before we hit one of the constraints? We can find this out by computing the ratios of the constants in the rightmost column of the tableau, and the y′^ coefficients of each row. The first constraint does not contain y′^ at all, so it allows any increase in y′. The second constraint will become active when y′^ = 4. The third constraint allows y′^ to go up to y′^ = 1, and the fourth constraint becomes active when y′^ = 2. The largest increase in y′^ that satisfies all the constrains is y′^ = 1. We’ll use the third row of the tableau as the pivot row, which means we de-slack s′ 3 from 1 to 0 and correspondingly increasing y′^ from 0 to 1. After pivoting, the values of the six variables are
x′^ = 3, y′^ = 1, s′′ 1 = 0, s′′ 2 = 4, s′′ 3 = 0, s′′ 4 = 2.
Change of variables
We can now subtract the equation y′^ = 1 from the other y′-containing rows. The required row operations are
- R 2 ← R 2 − R 3
- R 4 ← R 4 − R 3
- R 5 ← R 5 + 2000R 3
After applying these row operations, the resulting tableau is
The geometrical interpretation of subtracting the equation y′^ = 1 from the other con- straints is a change to a new coordinate system (x′′, y′′) with origin at (3, 1), as illustrated in Figure 5.
Choose a pivot
The third column of the tableau contains a negative number, which means the objective function will increase if we increase s′′ 1. We pick s′′ 1 as the next pivot variable. We now check the different constraints containing s′′ 1 to see which will become active first when we increase s′′ 1. The first inequality imposes s′′ 1 ≤ 3, the second imposes s′′ 1 ≤ 3 /2 = 1.5, the third equation contains a negative amount of s′′ 1 and thus can’t be used as a pivot. The equation in the fourth row, s′′ 1 − s′′ 3 + s′ 4 = 1, imposes the constraint
1.1 Simplex algorithm 11
5 1 1. 5 2 2. 5 3 3. 5 x
5
1
- 5
2
- 5
3
5
0
x′
y′
x′′
y′′
Figure 5: The origin of the (x′′, y′′) coordinate system is at (3, 1).
s′′ 1 ≤ 1, and is the tightest constraint. Therefore, we’ll use the fourth row equation as the pivot. After pivoting, the new values of the variables are
x′′^ = 3, y′′^ = 1, s′′ 1 = 1, s′′ 2 = 4, s′′ 3 = 0, s′′ 4 = 0.
We now carry out the following row operations to eliminate s′′ 1 from the other eqations:
- R 1 ← R 1 − R 4
- R 2 ← R 2 − 2 R 4
- R 3 ← R 3 + 2R 4
- R 5 ← R 5 + 1000R 4
The final tableau is (^)
We state variables that correspond to this tableau are
x′′′^ = 2, y′′′^ = 3, s′′′ 1 = 1, s′′′ 2 = 1, s′′′ 3 = 0, s′′′ 4 = 0.
We know this tableau is final because there are no more negative numbers in the last row. This means there are no more directions we can move in that would increase the objective function.
1.2 Using SymPy to solve linear programming problems 13
The full system of constrains that correspond to the feasible region is
x + s 1 = 3 y + s 2 = 4 2 x + y + s 3 = 7 x + y + s 4 = 5 −x + s 5 = 0 −y + s 6 = 0.
We can create a tableau as a SymPy matrix object whose coefficients correspond to the above system of equations. Use the following command:
M = Matrix([ [ 1, 0, 1, 0, 0, 0, 0, 0, 3] [ 0, 1, 0, 1, 0, 0, 0, 0, 4] [ 2, 1, 0, 0, 1, 0, 0, 0, 7] [ 1, 1, 0, 0, 0, 1, 0, 0, 5] [ -1, 0, 0, 0, 0, 0, 1, 0, 0] [ 0, -1, 0, 0, 0, 0, 0, 1, 0] [-3000,-2000, 0, 0, 0, 0, 0, 0, 0]] )
Note constructing the tableau did not require any special command—a tableau is just a regular Matrix object. The first six rows of the matrix M correspond to the constraints on the feasible region, while the last row contains the negatives of the coefficients of the objective function −g(x, y) = − 3000 x − 2000 y.
Vertices and sides
The simplex algorithm starts from a vertex of the feasible region, and moves along the edges of the feasible region. For the two-dimensional Xapper–Yapper production problem, the vertices of the region are the corners, and the edges are the sides of a two-dimensional polygon. A polygon is the general math term for describing a subset of the xy-plane delimited by a finite chain of straight line segments. More generally, the constraint region of an n-dimensional linear programming prob- lem has the shape of an n-dimensional polytope. A vertex of the polytope corresponds to a place where n constraint inequalities are satisfied, while an edge corresponds to the intersection of n − 1 inequalities. In each step of the simplex algorithm, we’ll keep track of which constraints in the tableau are active (tight) and which are inactive (loose). The simplex algorithm must be started from one of the vertices of the rate region. For the Xapper–Yapper production problem, we initialized the tableau from the corner (0, 0), which is the vertex where the two nonnegativity constraints intersect.
M [ 1, 0, 1, 0, 0, 0, 0, 0, 3] [ 0, 1, 0, 1, 0, 0, 0, 0, 4] [ 2, 1, 0, 0, 1, 0, 0, 0, 7] [ 1, 1, 0, 0, 0, 1, 0, 0, 5] [ -1, 0, 0, 0, 0, 0, 1, 0, 0] (active) [ 0, -1, 0, 0, 0, 0, 0, 1, 0] (active) [-3000, -2000, 0, 0, 0, 0, 0, 0, 0]
The current state of the simplex algorithm is completely specified by the two (active) flags. We can deduce the value of all state variables from the knowledge that the
1.2 Using SymPy to solve linear programming problems 14
inequalities −x ≤ 0 and −y ≤ 0 are tight. The fifth and sixths constraints are active, which means s 5 = 0 and s 6 = 0, from which we deduce that x = 0 and y = 0. If x = 0 and y = 0, then the other inequalities are maximally slack and thus s 1 = 3, s 2 = 4, s 3 = 7, s 4 = 5.
Choose the pivot 1
In the first step we choose to increase the x-variable since it leads to the bigger gain in g(x, y) (3000 per unit step), as compared to an increase in the y-direction (2000 per unit step). Next we must choose the pivot row. How big of a step can we make in the x- direction before hitting some constraint? To calculate the maximum allowed step in each direction, you must divide the current slack available by the x coefficient in each inequality:
[ M[i,8]/M[i,0] for i in range(0,4) ] [3, oo, 7/2, 5]
The above magical python invocation computes the largest step in the x direction allowed by constant in each constraint. Since the step size of the first row (index 0) is the smallest, we must choose the first row as our pivot.
Change of variables 1
Carry out the necessary row operations to eliminate all other numbers in the x column.
M.row(2, lambda v,j: v-2M[0,j] ) M.row(3, lambda v,j: v-1M[0,j] ) M.row(4, lambda v,j: v+1M[0,j] ) M.row(6, lambda v,j: v+3000M[0,j] ) M [1, 0, 1, 0, 0, 0, 0, 0, 3] (active) [0, 1, 0, 1, 0, 0, 0, 0, 4] [0, 1, -2, 0, 1, 0, 0, 0, 1] [0, 1, -1, 0, 0, 1, 0, 0, 2] [0, 0, 1, 0, 0, 0, 1, 0, 3] [0, -1, 0, 0, 0, 0, 0, 1, 0] (active) [0, -2000, 3000, 0, 0, 0, 0, 0, 9000]
Again, the state of the linear program can be deduced from the information about which constraints are currently active. Since the first constraint is active (s 1 = 0), we know x = 3 and sixth inequality hasn’t changed so we still have y = 0.
Choose the pivot 2
This time we see there are gains to be had if we increase the y-variable, so we check which constraint will become active first as we increase the y-variable. Note we M[i,1] corresponds to the second column of the Matrix object M because Python uses 0-based indexing.
[ M[i,8]/M[i,1] for i in range(0,4) ] [oo, 4, 1, 2].
The third row (index 2 ) is the corresponds to the constrain that allows the smallest step in the y-direction, so we must use the third row as our pivot.
1.3 Using a linear program solver 16
[0, 1, 0, 0, -1, 2, 0, 0, 3] (active) [0, 0, 1, 0, -1, 1, 0, 0, 1] (active) [0, 0, 0, 0, 1, -1, 1, 0, 2] [0, 0, 0, 0, -1, 2, 0, 1, 3] [0, 0, 0, 0, 1000, 1000, 0, 0, 12000]
The simplex algorithm now terminates, since there are no more negative numbers in the last row. The final state of the algorithm is
x = 2, y = 3, s 1 = 1, s 2 = 1, s 3 = 0, s 4 = 0.
The value of the function at this vertex is g(2, 3) = $12000.
1.3 Using a linear program solver
Using the Matrix row operations provided by SymPy is certainly a fast method to per- form the steps of the simplex algorithm. But there is an even better approach: we could outsource the work of solving the linear program completely to the computer by using the function linprog that comes with the Python package scipy. These functions are not available in the web interface at live.sympy.org, so you’ll need to install Python and the packages numpy and scipy on your computer. Once this is done, solving the linear program is as simple as setting up the problem description and calling the function linprog:
from scipy.optimize import linprog c = [-3000, -2000] # coefficients of f(x) A = [[ 1, 0], # coefficients of the constraints [ 0, 1], [ 2, 1], [ 1, 1], [ -1, 0], [ 0, -1]] b = [3, 4, 7, 5, 0, 0] # constraint constants
linprog(c, A_ub=A, b_ub=b) # find min of f(x) s.t. Ax <= b success: True x: [ 2, 3] fun: - slack: [ 1, 1, 0, 0, 2, 3] message: Optimization terminated successfully. nit: 3
The function linprog solves the minimization problem min f (~x), subject to the con- straints A~x ≤ ~b. The objective function f (x) ≡ ~c · ~x is specified in terms of its vector of coefficients ~c. We can’t use linprog directly since the linear program we’re trying to solve is a maximization problem max g(~x). To work around this mismatch between what we want to do and what the function can do for us, we can multiply the objective function by a negative sign. Finding the maximum of g(~x) is equivalent to finding the minimum of f (~x) ≡ −g(~x), since max g(~x) = min −g(~x) = min f (~x). This is the reason why we specified the coefficients of the objective function as ~c = (− 3000 , −2000). We must also keep in mind the “adaptor” negative sign when interpreting the result: the minimum value of f (~x) is −12 000, which means the maximum value of g(~x) = 12 000.
1.4 Duality 17
For more information about other linear program options you can supply, run the command help(linprog) from the Python command prompt. The package scipy supplies a number of other useful optimization functions, for quadratic programming, and optimization of non-linear functions. It’s good if you understand details of the simplex algorithm (especially if you’ll be asked to solve linear programs by hand on your exam), but make sure also know how to check your answers using the computer. Another useful computer tool is Stefan Waner’s linear program solver, which can be accessed here bit.ly/linprog solver. This solver not only shows you the solution to the linear program, but also displays the intermediate tableaus used to obtain the solution. This is very useful if you want to check your work.
1.4 Duality
The notion of duality is a deep mathematical connection between a given linear program and another related linear program called the dual program. We’ll now briefly discuss the general idea of duality. I want to expose you to the idea mainly because it gives us a new way of looking at linear programming.
Shadow prices
Instead of thinking of the problem in terms of the production rates x and y, and slack variables {si}, we can think of the problem in terms of the shadow prices (also known as opportunity costs) associated with each of the resources the company uses. The opportunity cost is an economics concept that describes the profits your company is foregoing by virtue of choosing to use a given resource in production. We treat these as “costs” because when choosing to use certain production facilities, it means you won’t be renting out the production facilities, and thus missing out on potential profits. If your company chooses not to use a given resource, it could instead sell this resource to the market—as part of an outsourcing deal, for example. The assumption is that a market exists where each type of resource (production facility, labour, and logistics) can be bought or sold. The shadow prices represent the price of each resource in such a market. The company has the opportunity to rent out their production facilities instead of using them. Alternatively, the company could increase its production by renting the necessary facilities from the market. We’ll use the variables λ 1 , λ 2 , λ 3 , and λ 4 to describe the shadow prices:
- λ 1 : the price of one unit of the Xapper-machine producing facilities
- λ 2 : the price of one unit of the Yapper-machine producing facilities
- λ 3 : the price of one unit of human labour
- λ 4 : the price of one using one shipping dock
Note these prices are not imposed on the company by the market but are directly tied to the company’s production operations, and the company is required to buy or sell at the “honest” price, after choosing to maximize its profit. Being “honest” means the company should agree to sell resource i at the price λi at the same price at which it would be willing to buy resource i to increase its revenue. This implies that the shadow price associated with each non-active resource constraint is zero. A zero shadow price for a given resource means the company has an excess amount of this resource, and is not interested (at all) in procuring more of this resource. Alternatively, we can say a zero shadow price means the company is willing to give away this resource to be used by
1.4 Duality 19
General formulation Consider the n-dimensional linear programming problem of maximizing g(~x) ≡ ~c · ~x ≡ ~cT~x. The vector of constants ~c ∈ Rn^ contains the coefficients of the objective function. The constraints of the problem can be expressed as a matrix equation A~x ≤ ~b, where the matrix A ∈ Rm×n^ contains the coefficients of the m constraints, and the vector ~b ∈ Rm^ contains the constants of the constraint equations. We will call this the primal
problem. To every primal problem max~x ~cT~x subject to A~x ≤ ~b, there exists a dual optimiza- tion problem, which is also a linear program:
max ~x
~c · ~x
subject to A~x ≤ ~b
min ~λ
~b · ~λ
subject to ~λTA ≥ ~c T.
Note the dual problem involves a minimization, and the constraint region is obtained from the matrix AT^ and greater-than-or-equal inequalities. The variables of the dual problem ~λ = (λ 1 , λ 2 ,... , λm) correspond to the coefficients of a linear combination of the m constraint equations. As a mathematical convention, when we discuss linear combinations of constraint equations in optimization problems, we call the coefficients of the linear combination Lagrange multipliers and denote them λi (the Greek letter lambda). The m Lagrange multipliers λi are complementary to the m slack variables si. The slack variable si is nonzero when the ith^ constraint is loose, and zero when the constraint is tight. Inversely, the Lagrange multiplier λi is zero when the ith^ constraint is loose, and nonzero when a constraint is tight. We can understand the dual problem as the search for the minimum value for the linear combination of the constraint equations for the rate region, which satisfies the constraint ~c ≤ ~λTA. The set of linear combinations ~λ that satisfy this constraint are called dual-feasible. Using the transitivity property of inequalities, we’ll now show the function h(~λ) = ~λ · ~b is an upper bound on the objective function g(~x) ≡ ~c · ~x for the primal problem. The following chain of inequalities is true for all primal-feasible ~x and dual-feasible ~λ:
g(~x) = ~cT~x ≤
~λTA
· ~x ≤ ~λ · ~b = h(~λ).
The first equality follows from definition of the objective function g(~x) = ~c · ~x. The first inequality follows from the assumption that ~λ is dual-feasible (~c ≤ ~λTA). The second inequality is true because ~x is assumed to be primal-feasible (A~x ≤ ~b). The last equality follows from the definition of h(~λ) = ~λ · ~b. The weak duality theorem states that if we have find a primal-feasible ~x∗^ and dual-feasible ~λ∗^ such that ~c · ~x∗^ = ~λ∗^ · ~b, then
g(~x∗) = h(~λ∗).
Since g(~x∗) is equal to its upper bound, the point ~x∗^ must be where g(~x) attains its maximum: ~x∗^ = argmax ~x
g(~x). Since g(~x) ≤ h(~λ) for all ~x and ~λ that are feasible, and
since g(~x∗) is the maximum value of the primal, it means ~λ∗^ = argmin ~λ
h(~λ).
The dual problem is the search for the minimum value of h(~λ). Studying the dual of a linear programming problem can help us characterize the solution to of the primal problem, even before we have found the actual solution. Switching from searching for the maximum of the function g(~x) to searching for the minimum of h(~λ) sometimes makes the problem easier to solve.
1.5 Practice problems 20
Complementary slackness conditions If a variable is positive, its corresponding (complementary) dual constraint holds with equality. If a dual constraint holds with strict inequality, then the corresponding (com- plementary) primal variable must be zero. The shadow price is the instantaneous change, per unit of the constraint, in the objective value of the optimal solution of an optimization problem obtained by relaxing the constraint. In other words, it is the marginal utility of relaxing the constraint, or, equivalently, the marginal cost of strengthening the constraint. More formally, the shadow price is the value of the Lagrange multiplier at the optimal solution, which means that it is the infinitesimal change in the objective function arising from an infinitesimal change in the constraint. This follows from the fact that at the optimal solution the gradient of the objective function is a linear combination of the constraint function gradients with the weights equal to the Lagrange multipliers.
Quadratic programming When the objective function is quadratic and constraints are linear functions the optimization problem is called quadratic programming.
1.5 Practice problems
P1 Use SymPy to reproduce the steps in the following example video http://youtu.be/XK26I9eoSl Confirm you obtain the same answer.
1 maxx,y,z P (x, y, z) = 708 at x = 48, y = 84, z = 0.
P2 Solve the following linear program:
max x,y g(x, y) = x + 3y,
subject to the constraints
2 x + 3y ≤ 24 , x − y ≤ 6 , y ≤ 6 , x ≥ 0 , y ≥ 0.
2 max g(x, y) = 21 and occurs at x = 3, y = 6.
2 Use the simplex method if solving the problem by hand. The code to solve using linprog is as follows:
from from scipy.optimize import linprog from numpy import array c = [-1,-3] D = array([[ 2, 3, 24], [ 1,-1, 6], [ 0, 1, 6], [-1, 0, 0], [ 0,-1, 0]] A = D[:,0:2] b = D[:,2] linprog(c, A_ub=A, b_ub=b)
