Download The Simplex Algorithm - Lecture Notes | ISYE 6669 and more Study notes Systems Engineering in PDF only on Docsity!
1.3 The Simplex Algorithm
The linear programming examples discussed in the introduction describe
situations where one is required to minimize or maximize a linear function of several
variables, subject to linear equality or inequality constraints on the variables. Many of the
methods that have been developed for solving such problems are based on geometric
concepts. So we begin our discussion of solution techniques with a geometric description
of linear programming problems in two and three variables.
Consider the problem
maximize 3 x 1^ ^2 x 2 (3.1)
subject to x 1 ^ x 2 ^6
1 2
1 2
1 2
x x
x x
x x
The region of the x 1^ x 2 plane described by these constraint inequalities is shown shaded
in the figure 3.1. This shaded region is called the feasible region for problem (3.1). It is
the set of values of the variables that satisfy the constraints. The feasible region for a
linear programming problem is always a polyhedron. In Fig. 3.1 we have also plotted the
equation of the objective function
1 2
z 3 x 2 x
for several values of z. As z increases the line described by this equation moves across
Figure 3.
the feasible region in a direction normal to the line. From this it is clear that the
maximum value z can attain on the feasible region occurs at a point where two of the
lines describing the feasible region intersect. We call such a point a vertex of the
0 1 2 3 4 5 6 7
0
1
2
3
4
5
6
x x
x x
x x
x x
x x
x x
x
x
constraint polyhedron. In general, at least one solution of a linear programming problem,
if solutions exists, occurs at a vertex of the constraint polyhedron. If this polyhedron is
unbounded a solution may not exist.
The problem (3.1) can be stated in matrix notation as
Maximize cx
Subject to Ax ^ b ,^ x ^0.
For the purpose of describing the simplex algorithm for solving such problems it is
convenient to convert the inequality constraints Ax^ b^ to equality constraints. We do
this by adding slack variables to the model, which, in the case of problem (3.1), converts
it to
maximize 3 x 1^ ^2 x 2 (3.2)
subject to x 1 ^ x 2 x 3 ^6
1 2 3 4 5
1 2 5
1 2 4
x x x x x
x x x
x x x
When (3.1) is written in this form we see that each of the lines in Fig. 3.1 is represented
by one of the equations
1 2 3 4 5
x x x x x
With this in mind, we rephrase our statement that a solution occurs at a vertex of the
feasible region, to say “a solution occurs at a feasible point where two of the variables in
(3.2) are zero”. At each feasible point in Fig. 3.1 where two of the variables in (3.2) are
zero, the remaining three variables are positive. So we can also characterize a solution of
(3.2) as one of the feasible points where three of the variables in (3.2) are positive and the
remaining ones are zero. In general, for a linear programming problem stated in the form
minimize cx^ (3.3)
subject to Ax ^ b ,^ x ^0 ,
where A is an m^ n^ matrix, an optimal solution occurs at a point where n-m of the
variables are zero. Thus in a problem like (3.3) involving m equality constraints and n
variables, we can confine our search for an optimal solution to feasible points where n-m
of the variables have the value zero. This fact will aid us in our search for an optimal
solution. But as we will see later, this search will be complicated by the fact that the
remaining m variables need not all be positive. For example if we were to add the
constraint
1 2
x x (3.4)
to the model (3.1) and change the objective slightly to
maximize 4 x 1^ ^2 x 2 , (3.5)
variables to the model we obtain a case of (3.3) with three equality constraints. However,
the optimal solution of this problem has x^ 3 ^1 and all other variables equal to zero.
Moreover, each of the constraints are essential in defining the feasible region. In
particular, if the constraint x 1^ ^ x 2 x 3 ^1 is dropped the optimal solution changes to
1 2 3
x x x
The above examples demonstrate that for the linear programming problem (3.3),
When an optimal solution exists, there is always one having at most m positive variables.
We will now give a more precise description of optimal solutions of (3.3).
A point x satisfying the constraints in (3.3) is said to be a feasible point for (3.3).
Let B be a m^ m^ nonsingular submatrix of A. Such a matrix is said to be a basis matrix,
or simply a basis for (3.3). For any basis matrix B we will denote the components of x
corresponding to columns in B by x^ B .The columns in A that are not in B will be
denoted by AN^ ,and the corresponding components of x will be denoted by xN.
Similarly the components of c corresponding to x^ B will be denoted by c^ B and those
corresponding to x^ N will be denoted by c^ N .The components of x^ B are called basic
variables , and those of x^ N are called nonbasic variables. In general upper case letters
will refer to sets and lower case numbers to elements. Thus A^ j will refer to the
th j
column of A. If Bx^ B ^ b ,^ and xN ^0 , x is said to be a basic solution of (3.3). If further,
B
x , x is said to be a basic feasible solution of (3.3). If 0
B
x x is said to be a
nondegenerate basic feasible solution of (3.3). Finally, if x^ B ^0 but has at least one zero
component, x is said to be a degenerate basic feasible solution. We have just seen two
examples of linear programming problems whose optimal solutions are degenerate basic
feasible solutions. It turns out that among the solutions of a linear programming problem
there is always one which is a basic feasible solution. This is the essence of the following
theorem.
Theorem3.1. If problem (3.3) has a feasible solution it has a basic feasible
solution. Moreover, if (3.3) has an optimal solution it has a basic feasible solution which
is optimal.
The importance of this result is that it allows us to restrict our search for an
optimal solution of a linear programming problem to the set of basic feasible solutions.
Clearly (3.3) has only a finite number of basic feasible solutions since there are at most
m
n
ways of selecting a basis matrix from the columns of A. Thus in principle we
could find an optimal solution of (3.3) by examining all of its basic feasible solutions.
But this would be inefficient. Especially if we had no way of checking whether or not a
given basic feasible solution is optimal. In this naïve approach we could start with an
optimal solution and not know it until we had compared it with all others. The simplex
algorithm is a much more efficient way of finding an optimal solution. One of its most
important features is a method of testing a given basic feasible solution for optimality.
Before we describe the simplex we explain this optimality test.
Let x be any feasible solution for (3.3) and let B be any basis matrix. We will
denote the vector by.
1 B b b
The equation Ax^ b^ can be written as
Bx (^) B ANxN b.
Solving for x^ B gives
1 1 1
j N
B N N j j
x B b B A x b B Ax
Substituting this in cx gives
1 1
B B N N B N B N N
z cx c x c x c B b c c B A x
This expresses z in terms of the nonbasic variables. Let
1
c B
B
. We can then write
this equation as
j N
z b cj Aj xj
The following theorem gives a simple optimality test for a basic feasible solution.
Theorem 3.2. Let
x denote a basic feasible solution for (3.3) corresponding to
the basis matrix B, and define.
1
c B
B
If
j j
A c
for each nonbasic variable j
x
x is optimal for (3.3).
Proof. By definition ,^0.
B N
x B b and x Therefore
cx c x c x c B b b
B B N N B
Equation (3.7) shows that
cx b cx for any feasible solution x. This proves the
optimality of.
x
This theorem gives a sufficient condition for a basic feasible solution of (3.3) to
be optimal. It is by no means a necessary condition. If
x is a degenerate basic feasible
solution it may be associated with several basis matrices. Some of them may not satisfy
the optimality test (3.8).
Example. Consider the following problem, which is the modification of (3.1)
discussed above.
minimize ^4 x 1^ ^2 x 2 (3.9)
subject to x 1 ^ x 2 x 3 ^6
1 2 3 4 5 6
1 2 6
1 2 5
1 2 4
x x x x x x
x x x
x x x
x x x
The reader should observe that we have added the constraint 3 x 1^ ^2 x 2 ^15 to the
problem (3.1). This does not decrease the feasible region shown in Fig. 3.1, but
introduces a degenerate vertex at 3 3 . 1 2
x x
Compute d^ B As
1
and observe that for any value of
B ( b d ) A b. s
Let x^ denote the vector with xs^ ^ and whose components corresponding to the
columns in B are given by b ^ d , and whose remaining components are zero. (3.10) can
then be written as Ax^ b^. Moreover
cx c B ( b d ) cs cBb ( cs As ) . (3.11)
Thus if ^ ^0 , and^ b ^ d ^0 , x is feasible for (3.3) and cx ^ ^ b .If all components
of the vector d are ^0 then b ^ d^ ^0 for all values of ^ ^0 .In this case it follows
from (3.11) that the objective cx^ can be made arbitrarily small by taking^ ^ sufficiently
large. In this case we say problem (3.3) is unbounded.
If d has at least one positive component we choose as large as possible
subject to the condition b ^ d^ ^0. That is, we take
min.
i
i
d
b
Assume this minimum is attained at i^ r^ .Then equation (3.10) expresses b^ as a
nonnegative combination of the column As^ and the columns of B , excluding the one in
position r. Let B ˆ^ denote the matrix obtained from B by replacing the column in
position r with As^ .Assume for the moment that (^) B ˆ^ is nonsingular. Equation (3.10)
shows that the equation Bx^ b
B
has a nonnegative solution^ x ˆ^^ B ˆ which satisfies
ˆ ˆ ( ) ( )^ ,
B B B s s B B
c x c b d c A c x
with strict inequality holding if ^ ^0 .Thus B ˆ^ is a basis matrix whose corresponding
basic solution is feasible and has an objective value which is at least as good as the one
corresponding to B^ .The process by which we went from B^ toB
ˆ
constitutes one
iteration of the simplex algorithm. To complete the description of this iteration we must
show that B ˆ^ is nonsingular.
Let E denote the matrix obtained from the identity matrix by replacing column
r with d. Then clearly.
ˆ
B BE The element in the diagonal position (^ r^ , r )of E is
r
d .This means that E is nonsingular. In fact, it is easy to see that if
0 i
d
2
1
m
r
d
d
d
d
E and 0
r
d , then
2
1
1
m r
r
r
r
d d
d
d d
d d
E . (3.12)
It follows that ˆ^.
1 1 1
B E B Thus, in going from one iteration of the simplex
algorithm there is a simple procedure for updating the inverse of the basis matrix. We
will elaborate on this when we discuss implementation issues in Section 1..
The following table gives a formal description of the simplex algorithm.
The Primal Simplex Algorithm
Step 1. Let B be a feasible basis. Compute.
1 1
b B b and c B
B
Step 2. Let
c c A , j 1 , , n. j j j
If possible find an index s such that
s
c
If no such s exists stop. The current basis is optimal.
Step 3. Compute d^ B As
1
. If d^ ^0 ,stop. The problem is unbounded. If
d has some positive components, perform the following minimum ratio
test
i
i
d
b
min
Step 4. Let r^ denote a value of i^ where the minimum in Step 3 is attained.
Replace column r^ in B with column As^ and return to Step 1.
di 0
denote the new dual variable. We must show that ˆ A^ r^ cr. Actually, we will prove the
stronger result ˆ A^ r^ cr .Let d B As
1
denote the update of the entering column. Then
s m m
A d A d A d A
This equation implies that
1
1
s rr B s s s r r
i r
m
i
r r s i i s rr ii
c dc c B A c A d c
d A A d A c dc d c
Since cs ^ A^ s ^0 we have d^ r^ ˆ A^ r drcr .And since d^ r ^0 this implies that ˆ Ar^ cr
as claimed.
The Simplex Tableau
The version of the simplex algorithm we have described is known as the revised
simplex algorithm. Given a basis B it requires one to solve the equations
B B
Bx b , B c
and to find the smallest of the numbers c^ j ^ A^ j ,^ j ^1 ,, n .This determines the
column that wants to enter the basis. Given the entering column, call it As^ ,we must
compute d^ B As
1
and perform a ratio test to determine the column that leaves the
basis. For calculations on small problems it is sometimes more convenient to carry out
these calculations on a matrix containing the LP data A ,^^ b ,and, c .This matrix is
called a simplex tableau. The first tableau is given by
A b
c
z x RHS
and is interpreted to read
z cx and Ax b. (3.13)
Given a feasible basis B , the first step of the simplex algorithm essentially asks us to
write the basic variables in terms of the nonbasic ones. z^ is always treated as a basic
variable. From the equations
j N
B B j j j N
B j j
Bx Ax b , z c x c x
we have
j N
j j j j N
B j j
x B b B Ax ,and z b ( c A ) x
1 1
where.
1
c B
B
This says that when we solve the m 1 equations (3.13) for the
m 1
variables z ,^ xB we compute all the quantities required by one step of the simplex
algorithm. It is convenient to solve these equations directly as they appear in the simplex
tableau. For example, consider problem (3.2). Its first tableau is
1 2 3 4 5
1
z x x x x x RHS
T
Suppose we start the simplex algorithm with x 1^ ^4 , x 3 ^2 , x 4 ^6 in the basis.
The basic variables z and x 4^ each appears in only one equation. We can eliminate x 1
from all but the second equation by successively multiplying row 2 by 3, 1, -3 and
adding the result to rows 1, 3, and 4 respectively. This operation is known as pivoting on
the circled 1 in tableau 1. This gives the tableau
We can eliminate x 3^ from all but the last equation in this tableau by pivoting on the
circles element. This gives the tableau
2
T
This tableau expresses the basic variables in terms of the nonbasic ones. In
particular it shows that
z 12 x 2 x 5.
Thus in order to increase z we must increase x 2^ .If we increase
2
x while holding x 5 at
0 the equations in T 2^ state that
based on empirical evidence, that when this rule is used for problems having m 50
constraints, the simplex algorithm usually terminates in fewer than 3 m / 2 iterations. The
example we are about to give has m^ ^5 constraints, and the simplex algorithm using
Dantzig’s rule requires 2 m iterations for its solution. In the next Section we will discuss
a procedure for selecting the entering variable that, on average, performs better than the
Dantzig rule. The worst case behavior of the simplex algorithm will be discussed in
Section?
Consider the problem min^ cx ,^ Ax ^ b , x ^0 where
c [ 4 , 4 , 4 , 5 , 22 , 0 , 1 , 0 ,. 5 , 0 , 7 , 2 ],
A and b
Suppose we start the simplex algorithm with the basis
[ , , , , ]
1 5 6 7 8 9
B A A A A A
It is easy to check that the corresponding basic variables are
[ 3 / 2 , 13 , 3 , 4 , 5 ].
1 1
T b B b
Thus B 1^ is a feasible basis. We have
[ 22 , 0 , 1 , 0 , 1 / 2 ]
1
c B and the vector [ 0 , 1 , 0 , 1 / 2 , 11 ]
1 satisfies
1 1 B 1
B c
The objective value corresponding to B 1^ is
1 1 1
z c b b B
To find an improved basis we compute
min { c (^) j 1 Aj | j N } c 4 1 A 4 18.
This shows that x^ 4 wants to enter the basis. The vector 4
1 1
d B A
is given by
[ 1 / 2 , 12 , 0 , 0 , 4 ].
T
d The ratio test gives / 13 / 12.
2 2
b d Thus A 6^ leaves
the basis. The new basis is
[ , , , , ]
2 5 4 7 8 9
B A A A A A
and the corresponding values of the basic variables are
8 4 4 9 5 5
5 1 1 4 7 3 3
x b d x b d
x b d x x b d
We combine these values in the vector b^ [^23 /^24 ,^13 /^12 ,^3 ,^4 ,^2 /^3 ]. The cost
coefficients associated with these variables are
[ 22 , 5 , 1 , 0 , 1 / 2 ]
2
B
c
and the
new objective value is
2 1
z z
The solution of the equation
2 B 2
yB c is y [ 3 / 2 1 0 1 / 2 7 ].A brief
calculation gives
c c y * A [ 3. 5 6 4. 5 0 0 1. 5 0 0 0 7 2. 5 ].
Following Dantzig’s rule we allow x^ 2 to enter the basis. Let
[ 0 , 0 , 2 , 1 , 0 ].
2
1
2
T
d B A
The ratio test in Step 3 gives ^ ^ b 3 /^ d 3 ^3 /^2 .Thus x 7^ leaves the basis, and the new
basis matrix is
[ , , , , ]
3 5 4 2 8 9
B A A A A A
The corresponding basic variables are
8 4 4 9 5 5
5 1 1 4 2 2 2
x b d x b d
x b d x b d x
The corresponding cost coefficients are
c yA [ 011 / 6 0 4 0 11 / 6 19 / 6 3 / 2 0 11 0 7 / 6 ]
so that
B is indeed an optimal basis.
