Download Analysis of a Recurrence Relation using the Iteration Method and Master Theorem and more Study notes Algorithms and Programming in PDF only on Docsity!
CMPS 201
Analysis of Algorithms
Spring 2007
Recurrence Relations
When analyzing the run time of recursive algorithms we are often led to consider functions T^ ( n ) which
are defined by recurrence relations of a certain form. A typical example would be
T n T n dn n
c n
T n
where c , d are fixed constants. The specific values of these constants are important for determining the
explicit solution to the recurrence. Often however we are only concerned with finding an asymptotic
(upper, lower, or tight) bound on the solution. We call such a bound an asymptotic solution to the
recurrence. In the above example the particular constants c and d have no effect on the asymptotic
solution. We may therefore write our recurrence as
T n T n n n
n
T n
In what follows we'll show that a tight asymptotic solution to the above recurrence is
T ( n ) ( n log n )
We will study the following methods for solving recurrences.
Substitution Method. This method consists of guessing an asymptotic (upper or lower) bound on
the solution, and trying to prove it by induction.
Recursion Tree Method – Iteration Method. These are two closely related methods for
expressing T^ ( n ) as a summation which can then be analyzed. A recursion tree is a graphical
depiction of the entire set of recursive invocations of the function T. The goal of drawing a
recursion tree is to obtain a guess which can then be verified by the more rigorous substitution
method. Iteration consists of repeatedly substituting the recurrence into itself to obtain an iterated
(i.e. summation) expression.
Master Method. This is a cookbook method for determining asymptotic solutions to recurrences
of a specific form.
The Substitution Method
We begin with the following example.
T n n n
n
T n
We will see later that T^ (^ n )^ ( n log( n )), but for the time being, suppose that we are able to guess
somehow that T^ (^ n ) O^ ( n log( n )). In order to prove this we must find positive numbers c and n 0^ such
that T^ (^ n ) cn^ log( n ) for all n^ n^0. If we knew appropriate values for these constants, we could prove
this inequality by induction. Our goal then is to determine c and 0
n
such that the induction proof will
work. In what follows it will be useful to take log to mean 3
log
. We begin by mimicking the induction
step (IId) to find c :
Let n^ ^ n 0 and assume T^ (^ k ) ck^ log( k ) for all k in the range n^ 0 k n. In particular, when
k n / (^3) we have T (^) n / (^3) c (^) n / (^3) log (^) n / (^3) . We must show that T ( n ) cn log( n ). Observe
T ( n ) 3 T n / 3 n
(by the recurrence for T )
3 c n / 3 log n / 3 n
(by the induction hypothesis)
3 c ( n / 3 )log( n / 3 ) n (since x^ x^ for all x )
cn (log( n ) 1 ) n
cn log( n ) cn n
To obtain T^ (^ n ) cn^ log( n )we need to have cn n 0 , which will be true if c 1. Thus as long as c
is chosen to satisfy the constraint c 1 , the induction step will go through. It remains to determine the
constant n^ 0. The base step requires T^ (^ n 0 ) cn^0 log( n 0 ). This can be viewed as a single constraint on
the two parameters c and n 0^ , which indicates that we have some freedom in choosing them. In other
words, the constraints do not uniquely determine both constants, so we must make some arbitrary choice.
Choosing n 0^ ^3 is algebraically convenient. This yields T^ (^3 )^3 c , whence c^ T^ (^3 )/^3. Since
T ( 3 ) 9 (using the recurrence) we have the constraint c 3. Clearly if we choose c 3 , both
constraints ( c 1 and c 3 ) will be satisfied.
It is important to note that we have not proved anything yet. Everything we have done up to this point has
been scratch work with the goal of finding appropriate values for c and 0
n
. It remains to present a
complete induction proof of the assertion: T^ (^ n )^3 n log( n ) for all n 3.
Proof:
I. Since T ( 3 ) 3 T 3 / 3 3 3 T ( 1 ) 3 3 2 3 9
and
3 3 log( 3 ) 9
the base case reduces to
9 9 which is true.
II. Let n 3 and assume for all k n that T^ (^ k )^3 k log( k ). Then
T ( n ) 3 T n / 3 n (by the recurrence for T )
3 3 n / 3 log n / 3 n (by the induction hypothesis letting k n / (^3) )
9 ( n / 3 )log( n / 3 ) n
(since
x x
for all x )
3 n (log( n ) 1 ) n
3 n log( n ) 2 n
3 n log( n )
It now follows that T^ (^ n )^3 n log( n ) for all n 3.
It is a somewhat more difficult exercise to prove by the same technique that T^ (^ n )^ ( n log( n )), and
hence
T ( n ) ( n log( n ))
The Recursion Tree – Iteration Method
The recursion tree method can be used to generate a good guess for an asymptotic bound on a recurrence.
This guess can then be verified by the substitution method. Since our guess will be verified, we can take
some liberties in our calculations, such as dropping floors and ceilings or restricting the values of n.
Let us begin with the example
T n n n
n
T n
Each node in a recursion tree represents one term in the calculation of T^ ( n ) obtained by recursively
substituting the expression for T into itself. We construct a sequence of such trees of increasing depths.
th
tree 1
st
tree 2
nd
tree
T ( n ) n n
T ( n / 3 ) T ( n / 3 ) ( n / 3 ) ( n / 3 )
2 T n ( / 3 )
2 T n ( / 3 )
2 T n ( / 3 )
2 T n
rd
tree
n
( n / 3 ) ( n / 3 )
2 n ( / 3 )
2 n ( / 3 )
2 n ( / 3 )
2 n
3 T n ( / 3 )
3 T n ( / 3 )
3 T n ( / 3 )
3 T n ( / 3 )
3 T n ( / 3 )
3 T n ( / 3 )
3 T n ( / 3 )
3 T n
By summing the nodes in any one of these trees, we obtain an expression for
T ( n )
. After k iterations of
this process we reach a tree in which all bottom level nodes are (^ /^3 )
k
T n.
k
th
tree
n
( n / 3 ) ( n / 3 )
2 n ( / 3 )
2 n ( / 3 )
2 n ( / 3 )
2 n
k T n ( / 3 )
k T n ( / 3 )
k
T n ……………………………………..……… ( / 3 )
k T n ( / 3 )
k T n
Note that there are
i
2 nodes at depth^ i , each of which has value^
i
n / 3 (for^0 ^ i^ k ^1 ). The sequence of
trees terminates when all bottom level nodes are
T ( 1 )
, i.e. when / 3 1
k
n , which implies
log ( ) 3
k n
The number of nodes at this bottom level is therefore
log () log( 2 ) 3 3 2 2 n
k n .^ Summing all nodes in the
final recursion tree gives us the following expression for
T ( n )
log( 2 )
1
0
3
T n n n T
k
i
i i
( 2 / 3 ) ( 1 )
log( 2 )
1
0
3 n n T
k
i
i
log 3 ( 2 ) n n T
k
log( 2 ) 3 n n
k
If we seek an asymptotic upper bound we may drop the negative term to obtain (^ )^3 ( )
log 3 ( 2 )
T n n n.
Since
log ( 2 ) 1 3
the first term dominates, and so we guess:
T ( n ) O ( n )
Exercise Prove that this guess is correct using the substitution method.
It is sometimes possible to push these computations a little further to obtain an asymptotic estimate
directly, with no simplifying assumptions. We call this the iteration method. We illustrate on the very
same example.
T n n n
n
T n
Here we do not assume that n is an exact power of 3, and keep the floor. Substituting this expression into
itself k times yields:
T ( n ) n 2 T n / 3
n 2 n / 3 2 T ^ n / 3 / 3
2 2
n 2 n / 3 2 T n / 3
2 2 2 n n n T n
^ ^ ^
2 2 3 3
n 2 n / 3 2 n / 3 2 T n / 3
log( 2 ) 3
n 1 n
( n ) (since log ( 2 ) 1
3
We've shown T^ (^ n )^ ( n ), and since T^ (^ n ) O^ ( n ), we now have T^ (^ n )^ ( n ).
The iteration method can even be used to find exact solutions to some recurrences, as opposed to the
asymptotic solutions we have been satisfied with up to now. For example define the function T^ ( n ) by
T ( 0 ) T ( 1 ) 5
, and
T ( n ) T ( n 2 ) n
for n^ ^2. Iterating k times we find
T ( n ) n T ( n 2 )
n ( n 2 ) T ( n 4 )
n ( n 2 )( n 4 ) T ( n 6 )
1
0
n i T n k
k
i
1
0
1
0
n i T n k
k
i
k
i
2 T n k
k k
kn (^)
kn k ( k 1 ) T ( n 2 k )
This process must terminate when k is chosen so that either n 2 k 0 or n 2 k 1 , which is where the
recurrence 'bottoms out' so to speak. This implies
0 n 2 k 2
2 k n 2 k 2
k
n k
k n / 2
Thus if k^ ^ n /^2 we have T^ (^ n ^2 k )^5 , and therefore the exact solution to the above recurrence is
given by
T ( n ) n / 2 n n / 2 n / 2 1 5
The asymptotic solution is now readily seen to be ( ) ( )
2
T n n.
Exercise Define T^ ( n ) by T^ (^1 )^0 and T^ (^ n ) T^ ^ n /^2 ^ ^1 for n 2. (Recall this was example 3 in
the induction handout.) Use the iteration method to show T^ (^ n )^ lg( n ) for all n , whence
T ( n ) (log( n ))
The Master Method
This is a method for finding (asymptotic) solutions to recurrences of the form
T ( n ) aT ( n / b ) f ( n )
where a 1 , b 1 , and the function f^ ( n ) is asymptotically positive. Here T^ (^ n / b ) denotes either
T ( n / b )
or
T ( n / b )
, and it is understood that
T ( n )( 1 )
for some finite set of initial terms. Such
a recurrence describes the run time of a 'divide and conquer' algorithm which divides a problem of size n
into a subproblems, each of size n / b. In this context f^ ( n ) represents the cost of doing the dividing and
re-combining.
Master Theorem
Let a^ ^1 , b^ ^1 ,
f ( n )
be asymptotically positive, and let
T ( n )
be defined by
T ( n ) aT ( n / b ) f ( n )
Then we have three cases:
(1) If
log() ( )
a b f n O n for some ^ ^0 , then
log() ( )
a b T n n.
(2) If
log() ( )
a b f n n , then ( ) log( )
log () T n n n
a b .
(3) If
log() ( )
a b f n n for some ^ ^0 , and if af ( n / b ) cf ( n )
for some 0 ^ c^ ^1 and for all
sufficiently large n , then T^ (^ n )^ ^ f ( n ).
Remarks In each case we compare f^ ( n ) to the polynomial
log ( a ) b n , and the solution is determined by
which function is of an asymptotically higher order. In case (1)
log ( a ) b n is polynomially larger than
f ( n ) and the solution is in the class
log ( a ) b n. In case (3) f ( n )
is polynomially larger (and an
additional regularity condition is met) so the solution is ^ f ( n ). In case (2) the functions are
asymptotically equivalent and the solution is in the class log( )
log () n n
a b , which is the same as
f ( n )log( n ). To say that
log ( a ) b n is^ polynomially larger^ than^
f ( n )
as in (1), means that
f ( n )
is
bounded above by a function which is smaller than
log (^) b ( a ) n
by a polynomial factor, namely
n for some
0. Note that the conclusion reached by the master theorem does not change if we replace f^ ( n ) by a
function asymptotically equivalent to it. For this reason the recurrence may simply be given as
T ( n ) aT ( n / b ) f ( n )
. Notice also that there is no mention of initial terms. It is part of the content
of the master theorem that the initial values of the recurrence do not effect it’s asymptotic solution.
Examples
Let
3
T ( n ) 8 T ( n / 2 ) n so that a 8 , b 2 , and log^ ( a )^3
b.^ Hence^ ^
3 log() ( )
a b f n n n , so we
are in case (2). Therefore (^ ) ( log( ))
3
T n n n.
Now let T^ ( n^ )^5 T ( n /^4 ) n. Here a 5 , b 4 , log^ b ( a )^1.^609 ...^1. Let log (^5 )^1
4
so that
0 , and
log( 5 ) 4 f ( n ) n On. Therefore we are in case (1) and
log( 5 ) 4 T ( n ) n.
Next consider
2
T ( n ) 5 T ( n / 4 ) n. Again a 5 , b 4 , so log ( a ) 1. 609 ... 2
b.^ Let
2 log ( 5 ) 4
, so that (^0) , and
2 log( 5 ) 4 f ( n ) n n. We appear to be in case (3), but we must
still check the regularity condition:
5 f ( n / 4 ) cf ( n )
for some 0 c 1 and all sufficiently large n. This
inequality says
2 2
5 ( n / 4 ) cn , i.e.
2 2
( 5 / 16 ) n cn , which is true as long as c is chosen to satisfy
5 / 16 c 1. By case (3) of the Master Theorem (^ ) ( )
2
T n n.
