Download Master Theorem for Recurrence Relations and more Exams Design and Analysis of Algorithms in PDF only on Docsity!
Lecture Notes, CS Kenrick Mock
Chapter 4: Recurrence Relations : Iterative and The Master Method
Iteration Method: Expand the terms into a summation, and solve algebraically
Example:
T(n)= Theta(1) for n= T(n) = 3T(n/4) + n for n>
T
n T
n n 4
^
=^
^
We can plug this back into the original recurrence relation:
T n ( ) T
n n = n ^
^
We can keep on going:
T n^ (^ ) = T n^ n^ n^ n
^
^
If we stop at this point and do some math:
T(n) = 27T(n/64) + 9(n/16) + 3(n/4) + n
T n n
n n T
n ( ) = + + +
^
There’s a pattern here! If we consider i as the index, where i=1 gives us n+(3/4)n, then we can generalize this as i increases:
i
i j
j (^) n T
n n n T n n 4
How far does i go? Does it increase to infinity? NO at some point we will have to stop. But we already know when we stop – we stop at T(1) because at this point there is no more recursion, we just return a constant number for the amount of work to do.
If we stop at T(1), this means we will stop when 1=(n/4i).
i
n 4
1 = n = 4 i log 4 n = i
So we can now express the recurrence relation as:
...^3
( )^3 + log^4 Θ
= + + + + n
i T n n n n n
substituting Θ( 1 ) for T(n/4i) since we will only do a constant amount of work on the last
iteration.
We can summarize this as a single summation. First recall that
3 log^4 n^ = n log^4 3 ; this is sublinear since log 4 3 < 1
( ) log^13 log^3 0
4
4 T n n n
n
i
i +^ Θ
∑
−
=
( )^3 log^3 0
T n n n^4 i
i +^ Θ
∑
∞
=
; up to infinity bigger, so <= applies
recall that x x
k k
∞ ∑
; for x<
T n ( ) ≤ n ( n log ) −
Θ^4
T n ( ) ≤ 4 n + Θ( n log^4 3 ) ; T ( n ) ≤ 4 n + o ( n ) ; loose upper bound so use little-o
This means that the recurrence is O(n).
This method is accurate but can result in a lot of algebra to keep track of; can also get very challenging for more complicated recurrence relations.
Second Example: T(n)=1 if n= T(n)=4T(n/2)+n if n>
T(n) =4T(n/2) + n =4(4T(n/4)+n/2)+n =4(4(4T(n/8)+n/4)+n/2)+n =64T(n/8) + 4n +2n +n =n + 2n +4n + 64T(n/8)
1=(n/2i)^2 so n^2 =22i^ ; n=2i^ ; i=lgn
The amount of work done is then:
n i i
n
0 2
2 ^
lg = Θ( n^2 ) ; this is geometrically decreasing in size, so it won’t get any
bigger than n^2.
One more example: T(n) = T(n/3) + T(2n/3) + n
Each level does work of size n; if we just know the height of the tree, i, the total work is ni.
The tree stops when the leaf is of size 1. The hard part is to figure out the formula based on the height:
n
2^ i 3
= 1 (why pick the 2/3 branch and not 1/3?)
n (^) i
i
^
i = log (^) 3 2/ n
So the total work is (log (^) 3 2/ n ) n or O(nlog (^) 3/2 n).
Master Method :
n
(n/3) (2n/3)
(n/9) (2n/9) (2n/9) (4n/9)
n
n
n
If the form of a recurrence is: T n (^ )^ aT (^ )
n b
= f n a b
^
+^ ,^ ≥^1 ,^ >^1
then we can use the Master Method, which is a cookbook-style method for proving the runtime of recurrence relations that fit its parameters. Note that not all recurrence of the above form can be solved through the master method. We won’t prove the master method, but will give an argument as to how it works.
In the master method:
- a is the number of subproblems that are solved recursively; i.e. the number of recursive calls.
- b is the size of each subproblem relative to n; n/b is the size of the input to the recursive call.
- f(n) is the cost of dividing and recombining the subproblems.
Recursion tree example: T(n)=aT(n/b)+f(n)
What is the height of the tree? When f
n b
f
n b i i n^ b^ i i^ bn
^
=^ ( )^1 →^ =^1 →^ =^ →^ =log
How many leaves are there? a height^ = NumberLeaves
F(n)
F(n/b) F(n/b) F(n/b)
a
F(n/b^2 ) F(n/b^2 ) F(n/b^2 )
a
... (^) F(n/b (^2) ) F(n/b (^2) ) F(n/b (^2) )
Θ( 1 ) Θ(^1 )
log b n
F(n)
aF(n/b)
a^2 F(n/b^2 )
Θ( n log^ ba )
−
=
=Θ +log^1
0
log^ n j
j ba^ b j
b
n
Total n a f
f ( n ) = O n ( log b^ a − ε ) n = O n ( log^3 9 −^ ε^ ) = O n ( 2 − ε ) if ε = 1 then n=O(n) and case 1 is satisfied. Therefore: T n ( ) = Θ ( n log^ b^ a ) = Θ( n log^3 9 ) =Θ( n^2 )
In this example, the cost of the leaves has dominated the runtime.
Example:
T n T
n ( ) = 2 ( )+ n 2
; Merge Sort
So a=2, b=2, f(n)=n Check case 1: Is f ( n ) = O n ( log b^ a − ε^ )? n = O n ( log^2 2 − ε ) n = O n ( 1 −^ ε ) For any epsilon>0, n is bigger, so case 1 does not work.
Check case 2: Is f n ( ) = Θ( n log b^ a ) n = Θ( n log^2 2 ) =Θ( n ) YES therefore: T n ( ) = Θ ( n log^ b^ a lg n ) = Θ( n log^2 2 lg n ) =Θ( n lg n )
Cost is evenly distributed among leaves and upper part of tree.
Example:
T n T
n ( ) = ( )+
So a=1, b=3/2, f(n)=
Case 1 does not work (exercise for the reader) Case 2: Is f n ( ) = Θ( n log b^ a )? 1 = Θ ( n log 3 2^ /^1 ) = Θ ( n^0 ) =Θ( ) 1 YES therefore:
T n ( ) = Θ( n log b^^ a lg n ) = Θ( n log^ 3 2/^1 lg n ) = Θ( n^0 lg n ) =Θ(lg n )
Cost is again evenly distributed.
Example:
T n T
n ( ) = 3 ( ) + n lg n 4
a=3,b=4,f(n)=nlgn
Case 1 and 2 don’t fit (exercise for the reader) Case 3: Is f n ( ) = Ω( n log b^ a + ε^ )? n lg n = Ω( n log^4 3 +^ ε^ ) =Ω( n 0 79. + ε ) YES, if epsilon =0.21, then n lg n = Ω( n )
We also need to show the extra condition:
Is af
n b
( ) ≤ cf ( n ) for c<1?
f
n cf n
n n cn n
n n cn n
n n cn n
lg lg
(lg lg ) lg
(lg ) lg
^
YES, if c=¾ then 3 4
n (lg n − ) ≤ n lg n
therefore: T ( n ) = Θ ( f ( n )) =Θ( n lg n )
Example:
T n T
n n n
lg
2
How many comparisons are necessary to determine the selection? Say we want to find the minimum:
Lower bound of at least n-1 comparisons to see every other element Think as a tournament:
Pick contender Contender competes with another (comparison) Winner is the smallest element
Every element except the winner must lose one match. This is a simple example to show that we need at least n-1 comparisons, we will use this technique later in more complex examples to show a lower bound. Selecting the ith smallest element:
Can do in Θ( n lg n )time easily by sorting with Merge Sort,
and then pick A[i]. But can do better!
Consider if the set of n numbers is divided as follows:
S1: < p p S2: > p
Note that the elements in S1 are not sorted, but all of them are smaller than element p (partition). We know that p is the (|S1| +1)th smallest element of n. We will use this idea later to also sort numbers (known as quicksort).
Now consider the following algorithm to find the ith smallest element from Array A:
- Select a pivot point, p, out of array A.
- Split A into S1 and S2, where all elements in S1 are
p
- If i=|S1|+1 then p is the ith smallest element.
- Else if i<=|S1| then the ith smallest element is somewhere in S1. Repeat the process recursively on S1 looking for the ith smallest element.
- Else i is somewhere in S2. Repeat the process recursively looking for the i-|S1|- smallest element.
Question: How do we select p? Best if p is close to the median. If p is the largest element or the smallest, the problem size is only reduced by 1.
- Always pick the same element, n or 1 - Pick a random element - Pick 3 random elements, and pick the median - Other method we will see later
How do we partition once we have p?
If A contains: [5 12 8 6 2 1 4 3]
Can create two subarrays, S1 and S2. For each element x in A, if x
=p put it in S2.
p= S1: [2 1 4 3] S2: [5 12 8 6]
This certainly works, but requires additional space to hold the subarrays. We can also do the partitioning in-place, using no additional space:
Partition(A,p,r) ; Partitions array A[p..r] x← A[p] ; Choose first element as partition element i← p- j← r+ while true do repeat j← j- until A[j]≤ x repeat i← i+ until A[i] ≥ x if i
largest or smallest value, it will also be guaranteed to split off at least one value). This routine makes only one pass through the array A, so it takes time Θ(
n ). No extra space required
except to hold index variables.
Worst case running time of selection: Pick min or max as partition element, producing region of size n-1.
T ( n ) = T ( n − 1 ) +Θ( n ) subprob time to split
Evaluate recurrence by iterative method:
T ( 1 ) = Θ ( 1 ), T ( 2 ) = Θ( 1 ) + Θ( 2 ), T ( 3 ) = Θ( 1 ) + Θ( 2 ) +Θ( 3 ),... Θ
Θ
( ) i
i
i
n
i
n
=
=
∑
∑
1
1 = Θ( n^2 )
Recursion tree for worst case:
Best-case Partitioning:
In the best case, we pick the median each time.
T n T
n ( ) = ( ) + ( n ) 2
Using the master method: a=1, b=2, f(n)=Θ( n )
Case 3: Is f n ( ) = Ω( n log b^ a + ε^ )?
n
1 n-
1 n-
n
n
n-
n-
Total =Θ( n^2 )
n n^ log n n
2 1 0
ε ε YES if epsilon between 0 and 1, say 0.
Also is af
n b
( ) ≤ cf ( n ) for c<1?
n c n 2
YES if c > ½
So T ( n ) = Θ( f ( n )) =Θ( n )
New partition selection algorithm:
- Arrange the n elements into n/5 groups of 5 elements each, ignoring the at most four extra elements. (Constant time)
- Find the median of each group. This gives a list M of n/5 medians. (time Θ( n )if we use
the same median selection algorithm as this one or hard-code it)
- Find the median of M. Return this as the partition element. (Call partition selection recursively using M as the input set)
See picture of median of medians:
Guarantees that at least 30% of n will be larger than pivot point p, and can be eliminated each time!
Runtime: T n T
n T
n ( ) = ( ) + ( ) + O n ( ) 5
select recurse overhead of split/select pivot subprob
The O(n) time will dominate the computation by far resulting in O(n) run time.
x
Quicksort
We can also use the Partition selection algorithm to do sorting, this is called Quicksort.
QuickSort(A,p,r) ; Sort A[p..r] if p
