Download Recurrences-handout notes and more Lecture notes Design and Analysis of Algorithms in PDF only on Docsity!
Recurrences
Computer Science & Engineering 235: Discrete Mathematics
Christopher M. Bourke [email protected]
Recursive Algorithms
A recursive algorithm is one in which objects are defined in terms of other objects of the same type.
Advantages:
I (^) Simplicity of code I (^) Easy to understand
Disadvantages:
I (^) Memory I (^) Speed I (^) Possibly redundant work
Tail recursion offers a solution to the memory problem, but really, do we need recursion?
Recursive Algorithms
Analysis
We’ve already seen how to analyze the running time of algorithms. However, to analyze recursive algorithms, we require more sophisticated techniques.
Specifically, we study how to define & solve recurrence relations.
Motivating Example
Factorial Recall the factorial function.
n! =
1 if n = 1 n · (n − 1)! if n > 1
Consider the following (recursive) algorithm for computing n!:
Algorithm (Factorial)
Input : n ∈ N Output : n! 1 if n = 1 then 2 return 1 3 end 4 else 5 return Factorial(n − 1) × n 6 end
Motivating Example
Factorial - Analysis?
How many multiplications M (n) does Factorial perform?
I (^) When n = 1 we don’t perform any. I (^) Otherwise we perform 1. I (^) Plus how ever many multiplications we perform in the recursive call, Factorial(n − 1). I (^) This can be expressed as a formula (similar to the definition of n!. M (0) = 0 M (n) = 1 + M (n − 1) I This is known as a recurrence relation.
Recurrence Relations I
Definition
Definition
A recurrence relation for a sequence {an} is an equation that expresses an in terms of one or more of the previous terms in the sequence, a 0 , a 1 ,... , an− 1 for all integers n ≥ n 0 where n 0 is a nonnegative integer. A sequence is called a solution of a recurrence relation if its terms satisfy the recurrence relation.
Recurrence Relations II
Definition
Example
The Fibonacci numbers are defined by the recurrence,
F (n) = F (n − 1) + F (n − 2) F (1) = 1 F (0) = 1
The solution to the Fibonacci recurrence is
fn =
)n −
)n
(your book derives this solution).
Recurrence Relations III
Definition
More generally, recurrences can have the form
T (n) = αT (n − β) + f (n), T (δ) = c
or T (n) = αT
n β
Note that it may be necessary to define several T (δ), initial conditions.
Recurrence Relations IV
Definition
The initial conditions specify the value of the first few necessary terms in the sequence. In the Fibonacci numbers we needed two initial conditions, F (0) = F (1) = 1 since F (n) was defined by the two previous terms in the sequence.
Initial conditions are also known as boundary conditions (as opposed to the general conditions). From now on, we will use the subscript notation, so the Fibonacci numbers are fn = fn− 1 + fn− 2 f 1 = 1 f 0 = 1
Recurrence Relations V
Definition
Recurrence relations have two parts: recursive terms and non-recursive terms.
T (n) = 2T (n − 2) ︸ ︷︷ ︸ recursive
- n︸ 2 ︷︷− (^10) ︸ non-recrusive
Recursive terms come from when an algorithm calls itself.
Non-recursive terms correspond to the “non-recursive” cost of the algorithm—work the algorithm performs within a function. We’ll see some examples later. First, we need to know how to solve recurrences.
Solving Recurrences
There are several methods for solving recurrences.
I (^) Characteristic Equations I (^) Forward Substitution I (^) Backward Substitution I (^) Recurrence Trees I (^) Maple!
Linear Homogeneous Recurrences
Definition
A linear homogeneous recurrence relation of degree k with constant coefficients is a recurrence relation of the form
an = c 1 an− 1 + c 2 an− 2 + · · · + ckan−k
with c 1 ,... , ck ∈ R, ck 6 = 0.
Second Order Linear Homogeneous Recurrences
Example Continued
I (^) Solving for α 1 = (1 − α 2 ) in (1), we can plug it into the second. 4 = 2 α 1 + 3α 2 4 = 2(1 − α 2 ) + 3α 2 4 = 2 − 2 α 2 + 3α 2 2 = α 2 I (^) Substituting back into (1), we get
α 1 = − 1
I (^) Putting it all back together, we have
an = α 1 (2n) + α 2 (3n) = − 1 · 2 n^ + 2 · 3 n
Second Order Linear Homogeneous Recurrences
Another Example
Example
Solve the recurrence
an = − 2 an− 1 + 15an− 2
with initial conditions a 0 = 0, a 1 = 1.
If we did it right, we have
an =
(3)n^ −
(−5)n
How can we check ourselves?
Single Root Case
Recall that we can only apply the first theorem if the roots are distinct, i.e. r 1 6 = r 2. If the roots are not distinct (r 1 = r 2 ), we say that one characteristic root has multiplicity two. In this case we have to apply a different theorem.
Theorem (Theorem 2, p416)
Let c 1 , c 2 ∈ R with c 2 6 = 0. Suppose that r^2 − c 1 r − c 2 = 0 has only one distinct root, r 0. Then {an} is a solution to an = c 1 an− 1 + c 2 an− 2 if and only if
an = α 1 rn 0 + α 2 nr 0 n
for n = 0, 1 , 2 ,... where α 1 , α 2 are constants depending upon the initial conditions.
Single Root Case
Example
Example
What is the solution to the recurrence relation
an = 8an− 1 − 16 an− 2
with initial conditions a 0 = 1, a 1 = 7?
I (^) The characteristic polynomial is
r^2 − 8 r + 16
I (^) Factoring gives us
r^2 − 8 r + 16 = (r − 4)(r − 4)
so r 0 = 4
Single Root Case
Example
I (^) By Theorem 2, we have that the solution is of the form
an = α 14 n^ + α 2 n 4 n
I (^) Using the initial conditions, we get a system of equations;
a 0 = 1 = α 1 a 1 = 7 = 4 α 1 + 4α 2
I (^) Solving the second, we get that α 2 = (^34) I And so the solution is
an = 4n^ +
n 4 n
I (^) We should check ourselves...
General Linear Homogeneous Recurrences
There is a straightforward generalization of these cases to higher order linear homogeneous recurrences. Essentially, we simply define higher degree polynomials.
The roots of these polynomials lead to a general solution. The general solution contains coefficients that depend only on the initial conditions.
In the general case, however, the coefficients form a system of linear inequalities.
General Linear Homogeneous Recurrences I
Distinct Roots
Theorem (Theorem 3, p417)
Let c 1 ,... , ck ∈ R. Suppose that the characteristic equation
rk^ − c 1 rk−^1 − · · · − ck− 1 r − ck = 0
has k distinct roots, r 1 ,... , rk. Then a sequence {an} is a solution of the recurrence relation
an = c 1 an− 1 + c 2 an− 2 + · · · + ckan−k
if and only if
an = α 1 rn 1 + α 2 rn 2 + · · · + αkrnk
for n = 0, 1 , 2 ,.. ., where α 1 , α 2 ,... , αk are constants.
General Linear Homogeneous Recurrences
Any Multiplicity
Theorem (Theorem 4, p418)
Let c 1 ,... , ck ∈ R. Suppose that the characteristic equation
rk^ − c 1 rk−^1 − · · · − ck− 1 r − ck = 0
has t distinct roots, r 1 ,... , rt with multiplicities m 1 ,... , mt.
General Linear Homogeneous Recurrences
Any Multiplicity
Theorem (Continued)
Then a sequence {an} is a solution of the recurrence relation
an = c 1 an− 1 + c 2 an− 2 + · · · + ckan−k
if and only if
an = (α 1 , 0 + α 1 , 1 n + · · · + α 1 ,m 1 − 1 nm^1 −^1 )rn 1 + (α 2 , 0 + α 2 , 1 n + · · · + α 2 ,m 2 − 1 nm^2 −^1 )rn 2 + .. . (αt, 0 + αt, 1 n + · · · + αt,mt− 1 nmt−^1 )rnt +
for n = 0, 1 , 2 ,.. ., where αi,j are constants for 1 ≤ i ≤ t and 0 ≤ j ≤ mi − 1.
Linear Nonhomogeneous Recurrences
For recursive algorithms, cost functions are often not homogenous because there is usually a non-recursive cost depending on the input size. Such a recurrence relation is called a linear nonhomogeneous recurrence relation. Such functions are of the form
an = c 1 an− 1 + c 2 an− 2 + · · · + ckan−k + f (n)
Linear Nonhomogeneous Recurrences
Here, f (n) represents a non-recursive cost. If we chop it off, we are left with
an = c 1 an− 1 + c 2 an− 2 + · · · + ckan−k
which is the associated homogenous recurrence relation.
Every solution of a linear nonhomogeneous recurrence relation is the sum of a particular solution and a solution to the associated linear homogeneous recurrence relation.
Linear Nonhomogeneous Recurrences
Theorem (Theorem 5, p420)
If {a (p) n }^ is a particular solution of the nonhomogeneous linear recurrence relation with constant coefficients
an = c 1 an− 1 + c 2 an− 2 + · · · + ckan−k + f (n)
then every solution is of the form {a (p) n +^ a
(h) n }, where^ {a
(h) n }^ is a solution of the associated homogenous recurrence relation
an = c 1 an− 1 + c 2 an− 2 + · · · + ckan−k
Linear Nonhomogeneous Recurrences II
Application
thus c = − 1 ; altogether:
an = a( nh )+ a( np )= α 1 (−φ + 1)n^ + α 2 (φ)n^ − 1
For A(0) = A(1) = 0 we get the regular solution to the fibonacci recurrence minus 1: an = fn − 1
which is exponential in n.
Other Methods
When analyzing algorithms, linear homogenous recurrences of order greater than 2 hardly ever arise in practice.
We briefly describe two “unfolding” methods that work for a lot of cases. Backward substitution – this works exactly as its name implies: starting from the equation itself, work backwards, substituting values of the function for previous ones.
Recurrence Trees – just as powerful but perhaps more intuitive, this method involves mapping out the recurrence tree for an equation. Starting from the equation, you unfold each recursive call to the function and calculate the non-recursive cost at each level of the tree. You then find a general formula for each level and take a summation over all such levels.
Backward Substitution
Example
Example
Give a solution to
T (n) = T (n − 1) + 2n
where T (1) = 5.
We begin by unfolding the recursion by a simple substitution of the function values. Observe that T (n − 1) = T ((n − 1) − 1) + 2(n − 1) = T (n − 2) + 2(n − 1)
Substituting this into the original equation gives us T (n) = T (n − 2) + 2(n − 1) + 2n
Backward Substitution
Example – Continued
If we continue to do this, we get the following.
T (n) = T (n − 2) + 2(n − 1) + 2n = T (n − 3) + 2(n − 2) + 2(n − 1) + 2n = T (n − 4) + 2(n − 3) + 2(n − 2) + 2(n − 1) + 2n .. . = T (n − i) +
∑i− 1 j=0 2(n^ −^ j)
I.e. this is the function’s value at the i-th iteration. Solving the sum, we get
T (n) = T (n − i) + 2n(i − 1) + 2
(i − 1)(i − 1 + 1) 2
Backward Substitution
Example – Continued
We want to get rid of the recursive term. To do this, we need to know at what iteration we reach our base case; i.e. for what value of i can we use the initial condition, T (1) = 5?
We can easily see that when i = n − 1 , we get the base case. Substituting this into the equation above, we get
T (n) = T (n − i) + 2n(i − 1) − i^2 + i + 2n = T (1) + 2n(n − 1 − 1) − (n − 1)^2 + (n − 1) + 2n = 5 + 2n(n − 2) − (n^2 − 2 n + 1) + (n − 1) + 2n = n^2 + n + 3
Recurrence Trees
When using recurrence trees, we graphically represent the recursion.
Each node in the tree is an instance of the function. As we progress downward, the size of the input decreases. The contribution of each level to the function is equivalent to the number of nodes at that level times the non-recursive cost on the size of the input at that level.
The tree ends at the depth at which we reach the base case. As an example, we consider a recursive function of the form
T (n) = αT
n β
Recurrence Trees
T (n)
T (n/β)
T (n/β^2 ) · · ·^ α^ · · ·^ T (n/β^2 )
· · · α · · · T (n/β)
T (n/β^2 ) · · ·^ α^ · · ·^ T (n/β^2 )
Iteration 0
1
2 .. . i .. . logβ n
Cost f (n)
α · f ( (^) n β
)
α^2 · f
( n β^2
)
.. . αi^ · f
( β^ ni
)
.. . αlogβ n^ · T (δ)
Recurrence Trees
Example
The total value of the function is the summation over all levels of the tree:
T (n) =
logβ n ∑
i=
αi^ · f
n βi
We consider the following concrete example.
Example
T (n) = 2T
( (^) n 2
Recurrence Trees
Example – Continued
T (n)
T (n/2)
T (n/4)
T (n/8) T (n/8)
T (n/4)
T (n/8) T (n/8)
T (n/2)
T (n/4)
T (n/8) T (n/8)
T (n/4)
T (n/8) T (n/8)
Iteration 0
1
2
3 .. . i .. . log 2 n
Cost n
n 2 + n 2
4 · (^ n 4 )
8 · (^ n 8 ) .. . 2 i^ · n 2 i
.. . 2 log^2 n^ · T (1)
Recurrence Trees
Example – Continued
The value of the function then, is the summation of the value of all levels. We treat the last level as a special case since its non-recursive cost is different.
T (n) = 4n +
(log ∑ 2 n)− 1
i=
2 i^ n 2 i^
= n(log n) + 4n
Smoothness Rule I
In the previous example we make the following assumption: that n was a power of two; n = 2k. This was necessary to get a nice depth of log n and a full tree.
We can restrict consideration to certain powers because of the smoothness rule.
Definition
A function f : N 7 → R is called smooth if it is monotonically nondecreasing and f (2n) ∈ Θ(f (n))
Most “slow” growing functions (logarithmic, polylogarithmic, polynomial) are smooth while exponential functions are not.
Smoothness Rule II
Theorem
For a smooth function f (n) and a fixed constant b ∈ Z such that b ≥ 2 , f (bn) ∈ Θ(f (n))
Thus the order of growth is preserved.
