Download Introduction to Algorithms Analysis - Lecture Notes | CSCE 350 and more Study notes Computer Science in PDF only on Docsity!
Lecture 1
Introduction to Algorithms Analysis
I’m assuming you’ve all had CSCE 350 or the equivalent. I’ll assume some basic things from there, and sometimes quickly review the more important/subtle points. We’ll start with Appendix A in CLRS — Summations. Why? They are essential tools in analyzing the complexity of algorithms.
A First Example
Consider the following two C code fragments:
/* Fragment 1 / sum = 0; for (i=1; i<n; i=2) for (j=0; j<i; j++) sum++;
and
/* Fragment 2 / sum = 0; for (i=1; i<n; i=2) for (j=i; j<n; j++) sum++;
Note the subtle difference. Is there a difference in running time (order-of-magnitude as a function of n)? Yes there is. Sample 1 runs in time Θ(n) and Sample 2 runs in time Θ(n log n), so Sample 2 runs significantly longer. [Recall:
- f = O(g) means f (n) is at most a constant times g(n) for all n large enough.
- f = Ω(g) means f (n) is at least a (positive) constant times g(n) for all n large enough. (Equivalently, g = O(f ).)
- f = Θ(g) means both f = O(g) and f = Ω(g). “f = Θ(g)” is an equivalence relation between f and g.
Also, log n means log 2 n.] Here’s the intuition: in both fragments, the variable i does not run from 1 to n at an even pace. Since it doubles each time, it spends most of its time being very small compared to n, which makes the first j-loop run faster and the second j-loop run slower. Let’s analyze the running times more rigorously. We generally don’t care about constant factors, so it is enough to find, for each fragment, an upper bound and a lower bound that are within a constant factor of each other. This looseness usually makes life a lot easier for us, since we don’t have to be exact.
Claim 1 The running time for Fragment 2 is O(n log n).
Proof The body of the inner loop (j-loop) takes O(1) time. Each time it runs, the j-loop iterates n − i ≤ n times, for a time of O(n) per execution. The outer i-loop runs about log n many times (actually, exactly dlog ne many times). So the total time for the fragment (including initialization, loop testing, and increment) is O(n log n). 2
Claim 2 The running time for Fragment 2 is Ω(n log n).
Proof Note that for all iterations of the i-loop except the last one, the j-loop iterates at least n/ 2 times (because i < n/2 and thus n − i > n − n/2 = n/2). Thus the sum variable is incremented at least n 2 × (log n − 1) times total, which is clearly Ω(n log n). 2
Claim 3 The running time for Fragment 1 is Ω(n).
Proof To get a lower bound, we only need to look at the last iteration of the i-loop! (Digression: the late Paul Erd˝os, arguably the greatest mathematician of the 20th century, once described the art of mathematical analysis as “knowing what information you can throw away.”) The value of i in the last i-loop iteration must be at least n/2. In this iteration (as in all iterations), j runs from 0 through i − 1, so the sum variable is incremented i ≥ n/2 times. 2 But maybe we threw too much away in ignoring the previous i-loop iterations, so that our lower bound of Ω(n) is not tight enough. Actually, it is tight.
Claim 4 The running time for Fragment 1 is O(n).
Proof Now we must be more careful; the simple analysis that worked for Fragment 2 is not good enough here. We must bite the bullet and take a sum of the running times of the j-loop as i increases. For all 0 ≤ k < dlog ne, during the (k + 1)st iteration of the i-loop we clearly have i = 2k. For this (k + 1)st iteration of the i-loop, the j-loop iterates i = 2k^ times, so it takes time at most C · 2 k^ for some constant C > 0. So the total time for the fragment is
T ≤
dlog ∑ ne− 1
k=
C · 2 k.
This is a finite geometric series (see below) whose exact value is
C ·
2 dlog^ ne^ − 1 2 − 1
= C · (2dlog^ ne^ − 1).
But dlog ne < log n + 1, so
T ≤ C · (2log^ n+1^ − 1) < C · 2 log^ n+1^ = 2Cn,
which is O(n) as claimed. Even if we add the extra bits (initialization, loop testing, incrementing), these can be “absorbed” into the O(n) bound. 2 headSeries
Arithmetic Series
How to compute a closed form for
∑n i=1 i?^ This method is attributed to Gauss as a child:^ Let S =
∑n i=1 i. Then reversing the terms, we see that^ S^ =^
∑n i=1(n^ + 1^ −^ i). Then adding these two sums term by term, we get
2 S =
∑^ n
i=
(i + (n + 1 − i)) =
∑^ n
i=
(n + 1) = n(n + 1),
so S = n(n + 1)/2. Notice that this means that
∑n i=1 i^ = Θ(n
Higher Powers
Closed forms can be computed for
∑n i=1 i k (^) for any integer k ≥ 0. We have ∑n i=1 i (^2) = n(n + 1)(2n +
1)/6 and
∑n i=1 i (^3) = n (^2) (n + 1) (^2) /4. More generally, we know that
∑^ n
i=
ik^ = Θ(nk+1)
for all integers k ≥ 0.
Supplement: Finding Closed Forms for
∑n− 1 i=0 i k
The sum
∑n− 1 i=0 i k (^) has a closed form for every integer k ≥ 0. The closed form is always a polynomial
in n of degree k + 1. Here’s how to find it using induction on k. For each i and k ≥ 0 define i∑k^ := i(i − 1)(i − 2) · · · (i − k + 1) (k many factors). First, we’ll find a closed form for the sum n− 1 i=0 i k, and from this we can easily get a closed form for ∑n−^1 i=0 i k. Consider the sum
S :=
n∑− 1
i=
[
(i + 1)k+1^ − ik+
]
This sum telescopes in two different ways. First of all, we clearly have that S = nk+1^ − 0 k+1^ = nk+1. Second, each term of the sum can be factored:
n∑− 1
i=
[
(i + 1)k+1^ − ik+
]
n∑− 1
i=
[((i + 1)i(i − 1) · · · (i − k + 1)) − (i(i − 1) · · · (i − k + 1)(i − k))]
i
[(i + 1) − (i − k)] i(i − 1) · · · (i − k + 1)
= (k + 1)
i
ik.
Equating these two expressions for S and dividing by k + 1, we get
n∑− 1
i=
ik^ = nk+ k + 1
k + 1
n(n − 1)(n − 2) · · · (n − k),
which is a closed form for
∑n− 1 i=0 i k. Now to get a closed form for ∑n−^1 i=0 i k, we first expand the
product ik^ completely. For example, for k = 2 we would get i^2 = i(i − 1) = i^2 − i. We then sum this for i going from 0 to n − 1 and use the closed form we got above:
n∑− 1
i=
(i^2 − i) =
i
i^2 =
n(n − 1)(n − 2) = n(n − 1)(n − 2) 3
The left-hand side is equal to
∑n− 1 i=0 i
(^2) −∑n−^1 i=0 i. We already know the second term—it is^ n(n−1)/2, and so we move it to the other side:
n∑− 1
i=
i^2 =
n(n − 1)(n − 2) 3
n(n − 1) 2
n(n − 1)(2n − 1) 6
So we have a closed form for the sum of squares. For more general k, when we expand ik^ and sum, we’ll get some combination
i i k (^) + · · · , where the other terms all involve sums of smaller powers
of i. Assuming by induction that we have closed forms for all the smaller power sums, we then get a closed form for
i i k. As an exercise, apply the ideas above to find a closed form for
∑n− 1 i=0 i
Harmonic Series
What about k = −1? We show that
∑n i=1 1 /i^ = Θ(log^ n) (more accurately,^
∑n i=1 1 /i^ = ln^ n+O(1)). We use the technique of nested sums. Assume n ≥ 1 and let m = blog(n + 1)c, that is, m is the largest integer such that 2m^ ≤ n + 1. Then,
∑^ n
i=i
i
m∑− 1
b=
(^2) ∑b− 1
i=
2 b^ + i
∑^ n
i=2m
i
The inner sum on the right-hand side gives a block of contiguous terms in the sum on the left-hand side. The last sum on the right gives the terms left over after all the complete blocks are counted. We use (1) to get both lower and upper bounds on the harmonic series. For the upper bound, we see that m∑− 1
b=
(^2) ∑b− 1
i=
2 b^ + i
m∑− 1
b=
(^2) ∑b− 1
i=
2 b^
m∑− 1
b=
1 = m,
and (^) n ∑
i=2m
i
∑^ n
i=2m
2 m^
since there are at most 2m^ terms in this sum. Thus
∑n i=1 1 /i^ ≤^ m^ + 1 =^ O(log^ n). We do something similar for the lower bound.
m∑− 1
b=
(^2) ∑b− 1
i=
2 b^ + i
m∑− 1
b=
(^2) ∑b− 1
i=
2 b+^
m∑− 1
b=
m 2
so
∑n i=1 1 /i^ ≥^ m/2 = Ω(log^ n).
Note that (n + 1)rn+1^ tends to zero, because n + 1 increases only linearly in n, but rn+1^ decreases exponentially in n. Integrating a series term by term can also be useful. Exercise Find a closed form for the series
i=
ri i , where^ |r|^ <^ 1.^ [Hint: integrate the infinite geometric series term by term. Doing this is legal because the series
i=
∣∣ xi i
∣∣ converges uniformly
for x in some neighborhood of r.]
Lecture 3
More Math
Products A product is like a sum, but the terms are multiplied instead of added. For example,
n! =
∏^ n
i=
i.
By convention, an empty sum has value 0, and an empty product has value 1. So in particular,
∏^0
i=
i = 1.
There are tricks for getting a closed form for a product, but a good fall-back method is to just take the logarithm (to any convenient base b). This turns the product into a sum. Simplify the sum, then take b to the power of the result, then simplify further if possible. For example, suppose we want to evaluate
S =
∏^ n
i=
2 · 3 i.
Taking the log (to base two) of both sides we have
log S =
∑^ n
i=
log(2 · 3 i)
∑^ n
i=
(log 2 + i log 3)
= n + 1 + log 3
∑^ n
i=
i
= n + 1 + log 3
n(n + 1) 2
Taking 2 to the power of each side, we get
S = 2 n+1^ · 2 (log 3)n(n+1)/^2 = 2 n+1^ · (2log 3)n(n+1)/^2 = 2 n+1^ · 3 n(n+1)/^2.
A quicker (but essentially equivalent) way to get the same closed form is to split the product, then use the fact that the product of b to the power of some term is equal to b to the power of the sum of the terms. So we have
∏^ n
i=
2 · 3 i^ =
∏^ n
i=
∏^ n
i=
3 i
= 2 n+1^ · 3
Pn i=0 i = 2 n+1^ · 3 n(n+1)/^2 ,
as we had before. Bounding Sums We’ve already done a few of these. Here’s one more: we show that
∑n i=1 i k (^) = Θ(nk+1) for all
fixed k ≥ 0. For the upper bound, we have ∑^ n
i=
ik^ ≤
∑^ n
i=
nk^ = nk+1.
For the lower bound, we have ∑^ n
i=
ik^ ≥
∑^ n
i=dn/ 2 e
ik
∑^ n
i=dn/ 2 e
( (^) n 2
)k
n 2
( (^) n 2
)k
nk+ 2 k+^
Integral Approximation Sums can be approximated closely by integrals in many cases.
Definition 5 Let I ⊆ R be an interval with endpoints a < b (a could be −∞, and b could be ∞). A function f : I → R is monotone increasing on I if, for all x, y ∈ I, if x ≤ y then f (x) ≤ f (y). The function f is monotone decreasing on I if we have x ≤ y implies f (x) ≥ f (y).
Theorem 6 Let I, a, and b be in Definition 5, and let f : I → R be some function integrable on I. For any integers c and d such that a ≤ c − 1 < d + 1 ≤ b, we have ∫ (^) d
c− 1
f (x)dx ≤
∑^ d
i=c
f (i) ≤
∫ (^) d+
c
f (x)dx
if f is monotone increasing on I, and
∫ (^) d+
c
f (x)dx ≤
∑^ d
i=c
f (i) ≤
∫ (^) d
c− 1
f (x)dx
if f is monotone decreasing on I.
f (n) = o(g(n)) implies f (n) = O(g(n)) and f (n) 6 = Ω(g(n)), but is strictly stronger than both combined. f (n) = ω(g(n)) means that f (n) grows strictly faster than any positive constant times g(n). That is, for any c > 0 there is an n 0 such that
f (n) > c · g(n)
for all n ≥ n 0. Equivalently,
lim n→∞
f (n) g(n)
or equivalently, g(n) = o(f (n)). f (n) = ω(g(n)) implies f (n) = Ω(g(n)) and f (n) 6 = O(g(n)), but is strictly stronger than both combined. Modular Arithmetic For integers a and n with n > 0, we define a mod n to be the remainder after dividing a by n. That is, a mod n = a − ba/ncn.
If a mod n = b mod n, then we may write
a ≡ b (mod n).
This is true if and only if n is a divisor of a − b. Exponentials
nlim→∞
nb an^
for any real constants a and b with a > 1. That is, nb^ = o(an).
ex^ = 1 + x + x^2 2!
x^3 3!
∑^ ∞
i=
xi i!
In particular, ex^ ≥ 1 + x for any real x. We also have
ex^ = lim n→∞
x n
)n .
Two useful identities are
- 1 + x ≤ ex^ for all x ∈ R.
- ex^ ≤ 1 + x + x^2 for all x ∈ R such that |x| ≤ 1.
Logarithms
For all real a > 0, b > 0, c > 0 and n,
a = blogb^ a, logc(ab) = logc a + logc b, logb an^ = n logb a,
logb a =
logc a logc b
logb(1/a) = − logb a, logb a =
loga b
alogb^ c^ = clogb^ a,
where we assume none of the logarithm bases are 1. The natural logarithm ln x = loge x =
∫ (^) x 1 dr/r^ satisfies
- ln(1 + x) = x − x^2 /2 + x^3 / 3 − x^4 /4 + · · · = −
i=1(−1) ixi/i (Taylor series),
- ln x ≤ x − 1 for all x > 0.
In mathematics, ln x is often denoted as log x. We will use lg x to denote log 2 x.
Lecture 4
Starting Algorithms
Problems, Algorithms, and the RAM Model Types of problems: decision, search, arrangement, optimization. We have examples for each, and there can be overlap. An algorithm is a careful, step-by-step procedure for performing some task. It should allow no room for judgment or intuition. To analyze algorithms rigorously, we must express them concretely as programs on a completely specified mathematical model of computation. One such popular model (which is essentially equiva- lent to several others) is the RAM (Random Access Machine) model. Defining this model rigorously would be too tedious, so we’ll only describe it. A RAM algorithm is a sequence of RAM instructions that act on (read and write) individual registers in a one-way infinite array (indexed 0, 1 , 2 ,.. .). Each register is a finite sequence of bits. Typical allowed instructions include arithmetic (addition, substraction, multiplication, and division) on registers (both integer and floating point), loading and storing values from/in registers, indirect addressing, testing (equality, comparison) of registers (both integer and floating point), and branching. Thus the RAM model looks like some generic assembly language. We count each instruction execution as costing one unit of time. This is only realistic, however, if we limit the range of possible values that can be stored in a register. A restriction that is commonly agreed upon is that for problems of size n, each register is allowed to be only c log n bits, where c ≥ 1 is some constant that we can choose as big as we need for the problem at hand, so long as it is independent of n. If we didn’t restrict the size of registers, then we could manipulate huge amounts of data in a single register in a single time step (say, by repeated squaring). This is clearly unrealistic.
Then we know that
T (1) = O(1), T (n) = 2 T (bn/ 2 c) + Θ(n),
where the second equation assumes n > 1. This is an example of a recurrence relation. It uniquely determines the asymptotic growth rate of T (n). The time for Mergesort roughly satisfies this recurrence relation. More accurately, it satisfies
T (1) = O(1), T (n) = T (bn/ 2 c) + T (dn/ 2 e) + Θ(n).
To get a relation that uniquely defines values (not just the asymptotics) of a function bounding the run time, we supply constants in place of the asymptotic notation:
T (1) = d, T (n) = 2 T (bn/ 2 c) + cn,
where c and d are positive constants. This uniquely determines the value of T (n) for all positive integers n, and allows us to manipulate it more directly. When we only care about asymptotics, we often omit c (tacitly assuming c = 1), as well as omitting the base case, which is always of the form, “if n is bounded by (some constant), then T (n) is bounded by (some constant).” The actual choice of these constants does not affect the asymptotic growth rate of T (n), so we can just ignore this case altogether. Solving Recurrence Relations We now concentrate on techniques to solve recurrence relations. There are three main ap- proaches: the substitution method, the tree method, and the master method. The Substitution Method This method yields a rigorous proof by induction that a function given by a recurrence relation has a given asymptotic growth rate. It is, in some sense, the final word in the analysis of that function. A drawback is that you have to guess what the growth rate is before applying the method. Consider the relation
T (0) = 1 , T (n) = 3 T (bn/ 4 c) + n^2 (for n > 0),
where c > 0 is a constant. Clearly T (n) = Ω(n^2 ), because T (n) ≥ n^2 for all n > 0. We guess that this is tight, that is, T (n) = O(n^2 ). Guessing the asymptotic growth rate is the first step of the method. Now we try to prove our guess; we must show (by induction) that there is an n 0 and constant c > 0 such that T (n) ≤ cn^2 (4)
for all n ≥ n 0. For the inductive step, we assume that (4) holds for all values less than n, and show
that it therefore holds for n (where n is large enough). We have
T (n) = 3 T (bn/ 4 c) + n^2 ≤ 3(cbn/ 4 c^2 ) + n^2 (inductive hyp.) ≤ 3 c(n/4)^2 + n^2
=
· c + 1
n^2.
We’re done with the inductive case if we can get the right-hand side to be ≤ cn^2. This will be the case provided 3 16 · c + 1 ≤ c,
or equivalently, c ≥ 16 /13. Thus, so far we are free to choose c to be anything at least 16/13. The base case might impose a further constraint, though. Now the base case, where we decide n 0. Note that T (0) = 1 > c · 02 for any c, so we cannot choose n 0 = 0. We can choose n 0 = 1, though, as long as we establish (4) for enough initial values of n to get the induction started. We have
T (1) = 4 , T (2) = 7 , T (3) = 12.
All higher values of n reduce inductively to these three, so it suffices to pick n 0 = 1 and c = 4, so that (4) holds for n ∈ { 1 , 2 , 3 }. Since 4 ≥ 16 /13, our choice of c is consistent with the constraint we established earlier. By this we have shown that T (n) = O(n^2 ), and hence T (n) = Θ(n^2 ). In the future, we will largely ignore floors and ceilings. This almost never gets us into serious trouble, but doing so make our arguments less than rigorous. Thus after analyzing a recurrence by ignoring the floors and ceilings, if we want a rigorous proof, we should use our result as the guess in an application of the substitution method on the original recurrence. Changing Variables Sometimes you can convert an unfamiliar recurrence relation into a familiar one by changing variables. Consider T (n) = 2T (
n) + lg n. (5)
Let S be the function defined by S(m) = T (2m) for all m. Then from (5) we get
S(m) = T (2m) = 2T (2m/^2 ) + lg 2m^ = 2S(m/2) + m. (6)
So we get a recurrence on S without the square root. This is essentially the same recurrence as that for MergeSort. By the tree or master methods (later), we can solve this recurrence for S to get S(m) = Θ(m lg m). Thus,
T (n) = S(lg n) = Θ(lg n lg lg n).
The Tree Method The execution of a recursive algorithm can be visualized as a tree. Each node of the tree corresponds to a call to the algorithm: the root being the initial call, and the children of a node
depth logb n, where the total for level i is aif (n/bi). Thus,
T (n) =
log ∑b n
i=
aif (n/bi)
log ∑b n
i=
ai^
( (^) n bi
)d
= nd
log ∑b n
i=
( (^) a bd
)i
= nd
log ∑b n
i=
ri,
where r = a/bd. This is clearly a finite geometric series. Noting that bd^ = a/r, and thus logb r = logb a − d = t − d, we see that r > 1, r = 1, or r < 1 just as t > d, t = d, or t < d, respectively. We consider the three possibilities:
t > d. This corresponds to Case 1 of the theorem. We have,
T (n) = nd
rlogb^ n+1^ − 1 r − 1
r − 1
nd(r · rlogb^ n^ − 1)
r r − 1 nd
rlogb^ n^ −
r
r r − 1 nd
nlogb^ r^ −
r
r r − 1 nd
nt−d^ −
r
r r − 1
nt^ − nd r
= Θ(nt)
as in Case 1. Note that nd^ = o(nt).
t < d. This corresponds to Case 3 of the theorem. We first verify the regularity property of f (n) = nd: af (n/b) = (a/bd)nd^ = r · nd^ = r · f (n), and thus we can take c = r < 1 to make the regularity property hold. Next, since 0 < r < 1,
we see that for all n ≥ 1,
nd^ = nd
∑^0
i=
ri
≤ nd
log ∑b n
i=
ri
= T (n)
≤ nd
∑^ ∞
i=
ri
nd 1 − r
and thus T (n) = Θ(nd) = Θ(f (n)), as in Case 3.
t = d. This corresponds to Case 2 of the theorem. We have r = 1, so
T (n) = nd
log ∑b n
i=
1 = nd(logb n + 1) = Θ(nd^ lg n),
as in Case 3.
2 Remark. In Case 1 of the theorem, the tree is bottom-heavy, that is, the total time is dominated by the time spent at the leaves. In Case 3, the tree is top-heavy, and the time spent at the root dominates the total time. Case 2 is in between these two. In this case, the tree is “rectangular,” that is, each level gives about the same total time. Thus the total time is the time spent at level 0 (the root, i.e., Θ(nt)) times the number of levels (i.e., Θ(lg n)). The Master Method Use the master theorem to solve the following recurrences:
- T (n) = 2T (n/2) + n (Case 2),
- T (n) = 8T (n/3) + n^2 (Case 3),
- T (n) = 9T (n/3) + n^2 (Case 2),
- T (n) = 10T (n/3) + n^2 (Case 1),
- T (n) = 2T (2n/3) + n^2 (Case 3),
- T (n) = T (9n/10) + 1 (Case 2).
The first recurrence is that of MergeSort. The Master Theorem gives a time of Θ(n lg n). In each case, go through the following steps:
- What is a?
- What is b?
We then have
bc = (b+ bh 2 m)(c + ch 2 m) = bc + (bch + bhc)2m^ + bhch 2 n.
This suggests that we can compute bc with four recursive multiplications of pairs of m-bit numbers— bc, bch, bhc, and bhch—as well as Θ(n) time spent doing other things, namely, some additions and multiplications by powers of two (the latter amounts to arithmetic shift of the bits, which can be done in linear time.) The time for this divide-and-conquer multiplication algorithm thus satisfies the recurrence T (n) = 4T (m) + Θ(n) = 4T (n/2) + Θ(n).
The Master Theorem (Case 1) then gives T (n) = Θ(n^2 ), which is asymptotically no better than the naive algorithm. Can we do better? Yes. Split b and c up into their low and high halves as above, but then recursively compute only three products:
x := bc, y := bhch, z := (b+ bh)(c + ch).
Now you should verify for yourself that
bc = x + (z − y − x)2m^ + y 2 n,
which the algorithm then computes. How much time does this take? Besides the recursive calls, there’s a linear time’s worth of overhead: additions, subtractions, and arithmetic shifts. There are three recursive calls—computing x, y, and z. The numbers x and y are products of two m-bit integers each, and z is the product of (at most) two (m + 1)-bit integers. Thus the running time satisfies T (n) = 2T (n/2) + T (n/2 + 1) + Θ(n).
It can be shown that the “+1” doesn’t affect the result, so the recurrence is effectively
T (n) = 3T (n/2) + Θ(n),
which yields T (n) = Θ(nlg 3) by the Master Theorem.^1 Since lg 3 = 1.. 585 < 2, the new approach runs significantly faster asymptotically. This approach dates back at least to Gauss, who discovered (using the same trick) that multi- plying two complex numbers together could be done with only three real multiplications instead of the more naive four.
Lecture 8
Sorting and Order Statistics: Heapsort
(^1) If you’re really worried about the “+1,”, you should verify that T (n) = Θ(nlg 3) directly using the substitution method. Alternatively, you can modify the algorithm a bit so that only m-bit numbers are multiplied recursively and the overhead is still Θ(n).
Finally, we look at algorithms for some specific problems. Note that we are skipping Chapter 5, although we will use some of its techniques eventually. The Sorting Problem
Input: A sequence of n numbers 〈a 1 , a 2 ,... , an〉.
Output: A permutation (reordering) 〈a′ 1 , a′ 2 ,... , a′ n〉 of the input sequence such that a′ 1 ≤ a′ 2 ≤ · · · ≤ a′ n.
Items usually stored in an array (or a linked list), perhaps as keys of records with additional satellite data. If the amount of satellite data in each record is large, we usually only include a pointer to it with the key, in order to minimize data movement while sorting. These are programming details rather than algorithmic—i.e., methodological—issues, so we usually consider the array to consist of numbers only. Sorting is fundamental in the study of algorithms, with a wide variety of applications. We’ve seen Insertion Sort (Θ(n^2 ) time in the worst case, but sorts in place) and MergeSort (Θ(n lg n) time), but the Merge subroutine does not rearrange data in place. We’ll see two other in-place sorting routines: HeapSort (Θ(n lg n) worst-case time) and Quick- Sort (Θ(n^2 ) worst-case time, but Θ(n lg n) average-case time). Both sort in place (O(1) extra space used). The Order Statistics Problem The ith order statistic of a (multi)set of n numbers is the ith smallest number in the set. We can find the ith order statistic by sorting the numbers in the set, then indexing the ith element. This takes time asymptotically equal to that of sorting, e.g., Θ(n lg n) for MergeSort. But there is actually a linear time algorithm for finding the ith order statistic. Heaps A heap is used to implement a priority queue, and is also used in Heapsort Heaps are collections of objects with keys taken from a totally ordered universe. There are two types of heap: min heap and max heap. Min heaps support the following operations:
Insertion Adding an element to the heap
Deletion Removing an element with minimum key value
Finding the minimum Returing an object with minimum key value (the heap is unchanged)
Decrease key Decreasing the key value of an element of the heap
Max heaps support the same operations with “minimum” replaced with “maximum” and “decrease” replaced with “increase.” An Implementation Binary Heap: array A with two attributes: length[A] and heap-size[A], which is at most length[A]. The elements of the heap reside in A[1... heap-size[A]] and can be pictured as an almost full binary tree. (A binary tree is almost full if all its leaves are either on the same level or on two adjacent levels with the deeper leaves as far left as possible. The correspondance between node locations on the tree and array indices is obtained by travers- ing the tree in level order, left to right within each level. The root is at index 1. The left and right children of the node with index i have indices 2i and 2i + 1, respectively, and for any nonroot note
