Download Efficient Multiplication Algorithms: Divide-and-Conquer and Matrix Multiplication and more Study notes Algorithms and Programming in PDF only on Docsity!
Divide-and-conquer algorithms
1
1 Multiplication
The mathematician Gauss once noticed that although the product of two complex numbers
seems to involve four real-number multiplications, it can in fact be done with just three :
, , and
, since
This speeds up the computation, but only by a constant factor, which to our big-) way of
thinking is negligible. Can something more substantial be salvaged from Gauss’ observation?
The seemingly modest improvement in computation time becomes very significant if this
same trick is applied recursively. Let’s move away from complex numbers and see how this
helps with regular multiplication. Suppose * and + are two , -bit integers. As a first step
towards multiplying them, split each of them into their left and right halves, which are ,.-0/
bits long:
For instance, if *
then * 1
and * 3
. The product is
We will compute *2+ via the expression on the right-hand side. The additions take linear time,
as do the multiplications by powers of two (which are left-shifts). The significant operations
are the four ,.-0/ -bit multiplications, * 1 + 1
- 3 + 3 ; these we can handle by four re-
cursive calls. If A
, denotes the time taken to multiply , -bit numbers by this method, then
A
, can be written as a recurrence relation ,
A
BDC
A
We will soon examine general strategies for solving such equations. In the meantime, this
particular one works out to )
, 9 , which is disappointing because it is no better than the
traditional grade-school multiplication technique. So we would like to speed up our recursive
algorithm somehow, and now Gauss’ trick comes to mind. Although the expression for *2+
seems to demand four ,.-0/ -bit multiplications, as before just three will do: * 1 + 1
+E
+3 , since *F1(+
*43G+E
H
*F
+E
F1(+E
*F3I+
The improved
running time is
A
BKJ
A
1 Copyright c
L
2004 S. Dasgupta, C.H. Papadimitriou, and U.V. Vazirani.
Figure 1.1 A divide-and-conquer algorithm for integer multiplication.
function multiply ( *
Input: Two , -bit numbers * and +.
Output: Their product.
if ,
: return *?+
leftmost, rightmost
,.-0/ bits of *
leftmost, rightmost
,.-0/ bits of +
multiply
multiply
multiply
return
At first glance this seems only a constant factor better than the previous attempt. But the
improvement occurs at every level of the recursion, and this compounding effect leads to a
dramatically lower time bound of )
.
The algorithmic strategy we have been using (see Figure 1.1) is called divide-and-conquer :
it tackles a problem by selecting subproblems, recursively solving them, and then gluing to-
gether these partial answers. All the work is done in the selecting and gluing, and in the base
case of the recursion.
The running time of our algorithm follows from its pattern of recursive calls, which form
a tree structure, as in Figure 1.2. Let’s try to understand the shape of this tree. At each
successive level of recursion the subproblems get halved in size. At the
level, the
subproblems get down to size one, and so the recursion ends. Therefore, the height of the
tree is
. The branching factor is three – each problem recursively produces three smaller
ones – with the result that at depth in the tree there are
J
subproblems, each of size ,.-0/
.
For each subproblem, a linear amount of work is done in selecting further subproblems
and gluing together answers. Therefore the total time spent at depth in the tree is
J
J
At the very top level, when
, this works out to )
,. At the bottom, when
, it
is )
J)(+* ,.-
, which can be rewritten as )
(check!). Between these two endpoints, the
work done increases geometrically from )
, to )
, by a factor of
J
sum of any increasing geometric series is, within a constant factor, simply the last term of the
series: such is the rapidity of the increase. Therefore the overall running time is )
,
which is about )
.
In the absence of Gauss’ trick, the recursion tree would have the same height, but with a
branching factor of four. There would be
C
G9 leaves, and therefore the running time
would be at least this much. In divide-and-conquer algorithms, the number of subproblems
translates into the branching factor of the recursion tree; small changes in this coefficient can
, is dominant. Finally, it could be that the ratio is exactly one, in
which case all )
terms of the series are equal to )
These cases translate directly into the three contingencies of the theorem.
3 Matrix multiplication
The product of two , , matrices
and is a third , , matrix
, with
entry
In general
is not the same as
; matrix multiplication is not commutative.
The formula above implies an )
algorithm for matrix multiplication: there are , 9 en-
tries to be computed, and each takes linear time. For quite a while, this was widely believed
to be the best running time possible, and it was even proved that no algorithm which used
just additions and multiplications could do better. It was therefore a source of great excite-
ment when in 1969, Strassen announced a significantly more efficient algorithm, based upon
divide-and-conquer.
Matrix multiplication is particularly easy to break into subproblems, because it can be
performed blockwise. To see what this means, carve
into four ,.-0/ ,.-0/ blocks, and also :
^ ^
Then their product can be expressed in terms of these blocks, and is exactly as if the blocks
were single elements.
We now have a divide-and-conquer strategy: to compute the size-, product
, recursively
compute eight size-, .-0/ products ^
, and then do a few )
time additions. The total running time is described by the recurrence relation
A
A
which comes out to )
, the same as for the default algorithm. However, an improvement in
the time bound is possible, and as with integer multiplication, it relies upon algebraic tricks.
It turns out that
can be computed from just seven subproblems.
^ ^
(% ^9 )
This translates into a running time of
A
A
which by the result of the previous section is )
(^) )
G
.
4 Mergesort
The problem of sorting a list of numbers lends itself immediately to a divide-and-conquer
strategy: split the list into two halves, recursively sort each half, and then merge the two
sorted sublists.
function mergesort (
)
Input: An array of numbers
Output: A sorted version of this array
if ,
return merge(mergesort(
.-0/ ), mergesort(
, .-0/
))
else
return
The correctness of this algorithm is self-evident, as long as a correct merge subroutine is
specified. If we are given two sorted arrays *
; %'$'$'
and +
; %'$'$' , how do we efficiently merge
them into a single sorted array
; %'$'$'
? Well, the very first element of is either *
; or
, whichever is smaller. The rest of
can then be constructed recursively.
function merge (*
; %'$'$'
)
if
: return +
; %'$'$'
if
G = : return *
; %'$'$'
if *
; +
; :
return *
; merge
/
else:
return +
; merge
; %'$'$'
/
This kind of tail recursion can be unraveled into a purely iterative algorithm, as shown in
Figure 4.1. It performs a constant number of operations for each element of the combined
array , and therefore takes linear time, )
, in all. The overall running time of mergesort
is then
A
A
which works out to )
.
5 Medians
The median of a list of numbers is its 50th percentile: half the numbers are bigger than it,
and half are smaller. In other words, it is the middle element when the numbers are arranged
in order. For instance, the median of
C ;0 ;$= J0= /
is /
. If the list has even length, there
are two choices for the middle element, and the median is defined to be their average.
The purpose of the median is to summarize a set of numbers by a single, typical value. The
mean , or average, is also commonly used for this, but it has the tremendous disadvantage that
The choice of is crucial. It should be picked quickly, and it should shrink the array
substantially, the ideal situation being
. If we could always guarantee this
situation, we would get a running time of
A
A
I
which is linear as desired. But this requires picking to be the median, which is what we are
trying to do in the first place! Instead, our method of choosing is much simpler: we just pick
it randomly from .
The running time of our algorithm depends on the random choices of. It is possible that
due to persistent bad luck we keep picking to be the smallest element of the array (or the
largest element), and thereby shrink the array by only one element each time. This worst
case scenario takes )
, 9 steps, but is extremely unlikely to occur. Equally unlikely is the
best possible case, in which each randomly chosen just happens to be the median, so that as
above the running time is )
,. Where, in this spectrum from )
, to )
, 9 , does the average
running time lie? Fortunately, it lies very close to the best-case time.
To distinguish between lucky and unlucky choices of , we will call good if it lies any-
where in /
to
percentile of the array that it is chosen from. The following property
follows from the definition of percentile.
Property. When is chosen randomly from an array, it has a probability
being good.
A good causes the array to shrink to at most
J
C
of its size (why?). This is very promising,
but how many times, on average, do we need to pick before we get a good one? There is an
equivalent question which might be more familiar: “on average how many times must we toss
a fair coin before getting heads?”. Let be the expected number of tosses. We certainly need
at least one toss, and if it’s heads, we’re done. Otherwise (and this occurs with probability ;
- 0/ ), we need to repeat. In other words,
, so
Therefore, after every two recursive calls on average, the array will shrink to
J
C
of its
size. Let A
, be the expected running time. Then
A
, A
J
C
I
which means A
, is linear. In short, on any input, our algorithm returns the correct answer
in an average of )
, steps.
