Download Dynamic Programming: String Reconstruction and Edit Distance and more Study notes Algorithms and Programming in PDF only on Docsity!
UC Berkeley—CS 170: Efficient Algorithms and Intractable Problems Handout 13 Lecturer: Shyam Lakshmin October 14, 2003
Notes 13 for CS 170
1 Introduction to Dynamic Programming
Recall our first algorithm for computing the n-th Fibonacci number Fn; it just recursively applied the definition Fn = Fn− 1 + Fn− 2 , so that a function call to compute Fn resulted in two functions calls to compute Fn− 1 and Fn− 2 , and so on. The problem with this approach was that it was very expensive, because it ended up calling a function to compute Fj for each j < n possibly very many times, even after Fj had already been computed. We improved this algorithm by building a table of values of Fibonacci numbers, computing Fn by looking up Fn− 1 and Fn− 2 in the table and simply adding them. This lowered the cost of computing Fn from exponential in n to just linear in n. This worked because we could sort the problems of computing Fn simply by increasing n, and compute and store the Fibonacci numbers with small n before computing those with large n. Dynamic programming uses exactly the same idea:
- Express the solution to a problem in terms of solutions to smaller problems.
- Solve all the smallest problems first and put their solutions in a table, then solve the next larger problems, putting their solutions into the table, solve and store the next larger problems, and so on, up to the problem one originally wanted to solve. Each problem should be easily solvable by looking up and combining solutions of smaller problems in the table.
For Fibonacci numbers, how to compute Fn in terms of smaller problems Fn− 1 and Fn− 2 was obvious. For more interesting problems, figuring out how to break big problems into smaller ones is the tricky part. Once this is done, the the rest of algorithm is usually straightforward to produce. We will illustrate by a sequence of examples, starting with “one-dimensional” problems that are most analogous to Fibonacci.
2 String Reconstruction
Suppose that all blanks and punctuation marks have been inadvertently removed from a text file, and its beginning was polluted with a few extraneous characters, so the file looks something like ”lionceuponatimeinafarfarawayland...” You want to reconstruct the file using a dictionary. This is a typical problem solved by dynamic programming. We must define what is an appropriate notion of subproblem. Subproblems must be ordered by size, and each subproblem must be easily solvable, once we have the solutions to all smaller subproblems. Once we have the right notion of a subproblem, we write the appropriate recursive equation expressing how a subproblem is solved based on solutions to smaller subproblems, and the
program is then trivial to write. The complexity of the dynamic programming algorithm is precisely the total number of subproblems times the number of smaller subproblems we must examine in order to solve a subproblem. In this and the next few examples, we do dynamic programming on a one-dimensional object—in this case a string, next a sequence of matrices, then a set of strings alphabetically ordered, etc. The basic observation is this: A one-dimensional object of length n has about n^2 sub-objects (substrings, etc.), where a sub-object is defined to span the range from i to j, where i, j ≤ n. In the present case a subproblem is to tell whether the substring of the file from character i to j is the concatenation of words from the dictionary. Concretely, let the file be f [1... n], and consider a 2-D array of Boolean variables T (i, j), where T (i, j) is true if and only if the string f [i... j] is the concatenation of words from the dictionary. The recursive equation is this:
T (i, j) = dict(x[i... j]) ∨
∨
i≤k<j
[T (i, k) ∧ T (k + 1, j)]
In principle, we could write this equation verbatim as a recursive function and execute it. The problem is that there would be exponentially many recursive calls for each short string, and 3n^ calls overall. Dynamic programming can be seen as a technique of implementing such recursive pro- grams, that have heavy overlap between the recursion trees of the two recursive calls, so that the recursive function is called once for each distinct argument; indeed the recursion is usually “unwound” and disappears altogether. This is done by modifying the recursive program so that, in place of each recursive call a table is consulted. To make sure the needed answer is in the table, we note that the lengths of the strings on the right hand side of the equation above are k − i + 1 and j − k, a both of which are shorter than the string on the left (of length j − i + 1). This means we can fill the table in increasing order of string length.
for d := 0 to n − 1 do ... d + 1 is the size (string length) of the subproblem being solved for i := 1 to n − d do ... the start of the subproblem being solved j = i + d if dict(x[i... j]) then T (i, j) :=true else for k := i to j − 1 do if T (i, k) =true and T (k + 1, j) =true then do {T (i, j) :=true}
The complexity of this program is O(n^3 ): three nested loops, ranging each roughly over n values. Unfortunately, this program just returns a meaningless Boolean, and does not tell us how to reconstruct the text. Here is how to reconstruct the text. Just expand the innermost loop (the last assignment statement) to {T [i, j] :=true, first[i, j] := k, exit for} where first is an array of pointers initialized to nil. Then if T [i, j] is true, so that the substring from i to j is indeed a concatenation of dictionary words, then first[i, j] points to a breaking point in the interval i, j. Notice that this improves the running time, by exiting the for loop after the first match; more optimizations are possible. This is typical of dynamic programming algorithms: Once the basic algorithm has been derived using
This suggests a recursive scheme where the sub-problems are of the form “how many operations do we need to transform x 1 · · · xi into y 1 · · · yj. Our dynamic programming solution will be to define a (n + 1) × (m + 1) matrix M [·, ·], that we will fill so that for every 0 ≤ i ≤ n and 0 ≤ j ≤ m, M [i, j] is the minimum number of operations to transform x 1 · · · xi into y 1 · · · yj. The content of our matrix M can be formalized recursively as follows:
- M [0, j] = j because the only way to transform the empty string into y 1 · · · yj is to add the j characters y 1 ,... , yj.
- M [i, 0] = i for similar reasons.
- For i, j ≥ 1,
M [i, j] = min{ M [i − 1 , j] + 1, M [i, j − 1] + 1, M [i − 1 , j − 1] + change(xi, yj )}
where change(xi, yj ) = 1 if xi 6 = yj and change(xi, yj ) = 0 otherwise.
As an example, consider again x = aabab and y = babb
λ b a b b λ 0 1 2 3 4 a 1 1 1 2 3 a 2 2 1 2 3 b 3 2 2 1 2 a 4 3 2 2 2 b 5 4 3 2 2
What is, then, the edit distance between x and y? The table has Θ(nm) entries, each one computable in constant time. One can construct an auxiliary table Op[·, ·] such that Op[·, ·] specifies what is the first operation to do in order to optimally transform x 1 · · · xi into y 1 · · · yj. The full algorithm that fills the matrices can be specified in a few lines algorithm EdDist(x,y) n = length(x) m = length(y) for i = 0 to n M [i, 0] = i for j = 0 to m M [0, j] = j for i = 1 to n for j = 1 to m if xi == yj then change = 0 else change = 1
M [i, j] = M [i − 1 , j] + 1; Op[i, j] = delete(x, i) if M [i, j − 1] + 1 < M [i, j] then M [i, j] = M [i, j − 1] + 1; Op[i, j] = insert(x, i, yj ) if M [i − 1 , j − 1] + change < M [i, j] then M [i, j] = M [i − 1 , j − 1] + change if (change == 0) then Op[i, j] = none else Op[i, j] = change(x, i, yj )
4 Longest Common Subsequence
A subsequence of a string is obtained by taking a string and possibly deleting elements. If x 1 · · · xn is a string and 1 ≤ i 1 < i 2 < · · · < ik ≤ n is a strictly increasing sequence of indices, then xi 1 xi 2 · · · xik is a subsequence of x. For example, art is a subsequence of algorithm. In the longest common subsequence problem, given strings x and y we want to find the longest string that is a subsequence of both. For example, art is the longest common subsequence of algorithm and parachute. As usual, we need to find a recursive solution to our problem, and see how the problem on strings of a certain length can be reduced to the same problem on smaller strings. The length of the l.c.s. of x = x 1 · · · xn and y = y 1 · · · ym is either
- The length of the l.c.s. of x 1 · · · xn− 1 and y 1 · · · ym or;
- The length of the l.c.s. of x 1 · · · xn and y 1 · · · ym− 1 or;
- 1 + the length of the l.c.s. of x 1 · · · xn− 1 and y 1 · · · ym− 1 , if xn = ym.
The above observation shows that the computation of the length of the l.c.s. of x and y reduces to problems of the form “what is the length of the l.c.s. between x 1 · · · xi and y 1 · · · yi?” Our dynamic programming solution uses an (n + 1) × (m + 1) matrix M such that for every 0 ≤ i ≤ n and 0 ≤ j ≤ m, M [i, j] contains the length of the l.c.s. between x 1 · · · xi and y 1 · · · yj. The matrix has the following formal recursive definition
- M [i, 0] = 0
- M [0, j] = 0
M [i, j] = max{ M [i − 1 , j] M [i, j − 1] M [i − 1 , j − 1] + eq(xi, yj )}
where eq(xi, yj ) = 1 if xi = yj , eq(xi, yj ) = 0 otherwise.
