Download Algorithm Analysis: Determining Algorithm Complexity and Time Requirements and more Exams Computer Science in PDF only on Docsity!
Computational Complexity
Algorithm Analysis Algorithm - a clearly specified set of instructions that the computer will follow to solve a problem. Algorithm Analysis - determining the amount of resources that the algorithm will require, typically in terms of time and space. Areas of study include: Estimation techniques for determining the running time of an algorithm. Techniques to reduce the running time of an algorithm. Mathematical framework for the accurate determination of the running time of an algorithm. Algorithm Analysis The running time of an algorithm is a function of the size of the input. Example: It takes longer to sort 1000 numbers than it does to sort 10 numbers. The value of this function depends upon many things including:
- The speed of the host computer.
- The size of the host computer.
- The compilation process (quality of the compiler generated code).
- The quality of the original source code which implements the algorithm. Illustration of running time vs input size for small input sets (corrected Figure 5.1) linear N log N quadratic time cubic constant N
Computational Complexity
When comparing two functions F(N) and G(N), it does not make sense to state that: F < G, F = G, or G < F. Example: At some point x , F may be smaller than G, yet at some other point y , F may be equal to or greater than G. Instead, the growth rates of the functions need to be determined. Definitions (based on the growth rate of the function):
- constant function – function whose dominant term is a constant (c)
- logarithmic func. – dominant term is log N
- log-squared func. – dominant term is log^2 N
- linear func. – dominant term is N
- N log N func. – dominant term is N log N
- quadratic func. – dominant term is N 2
- cubic func. – dominant term is N 3
- exponential func. – dominant term is 2 N There is a three-fold reason for basing our analysis on the growth rate of the function rather than its specific value at some point:
- For sufficiently large values of N, the value of the function is primarily determined by its dominant term (sufficiently large varies by function). Example: Consider the cubic function where the function is expressed by 15N^3 + 20N^2 - 10N + 4. For large values of N, say 1000, the value of this function is: 15,019,990,004 of which 15,000,000,000 is due entirely to the N^3 term. Using only the N^3 term to estimate the value of this function introduces an error of only 0.1% which is typically close enough for estimation purposes.
- Constants associated with the dominant term are usually not meaningful across different machines (maybe though for identically growing functions).
- Small values of N are generally not important. Big-Oh Notation Used to represent the growth rate of a function. Allows algorithm designers to establish a relative order among functions by comparison of their dominant terms. Denoted as O(N^2 ), read as "order N squared". Examine this notation and related notation more formally a bit later. (section 5.4).
Asymptotic Notation Big Oh Notation Definition: Let p(n) and q(n) be two nonnegative functions. The function p(n) is asymptotically bigger [p(n) asymptotically dominates q(n)] than the function q(n) iff The function q(n) is asymptotically smaller than p(n) iff p(n) is asympotically bigger than q(n). Functions p(n) and q(n) are asymptotically equal iff neither is asymptotically bigger than the other. Example 1: Let p(n) = 3n^2 + 2n + 6 and q(n) = 10n + 7. divide both functions by n 2 (to reduce dominant term to a constant) which will produce: Thus, 3n^2 + 2n + 6 is asymptotically bigger than 10n + 7. Similarly, 10n + 7 is asymptotically smaller than 3n 2
- 2n + 6. Example 2: Let p(n) = 6n + 2 and q(n) = 12n + 6 Try: Show that 8n^4 + 9n^2 is asymptotically bigger than 100n^3 – 3. Also do the same for: 2n^2 + 3n and 83n. 0 p(n ) q(n ) lim n 6 / 12 1 / 2 12 6 / n 6 / 2 n (dividebyn ) 12 n 6 6 n 2 lim 6 / 3 2 6 2 / n 12 6 / n (dividebyn ) 6 n 2 12 n 6 lim n n 3 n 2 n 6 10 n 7 lim (^2) n (^) 0 / 3 0 3 2 /n 6 / n 10 /n 7 / n 2 2
0 / 8 0 8 9 / n 100 /n 3 / n 8 n 9 n 100 n 3 lim (^2) 4 4 2 3 n A similar technique will show the same is true for the functions 2n^2 + 3n and 83n. Term Name 1 Constant Log N Logarithmic N Linear N log N N log N N^2 Quadratic N^3 Cubic 2 N^ Exponential N! Factorial Asymptotic notation describes the behavior of the time or space complexity for large instance characteristics. When the instance characteristic is described by a single variable, say n , asymptotic notation describes the complexity using a single term, the asymptotically biggest term in the function (step count of algorithm steps). Big-Oh Notation (O) Notation: f(n) = O(g(n)) [read as f(n) is big-oh of g(n)] means that f(n) is asymptotically smaller than or equal to g(n). Meaning: g(n) establishes an upper bound on f(n). The asymptotic growth rate of the function f(n) is bounded from above by g(n). t cg(n) m f(n) n g(n) is an upper bound on f(n) Omega Notation ( )
Algorithm Assumptions: an array named a that holds the values and is of size SIZE. min = a[0] for ( i = 1; i < SIZE; i++) if( a[i] < min) min = a[i] return min Analysis Makes N-1 iterations of the loop and is therefore O(N). (g(n)) = {f(n) | f(n) = (g(n))} (g(n)) = {f(n) | f(n) = (g(n))} Under this interpretation, statements such as O(g1(n)) = O(g2(n)) and (g1(n)) = (g2(n)) are meaningful. This notation also allows one to read f(n) = O(g(n)) as “f of n is in big oh of g of n”, and so on. Little Oh Notation (o) Little Oh notation, o(f(n)), is commonly used in step count analysis and provides a strict upper bound on the asymptotic growth rate of the function. This means that f(n) is o(g(n)) iff f(n) is asymptotically smaller than g(n). [Note that the equal to case provided by the Big-Oh notation is not present in Little-Oh notation, thus the strict upper bound.] Definition: f(n) = o(g(n)) iff f(n) = O(g(n)) and f(n) (g(n)). Examples of Algorithm Running Times
- Minimum Element in an Array - given an array of N items, find the smallest item. This is problem can be solved with the following algorithm: maintain a single variable called minimum in which the value of the smallest element yet seen is stored. Initialize this variable to the value of the first element in the array. Make a single sequential pass through the array and change the value of minimum whenever a value smaller than the value of minimum is encountered. The running time of this algorithm will be O(N). Reason: the same "work" will be done for each and every item in the array of N items. A better algorithm is not possible since each item in the array must be examined one time.
- Closest Points in the Plane - given N points in a plane (an x-y coordinate system), find the pair of points that are closest together. The algorithm for solving this problem requires that the distance between all possible pairs of points in the plane be calculated and store the smallest value (i.e., remember the minimum distance). Note that there are N(N-1)/2 pairs of points or "on the order of N 2 " pairs of points. [N points can be paired with (N-1) points for a total of N(N-1) pairs - this however double counts pairs like (A,B) and (B,A) which are the same pair so the total number is divided in half] Calculating all of the pairs of points and storing the minimum distance will require quadratic time (O(N 2 )). [Note: there are algorithms for this problem which are O(NlogN) and O(N) but they are beyond the scope of our analysis.] The distance between two points in an x-y coordinate system of the form (xi, yi) and (xj, yj) is given by the formula: Shown below are two algorithms which solve this problem. 2 i j 2 (x (^) i x j) (y y)
Example – Better Algorithm – Closest Points in a Plane 4 3 Let a = y 2 a[0] a[1] a[2] a[3] 1 0 1 2 3 4 5 x Assume that a is an array (1dimension) that holds the points from the plane. Assume that the function dist(r, s) returns the distance between two points r and s. Assume that the constant SIZE is the number of elements in the array of points a. Operation of the Algorithm for the Example Case Recall that: distance (a[0], a[1]) = 2 distance (a[0], a[2]) = 5 1/ distance (a[0], a[3]) = 3 distance (a[1], a[2]) = 5 1/ distance (a[1], a[3]) = 13 1/ distance (a[2], a[3]) = 2 1/ note: the first algorithm would have also calculated distances for the following pairs of points: (a[1],a[0]), (a[2], a[0]), (a[3],a[0]), (a[2],a[1]), (a[3],a[1]), and (a[3],a[2]) or twice as many points. However, realize that the distance from a[0] to a[1] is exactly the same as the distance from a[1] to a[0]! outer loop iteration #1: i = 0 inner loop iterates from 1 to 3 for this outer iteration checks distances for the following point pairs: (a[0], a[1]), (a[0], a[2]), (a[0], a[3]) {minimum = 2, x = 0, y = 1} outer loop iteration #2: i = 1 inner loop iterates from 2 to 3 for this outer iteration checks distances for the following point pairs: (a[1], a[2]), (a[1], a[3]) {minimum = 2, x = 0, y = 1} outer loop iteration #3: i = 2 inner loop iterates from 3 to 3 for this outer iteration checks distances for the following point pairs:
2 i j 2 (x (^) i x j) (y y)
(a[2], a[3]) {minimum = 21/2, x = 2, y = 3} outer loop terminates minimum returned as 2 1/ with the closest pair of points being a[2] and a[3]
- Colinear Points in a Plane - given N points in a plane, determine if any three form a straight line. This problem suffers from the fact that the existence of colinear points in the plane introduces a degenerate case that requires special handling. Direct solution requires the enumeration of all sets of three points from the plane. The number of different groups of three points is N(N-1)(N-2)/6. This yields a dominate term which is on the order of N 3 (a cubic function). This algorithm is even more computationally expensive than the previous quadratic algorithm. [Note: there is a quadratic time algorithm for this problem.] [Note: three items can be grouped together in 6 different ways, consider (a,b,c) which can be grouped as (a,b,c), (a,c,b), (b,a,c), (b,c,a), (c,a,b), or (c,b,a) which all represent the same sequence of points – so the number of different groups is divided by 6.
- Sorted List Matching Problem – given two sorted lists of names, output the names common to both lists. Perhaps the standard way to attack this problem is the following: For each name on list #1, do the following: a) Search for the current name in list #2. b) If the name is found, output it. If a list is unsorted, steps a and b may take O(n) time. Can you tell me why? BUT, we know that both lists are already sorted. Thus we can use a binary search in step a. We know that this takes O(log n) time, where n is the total number of names in the list. For the moment, if we assume that both lists are of equal size, then we can safely say that the size of list #2 is about ½ the total input size, so technically, our search would take O(log n/2) time, where n is the TOTAL SIZE of our input to the problem. Using our log rules however, we find that log 2 n = (log 2 n/2) + 1. Thus, it’s fairly safe to assume for large n that our running time is simply O(log 2 n).
There are some slight mistakes in this algorithm, but the overall idea is here and you will “fix” the mistakes when you write a program to implement this algorithm for your homework. Now, before we finish the lecture, the last thing we must do is analyze the running time of this algorithm. Here is what we see: For each loop iteration, we advance at least one marker. The maximum number of iterations then, would be the total number of names on both lists, which is n, using our previous interpretation. For each iteration, we are doing a constant amount of work. (Essentially a comparison, and sometimes outputting a single name...) Thus, our algorithm runs in O(n) time – an improvement over our previous algorithm. A final question one must ask is, can we solve this question in even less time? If yes, what is such an algorithm, if no, how can we prove it? Our proof goes along these lines: In order to have an accurate list, we must read every name on one of the two lists. If we skip names on BOTH lists, we can NOT deduce whether we would have matches between those names or not. In order to simply “read” all the names on one list, we would take O(n/2) time. But, in order notation, this is still O(n), the running time of our second algorithm. Thus, we know we can not do better in terms of time, (within a constant factor), of our second algorithm.
