Download Analysis of Algorithms: Time Complexity and Implementation Examples and more Lecture notes Data Structures and Algorithms in PDF only on Docsity!
Data Structures and Algorithms Analysis
1. Introduction to Data Structures and Algorithms Analysis
A program is written in order to solve a problem. A solution to a problem actually consists of two things: A way to organize the data Sequence of steps to solve the problem
The way data are organized in a computers memory is said to be Data Structure and the sequence of computational steps to solve a problem is said to be an algorithm. Therefore, a program is nothing but data structures plus algorithms.
1.1. Introduction to Data Structures
Given a problem, the first step to solve the problem is obtaining ones own abstract view, or model , of the problem. This process of modeling is called abstraction.
The model defines an abstract view to the problem. This implies that the model focuses only on problem related stuff and that a programmer tries to define the properties of the problem.
These properties include
The data which are affected and The operations that are involved in the problem.
With abstraction you create a well-defined entity that can be properly handled. These entities define the data structure of the program.
An entity with the properties just described is called an abstract data type (ADT).
1.1.1. Abstract Data Types
An ADT consists of an abstract data structure and operations. Put in other terms, an ADT is an abstraction of a data structure.
The ADT specifies:
- What can be stored in the Abstract Data Type
- What operations can be done on/by the Abstract Data Type. For example , if we are going to model employees of an organization: This ADT stores employees with their relevant attributes and discarding irrelevant attributes. This ADT supports hiring, firing, retiring, … operations.
A data structure is a language construct that the programmer has defined in order to implement an abstract data type.
There are lots of formalized and standard Abstract data types such as Stacks, Queues, Trees, etc.
Do all characteristics need to be modeled? Not at all It depends on the scope of the model It depends on the reason for developing the model
1.1.2. Abstraction
Abstraction is a process of classifying characteristics as relevant and irrelevant for the particular purpose at hand and ignoring the irrelevant ones.
Applying abstraction correctly is the essence of successful programming
How do data structures model the world or some part of the world? The value held by a data structure represents some specific characteristic of the world The characteristic being modeled restricts the possible values held by a data structure The characteristic being modeled restricts the possible operations to be performed on the data structure. Note: Notice the relation between characteristic, value, and data structures
Where are algorithms, then?
1.2. Algorithms
An algorithm is a well-defined computational procedure that takes some value or a set of values as input and produces some value or a set of values as output. Data structures model the static part of the world. They are unchanging while the world is changing. In order to model the dynamic part of the world we need to work with algorithms. Algorithms are the dynamic part of a program’s world model.
An algorithm transforms data structures from one state to another state in two ways:
Running time is usually treated as the most important since computational time is the most precious resource in most problem domains.
There are two approaches to measure the efficiency of algorithms:
- Empirical: Programming competing algorithms and trying them on different instances.
- Theoretical: Determining the quantity of resources required mathematically (Execution time, memory space, etc.) needed by each algorithm.
However, it is difficult to use actual clock-time as a consistent measure of an algorithm’s efficiency, because clock-time can vary based on many things. For example,
Specific processor speed Current processor load Specific data for a particular run of the program o Input Size o Input Properties Operating Environment
Accordingly, we can analyze an algorithm according to the number of operations required, rather than according to an absolute amount of time involved. This can show how an algorithm’s efficiency changes according to the size of the input.
1.2.3. Complexity Analysis
Complexity Analysis is the systematic study of the cost of computation, measured either in time units or in operations performed, or in the amount of storage space required.
The goal is to have a meaningful measure that permits comparison of algorithms independent of operating platform. There are two things to consider: Time Complexity : Determine the approximate number of operations required to solve a problem of size n. Space Complexity: Determine the approximate memory required to solve a problem of size n.
Complexity analysis involves two distinct phases: Algorithm Analysis : Analysis of the algorithm or data structure to produce a function T (n) that describes the algorithm in terms of the operations performed in order to measure the complexity of the algorithm. Order of Magnitude Analysis : Analysis of the function T (n) to determine the general complexity category to which it belongs.
There is no generally accepted set of rules for algorithm analysis. However, an exact count of operations is commonly used.
1.2.3.1. Analysis Rules:
- We assume an arbitrary time unit.
- Execution of one of the following operations takes time 1: Assignment Operation Single Input/Output Operation Single Boolean Operations Single Arithmetic Operations Function Return
- Running time of a selection statement (if, switch) is the time for the condition evaluation + the maximum of the running times for the individual clauses in the selection.
- Loops: Running time for a loop is equal to the running time for the statements inside the loop * number of iterations. The total running time of a statement inside a group of nested loops is the running time of the statements multiplied by the product of the sizes of all the loops. For nested loops, analyze inside out. Always assume that the loop executes the maximum number of iterations possible.
- Running time of a function call is 1 for setup + the time for any parameter calculations + the time required for the execution of the function body. Examples:
- int count(){ int k=0; cout<< “Enter an integer”; cin>>n; for (i=0;i<n;i++) k=k+1; return 0;} Time Units to Compute
1 for the assignment statement: int k= 1 for the output statement. 1 for the input statement. In the for loop: 1 assignment, n+1 tests, and n increments. n loops of 2 units for an assignment, and an addition. 1 for the return statement.
T (n)= 1+1+1+(1+n+1+n)+2n+1 = 4n+6 = O(n)
- int total(int n) { int sum=0; for (int i=1;i<=n;i++) sum=sum+1; return sum; } Time Units to Compute
Time Units to Compute
1 for the assignment. 1 assignment, n+1 tests, and n increments. n loops of 4 units for an assignment, an addition, and two multiplications. 1 for the return statement.
T (n)= 1+(1+n+1+n)+4n+1 = 6n+4 = O(n)
1.2.3.2. Formal Approach to Analysis
In the above examples we have seen that analysis is a bit complex. However, it can be simplified by using some formal approach in which case we can ignore initializations, loop control, and book keeping.
for Loops: Formally
• In general, a for loop translates to a summation. The index and bounds of the summation are the
same as the index and bounds of the for loop.
for (int i = 1; i <= N; i++) { sum = sum+i; }
N
N
i
^
• Suppose we count the number of additions that are done. There is 1 addition per iteration of the
loop, hence N additions in total.
Nested Loops: Formally
• Nested for loops translate into multiple summations, one for each for loop.
for (int i = 1; i <= N; i++) {
for (int j = 1; j <= M; j++) {
sum = sum+i+j;
M MN
N
i
N
i
M
j
2 2 2 1 1 1
^ ^
• Again, count the number of additions. The outer summation is for the outer for loop.
Consecutive Statements: Formally
• Add the running times of the separate blocks of your code
for (int i = 1; i <= N; i++) { sum = sum+i; } for (int i = 1; i <= N; i++) { for (int j = 1; j <= N; j++) { sum = sum+i+j; } }
2
1 1 1
1 2 N 2 N
N
i
N
j
N
i
^
Conditionals: Formally
- If (test) s1 else s2: Compute the maximum of the running time for s1 and s2.
if (test == 1) { for (int i = 1; i <= N; i++) { sum = sum+i; }} else for (int i = 1; i <= N; i++) { for (int j = 1; j <= N; j++) { sum = sum+i+j; }}
2 2
1 1 1
max , 2 2
max 1 , 2
N N N
N
i
N
j
N
i
Example:
Suppose we have hardware capable of executing 10^6 instructions per second. How long would it take to execute an algorithm whose complexity function was: T (n) = 2n^2 on an input size of n=10^8? The total number of operations to be performed would be T (10^8 ):
T(10^8 ) = 2(10^8 )^2 =210^16 The required number of seconds required would be given by T(10^8 )/10^6 so:
Running time =210^16 /10^6 = 210^10 The number of seconds per day is 86,400 so this is about 231,480 days (634 years).
Exercises Determine the run time equation and complexity of each of the following code segments.
- for (i=0;i<n;i++) for (j=0;j<n; j++) sum=sum+i+j;
- for(int i=1; i<=n; i++) for (int j=1; j<=i; j++) sum++; What is the value of the sum if n=20?
1.4. Asymptotic Analysis
Asymptotic analysis is concerned with how the running time of an algorithm increases with the size of the input in the limit, as the size of the input increases without bound.
There are five notations used to describe a running time function. These are:
Big-Oh Notation (O) Big-Omega Notation () Theta Notation () Little-o Notation (o) Little-Omega Notation ()
1.4.1. The Big-Oh Notation
Big-Oh notation is a way of comparing algorithms and is used for computing the complexity of algorithms; i.e., the amount of time that it takes for computer program to run. It’s only concerned with what happens for very a large value of n. Therefore only the largest term in the expression (function) is needed. For example, if the number of operations in an algorithm is n^2 – n , n is insignificant compared to n^2 for large values of n. Hence the n term is ignored. Of course, for small values of n, it may be important. However, Big-Oh is mainly concerned with large values of n.
Formal Definition : f (n)= O (g (n)) if there exist c, k ∊ ℛ+^ such that for all n≥ k, f (n) ≤ c.g (n).
Examples: The following points are facts that you can use for Big-Oh problems:
1<=n for all n>= n<=n^2 for all n>= 2 n^ <=n! for all n>= log 2 n<=n for all n>= n<=nlog 2 n for all n>=
- f(n)=10n+5 and g(n)=n. Show that f(n) is O(g(n)).
To show that f(n) is O(g(n)) we must show that constants c and k such that
f(n) <=c.g(n) for all n>=k
Or 10n+5<=c.n for all n>=k
Try c=15. Then we need to show that 10n+5<=15n
Solving for n we get: 5<5n or 1<=n.
So f(n) =10n+5 <=15.g(n) for all n>=1.
(c=15,k=1).
- f(n) = 3n^2 +4n+1. Show that f(n)=O(n^2 ).
4n <=4n^2 for all n>=1 and 1<=n^2 for all n>=
3n^2 +4n+1<=3n^2 +4n^2 +n^2 for all n>=
<=8n^2 for all n>=
So we have shown that f(n)<=8n^2 for all n>=
Therefore, f (n) is O(n^2 ) (c=8,k=1)
Typical Orders
Here is a table of some typical cases. This uses logarithms to base 2, but these are simply proportional to logarithms in other base.
N O(1) O(log n) O(n) O(n log n) O(n^2 ) O(n^3 ) 1 1 1 1 1 1 1
2 1 1 2 2 4 8 4 1 2 4 8 16 64 8 1 3 8 24 64 512
16 1 4 16 64 256 4, 1024 1 10 1,024 10,240 1,048,576 1,073,741,
Demonstrating that a function f(n) is big-O of a function g(n) requires that we find specific constants c and k for which the inequality holds (and show that the inequality does in fact hold).
Big-O expresses an upper bound on the growth rate of a function, for sufficiently large values of n.
An upper bound is the best algorithmic solution that has been found for a problem. “ What is the best that we know we can do?”
Exercise:
f(n) = (3/2)n^2 +(5/2)n- Show that f(n)= O(n^2 )
In simple words, f (n) =O(g(n)) means that the growth rate of f(n) is less than or equal to g(n).
1.4.1.2. Properties of the O Notation
Higher powers grow faster
- nr^ O( ns) if 0 <= r <= s
Fastest growing term dominates a sum
- If f(n) is O(g(n)), then f(n) + g(n) is O(g)
E.g 5n^4 + 6n^3 is O (n^4 )
n (^) b > 1 and k >= 0 E.g. n^20 is O( 1.05n)
Logarithms grow more slowly than powers
E.g. log 2 n is O( n0.5)
1.4.2. Big-Omega Notation
Just as O-notation provides an asymptotic upper bound on a function, notation provides an asymptotic lower bound. Formal Definition: A function f(n) is ( g (n)) if there exist constants c and k ∊ ℛ+ such that f(n) >=c. g(n) for all n>=k.
f(n)= ( g (n)) means that f(n) is greater than or equal to some constant multiple of g(n) for all values of n greater than or equal to some k.
Example : If f(n) =n2,^ then f(n)= ( n)
In simple terms, f(n)= ( g (n)) means that the growth rate of f(n) is greater that or equal to g(n).
1.4.3. Theta Notation
A function f (n) belongs to the set of (g(n)) if there exist positive constants c1 and c2 such that it can be sandwiched between c1.g(n) and c2.g(n), for sufficiently large values of n.
Formal Definition: A function f (n) is (g(n)) if it is both O( g(n) ) and ( g(n) ). In other words, there exist constants c1, c2, and k >0 such that c1.g (n)<=f(n)<=c2. g(n) for all n >= k
If f(n)= (g(n)), then g(n) is an asymptotically tight bound for f(n).
In simple terms, f(n)= (g(n)) means that f(n) and g(n) have the same rate of growth.
Example:
- If f(n)=2n+1, then f(n) = (n)
- f(n) =2n^2 then
f(n)=O(n^4 )
f(n)=O(n^3 )
f(n)=O(n^2 )
All these are technically correct, but the last expression is the best and tight one. Since 2n^2 and n^2 have the same growth rate, it can be written as f(n)= (n^2 ).
1.4.4. Little-o Notation
Big-Oh notation may or may not be asymptotically tight, for example:
2n^2 = O(n^2 )
=O(n^3 )
f(n)=o(g(n)) means for all c>0 there exists some k>0 such that f(n)<c.g(n) for all n>=k. Informally, f(n)=o(g(n)) means f(n) becomes insignificant relative to g(n) as n approaches infinity.
Example : f(n)=3n+4 is o(n^2 )
In simple terms, f(n) has less growth rate compared to g(n).
g(n)= 2n^2 g(n) =o(n^3 ), O(n^2 ), g(n) is not o(n^2 ).
1.4.5. Little-Omega ( notation)
Little-omega () notation is to big-omega () notation as little-o notation is to Big-Oh notation. We use notation to denote a lower bound that is not asymptotically tight.
Formal Definition : f(n)= (g(n)) if there exists a constant no>0 such that 0<= c. g(n)<f(n) for all n>=k.
Example : 2n^2 =(n) but it’s not (n^2 ).
Time is proportional to the size of input ( n) and we call this time complexity O(n).
Example Implementation:
int Linear_Search(int list[], int key) { int index=0; int found=0; do{ if(key==list[index]) found=1; else index++; }while(found==0&&index<n); if(found==0) index=-1; return index; }
2.1.2. Binary Search
This searching algorithms works only on an ordered list.
The basic idea is:
- Locate midpoint of array to search
- Determine if target is in lower half or upper half of an array. o If in lower half, make this half the array to search o If in the upper half, make this half the array to search
- Loop back to step 1 until the size of the array to search is one, and this element does not match, in which case return – 1.
The computational time for this algorithm is proportional to log 2 n. Therefore the time complexity is
O ( log n)
Example Implementation:
int Binary_Search(int list[],int k) { int left=0; int right=n-1; int found=0; do{ mid=(left+right)/2; if(key==list[mid]) found=1;
else{ if(key<list[mid]) right=mid-1; else left=mid+1; } }while(found==0&&left<right); if(found==0) index=-1; else index=mid; return index; }
2.2. Sorting Algorithms
Sorting is one of the most important operations performed by computers. Sorting is a process of reordering a list of items in either increasing or decreasing order. The following are simple sorting algorithms used to sort small-sized lists.
- Insertion Sort
- Selection Sort
- Bubble Sort
2.2.1. Insertion Sort
The insertion sort works just like its name suggests - it inserts each item into its proper place in the final list. The simplest implementation of this requires two list structures - the source list and the list into which sorted items are inserted. To save memory, most implementations use an in-place sort that works by moving the current item past the already sorted items and repeatedly swapping it with the preceding item until it is in place.
It's the most instinctive type of sorting algorithm. The approach is the same approach that you use for sorting a set of cards in your hand. While playing cards, you pick up a card, start at the beginning of your hand and find the place to insert the new card, insert it and move all the others up one place.
Basic Idea:
Find the location for an element and move all others up, and insert the element.
The process involved in insertion sort is as follows:
- The left most value can be said to be sorted relative to itself. Thus, we don’t need to do anything.
- Check to see if the second value is smaller than the first one. If it is, swap these two values. The first two values are now relatively sorted.
Analysis
How many comparisons?
1+2+3+…+(n-1)= O(n^2 )
How many swaps?
1+2+3+…+(n-1)= O(n^2 )
How much space?
In-place algorithm
2.2.2. Selection Sort
Basic Idea:
Loop through the array from i=0 to n-1. Select the smallest element in the array from i to n Swap this value with value at position i.
Implementation:
void selection_sort(int list[]) { int i,j, smallest; for(i=0;i<n;i++){ smallest=i; for(j=i+1;j<n;j++){ if(list[j]<list[smallest]) smallest=j; }//end of inner loop } //end of outer loop temp=list[smallest]; list[smallest]=list[i]; list[i]=temp;
}//end of selection_sort
Analysis
How many comparisons?
(n-1)+(n-2)+…+1= O(n^2 )
How many swaps?
n=O(n)
How much space?
In-place algorithm
2.2.3. Bubble Sort
Bubble sort is the simplest algorithm to implement and the slowest algorithm on very large inputs.
Basic Idea:
Loop through array from i=0 to n and swap adjacent elements if they are out of order.
Implementation: void bubble_sort(list[]) { int i,j,temp; for(i=0;i<n; i++){ for(j=n-1;j>i; j--){ if(list[j]<list[j-1]){ temp=list[j]; list[j]=list[j-1]; list[j-1]=temp; }//swap adjacent elements }//end of inner loop }//end of outer loop }//end of bubble_sort
Analysis of Bubble Sort
How many comparisons?
(n-1)+(n-2)+…+1= O(n^2 )
How many swaps?
