Download Algorithm Complexity I, Algorithms Efficiency - Notes | CMSC 132 and more Study notes Computer Science in PDF only on Docsity!
1
CMSC 132:
Object-Oriented Programming II
Algorithmic Complexity I
Department of Computer Science
University of Maryland, College Park
2
Algorithm Efficiency
Efficiency
Amount of resources used by algorithm Time, space
Measuring efficiency
Benchmarking Asymptotic analysis
3
Benchmarking
Approach
Pick some desired inputs Actually run implementation of algorithm Measure time & space needed
Industry benchmarks
SPEC – CPU performance MySQL – Database applications WinStone – Windows PC applications MediaBench – Multimedia applications Linpack – Numerical scientific applications
4
Benchmarking
Advantages
Precise information for given configuration Implementation, hardware, inputs
Disadvantages
Affected by configuration Data sets (usually too small) Hardware Software Affected by special cases (biased inputs) Does not measure intrinsic efficiency
5
Asymptotic Analysis
Approach
Mathematically analyze efficiency Calculate time as function of input size n T ≈ O[ f(n) ] T is on the order of f(n) “Big O” notation
Advantages
Measures intrinsic efficiency Dominates efficiency for large input sizes
6
Search Example
Number guessing game
Pick a number between 1…n Guess a number Answer “correct”, “too high”, “too low” Repeat guesses until correct number guessed
7
Linear Search Algorithm
Algorithm
- Guess number = 1
- If incorrect, increment guess by 1
- Repeat until correct
Example
Given number between 1… Pick 20 Guess sequence = 1, 2, 3, 4 … 20 Required 20 guesses
8
Linear Search Algorithm
Analysis of # of guesses needed for 1…n
If number = 1, requires 1 guess If number = n, requires n guesses On average, needs n/2 guesses Time = O( n ) = Linear time
9
Binary Search Algorithm
Algorithm
Set ∆ to n/ Guess number = n/ If too large, guess number – ∆ If too small, guess number + ∆ Reduce ∆ by ½ Repeat until correct
10
Binary Search Algorithm
Example Given number between 1… Pick 20 Guesses = 50, ∆ = 25, Answer = too large, subtract ∆ 25, ∆ = 12 , Answer = too large, subtract ∆ 13, ∆ = 6, Answer = too small, add ∆ 19, ∆ = 3, Answer = too small, add ∆ 22, ∆ = 1, Answer = too large, subtract ∆ 21, ∆ = 1, Answer = too large, subtract ∆ 20 Required 7 guesses
11
Binary Search Algorithm
Analysis of # of guesses needed for 1…n
If number = n/2, requires 1 guess If number = 1, requires log 2 ( n ) guesses If number = n, requires log 2 ( n ) guesses On average, needs log 2 ( n ) guesses Time = O( log 2 ( n ) ) = Log time
12
Search Comparison
For number between 1…
Simple algorithm = 50 steps Binary search algorithm = log 2 ( n ) = 7 steps
For number between 1…100,
Simple algorithm = 50,000 steps Binary search algorithm = log 2 ( n ) (about 17 steps)
Binary search is much more efficient!
19
Big-O Notation
Represents
Upper bound on number of steps in algorithm For sufficiently large input size Intrinsic efficiency of algorithm for large inputs
f(n)
O(…)
input size
# steps
20
Formal Definition of Big-O
Function f(n) is Ο( g(n) ) if
For some positive constants M, N 0 M × g(n) ≥ f(n), for all n ≥ N 0
Intuitively
For some coefficient M & all data sizes ≥ N 0 M × g(n) is always greater than f(n)
21
Big-O Examples
5n + 1000 ⇒ O(n)
Select M = 6, N 0 = 1000 For n ≥ 1000 6n ≥ 5n+1000 is always true Example ⇒ for n = 1000 6000 ≥ 5000 +
22
Big-O Examples
2n^2 + 10n + 1000 ⇒ O(n^2 )
Select M = 4, N 0 = 100 For n ≥ 100 4n 2 ≥ 2n 2 + 10n + 1000 is always true Example ⇒ for n = 100 40000 ≥ 20000 + 1000 + 1000
23
Observations
Big O categories
O(log(n)) O(n) O(n 2 )
For large values of n
Any O(log(n)) algorithm is faster than O(n) Any O(n) algorithm is faster than O(n^2 )
Asymptotic complexity is fundamental measure
of efficiency
24
Comparison of Complexity
25
Complexity Category Example
0
50
100
150
200
250
300
2 3 4 5 6 7 8 Problem Size
of Solution Steps
2^n n^ nlog(n) n log(n)
26
Complexity Category Example
1
10
100
1000
2 3 4 5 6 7 8 Problem Size
of Solution Steps
2^n n^ nlog(n) n log(n)
27
Calculating Asymptotic Complexity
As n increases
Highest complexity term dominates Can ignore lower complexity terms
Examples
2 n + 100 ⇒ O(n) n log(n) + 10 n ⇒ O(nlog(n)) ½ n^2 + 100 n ⇒ O(n^2 ) n 3 + 100 n 2 ⇒ O(n^3 ) 1/100 2 n^ + 100 n^4 ⇒ O(2 n)
28
Complexity Examples
2n + 100 ⇒ O(n)
0
100000
200000
300000
400000
500000
600000
700000
800000
(^1349120260533) (^10682118417582081611131602)
n nlog(n) 2 n + 100
29
Complexity Examples
½ n log(n) + 10 n ⇒ O(nlog(n))
0
100000
200000
300000
400000
500000
600000
700000
800000
(^22879) (^178373756150629755855) 115012256544252
n nlog(n) 1/2 n log(n) + 10 n
30
Complexity Examples
½ n^2 + 100 n ⇒ O(n^2 )
0
100000
200000
300000
400000
500000
600000
700000
800000
(^22879) (^178373756150629755855) 115012256544252
nlog(n) n^2 1/2 n^2 + 100 n
37
Types of Case Analysis
Average case
Number of steps required for “typical” case Most useful metric in practice Different approaches Average case Expected case
38
Approaches to Average Case
Average case
Average over all possible inputs Assumes all inputs have the same probability Example Case 1 = 10 steps, Case 2 = 20 steps Average = 15 steps
Expected case
Weighted average over all possible inputs Based on probability of each input Example Case 1 (90%) = 10 steps, Case 2 (10%) = 20 steps Average = 11 steps
39
Average Case Example
Example problem
Average # of comparisons needed to find a number in the (sorted) array A[ ] = {1, 4, 8, 12, 15} using
- Linear search Start from beginning, compare elements one at a time
- Binary search Start from middle of array at index k, compare element If not element, repeat for top or bottom half of remaining array depending on whether element is smaller or greater than A[k]
40
Average Case : Linear Search
Algorithm
- Find # of comparisons needed for each case 1 → 1 comparison (1) 4 → 2 comparisons (1, 4) 8 → 3 comparisons (1, 4, 8) 12 → 4 comparisons (1, 4, 8, 12) 15 → 5 comparisons (1, 4, 8, 12, 15)
- Calc average = total # of comparisons / # cases Total # comparisons = 1 + 2 + 3 + 4 + 5 = 15
cases = 5
Average = 3 comparisons / number
41
Average Case : Binary Search
Algorithm
- Find # of comparisons needed for each case 1 → 3 comparisons (8, 4, 1) 4 → 2 comparisons (8, 4) 8 → 1 comparisons (8) 12 → 2 comparisons (8, 12) 15 → 3 comparisons (8, 12, 15)
- Calc average = total # of comparisons / # cases Total # comparisons = 3 + 2 + 1 + 2 + 3 = 11
cases = 5
Average = 2.2 comparisons / number 42
Average Case Example
Example problem 2
Average # of comparisons needed to find a number in a sorted array A[ n ] of size n using
- Linear search
- Binary search
For simplicity, we assume elements are stored in A[1] … A[n]
43
Average Case : Linear Search
Algorithm
- Find # of comparisons needed for each case A[1] → 1 comparison (A[1]) A[2] → 2 comparisons (A[1], A[2]) … A[n] → n comparisons (A[1] … A[n])
- Calc average = total # of comparisons / # cases Total # comparisons = 1 + 2 + … + n = ½ n 2 + 1
cases = n
Average ≈ ½ n comparisons / number
44
Average Case : Binary Search
Algorithm
- Find # of comparisons needed for each case A[n/2] → 1 comp (A[n/2]) A[n/4], A[3n/4] → 2 comps (A[n/2], A[n/4]) … A[1], A[3]…A[n] → log 2 (n) comparisons (A[n/2], A[n/4], A[n/8]…A[1])
- Calc average = total # of comparisons / # cases Total # comparisons = n/2 * log 2 (n) + n/4 * log 2 (n)–1 + … + 1 = n log 2 (n)
cases = n
Average ≈ log 2 (n) comparisons / number
45
Amortized Analysis
Approach
Applies to worst-case sequences of operations Finds average running time per operation Example Normal case = 10 steps Every 10 th^ case may require 20 steps Amortized time = 11 steps
Assumptions
Can predict possible sequence of operations Know when worst-case operations are needed Does not require knowledge of probability
46
Amortization Example
Adding numbers to end of array of size k
If array is full, allocate new array Allocation cost is O(size of new array) Copy over contents of existing array
Two approaches
Non-amortized If array is full, allocate new array of size k+ Amortized If array is full, allocate new array of size 2k Compare their allocation cost
47
Amortization Example
Non-amortized approach
Allocation cost as table grows from 1..n
Total cost ⇒ n(n+1)/
Case analysis
Best case ⇒ allocation cost = k Worse case ⇒ allocation cost = k Amortized case ⇒ allocation cost = (n+1)/
6
6 Cost 1 2 3 4 5 7 8
Size (k) 1 2 3 4 5 7 8
48
Amortization Example
Amortized approach
Allocation cost as table grows from 1..n
Total cost ⇒ 2 (n – 1)
Case analysis
Best case ⇒ allocation cost = 0 Worse case ⇒ allocation cost = 2(k – 1) Amortized case ⇒ allocation cost = 2
An individual step might take longer, but faster
for any sequence of operations
0
6 Cost 2 0 4 0 8 0 0
Size (k) 1 2 3 4 5 7 8
