Week 5: Quicksort, Lower bound, Greedy, Schemes and Mind Maps of Data Structures and Algorithms

Download Week 5: Quicksort, Lower bound, Greedy and more Schemes and Mind Maps Data Structures and Algorithms in PDF only on Docsity!

Week 5: Quicksort, Lower bound, Greedy

Agenda:

• Quicksort: Average case

• Lower bound for sorting

• Greedy method

Week 5: Quicksort

Recall Quicksort:

• The ideas:
  • Pick one key
  • Compare to others: partition into ‘smaller’ and ‘greater’ sublists
  • Recursively sort two sublists
• Pseudocode:

procedure Quicksort(A, p, r)

if p < r then q ← Partition(A, p, r) Quicksort(A, p, q − 1) Quicksort(A, q + 1, r)

procedure Partition(A, p, r) ** A[r] is the key picked to do the partition

x ← A[r] i ← p − 1 for j ← p to r − 1 do if A[j] ≤ x then i ← i + 1 exchange A[i] ↔ A[j] exchange A[i + 1] ↔ A[r] return i + 1

Week 5: Quicksort

Quicksort correctness:

• It follows from the correctness of Partition.
• Partition correctness:
  • Loop invariant: At the start of for loop:

1. A[p..i] ≤ A[r] — A[s] ≤ A[r], p ≤ s ≤ i
2. A[(i + 1)..(j − 1)] > A[r]
3. x = A[r]
   • Proof of LI: (pages 147 – 148)
     1. Initialization
     2. Maintenance
     3. Termination
   • LI correctness implies Partition correctness

• Why we study QuickSort and its analysis:
  • very efficient, in use
  • divide-and-conquer, randomization
  • huge literature
  • a model for analysis of algorithms

Week 5: Quicksort analysis

Quicksort recursion tree:

• Observations:
  • (Again) key comparison is the dominant operation
  • Counting KC — only need to know (at each call) the rank of the split key
• An example: 06

04 09

02 05 07 12

01 03 08 11 13

10 14

15

• More observations:
  • In the resulting recursion tree, at each node (all keys in left subtree) ≤ (key in this node) < (all keys in right subtree)
  • 1-1 correspondence: quicksort recursion tree ←→ binary search tree

Week 5: Quicksort analysis

Quicksort BC running time:

• Notice that when both subarrays are non-empty, we will be saving 1 KC ...
• Best case: each partition is a bipartition !!! Saving as many KC as possible every level ... The recursion tree is as short as possible ...
• Recurrence:

T (n) = 2 × T (

n − 1 2

) + (n − 1),

• Solving the recurrence — apply Master Theorem? not exactly T (n) ∈ Θ(n log n)
• Question:
  • What is the best case array? for n = 7?
• Conclusion:
  • In order to save time, A[n] better BI-partitions the array ... — usually it might not bipartition ... we will push it by a technique called randomization (future lectures)

Week 5: Quicksort

Quicksort BC running time (cont’d):

• In the recursion tree, what is the number of KC at each level? Answer: - n − 1 at the top level - at most 2 nodes at the 2nd level, at least (n 1 − 1) + (n − 1 − n 1 − 1) = n − 3 KC - at most 4 nodes at the 3rd level, at least (n 1 − 3) + (n − 1 − n 1 − 3) = n − 7 KC -... - at kth level, at most 2k−^1 nodes, at least n − 2 k^ + 1 KC
• How many levels are there? Answer: - At least lg n levels — binary tree
• So, at least we need

∑lg n− 1

i=1 (n^ −^2

i (^) + 1) KC, and

lg ∑ n− 1

i=

(n − 2 i^ + 1) = (n + 1)(lg n − 1) − (n − 2) ∈ Θ(n log n)

• Try n = 2k^ − 1 to get the closed form for the following recur- rence

T (n) =

0 , if n = 1 (n − 1) + T (bn− 2 1 c) + T (dn− 2 1 e), if n ≥ 2

Week 5: Quicksort

Solving T (n):

T (n) = (^) n^1 (T (0) + T (n − 1))

• (^1) n (T (1) + T (n − 2))

+...

• (^1) n (T (n − 2) + T (1))

• (^1) n (T (n − 1) + T (0))

• (n − 1)

= (^) n^2

∑n− 1

i=0 T^ (i) + (n^ −^ 1)

• Therefore,
  • n × T (n) = 2

∑n− 1

i=0 T^ (i) +^ n(n^ −^ 1)

• (n − 1) × T (n − 1) = 2

∑n− 2

i=0 T^ (i) + (n^ −^ 1)(n^ −^ 2)

• Subtract the two terms:

n × T (n) − (n − 1) × T (n − 1) = 2T (n − 1) + 2(n − 1)

Rearrange it:

nT (n) = (n + 1)T (n − 1) + 2(n − 1)

Week 5: Quicksort

Solving T (n) (cont’d):

• Or we can say: T (n) n+1 =^

T (n−1) n +^

2(n−1) n(n+1)

= T^ (n n− 1)+ (^) n+1^2 − 2(^1 n − (^) n+1^1 )

= T^ (n n− 1)+ (^) n+1^4 − (^2) n

which gives you (iterated substitution)

T (n) n + 1

∑^ n

i=

i + 1

n + 1

Recall that

∑n

i=

1 i =^ Hn^ = ln^ n^ +^ γ^ — the Harmonic number where γ ≈ 0. 577 · · ·

• So, from

T (n) n + 1

∑^ n

i=

i + 1

n + 1

we have T (n) = 2(n + 1)Hn+1 − (4n + 2)

≈ 2(n + 1)(ln(n + 1) + γ) − (4n + 2)

∈ Θ(n log n)

• Conclusion: Quicksort AC running time in Θ(n log n).

Week 5: Lower Bounds for Comparison-Based Sorting

Two useful trees in algorithm analysis:

• Recursion tree
  • node ←→ recursive call
  • describes algorithm execution for one particular input by showing all calls made
  • one algorithm execution ←→ all nodes (a tree)
  • useful in analysis: sum the numbers of operations over all nodes

Week 5: Lower Bounds for Comparison-Based Sorting

Recursion tree example:

• Mergesort pseudocode

Merge(A; lo, mid, hi) **pre-condition: lo ≤ mid ≤ hi **pre-condition: A[lo, mid] and A[mid + 1, hi] sorted **post-condition: A[lo, hi] sorted

MergeSort(A; lo, hi)

if lo < hi then mid ← b(lo + hi)/ 2 c MergeSort(A; lo, mid) MergeSort(A; mid + 1, hi) Merge(A; lo, mid, hi)

[1]

[2] [3]

[4]

[5] [6]

• • For different input instance, the number of operations at each node could be different.

Week 5: Lower Bounds for Comparison-Based Sorting

Selectionsort decision tree:

• Assume input keys in array A[1..3] = {a, b, c}
• Tree node: if A[k] > A[j] — 2-way key comparison
• Node label A[j]

SelectionSort(A; n)

if n ≥ 1 then for j ← n downto 2 do psn ← j for k ← j − 1 downto 1 do if A[k] > A[psn] then psn ← k exchange A[j] ↔ A[psn] return A[2] > A[3]?

No. Yes.

abc bac bca acb cab cba

• In every case — whatever input instance is, 3 KC !!!

Week 5: Lower Bounds for Comparison-Based Sorting

Sorting lower bound:

• Comparison-based sort: keys can be (2-way) compared only!
• This lower bound argument considers only the comparison- based sorting algorithms. For example, - Insertionsort, Mergesort, Heapsort, Quicksort,
• Binary tree facts:
  • Suppose there are t leaves and k levels. Then,
  • t ≤ 2 k−^1
  • So, lg t ≤ (k − 1)
  • Equivalently, k ≥ 1 + lg t — binary tree with t leaves has at least (1 + lg t) levels
• Comparison-based sorting algorithm facts:
  • Look at its Decision Tree. We have,
  • It’s a binary tree.
  • It should contain every possible permutation of the posi- tions { 1 , 2 ,... , n}.
  • So, it contains at least n! leaves ...
  • Equivalently, it has at least 1 + lg(n!) levels.
  • A longest root-to-leaf path of length at least lg(n!).
  • The worst case number of KC is at least lg(n!).
  • lg(n!) ∈ Θ(n log n)
• Therefore, Mergesort and Heapsort are asymptotically optimal (comparison-based) sorting algorithms.

Week 5: Greedy method

Example 1: Fractional Knapsack

• Suppose we have a set S of n items, each with a profit/value bi and weight wi.
• We also have a knapsack of capacity W ,
• Assume that each item can be picked at any fraction, that is we can pick 0 ≤ xi ≤ wi amount of item i.
• Our goal is to fill the knapsack (without exceeding its capac- ity) with a combination of the items with maximum profit.
• Formally, find 0 ≤ xi ≤ wi for 1 ≤ i ≤ n such that

∑n

i=1 xi^ ≤^ W and

∑n

i=

xi wi ×^ bi^ is maximized.

• Greedy idea: start picking the items with more “value”:

value ≡

bi wi So let vi = (^) wbii. The algorithm will be as follows:

Week 5: Greedy method

• The pseudocode is:

Procedure Frac-Knapsack (S, W )

for i ← 1 to n do xi ← 0 vi ← (^) wbii CurrentW ← 0 While CurrentW < W do let ai be the next item in S with highest value xi ← min{wi, W − CurrentW } add xi amount of i to knapsack CurrentW ← CurrentW + xi

• How to find next highest value in each step?
• One way is to sort S at the begining based on vi’s in non- increasing order.
• Another way is to keep a PQ (max-heap) based on values.
• Since we check at most n items, the total time is O(n log n).

Correctness of the algorithm:

• We prove, for all i ≥ 0, that if we have picked x 1 ,... , xi from items 1,... , i in the first i iterations (respectively), then this partial solution can be extended to an optimal solution.
• In other words, there is some optimal solution call it OPT which has xj amount from item j for 1 ≤ j ≤ i.