Download Big O Notation Cheat Sheet and more Cheat Sheet Data Structures and Algorithms in PDF only on Docsity!
Big O notation (with a capital letter O, not a zero), also called Landau's symbol , is a symbolism used in complexity theory, computer science, and mathematics to describe the asymptotic behavior of functions. Basically, it tells you how fast a function grows or declines.
Landau's symbol comes from the name of the German number theoretician Edmund Landau who invented the notation. The letter O is used because the rate of growth of a function is also called its order.
For example, when analyzing some algorithm, one might find that the time (or the number of steps) it takes to complete a problem of size n is given by T ( n ) = 4 n^2 - 2 n + 2. If we ignore constants (which makes sense because those depend on the particular hardware the program is run on) and slower growing terms, we could say " T ( n ) grows at the order of n^2 " and write: T ( n ) = O( n^2 ).
In mathematics, it is often important to get a handle on the error term of an approximation. For instance, people will write
ex^ = 1 + x + x^2 / 2 + O( x^3 ) for x -> 0
to express the fact that the error is smaller in absolute value than some constant times x^3 if x is close enough to 0.
For the formal definition, suppose f ( x ) and g ( x ) are two functions defined on some subset of the real numbers. We write
f ( x ) = O( g ( x ))
(or f ( x ) = O( g ( x )) for x -> to be more precise) if and only if there exist constants N and C such that
| f ( x )| C | g ( x )| for all x > N.
Intuitively, this means that f does not grow faster than g.
If a is some real number, we write
f ( x ) = O( g ( x )) for x -> a
if and only if there exist constants d > 0 and C such that
| f ( x )| C | g ( x )| for all x with | x - a | < d.
The first definition is the only one used in computer science (where typically only positive functions with a natural number n as argument are considered; the absolute values can then be ignored), while both usages appear in mathematics.
Here is a list of classes of functions that are commonly encountered when analyzing algorithms. The slower growing functions are listed first. c is some arbitrary constant.
notation name O(1) constant O(log( n )) logarithmic O((log( n ))c) polylogarithmic O( n ) linear O( n^2 ) quadratic O( nc ) polynomial O( cn ) exponential
Note that O( nc ) and O( cn ) are very different. The latter grows much, much faster, no matter how big the constant c is. A function that grows faster than any power of n is called superpolynomial. One that grows slower than an exponential function of the form cn^ is called subexponential. An algorithm can require time that is both superpolynomial and subexponential; examples of this include the fastest algorithms known for integer factorization.
Note, too, that O(log n ) is exactly the same as O(log( nc )). The logarithms differ only by a constant factor, and the big O notation ignores that. Similarly, logs with different constant bases are equivalent.
The above list is useful because of the following fact: if a function f ( n ) is a sum of functions, one of which grows faster than the others, then the faster growing one determines the order of f ( n ).
Example: If f ( n ) = 10 log( n ) + 5 (log( n ))^3 + 7 n + 3 n^2 + 6 n^3 , then f ( n ) = O( n^3 ).
One caveat here: the number of summands has to be constant and may not depend on n. This notation can also be used with multiple variables and with other expressions on the right side of the equal sign. The notation:
f ( n , m ) = n^2 + m^3 + O( n + m )
represents the statement:
∃ C ∃ N ∀ n , m > N : f ( n , m ) n^2 + m^3 + C ( n + m )
A common error is to confuse these by using O when is meant. For example, one might say "heapsort is O( n log n )" when the intended meaning was "heapsort is ( n log n )". Both statements are true, but the latter is a stronger claim.
The notations described here are very useful. They are used for approximating formulas for analysis of algorithms, and for the definitions of terms in complexity theory (e.g. polynomial time).
How efficient is an algorithm or piece of code? Efficiency covers lots of resources, including:
- CPU (time) usage
- memory usage
- disk usage
- network usage
All are important but we will mostly talk about time complexity (CPU usage).
Be careful to differentiate between:
- Performance: how much time/memory/disk/... is actually used when a program is run. This depends on the machine, compiler, etc. as well as the code.
- Complexity: how do the resource requirements of a program or algorithm scale, i.e., what happens as the size of the problem being solved gets larger?
Complexity affects performance but not the other way around.
The time required by a function/procedure is proportional to the number of "basic operations" that it performs. Here are some examples of basic operations:
- one arithmetic operation (e.g., +, *).
- one assignment (e.g. x := 0)
- one test (e.g., x = 0)
- one read (of a primitive type: integer, float, character, boolean)
- one write (of a primitive type: integer, float, character, boolean)
Some functions/procedures perform the same number of operations every time they are called. For example, StackSize in the Stack implementation always returns the number of elements currently in the stack or states that the stack is empty, then we say that StackSize takes constant time.
Other functions/ procedures may perform different numbers of operations, depending on the value of a parameter. For example, in the BubbleSort algorithm, the number of elements in the array, determines the number of operations performed by the algorithm. This parameter (number of elements) is called the problem size / input size.
T(N) = 3 * N^2 + 5.
We can show that T(N) is O(N^2 ) by choosing c = 4 and n 0 = 2.
This is because for all values of N greater than 2:
3 * N^2 + 5 <= 4 * N^2
T(N) is not O(N), because whatever constant c and value n 0 you choose, There is always a value of N > n 0 such that (3 * N^2 + 5) > (c * N)
In general, how can you determine the running time of a piece of code? The answer is that it depends on what kinds of statements are used.
Sequence of statements
statement 1; statement 2; ... statement k;
The total time is found by adding the times for all statements:
total time = time(statement 1) + time(statement 2) + ... + time(statement k)
If each statement is "simple" (only involves basic operations) then the time for each statement is constant and the total time is also constant: O(1).
If-Then-Else
if (cond) then block 1 (sequence of statements) else block 2 (sequence of statements) end if;
Here, either block 1 will execute, or block 2 will execute. Therefore, the worst-case time is the slower of the two possibilities:
max(time(block 1), time(block 2))
If block 1 takes O(1) and block 2 takes O(N), the if-then-else statement would be O(N).
Loops
for I in 1 .. N loop sequence of statements end loop;
The loop executes N times, so the sequence of statements also executes N times. If we assume the statements are O(1), the total time for the for loop is N * O(1), which is O(N) overall.
Nested loops
for I in 1 .. N loop for J in 1 .. M loop sequence of statements end loop; end loop;
The outer loop executes N times. Every time the outer loop executes, the inner loop executes M times. As a result, the statements in the inner loop execute a total of N * M times. Thus, the complexity is O(N * M).
In a common special case where the stopping condition of the inner loop is instead
of (i.e., the inner loop also executes N times), the total complexity for the two loops
is O(N^2 ).
Statements with function/ procedure calls
When a statement involves a function/ procedure call, the complexity of the statement includes the complexity of the function/ procedure. Assume that you know that function/ procedure f takes constant time, and that function/procedure g takes time proportional to (linear in) the value of its parameter k. Then the statements below have the time complexities indicated.
f(k) has O(1) g(k) has O(k)
When a loop is involved, the same rule applies. For example:
for J in 1 .. N loop g(J); end loop;
