Lecture 2: Complexity - CM22008: Algorithms and Complexity

Measuring Program Runtime

Stopwatch (Idea 1): Ineffective due to the speed of modern computers.
Using the computer's time command (Idea 2): Provides insufficient resolution.
Instrumenting the code (Idea 3):
- Use System.nanoTime() to measure execution time.
- Example:
  java long start = System.nanoTime(); long end = System.nanoTime(); System.out.println("That took " + (end-start) +" ns.");
- Inconsistent results may occur due to various factors.
Analyzing the Code (Idea 4): Determine the number of operations the code uses.

Analyzing Code Operations

Example code:

for (int i=0; i<n; i++) {
  for (int j=0; j<n; j++) {
    System.out.println("Hello again, world!");
  }
}

Outer loop analysis:
- Initialization: 1
- Comparison: $n + 1$
- Increment: $n$
- Total: $2n + 2$
Inner loop analysis:
- Runs $n$ times.
- Each time:
 - Initialize j: 1
 - Compare j<n: $n + 1$
 - Increment j: $n$
 - Print: $n$
 - Total: $3n + 2$
- Repeated $n$ times: $3n^2 + 2n$
Overall total: $3n^2 + 4n + 2$
Focus on the dominant term for simplicity.

Graphing Runtime

Graphs illustrate the growth of runtime as input size increases.
The dominant term dictates the overall shape of the curve.
Comparing $3n^2 + 4n + 2$ to $3n^2$ shows that the dominant term is significant.

Dominant Terms and Constant Factors

Dominant terms (e.g., $3x^2$ in $3x^2 + 4x$ ) dictate the growth rate.
Constant factors are less important because:
- Operation counts do not precisely translate to runtime.
- Faster computers or optimized compilers can alter the time by a constant factor.

Asymptotic Complexity: Big-O Notation

Definition: Given a function $g(n)$ , $f(n)$ is $O(g(n))$ if there exist constants c > 0 and $n0 ≥ 0$ such that for all $n ≥ n0$ , $0 ≤ f(n) ≤ cg(n)$ .
$f(n) = O(g(n))$ means that eventually, $f(n)$ is smaller than $g(n)$ when $g(n)$ is scaled by a constant factor.

Big-O Examples

$n^2$ is $O(n^3)$ , $O(n^4)$ , etc.
$n^2$ is $O(n^2)$
$3n^2$ is $O(n^2)$ (constant factors don't matter).
$3n^2 + 2n + 1$ is $O(n^2)$
$1000n^2$ is not $O(n)$
$n^3$ is $O(2^n)$ , but $2^n$ is not $O(n^2)$
$\log n$ is $O(n)$
$n \log n$ is $O(n^2)$
$O(1)$ means bounded by a constant.

Heuristics for Big-O

Exponentials grow faster than polynomials.
Polynomials grow faster than logarithms.
Higher-base exponentials grow faster than lower-base exponentials.
Higher-degree polynomials grow faster than lower-degree polynomials.
Logarithms are asymptotically the same regardless of their base.

Big-O Growth Comparisons

$3^n$ grows faster than $2^n$ .
$2^n$ grows faster than $n^2$ .
$n^2$ grows faster than $\log n$ .
$(\log n)^2$ is even slower than $\log n^2$ .
When terms are added, only the fastest-growing term matters.
Constant factors do not matter.

Terrible Sorting Algorithm

Selection Sort:
- Find the minimum element in $a1, …, an$ and swap it with $a_1$ .
- Find the minimum element in $a2, …, an$ and swap it with $a_2$ .
- Continue this process.

Java Implementation:

for (int i=0; i<a.length; i++){
  int j = findMin(a,i);
  swap(a,i,j);
}

public static int findMin(int[] a, int i) {
  int m=i;
  for (int j=i; j<a.length; j++) {
    if (a[j]<a[m]) m = j;
  }
  return m;
}

Complexity Analysis of Selection Sort

The loop runs $n$ times.
Each time, it computes a minimum and performs constant-time operations (swap, increment).
Computing the minimum of $n$ elements takes $n$ steps.
Total time complexity: $T(n) = n + k + (n-1) + k + … + 2 + k + 1 + k$
Simplified: $T(n) = \frac{n(n+1)}{2} + nk$
Therefore, $T(n) = O(n^2)$

Big-O: Algorithm Complexity Definition

An algorithm has $O(g(n))$ complexity if the number of operations to compute a solution on input size $n$ is $O(g(n))$ .
Selection sort has $O(n^2)$ complexity (quadratic complexity).
$O(5n^2 + 9)$ , $O(n^2/1203 - 14n)$ are equivalent to $O(n^2)$ .
Big-O's robustness to constant factors means that the implementation language (Java, Python) or the speed of the computer does not affect the $O(n^2)$ complexity.

Shortcuts in Complexity Calculation

Constant-time operations within a loop are dominated by the $O(n)$ operation (finding the minimum).
Running an $O(n)$ operation $n$ times gives $O(n^2)$ complexity.

Overestimation in Big-O

Shortcuts may overestimate the complexity if the $O(n)$ operation inside the loop is faster than linear time.
Concluding that the algorithm is $O(n^2)$ remains correct because big-O provides an overestimate.
For selection sort, it's not an overestimate.

Big-Omega and Big-Theta

Big-O: Upper bound on the rate of growth.
Big-Omega (Ω): Lower bound on the rate of growth.
- Given $g(n)$ , $f(n)$ is $Ω(g(n))$ if there exist constants c > 0 and $n0 ≥ 0$ such that for all $n ≥ n0$ , $0 ≤ cg(n) ≤ f(n)$ .
Big-Theta (Θ): Exact (asymptotic) complexity.
- Given $g(n)$ , $f(n)$ is $Θ(g(n))$ if it is both $O(g(n))$ and $Ω(g(n))$ .
- There are positive constants $c1 > 0$ , c2 > 0, and $n0 ≥ 0$ such that for all $n ≥ n0$ , $0 ≤ c1g(n) ≤ f(n) ≤ c2g(n)$ .

Big-O, Big-Omega, and Big-Theta Examples

Big-O examples work as Big-Omega in reverse. For example, $n^3$ is $Ω(n^2)$ .
$2^n$ is $Ω(n^k)$ for any fixed $k$ .
A polynomial $n^3 + 2n + 7$ is $Θ(n^3)$ .
Selection sort's running time is $Θ(n^2)$ .

Summary

Asymptotic complexity is a more robust measure of runtime than direct measurement.
Big-O expresses upper bounds on running time.
Big-Omega expresses lower bounds.
Big-Theta expresses exact bounds (both upper and lower).
Selection sort is a $Θ(n^2)$ sorting algorithm.

Quiz Questions

Suppose we have two algorithms A and B. A is $O(n)$ ; B is $O(\log n)$ .

What’s the complexity of an algorithm that runs algorithm A exactly three times?
- $O(n)$
What’s the complexity of running A then B?
- $O(n)$
What’s the complexity of running B $n$ times?
- $O(n \log n)$
What’s the complexity of running A twice, then B $n$ times?
- $O(n \log n)$
What’s the complexity of running A twice, then B $n^2$ times?
- $O(n^2 \log n)$