Lecture 03 - Searching and Sorting Algorithms (Data Structures)

Built-in ADTs in Languages

Modern languages provide efficient, well-tested implementations of common data structures in standard libraries.
As professional programmers, you rarely implement these from scratch, but understanding how they work "under the hood" helps:
- Choose the right data structure for your needs
- Recognize and avoid performance pitfalls
- Build your own when built-ins fall short
Analogy: learning how an engine works makes you a better driver, mechanic, and problem solver, even if you don’t hand-build an engine.

Language Core Library vs ADT Examples

Python: Core Library includes list, set, dict
C++: STL (Standard Template Library) includes vector, list, stack, queue, deque, set, map
Java: Java Collections Framework (JCF) includes Collection, Set, List, Map, Queue, Deque

Searching and Sorting: Overview

Searching algorithm: a method to locate a specific item/value within a data collection or defined space.
Basic use cases: search for values (e.g., a number in an array, a key in a dictionary) using techniques like linear search, binary search, or hash lookups.
Beyond the list: searching can mean exploring a solution space (e.g., maze paths, puzzles, function optimization) where each position is a candidate solution.
Algorithmic goal: efficiently reduce the number of possibilities by exploiting structure (ordering, heuristics, probability) to find a valid or optimal solution within acceptable time/space bounds.

Linear Search

Definition: sequential scan through all elements.
Pseudocode (conceptual):
- LinearSearch(values, key):
- i = 0
- for each value in values:
  - if value == key: return i
  - else: i = i + 1
- return -1
Complexity:
- Best case: $O(1)$
- Average case: $O(N)$
- Worst case: $O(N)$

Binary Search

Prerequisite: data must be sorted.
Principle: each step halves the search space.
Pseudocode (iterative):
- BinarySearch(values, key):
- low = 0
- high = size(values) - 1
- while high >= low:
 - mid = (high - low) / 2 + low
 - if values[mid] < key: low = mid + 1
 - else if values[mid] > key: high = mid - 1
 - else: return mid
- return -1
Complexity:
- Best case: $O(1)$
- Average case: $O(abla N)$
- Worst case: $O(abla N)$
- Note: $ abla$ O(\, ext{log} \, N) $</li></ul></li></ul><h3 id="4e1eb666-1a32-442d-bd9f-d6b935bc033e" data-toc-id="4e1eb666-1a32-442d-bd9f-d6b935bc033e" collapsed="false" seolevelmigrated="true">Binary Search (Recursive)</h3><ul><li>Recursive version: BinarySearch(values, key, low, high)<ul><li>if high < low: return -1</li><li>mid = (high - low) / 2 + low</li><li>if values[mid] < key: return BinarySearch(values, key, mid + 1, high)</li><li>else if values[mid] > key: return BinarySearch(values, key, low, mid - 1)</li><li>else: return mid</li></ul></li><li>Call: BinarySearch(values, key, 0, size(values) - 1)</li></ul><h3 id="4f8812be-1d33-497f-a269-963323d52e7f" data-toc-id="4f8812be-1d33-497f-a269-963323d52e7f" collapsed="false" seolevelmigrated="true">Other Search Techniques (as listed in the slides)</h3><ul><li>Interpolation Search: heuristic that uses the value distribution to guess where the key might be (details not spelled out in the transcript).</li><li>Jump Search: jump ahead by fixed-size blocks and then linear search within the block (details not specified in the transcript).</li><li>Exponential Search: expands a range exponentially to locate a block where the key may reside (details not specified in the transcript).</li><li>Note: The transcript explicitly labels these strategies but provides little to no complexity details beyond names.</li></ul><h3 id="a9e126c0-30aa-4637-9141-212d1168389b" data-toc-id="a9e126c0-30aa-4637-9141-212d1168389b" collapsed="false" seolevelmigrated="true">Sorting: Divide and Conquer and Recursion Concepts</h3><ul><li>Theme: many sorting algorithms use divide and conquer, often via recursion, then combine sub-solutions.</li><li>Visual metaphor on the slides highlights building up from smaller pieces (e.g., cutting, merging, or partitioning treasure maps).</li><li>Core steps (Divide and Conquer):<ul><li>Divide the problem into smaller subproblems of the same type (ideally roughly equal in size).</li><li>Conquer the pieces recursively, down to a simple base case.</li><li>Combine the sub-solutions to form the overall solution.</li></ul></li><li>Efficiency comes from reducing problem size at each divide step.</li></ul><h3 id="e68c87ba-5d8f-4804-968c-acd36eaf76f3" data-toc-id="e68c87ba-5d8f-4804-968c-acd36eaf76f3" collapsed="false" seolevelmigrated="true">Insertion Sort</h3><ul><li>Builds a sorted list one item at a time.</li><li>It is the only quadratic runtime sort shown as a practical, non-educational example in production contexts (per slide).</li><li>Pseudocode concept:<ul><li>for i = 1 to array.size - 1:</li><li>j = i</li><li>while j > 0 and array[j] < array[j - 1]:<ul><li>swap array[j] with array[j - 1]</li><li>j = j - 1</li></ul></li></ul></li><li>Complexity:<ul><li>Best case:$ O(N) $</li><li>Average case:$ O(N^2) $</li><li>Worst case:$ O(N^2) $</li></ul></li></ul><h3 id="885e1b3f-c8b1-4264-9992-2eca4bff5aab" data-toc-id="885e1b3f-c8b1-4264-9992-2eca4bff5aab" collapsed="false" seolevelmigrated="true">Merge Sort</h3><ul><li>Concept: divide the array, sort halves, and merge.</li><li>Recurrence: $ T(n) = 2T(n/2) + O(n) $</li><li>Time complexity:$ O(n \, log\, n) $in all cases</li><li>Key operation: Merge two sorted halves in linear time.</li><li>Merge procedure (conceptual):<ul><li>Merge(left, right): merge two sorted subarrays into a single sorted array.</li><li>Example structure: while leftPos <= leftEnd and rightPos <= rightEnd: compare and copy the smaller element; then copy any remaining elements from either side.</li><li>Finally copy merged elements back into the original array.</li></ul></li><li>Master Theorem application (Merge Sort): a = 2, b = 2, f(n) = Θ(n) ⇒ Case 2 ⇒ T(n) = Θ(n log n).</li><li>Recursion tree visualization: cost per level doubles subproblems while total work per level remains Θ(n), yielding Θ(n log n) total.</li></ul><h3 id="5605d907-4c45-4746-a6aa-dc42a9a0575d" data-toc-id="5605d907-4c45-4746-a6aa-dc42a9a0575d" collapsed="false" seolevelmigrated="true">Quicksort</h3><ul><li>Core idea: choose a pivot, partition the array so that values less than the pivot go left and values greater go right.</li><li>Recursively sort the left and right partitions.</li><li>No separate merge step; ordering emerges from partitioning.</li><li>Pseudocode (partition):<ul><li>Partition(array, low, high):</li><li>mid = low + (high - low) / 2</li><li>pivot = array[mid]</li><li>done = false</li><li>while not done:<ul><li>while array[low] < pivot: low = low + 1</li><li>while pivot < array[high]: high = high - 1</li><li>if low >= high: done = true</li><li>else: swap array[low] with array[high]</li><li>low = low + 1</li><li>high = high - 1</li></ul></li><li>return high // last index of low partition</li></ul></li><li>Quicksort function:<ul><li>Quicksort(array, low, high):</li><li>if low >= high: return</li><li>endLow = Partition(array, low, high)</li><li>Quicksort(array, low, endLow)</li><li>Quicksort(array, endLow + 1, high)</li></ul></li><li>Complexity (average/best vs worst):<ul><li>Best/Average case:$ O(n \, log n) $</li><li>Worst case:$ O(n^2) $</li></ul></li><li>Recurrence approximation (via Master Theorem): best/average ~$ T(n) = O(n \, log n) $, worst ~$ T(n) = O(n^2) $</li></ul><h3 id="ca75f20a-856c-4c50-8864-74e61303ee76" data-toc-id="ca75f20a-856c-4c50-8864-74e61303ee76" collapsed="false" seolevelmigrated="true">Merge Sort vs Quicksort: Comparison (high-level)</h3><ul><li>Algorithm type: both are divide-and-conquer.</li><li>Worst-case time: Merge Sort$ O(n \, log n) $; Quicksort$ O(n^2) $(worst case).</li><li>Average-case time: both approx$ O(n \, log n) $(Quicksort is typically near this in practice).</li><li>Best-case time: both$ O(n \, log n) $(for typical well-behaved inputs).</li><li>Space complexity: Merge Sort uses$ O(n) $auxiliary space; Quicksort uses$ O(\,log n) $stack space on average.</li><li>Stability: Merge Sort is stable; Quicksort is not stable unless modified.</li><li>In-place: Merge Sort is not in-place by default; Quicksort is in-place (in typical implementations).</li><li>Use cases: Merge Sort for large datasets and linked lists; Quicksort for in-memory arrays and general-purpose sorting in libraries.</li><li>Library usage and implementation considerations: often used in parallel/distributed contexts (Merge) versus in-memory, general-purpose sorting (Quicksort).</li></ul><h3 id="57c8a428-4825-483d-90dd-380a43825c06" data-toc-id="57c8a428-4825-483d-90dd-380a43825c06" collapsed="false" seolevelmigrated="true">Radix Sort</h3><ul><li>Radix Sort processes digits from least significant to most significant (LSD first).<ul><li>(LSD the least significant digit is the number farthest to the right)</li><li>(MSD the most significant digit is the number farthest to the left)</li></ul></li><li>Stable sub-sorting per digit place (e.g., counting sort) is used to preserve the order of digits already sorted.</li><li>Does not rely on comparisons between keys; relies on digit bucketing.</li><li>Best for uniform, fixed-length keys (e.g., large lists of integers or strings with consistent key length).</li><li>Key components:<ul><li>RadixGetLength(value): returns number of digits in value (at least 1 for 0).</li><li>RadixGetMaxLength(array): maximum number of digits among all elements.</li><li>RadixSort(array): uses 10 buckets (for base-10) and iterates over digit positions from least to most significant, rebuilding the array after each pass. Also handles negative numbers by separating negatives and nonnegatives, reversing negatives, and concatenating.</li></ul></li><li>High-level pseudocode (outline):<ul><li>Buckets = array[0..9] of empty lists</li><li>maxDigits = RadixGetMaxLength(array)</li><li>pow10 = 1</li><li>for digitIndex = 0 to maxDigits - 1:</li><li>for i = 0 to array.size - 1:<ul><li>bucketIndex = abs(array[i] / pow10) % 10</li><li>Append array[i] to buckets[bucketIndex]</li></ul></li><li>arrayIndex = 0</li><li>for i = 0 to 9:<ul><li>for j = 0 to buckets[i].size - 1:</li><li>array[arrayIndex] = buckets[i][j]</li><li>arrayIndex = arrayIndex + 1</li></ul></li><li>pow10 = pow10 * 10</li><li>clear all buckets</li><li>Handle negatives: negatives = all negative values; nonnegatives = all nonnegative values; reverse negatives; concatenate negatives + nonnegatives.</li></ul></li><li>Runtime intuition:<ul><li>Step 1: number of passes = d, where d is the number of digits in the largest number.</li><li>Step 2: cost per pass ~ O(n) (scan elements, place into buckets, then flatten buckets).</li><li>Step 3: total cost ~ O(d × n).</li><li>For base 10, d = O(log10(k)) where k is the largest value; hence overall complexity ~$ O(n \log_{10}(k)) $.</li></ul></li><li>Practical takeaway: Radix Sort can outperform comparison-based sorts on large datasets with fixed-length keys but requires extra space and careful handling of negatives.</li></ul><h3 id="5328dd2f-a016-4cc5-a349-ac7a71704d37" data-toc-id="5328dd2f-a016-4cc5-a349-ac7a71704d37" collapsed="false" seolevelmigrated="true">Sorting Algorithm Comparison Chart (summary points)</h3><ul><li>Bubble Sort: Best$ O(n) $; Average$ O(n^2) $; Worst$ O(n^2) $; Stable: Yes; In-place: Yes. Notes: educational; not used in practice.</li><li>Selection Sort: Best$ O(n^2) $; Average$ O(n^2) $; Worst$ O(n^2) $; Stable: No; In-place: Yes. Notes: always does n^2 comparisons.</li><li>Insertion Sort: Best$ O(n) $; Average$ O(n^2) $; Worst$ O(n^2) $; Stable: Yes; In-place: Yes. Notes: great for nearly-sorted data; conceptually a bridge to Merge Sort.</li><li>Shell Sort: Best$ O(n \, ext{log} n)^* $~ roughly$ O(n^{1.3}) $; Average$ O(n^2) $; Stable: No; In-place: Yes. Notes: faster than insertion but not as commonly used; not stable.</li><li>Merge Sort: Best/Average/Worst$ O(n \, log n) $; Stable: Yes; In-place: No. Notes: recursive; requires$ O(n) $space; excellent performance.</li><li>Quick Sort: Best/Average$ O(n \, log n) $; Worst$ O(n^2) $; Stable: No; In-place: Yes. Notes: very fast in practice; pivot strategy is key; not stable by default.</li><li>Heap Sort: Best/Average/Worst$ O(n \, log n) $; Stable: No; In-place: Yes. Notes: based on heap; good performance but not stable.</li></ul><h3 id="ca0d0737-14a4-45d2-b9de-3e9331551db6" data-toc-id="ca0d0737-14a4-45d2-b9de-3e9331551db6" collapsed="false" seolevelmigrated="true">Notes on Master Theorem and Recurrence Relations</h3><ul><li>Recurrence form for divide-and-conquer algorithms:<ul><li>T(n) = a · T(n/b) + f(n)</li><li>a: number of subproblems</li><li>b: factor by which the problem size is reduced</li><li>f(n): time to combine the sub-solutions (the work outside recursive calls)</li></ul></li><li>Master Theorem applicability:<ul><li>Used to solve recurrences of the form T(n) = a T(n/b) + f(n).</li><li>Compute n^{log_b a} and compare with f(n) to determine the case.</li></ul></li><li>The three classical cases (informal):<ul><li>Case 1: f(n) = O(n^{log_b a - ε}) for some ε > 0</li><li>Then T(n) = Θ(n^{log_b a})</li><li>Case 2: f(n) = Θ(n^{log_b a} log^k n) for some k ≥ 0</li><li>Then T(n) = Θ(n^{log_b a} log^{k+1} n)</li><li>Case 3: f(n) = Ω(n^{log_b a + ε}) for some ε > 0, and regularity condition holds (often written as a f(n/b) ≤ c f(n) for some c < 1 and sufficiently large n)</li><li>Then T(n) = Θ(f(n))</li></ul></li><li>How to apply Master Theorem (workflow):<ul><li>Identify a, b, f(n) in the recurrence.</li><li>Compute n^{log_b a}.</li><li>Compare f(n) with n^{log_b a} (and any log factors) to select the correct case.</li><li>Conclude T(n) from the case.</li></ul></li><li>Classic examples:<ul><li>Merge Sort: T(n) = 2 T(n/2) + Θ(n) → a = 2, b = 2, f(n) = Θ(n) → Case 2 → T(n) = Θ(n log n).</li><li>Binary Search: T(n) = T(n/2) + Θ(1) → a = 1, b = 2, f(n) = Θ(1) → Case 2 → T(n) = Θ(log n).</li></ul></li></ul><h3 id="d9b1fe6a-652c-4bf6-b7ec-d9f4ac71301e" data-toc-id="d9b1fe6a-652c-4bf6-b7ec-d9f4ac71301e" collapsed="false" seolevelmigrated="true">Recurrence Examples and Master Theorem Flow</h3><ul><li>Master Theorem Flowchart (conceptual):<ul><li>If f(n) = O(n^{logb a - ε}) for some ε > 0 → Case 1 → T(n) = Θ(n^{logb a})</li><li>Else if f(n) = Θ(n^{logb a} log^k n) → Case 2 → T(n) = Θ(n^{logb a} log^{k+1} n)</li><li>Else if f(n) = Ω(n^{log_b a + ε}) and regularity condition holds → Case 3 → T(n) = Θ(f(n))</li><li>Else Master Theorem does not apply</li></ul></li></ul><h3 id="313e3b22-8c38-46be-98c2-2a16054d47ee" data-toc-id="313e3b22-8c38-46be-98c2-2a16054d47ee" collapsed="false" seolevelmigrated="true">Master Theorem: Worked Case Examples (conceptual)</h3><ul><li>Case 1 example outline: T(n) = 2 T(n/2) + Θ(1)<ul><li>a = 2, b = 2, f(n) = Θ(1) → n^{logb a} = n^{log2 2} = n^1 = n</li><li>Since f(n) = O(n^{1 - ε}) with ε = 1, Case 1 applies? (Note: the standard interpretation yields T(n) = Θ(n), but the slide framing is context-dependent; the key takeaway is using the comparison with n^{log_b a}.)</li></ul></li><li>Case 2 example outline: T(n) = 4 T(n/2) + Θ(n^2)<ul><li>a = 4, b = 2, f(n) = Θ(n^2) → n^{logb a} = n^{log2 4} = n^{2}</li><li>f(n) = Θ(n^{logb a}) → Case 2 with k = 0 → T(n) = Θ(n^{logb a} log n) = Θ(n^2 log n) (as per the general form). (The slide uses a variant; the essential point is that equality with a log factor escalates to Case 2 in the Master Theorem framework.)</li></ul></li><li>Case 3 example outline: T(n) = 2 T(n/2) + Θ(n^2)<ul><li>a = 2, b = 2, f(n) = Θ(n^2) → n^{log_b a} = n</li><li>Since f(n) grows faster than n^{log_b a}, Case 3 applies given the regularity condition, yielding T(n) = Θ(f(n)) = Θ(n^2).</li></ul></li><li>Final notes: The Master Theorem provides a structured way to deduce asymptotic growth for divide-and-conquer recurrences, with Merge Sort and Binary Search as canonical examples.</li></ul><h3 id="08bcfd81-aae9-473c-bdd0-a695d2ff37f8" data-toc-id="08bcfd81-aae9-473c-bdd0-a695d2ff37f8" collapsed="false" seolevelmigrated="true">Practical Takeaways</h3><ul><li>When choosing a sorting algorithm, consider:<ul><li>Input size and characteristics (random, nearly sorted, linked lists, in-memory arrays).</li><li>Stability requirements.</li><li>Memory constraints (in-place vs extra space).</li><li>Worst-case vs average-case performance.</li></ul></li><li>Radix Sort offers a non-comparison-based alternative with predictable linear passes in certain contexts, particularly with fixed-length keys, but requires extra space and handling of signs. Overall complexity scales with the number of digits of the largest key:$ O(n \, d) = O(n \, ext{log}_{b}(k)) $for base$ b $and largest key$ k $.</li></ul><h3 id="d68b5da7-e0e0-43fa-9496-aea8e3c6e566" data-toc-id="d68b5da7-e0e0-43fa-9496-aea8e3c6e566" collapsed="false" seolevelmigrated="true">Quick Reference: Key Notation</h3><ul><li>Time complexities:$ O(\,) $,$
 abla $$ (log base 2) denotation in context, and variations by base as needed.
- Recurrences: T(n) = a T(n/b) + f(n)
- Master Theorem cases summarized above.