Lecture5-Sorting Algorithms

Course Information

Course: CSCI 3230 Data StructuresTerm: Fall 2024Instructor: M Arif Rahman, Ph.D.Affiliation: Department of Computer Science, Georgia Southern UniversityEmail: marahman@georgiasouthern.eduAcknowledgment: Thanks to Dr. Tong

Table of Contents

  1. Comparison Sort

  2. Selection Sort

  3. Bubble Sort

  4. Insertion Sort

  5. Merge Sort

  6. Heap Sort

  7. Quick Sort

  8. Lower Bound on Comparison Sorting

  9. Linear Sorting

  10. Counting Sort

  11. Radix Sort

  12. Bucket Sort

Comparison Sort

Comparison sort refers to a category of algorithms that sort input data through repeated comparisons between pairs of elements. A critical feature of these algorithms is that they do not assume any prior knowledge about the order or distribution of the input values.

Characteristics

  • Sort in Place: Requires constant additional memory (O(1)), achieved by swapping elements directly within the main array, which is advantageous for space efficiency.

  • Stable Sort: Ensures that identical elements retain their relative order from the original input throughout the sorting process. This characteristic becomes crucial in applications where sorting by multiple criteria is necessary.

Example (Stability Demonstrated)

Input: [4, 3, 7, 11, 2, 2, 1, 3, 5, 6]Output: [1, 2, 2, 3, 3, 4, 5, 6, 7, 11](In the output, the relative order of the two '2's is preserved, illustrating stability.)

Selection Sort

Procedure:

  1. Identify the smallest element in the list.

  2. Move this element to the beginning of the output list.

  3. Repeat the process for the remaining elements, effectively narrowing the portion of the array left to sort until complete.

Example Sequence

Input: [4, 3, 7, 11, 2, 2, 1, 3, 5, 6]Output: [1, 2, 2, 3, 3, 4, 5, 6, 7, 11]

Selection Sort Pseudocode

for i = 0 to n−1
    jmin = i
    for j = i + 1 to n−1
        if A[j] < A[jmin]
            jmin = j
    swap A[i] with A[jmin]

Running Time:

The total running time for Selection Sort has a time complexity of O(n²), making it inefficient for larger datasets as the number of comparisons grows quadratically with the number of elements.

Bubble Sort

Procedure:

  1. Repeatedly compare adjacent elements in the array.

  2. Swap them if they are in the wrong order (i.e., the first is greater than the second).

  3. The process continues until the end of the array, thus allowing the largest elements to "bubble up" to the end with each pass.

Key Characteristics

  • Best Case: O(n) - This occurs if the input array is already sorted, resulting in no swaps being necessary.

  • Worst Case: O(n²) - Arises when the array is in completely reverse order, leading to maximum comparisons and swaps.

Bubble Sort Pseudocode

for i = n−1 down to 1
    for j = 0 to i−1
        if A[j] > A[j + 1]
            swap A[j] and A[j + 1]

Insertion Sort

Procedure:

  1. Maintain two sublists: one sorted and one unsorted.

  2. Gradually take elements from the unsorted list and place them into their correct position within the sorted list, maintaining order.

  3. This algorithm is especially effective for datasets that are already partially sorted.

Key Characteristics

  • Best Case: O(n) - This optimal scenario occurs when the input list is nearly sorted, requiring minimal movements.

  • Worst Case: O(n²) - In cases where elements are arranged in reverse order, the algorithm must perform extensive comparisons and movements.

Insertion Sort Pseudocode

for i = 1 to n - 1
    key = A[i]
    j = i - 1
    while j >= 0 and A[j] > key
        A[j + 1] = A[j]
        j = j - 1
    A[j + 1] = key

Merge Sort

Divide-and-Conquer Strategy:

  1. Split the input data into two smaller subsets (left and right).

  2. Recursively sort both subsets using merge sort.

  3. Merge the two sorted sublists back together into a single sorted list in a manner that preserves order.

Running Time: O(n log n)

Merge Sort guarantees this efficiency due to its logarithmic division and linear merging process.

Merge Algorithm Pseudocode

merge(A, B)
while !A.isEmpty() and !B.isEmpty()
    if A.first() < B.first()
        S.addLast(A.remove(A.first()))
    else
        S.addLast(B.remove(B.first()))
return S

Heap Sort

Mechanism:

  1. Construct a max heap structure from the input list, arranging elements according to the heap property.

  2. Repeatedly remove the maximum value from the heap and restore the heap property after each operation.

Running Time: O(n log n)

In the worst case, Heap Sort maintains its time complexity due to the logarithmic height of the heap.

Stable?:

Heap Sort is not stable as the process involves rearranging the order of elements while maintaining the heap structure.

Quick Sort

Divide-and-Conquer Strategy:

  1. Choose a pivot element from the array.

  2. Partition the remaining array into two subarrays: those less than the pivot and those greater than the pivot.

  3. Recursively apply Quick Sort to the two subarrays to sort all elements.

Running Time:

  • Worst Case: O(n²) - This scenario arises from consistently choosing the smallest or largest element as the pivot, particularly in sorted or reverse sorted data.

  • Best Case: O(n log n) - Achieved with random pivot selections that effectively split the dataset.

Partition Algorithm Pseudocode

partition(A, pivot)
while !A.isEmpty()
    y = A.remove(A.first())
    if y < pivot
        L.addLast(y)
    else if y == pivot
        E.addLast(y)
    else
        G.addLast(y)
return L, E, G

Linear Sorting

This category comprises sorting algorithms that can achieve a linear time complexity under specific conditions or data structures, including the following:

Counting Sort

Mechanism: Counts occurrences of each distinct element in the input, and then reconstructs the sorted output based on these counts.Running Time: O(n + k) - Here, k represents the range of the input values, making it efficient for datasets with a limited range.

Radix Sort

Sorts numbers by processing their individual digits, starting from the least significant digit to the most significant. This algorithm utilizes a stable sort (such as Counting Sort) for each digit. Expected Running Time: O(d(n + k)), where d is the number of digits in the maximum number.

Bucket Sort

Distributes elements into a finite number of buckets, sorts each bucket individually (often employing insertion sort), and concatenates the sorted buckets to form the final sorted output.Expected Running Time: O(n) when inputs are uniformly distributed across the buckets.

Summary of Comparison Sorts

Algorithm

Best

Worst

In Place

Stable

Comments

Selection Sort

O(n²)

O(n²)

Yes

Yes

Simple implementation but inefficient.

Bubble Sort

O(n)

O(n²)

Yes

Yes

Requires condition check for best case.

Insertion Sort

O(n)

O(n²)

Yes

Yes

Very efficient for small or almost sorted.

Merge Sort

O(n log n)

O(n log n)

No

Yes

Ideal for large datasets and linked lists.

Heap Sort

O(n log n)

O(n log n)

Yes

No

Guarantees O(n log n) in all cases.

Quick Sort

O(n log n)

O(n²)

Yes

Yes

Usually fastest in practice.

Conclusion

In conclusion, understanding the strengths and weaknesses of each sorting algorithm is essential for effectively choosing the appropriate one contingent upon the specific context and constraints, such as data size, distribution, and memory requirements.

Comparison Sort

Comparison sort is a category of algorithms that sort input data by repeatedly comparing pairs of elements. Crucially, these algorithms do not rely on any prior knowledge about the order or distribution of the input values.

Key Characteristics

  • Sort in Place: This characteristic requires constant additional memory (O(1)). It is achieved by swapping elements directly within the main array, making it space-efficient.

  • Stable Sort: This ensures that identical elements retain their initial relative order throughout the sorting process. Stability is vital in applications where sorting based on multiple criteria is necessary.

Example (Stability Demonstrated)

  • Input: [4, 3, 7, 11, 2, 2, 1, 3, 5, 6]

  • Output: [1, 2, 2, 3, 3, 4, 5, 6, 7, 11](In this output, the relative order of the two '2's is preserved, demonstrating stability.)

Selection Sort

Procedure:

  1. Identify the smallest element in the list.

  2. Move this element to the beginning of the output list.

  3. Repeat the process for the remaining elements until the list is fully sorted.

Example Sequence

  • Input: [4, 3, 7, 11, 2, 2, 1, 3, 5, 6]

  • Output: [1, 2, 2, 3, 3, 4, 5, 6, 7, 11]

Selection Sort Pseudocode:
for i = 0 to n−1
    jmin = i
    for j = i + 1 to n−1
        if A[j] < A[jmin]
            jmin = j
    swap A[i] with A[jmin]

  • Running Time: The overall time complexity is O(n²), making it inefficient for larger datasets due to the quadratic growth of comparisons with the number of elements.

Bubble Sort

Procedure:

  1. Compare adjacent elements in the array.

  2. Swap them if they are in the wrong order (first > second).

  3. Repeat until the end of the array. This process lets the largest elements "bubble up" to the end.

Key Characteristics:

  • Best Case: O(n) - Occurs when the array is already sorted.

  • Worst Case: O(n²) - Happens when the array is in completely reversed order.

Bubble Sort Pseudocode:
for i = n−1 down to 1
    for j = 0 to i−1
        if A[j] > A[j + 1]
            swap A[j] and A[j + 1]

Insertion Sort

Procedure:

  1. Maintain two sublists: one sorted and one unsorted.

  2. Gradually take elements from the unsorted list and place them into the correct position within the sorted list.

Key Characteristics:

  • Best Case: O(n) - When the input list is nearly sorted.

  • Worst Case: O(n²) - When elements are in reverse order.

Insertion Sort Pseudocode:
for i = 1 to n - 1
    key = A[i]
    j = i - 1
    while j >= 0 and A[j] > key
        A[j + 1] = A[j]
        j = j - 1
    A[j + 1] = key

Merge Sort

Divide-and-Conquer Strategy:

  1. Split the input into two smaller subsets (left and right).

  2. Recursively sort both subsets.

  3. Merge the two sorted sublists into a single sorted list, preserving order.

  • Running Time: O(n log n)

Merge Algorithm Pseudocode:
merge(A, B)
while !A.isEmpty() and !B.isEmpty()
    if A.first() < B.first()
        S.addLast(A.remove(A.first()))
    else
        S.addLast(B.remove(B.first()))
return S

Heap Sort

Mechanism:

  • Construct a max heap from the input list.

  • Repeatedly remove the maximum value from the heap and restore the heap property after each operation.

  • Running Time: O(n log n)

  • Stable?: No, the rearrangement of elements can change their relative order.

Quick Sort

Divide-and-Conquer Strategy:

  1. Choose a pivot element.

  2. Partition the array into two subarrays: less than pivot and greater than pivot.

  3. Recursively apply Quick Sort to sort all elements.

  • Running Time:

    • Worst Case: O(n²) - When the smallest or largest element is chosen consistently as a pivot.

    • Best Case: O(n log n) - With effective pivot selections.

Partition Algorithm Pseudocode:
partition(A, pivot)
while !A.isEmpty()
    y = A.remove(A.first())
    if y < pivot
        L.addLast(y)
    else if y == pivot
        E.addLast(y)
    else
        G.addLast(y)
return L, E, G

Linear Sorting Algorithms

These algorithms can achieve linear time complexity under specific conditions or data structures:

  • Counting Sort: Counts occurrences of elements and reconstructs the sorted output.

    • Running Time: O(n + k) where k is the range of input values.

  • Radix Sort: Sorts numbers by processing individual digits, leveraging a stable sort for each digit.

    • Expected Running Time: O(d(n + k)), where d is the number of digits in the maximum number.

  • Bucket Sort: Distributes elements into buckets, sorts each bucket, and concatenates the results.

    • Expected Running Time: O(n) for uniformly distributed inputs.

Summary of Comparison Sorts

Algorithm

Best

Worst

In Place

Stable

Comments

Selection Sort

O(n²)

O(n²)

Yes

Yes

Simple but inefficient.

Bubble Sort

O(n)

O(n²)

Yes

Yes

Requires best case condition check.

Insertion Sort

O(n)

O(n²)

Yes

Yes

Efficient for small/almost-sorted data.

Merge Sort

O(n log n)

O(n log n)

No

Yes

Ideal for large datasets and linked lists.

Heap Sort

O(n log n)

O(n log n)

Yes

No

Guarantees O(n log n) in all cases.

Quick Sort

O(n log n)

O(n²)

Yes

Yes

Fastest in practice generally.

Conclusion

Understanding the strengths and weaknesses of each sorting algorithm is crucial for selecting the most suitable one based on specific contexts, including data size, distribution, and memory requirements.

Questions to Ask the Professor:

  1. Can you explain the scenarios where one sorting algorithm would be preferable over another in practice?

  2. What are some examples where a stable sort is essential in real-world applications?

  3. Could you elaborate on how the choice of pivot in Quick Sort can significantly affect its efficiency?

  4. How does the memory usage in in-place sorting algorithms like Insertion and Bubble Sort compare in practical terms?