Lecture 04: Concurrency and Parallelism Notes

Foundations of Concurrency and Parallelism

Parallelism Overview: Parallelism is used to improve the performance of specific problems by having multiple workers solve tasks simultaneously. * Workers: These are the entities that execute the code. * Tasks: This represents the code being executed. * Requirements: Parallelism is possible in systems with concurrency and multiple cores. It occurs when multiple workers execute in cores simultaneously. * Amdahl’s Law: Students should be reminded of this law regarding the theoretical speedup in latency of the execution of a task at fixed workload that can be expected of a system whose resources are improved. * Operating System Considerations: General-use operating systems should make specific considerations for parallelism due to its utility in modern computing.
Problem Classifications in Parallelism: * Task Parallelism: Multiple workers work on entirely different tasks at the same time. Examples include one worker rendering graphics, another handling networking, and a third performing game logic. Co-processors are often used to solve these problems. * Data Parallelism: The same task is performed in parallel across different workers, but each worker uses different inputs. Examples include adding elements of a large array of integers or sorting many different arrays simultaneously.

Utilizing Processes for Parallelism

Conceptual Use: Processes are helpful when different parallel workers require the same code and variables but need different values for those variables or different control flows.
Code Example: Finding Relative Primes: * Objective: Find all positive integers relatively prime to a user-entered integer, $n$ . * Function:

void findRelativePrimes(int start, int end, int n) {
    for(int i = start; i < end; i++)
        if(n % i == 0) printf(’’%i’’, i);
    exit(0);
}

* Implementation with Forking:

const int PROCESSES = 10;
int main() {
    int n;
    printf(’’Please enter an integer to test’’);
    scanf(’’%i’’, &n);
    int partitionSize = (n - 1) / PROCESSES; // Size of partition for each process
    for(int i = 0; i < PROCESSES - 1; i++)
        if(fork() == 0) findRelativePrimes(i*partitionSize, (i+1)*partitionSize, n);
    findRelativePrimes(n - partitionSize, n, n); // Parent does the last partition
}

Limitations and Overhead of Processes: * Memory Isolation: Each process gets its own memory allocation. If a child process calculates a value (like a sum), that variable is deallocated upon termination and the parent cannot access it. * Scalability Issues: In an example like summing $100\text{K}$ integers in an array, using fork() creates a duplicate of the process. If $10$ children are made, there will be $10$ copies of the array in memory, which does not scale well. * Inter-Process Communication (IPC): To share results, memory must be communicated between processes using IPC (e.g., message passing). IPC can be facilitated by the OS or a library but incurs latency because duplicate values must be maintained across processes.

Threads: Lightweight Processes

Definition: Threads are alternatives to processes, often called "lightweight processes." They run concurrently but share some memory allocations with an existing parent process.
Resource Sharing: * Shared: Threads share static data, code, and the heap. * Unique: Each thread receives its own stack and register states to allow for different control flows. * Stack Allocation: A thread’s stack is allocated within the parent process’s heap.
Management: Each thread has a Thread Control Block (TCB) to store relevant metadata. Threads are managed by the OS and are schedulable like processes.
Implementation in C (pthread library): * Creation: Use pthread_create or a thread constructor. Threads are given a starting function and arguments. * Constraints: Threads do not have a return value. In C, they take exactly one parameter, typically a struct containing all necessary arguments. The function executed must be void. * Thread Join: A parent process can wait for its children threads to exit using a thread join. * Compilation: The -lpthread flag is required with gcc to specify the thread version.
Code Example: Parallel Summation with Threads: * Data Structure:

struct findSumArgs {
    int start, end;
    int* arr;
    int sum;
};

* Operation:

void findSum(void* args) {
    findSumArgs* fSArgs = (findSumArgs*)args;
    for(int i = fSArgs->start; i < fSArgs->end; i++)
        fSArgs->sum += fSArgs->arr[i];
}

// Main Logic
const int THREADS = 10;
const int ARR_SIZE = 100000;
const int PARTITION_SIZE = ARR_SIZE / THREADS;
int main(void) {
    pthread_t workers[THREADS];
    findSumArgs args[THREADS];
    for(int i = 0; i < THREADS; i++) {
        args[i].start = i * PARTITION_SIZE;
        args[i].end = args[i].start + PARTITION_SIZE;
        args[i].arr = arr;
        args[i].sum = 0;
        pthread_create(&workers[i], NULL, findSum, &args[i]);
    }
    int finalSum = 0;
    for(int i = 0; i < THREADS; i++) {
        pthread_join(workers[i], NULL);
        finalSum += args[i].sum;
    }
}

Race Conditions and Mutual Exclusion

Race Conditions: These occur when two or more threads run in parallel and share a common resource that at least one thread writes to. This can cause data corruption because the sequence of operations (Read-Add-Write) is interleaved between threads.
Critical Sections: A section of code that updates a shared resource. Execution must be mutually exclusive (only one thread at a time).
Solutions to Race Conditions: 1. Eliminate Shared Resources: Replace a single shared resource with multiple unique resources (like the args[i].sum in the code above). This requires a final merge step. 2. Mutual Exclusion: Control access so only one thread executes the critical section at a time.
Locks (Mutexes): * Type: pthread_mutex_t. * Mechanics: pthread_mutex_lock checks if a lock is available. If unavailable, the thread blocks. pthread_mutex_unlock releases the lock. * Performance Impact: Excessive locking can cause threads to spend significant time blocked, potentially leading to performance matching or trailing sequential execution due to overhead.
Code Example: Locking the Sum Variable:

pthread_mutex_t lock;
void findSum(void* args) {
    findSumArgs* fSArgs = (findSumArgs*)args;
    for(int i = start; i < end; i++) {
        pthread_mutex_lock(&lock);
        fSArgs.sum += fSArgs.arr[i];
        pthread_mutex_unlock(&lock);
    }
}

Note on Efficiency: The above implementation is inefficient because the lock is held for the duration of every addition. A better approach is to sum locally and then lock once to add the local sum to the global total.

Synchronization and Ordering

Interleaving: In parallel systems, operations can be interleaved in arbitrary orders. For an array of $12$ elements across $4$ threads, elements handled by a single thread maintain their internal order, but the order between threads is non-deterministic (e.g., arr[0] might be followed by arr[3] or arr[1]).
Condition Variables: Used to signal that a thread should be placed in the blocked queue until a specific event occurs. Unlike sleep(), which requires a fixed time duration, condition variables use wait() and signal()/broadcast() for event-based wake-ups. * wait(): Unlocks the associated mutex and blocks the thread. * signal(): Moves a waiting thread to the ready queue; the thread then re-acquires the lock.
Producer-Consumer Problem (Bounded Buffer): * Scenario: A fast writer (Producer) and a reader (Consumer) share a fixed-size buffer. * Synchronization: The reader must wait if the buffer is empty. The writer must wait if the buffer is full.

Software and Hardware Implementations of Locks

Spin Locks: A simple implementation where a thread checks a boolean in a while loop indefinitely. * Failure: Not guaranteed to work on preemptive systems. If a thread is preempted after the loop but before setting the lock, multiple threads can enter the critical section.
Peterson’s Solution: A software approach for exactly two threads. It uses a turn variable and a flag array. * Logic: while (lock[other] && turn == other) { /* wait */ }. * Modern Challenges: Features like out-of-order execution and cache write-back optimization can break software-only solutions.
Atomic Operations: CPU-level operations like xchg (exchange) allow a single core to execute a swap while blocking others and disabling interrupts, ensuring the operation is uninterruptible.

Semaphores

Definition: An atomic integer representing the count of available resources. Programs interact via an API, not direct assignment.
Operations: * Wait (P): Decrements the counter if non-zero; otherwise blocks. * Post (V): Increments the counter.
Binary Semaphores: A conceptual use of semaphores restricted to values $0$ and $1$ to act as a lock.
Bounded Buffer with Semaphores: * Uses an empty semaphore (initialized to BUFFER_SIZE) and a full semaphore (initialized to $0$ ).

Classical Problems in Synchronization

Santa Claus Problem: A barrier problem where Santa sleeps until either $9$ reindeer arrive (Christmas Eve) or $3$ elves arrive (toy questions). Reindeer have priority over elves.
Reader-Writer Problem: Multiple readers can access a resource simultaneously, but writers require exclusive access. * Reader Code Pattern:

wait(mutex);
read_count++;
if(read_count == 1) wait(rw_mutex);
signal(mutex);
/* perform reading */
wait(mutex);
read_count--;
if(read_count == 0) signal(rw_mutex);
signal(mutex);

Dining Philosophers Problem: Philosophers require two chopsticks (shared resources) to eat gyoza. This is used to illustrate deadlock and livelock.

Mutual Exclusion Related Bugs

Deadlock: Occurs when two or more threads are waiting on resources held by each other, and no progress can be made. * Example: Two threads acquire lock1 and lock2 in opposite orders. * Dependency Graph: Threads are circles, resources are rectangles. A cycle in the graph indicates potential deadlock.
Livelock: Threads constantly try to resolve a conflict (e.g., dropping a resource and trying again) but do so in synchronized lockstep, consuming CPU cycles without making meaningful progress. * Hallway Analogy: Two people meeting in a narrow hall. In deadlock, neither moves. In livelock, both keep stepping to the same side simultaneously to let the other pass, continuing to block each other.

Threading Models and Asynchronous Programming

User vs. Kernel Threads: * One-to-One: OS schedules each thread to a core. High overhead for thread creation via syscalls. * Many-to-One: A library manages threads within a single process. OS only sees the process. Concurrent but not parallel on multiple cores. * Many-to-Many: OS schedules sets of threads to cores; internal schedulers manage threads within those sets.
Asynchronous Programming: Concurrency handled independently of the OS. * Coroutines (e.g., Unity Engine): Workers scheduled within a single parent process. They share resources and do not require OS threads. * Use Case: Mitigating latency in server requests. A program sends a request and performs other work while waiting for the response, avoiding the need for full thread parallelism.