High-Performance Computing (HPC) and Parallel Programming

Comprehensive flashcards covering High-Performance Computing (HPC) fundamentals, machine architecture, parallel decomposition, Amdahl's Law, computer architecture basics, process/thread management, concurrency principles, MPI programming, and CUDA fundamentals.

What are the three core components of HPC defined in the lecture?

The core components are Hardware Systems (Multi-core CPUs, GPUs, and clusters), Software Tools (Compilers, libraries, debuggers), and Programming Paradigms (Parallel programming frameworks like MPI, OpenMP, and CUDA).

As of November 2025, which supercomputer is ranked #1 on the TOP500 list, and what is its country of origin?

El Capitan, United States.

What characterizes Exascale computing today?

Systems capable of $10^{18}$ calculations per second.

What are the two primary concerns in Interconnection Topology design for HPC clusters?

1. Minimizing the number of connections per node as the cluster scales. 2. Ensuring that communication time and bandwidth remain constant regardless of the number of processing nodes.

How does a Torus topology differ from a standard Mesh topology?

The Torus topology extends the mesh by connecting the edges of the grid, creating a wrap-around effect to reduce communication distance.

Describe the connection requirements and communication time scaling for a Hypercube topology.

It requires $\log_2(n)$ connections per node, with communication times scaling up to $(\log_2(n))$ where $n = 2^k$.

What is the purpose of a Fat-Tree topology in modern supercomputers?

It is a version of the tree topology with additional bandwidth in higher levels of the hierarchy to prevent bottlenecks and maintain consistent bandwidth across all bisections.

Match the levels of parallel granularity (Fine, Medium, Coarse) to their respective categories.

Fine-grained: SIMD (operations on variables); Medium-grained: MIMD shared memory (thread level); Coarse-grained: MIMD distributed memory (process level).

State the formula for Amdahl's Law speedup including communication overhead.

$S_{overall} = \frac{1}{(1 - P) + C + \frac{P}{N}}$ where $P$ is the parallelizable fraction, $C$ is the communications overhead, and $N$ is the speedup factor.

What is the Stored-Program Architecture, and who developed it?

The concept of representing instructions as data stored in main memory alongside other data, developed independently by John von Neumann and Alan Turing.

What are the two fundamental internal states a computer system alternates between?

1. Instruction Fetch (retrieving the next instruction from memory) and 2. Instruction Execution (decoding and carrying out the operation).

How is speedup ($S$) calculated for an N-stage pipeline compared to a non-pipelined version?

$S = \frac{t_{non-pipelined}}{t_{pipelined}}$ where for a large number of objects, the speedup ideally approaches an $N$-fold improvement ($S \approx N$).

What is the purpose of the write-through policy in multicore cache systems?

When a write occurs, all copies in L1 and L2 are marked as stale to ensure subsequent reads trigger a transfer from L3, which holds the most recent version of data.

Define a 'Process' in the context of computer architecture.

An active instance of a program characterized by addressing space, processor context, I/O context, and execution state.

Contrast User-Level Threads (ULTs) and Kernel-Level Threads (KLTs).

ULTs are managed by user libraries without kernel involvement but can block the whole process if one thread blocks. KLTs are managed directly by the kernel, allowing other threads to run if one blocks and enabling parallel execution on multicore processors.

What is a 'Race Condition'?

A scenario occurring when multiple threads access and modify shared data concurrently, so the final outcome depends on the timing or order of execution.

List the four necessary conditions for Deadlock to occur.

1. Mutual Exclusion, 2. Hold and Wait (Waiting with Retention), 3. No Preemption (Non-Liberation), and 4. Circular Wait (Vicious Circle).

What is 'Aging' in the context of resource allocation?

A policy to prevent starvation (indefinite postponement) by gradually increasing the priority of a waiting process over time.

Compare Blocking vs. Non-blocking synchronization in message exchange.

Blocking (synchronous) operations halt the process until the message is received/arrives. Non-blocking (asynchronous) operations return immediately, requiring processes to manage synchronization themselves.

Define MPI as per the transcript.

MPI (Message Passing Interface) is a library specification (not a language) for the message-passing programming model where data is explicitly transferred between processes through a standardized API.

What are the two mandatory MPI instructions for starting and ending an MPI program?

The first instruction must be MPI\_Init(&argc, &argv) and the last must be $MPI\_Finalize()$.

What information is contained in the MPI Message Header (Envelope)?

Communication context (communicator), Source Identification (rank), Destination Identification (rank), and Tag (integer label).

Distinguish between MPI_Scatter and MPI_Scatterv.

MPI_Scatter distributes uniform, distinct parts of a message from a root to all processes. MPI_Scatterv is an extended version that supports non-uniform message sizes using an array of displacements ($displs$).

How does MPI_Reduce function?

It performs a global element-wise computation (e.g., $MPI\_SUM$, $MPI\_MAX$) on value arrays from all processes and delivers the final result to a root process.

What are the two routines MPI provides to monitor non-blocking operations?

$MPI\_Wait$ (blocks until completion) and $MPI\_Test$ (checks completion status without blocking).

Explain the role of MPI_Barrier.

A collective synchronization mechanism where all participating processes block until every process in the communicator has reached the barrier.

What is the execution configuration syntax for launching a CUDA kernel?

<<

Define a 'Warp' in CUDA programming.

A warp is a batch of 32 threads that the GPU executes concurrently in SIMD/SIMT (Single Instruction, Multiple Threads) fashion.

What is Unified Memory (Managed Memory) in CUDA?

A single coherent memory space shared between the CPU and GPU that provides automatic data migration and simplifies programming by removing the need for manual memory copying.

What are the restrictions for CUDA Kernels?

They must return void, only access device memory, cannot use static variables, cannot use function pointers, and launches are asynchronous (control returns to the host immediately).