PDP Auto-generated

What is the defining characteristic of a Shared-Memory Parallel Computer?

A system in which multiple running CPUs/cores work on a single, common shared physical address space.

What is Uniform Memory Architecture (UMA)?

A shared-memory architecture where all processors access memory with equal access time and bandwidth through a centralized bus or interconnect.

What is Cache-Coherent Non-Uniform Memory Architecture (CC-NUMA)?

An architecture where memory is physically distributed among sockets/nodes, resulting in faster local memory access than remote access, while cache coherence protocols maintain a unified logical address space.

What hardware mechanisms connect sockets in modern CC-NUMA systems?

High-speed point-to-point interconnects such as Intel Ultra Path Interconnect (UPI / QPI) or AMD Infinity Fabric.

What is a multi-core processor?

A single physical chip (die) that integrates two or more independent execution units called cores.

What rule of thumb governs voltage, frequency, power, and performance scaling?

Reducing voltage by 1% and frequency by 1% reduces power consumption by ~3% while reducing performance by only ~0.66%.

Why did single-core CPU frequency scaling stagnate around 2005?

Power density and heat dissipation limits (the 'Power Wall') forced the industry to shift from increasing clock frequencies to multi-core parallelism.

What is a cluster in HPC?

A collection of interconnected independent computers (nodes) working together as a single unified parallel computing resource.

Are modern supercomputers purely shared-memory or distributed-memory?

They are hybrid systems: shared-memory within each node (multi-core, multi-socket) and distributed-memory across nodes connected via networks (e.g., InfiniBand).

What is the role of High-Bandwidth Memory (HBM) in modern HPC accelerators?

HBM provides ultra-wide 3D-stacked memory interfaces delivering terabytes-per-second memory bandwidth to feed massively parallel compute units.

What is Network-Attached Parallel Storage in HPC clusters?

Shared high-performance parallel file systems (e.g., Lustre, Spectrum Scale/GPFS) that allow all cluster compute nodes to read/write large data concurrently.

Why is local node storage (e.g., local NVMe SSDs) becoming prominent in HPC?

To reduce network traffic, avoid parallel file system bottlenecks, and act as fast burst buffers or temporary storage for big data and AI workflows.

What is Flynn's Taxonomy category for modern multi-core CPU clusters?

MIMD (Multiple Instruction, Multiple Data).

What is Flynn's Taxonomy category for SIMD/Vector units?

SIMD (Single Instruction, Multiple Data).

How does a socket differ from a node?

A socket is a physical slot/chip holding a processor die; a node is an independent computer containing one or more sockets, memory, and network interfaces.

What is cache coherence?

A hardware/software protocol ensuring that modifications to shared memory locations are instantly reflected in all processor caches holding a copy of that memory.

What is the memory wall problem?

The growing gap between processor execution speed and main memory access latency/bandwidth.

How does vectorization improve energy efficiency?

By executing a single instruction on $N$ data elements, instruction fetch and decode overhead is amortized across multiple operations.

What is the core focus of traditional HPC Architecture?

Compute-intensive numerical modeling and simulation on tightly coupled systems with low-latency interconnects.

What is the core focus of Big Data Architecture?

Data-intensive ingestion and analysis on loosely coupled commodity hardware using distributed file systems (e.g., HDFS) and resilient data frameworks.

Why can exascale simulation output no longer be moved to a separate analysis cluster?

Exascale simulations produce petabytes of data; moving this data across networks consumes prohibitive time and energy, requiring in-situ analytics.

What does 'in-situ analytics' mean?

Analyzing and visualizing simulation data in real-time as it is generated in node memory, without writing raw data to disk first.

How does energy cost compare between moving data and performing computation?

The energy cost of moving data across memory/networks is orders of magnitude higher than performing arithmetic operations on that data.

What was the 'Hourglass Architecture' vision from the 2015 US NSCI?

A model with a 'narrow waist' of standardized services (data management, scheduling, protocols) enabling diverse applications to run seamlessly across diverse hardware.

How are supercomputers evolving from 'calculators' to 'AI systems'?

Supercomputers now feature massive GPU clusters, high-bandwidth memory (HBM), and AI-optimized precision formats to train models with trillions of parameters.

What container technologies enabled widespread adoption of complex AI stacks on HPC?

Apptainer (Singularity) and Docker, allowing reproducible, dependency-packaged execution without root privileges on bare-metal supercomputers.

What is an AI Surrogate Model in scientific computing?

A neural network trained on previous simulation data to predict complex physics/climate outcomes orders of magnitude faster than full numerical solvers.

Compare Numeric Precision in HPC vs AI workflows.

HPC uses High Precision (FP64 double precision); AI training uses Low/Mixed Precision (FP16, BF16, FP8, INT8).

Compare Logic Types in HPC vs Big Data vs AI.

HPC = Deterministic Physics equations; Big Data = Statistical Patterns & Aggregations; AI = Neural Weight Optimization & Gradients.

Compare Data Movement in HPC vs Big Data vs AI.

HPC = Tightly Coupled inter-node communication; Big Data = Loosely Coupled batch/stream processing; AI = High-throughput matrix synchronization & gradient all-reduce.

What role does Big Data play in the converged HPC-AI workflow?

The Data Manager: managing data ingestion, ETL, storage, and cataloging.

What role does AI play in the converged HPC-AI workflow?

The Accelerator & Steerer: speeding up sub-computations via surrogates and steering parameter sweeps.

What role does HPC play in the converged HPC-AI workflow?

The Ground Truth Generator: producing high-fidelity physical ground-truth data via numerical first-principles simulation.

Why is FP16 or FP8 preferred over FP64 for Deep Learning?

Lower precision dramatically reduces memory bandwidth demands, footprint, and power consumption while increasing matrix multiplication throughput via Tensor Cores.

What is steering a simulation with Machine Learning?

Using ML models to analyze intermediate simulation results and dynamically adjust simulation parameters or focus resolution on regions of interest.

What technology connects GPUs together with ultra-high bandwidth in AI supercomputers?

Custom high-speed interconnects like NVIDIA NVLink / NVSwitch or AMD Infinity Fabric.

What are the three main phases of a Bulk-Synchronous Parallel (BSP) superstep?

"1. Local Computation Phase

2. Global Communication Phase

3. Barrier Synchronization Phase."

What happens during the Computation Phase of BSP?

Each process performs local calculations using data already available in its local memory.

What happens during the Communication Phase of BSP?

Processors exchange computed data and messages with other processors.

What happens at the Barrier in a BSP superstep?

All processors wait until every process reaches the barrier and all communications are completed before starting the next superstep.

Name three big data / parallel models inspired by BSP concepts.

MapReduce, Google Pregel (graph processing), and Apache Hama.

What is Hybrid Programming in HPC?

Combining multiple parallel programming paradigms (e.g., MPI + OpenMP, MPI + CUDA) across hierarchical cluster architectures.

In an MPI + OpenMP hybrid setup, what handles inter-node vs intra-node parallelism?

MPI handles message-passing inter-node communication; OpenMP handles shared-memory multithreading intra-node.

Why is hybrid MPI + OpenMP often better than pure MPI on high-core-count nodes?

It reduces memory footprint (fewer MPI buffers/structures), reduces communication endpoints, and handles load balancing within nodes better.

What is hierarchical parallelism?

Matching code parallelism to hardware hierarchy: cluster nodes -> sockets -> cores -> hyperthreads -> vector units / GPU warps.

What is a Processing Element (PE)?

A generic abstraction for an independent compute resource (e.g., CPU core, hardware thread, GPU lane).

What is the primary advantage of the BSP cost model?

It provides a simple, structured mathematical formula to estimate execution time based on computation cost, communication volume ($g$), and barrier latency ($L$).

What is the main drawback of rigid barrier synchronization in BSP?

Imbalance in computation causes faster processors to idle at the barrier, waiting for the slowest process (straggler effect).

How does CUDA + MPI hybrid programming function?

MPI handles data transfer between CPU hosts across cluster nodes; CUDA handles massively parallel offloaded computation on local node GPUs.

What is CUDA-aware MPI?

An MPI implementation that can directly pass GPU device memory pointers to MPI communication routines, bypassing manual host-staging copies.

What is straggler impact in BSP supersteps?

The slowest task in a superstep determines the completion time for all processes due to the mandatory global barrier.

How does asynchronous communication mitigate BSP barrier overhead?

By overlapping communication with local computation so data transfers occur concurrently while the CPU/GPU works on independent tasks.

What five major cluster computing challenges does MapReduce abstract away?

Parallelization, Data Distribution, Failure Handling, Load Balancing, and Fault Tolerance.

What are the core steps of the MapReduce processing model?

"1. Input Splitting

2. Map Phase

3. Shuffle & Sort Phase

4. Reduce Phase

5. Output Writing."

What is the signature of the Map function in MapReduce?

"$

ightarrow

ext{list}(k_2, v_2)$"

What is the signature of the Reduce function in MapReduce?

"$

ightarrow

ext{list}(k_3, v_3)$"

What happens during the Shuffle & Sort phase of MapReduce?

Worker nodes re-distribute and group all intermediate key-value pairs by key across the network so all values for key $k_2$ arrive at the same Reducer.

What is Apache Hadoop?

An open-source framework implementing distributed storage (HDFS), resource management (YARN), and processing (MapReduce).

What is a Combiner in MapReduce?

An optional local 'mini-reducer' that aggregates Map output on the local worker node prior to the Shuffle phase to save network bandwidth.

What mathematical properties must a function satisfy to be safely used as a Combiner?

Commutativity and Associativity (e.g., sum, count, min, max).

Why is Apache Spark faster than traditional MapReduce for iterative algorithms?

Spark keeps intermediate datasets in memory (RAM) via RDDs, whereas MapReduce writes intermediate results to disk (HDFS) after every Map/Reduce job.

What is a Resilient Distributed Dataset (RDD) in Spark?

An immutable, fault-tolerant, lazily evaluated, partitioned collection of records distributed across cluster nodes.

Difference between Spark Transformations and Actions?

Transformations (e.g., map, filter) create a new RDD lazily without computing results; Actions (e.g., count, collect) trigger computation and return values.

What is RDD Lineage in Apache Spark?

A Directed Acyclic Graph (DAG) tracking the exact sequence of transformations used to build an RDD, enabling fault recovery by recomputing missing partitions.

Name four high-level libraries built on Spark Core.

Spark SQL, Spark Streaming, MLlib (Machine Learning), and GraphX (Graph processing).

What is PyTorch?

An open-source deep learning framework providing dynamic computational graphs, multi-dimensional Tensor operations, and GPU acceleration.

What is autograd in PyTorch?

PyTorch's automatic differentiation engine that records operations on tensors to automatically compute gradients during loss.backward().

What are the four core steps in a PyTorch model training loop?

"1. Forward pass: y_pred = model(x)

2. Compute loss: loss = loss_fn(y_pred, y)

3. Zero gradients: optimizer.zero_grad()

4. Backward pass & update: loss.backward(), optimizer.step()."

What domain libraries belong to the PyTorch ecosystem?

torchvision (images/video), torchaudio (audio/signals), torchtext (NLP/vocabularies).

What is Narrow vs Wide Dependency in Spark RDDs?

Narrow: Each parent RDD partition is used by at most one child partition (no shuffle, e.g. map); Wide: Multiple child partitions depend on data from a parent partition (requires shuffle, e.g. groupByKey).

What is HDFS and how does it achieve fault tolerance?

Hadoop Distributed File System; splits large files into blocks (e.g., 128MB) and replicates each block across multiple nodes (default factor = 3).

What is YARN in Hadoop?

Yet Another Resource Negotiator: architectural layer separating cluster resource management (ResourceManager) from job scheduling/execution (ApplicationMaster).

In YARN, what is the role of the ResourceManager?

Cluster-wide authority that allocates memory and CPU resources across all running applications.

In YARN, what is the role of the ApplicationMaster?

A per-application process that negotiates resources with the ResourceManager and works with NodeManagers to execute and monitor tasks.

What is lazy evaluation in Apache Spark?

Spark delays executing RDD transformations until an Action is explicitly invoked, allowing the query optimizer (Catalyst) to optimize the overall DAG execution plan.

Define Speedup $S_p(N)$.

$S_p(N) = rac{T(1)}{T(N)}$, where $T(1)$ is sequential runtime and $T(N)$ is parallel execution time on $N$ processors.

Define Parallel Efficiency $E_p(N)$.

$E_p(N) = \frac{S_p(N)}{N} = \frac{T(1)}{N \cdot T(N)}$. Ideal linear efficiency is $1.0$ (or $100\%$).

What is ideal (linear) Speedup?

When speedup equals processor count: $S_p(N) = N$. Indicates perfect parallelization with zero overhead.

What core assumption underlines Amdahl's Law?

Fixed Data Set / Problem Size: execution time decomposes into an unparallelizable serial fraction $s$ and a parallelizable fraction $p$ ($s + p = 1$).

State Amdahl's Law formula for Speedup $S_p(N)$.

$S_p(N) = rac{1}{s + rac{1-s}{N}} = rac{1}{s + rac{p}{N}}$.

"According to Amdahl's Law, what is the maximum theoretical speedup as $N

ightarrow \infty$?"

"$\lim_{N

If a program has 5% serial work ($s = 0.05$), what is its maximum possible speedup under Amdahl's Law?

"$rac{1}{0.05} = 20

What type of scaling does Amdahl's Law model?

Strong Scaling (fixed total problem size, increasing processor count).

What type of scaling does Gustafson's Law model?

Weak Scaling (problem size grows proportionally with processor count to keep execution time constant).

State Gustafson's Law formula for Scaled Speedup $S_p(N)$.

$S_p(N) = s + N \cdot p = (1-p) + N \cdot p = N - s(N-1)$.

State Gustafson's Law formula for Efficiency $E_p(N)$.

$E_p(N) = \frac{(1-p) + N \cdot p}{N} = p + \frac{1-p}{N}$.

Why does Gustafson's Law show linear speedup potential for large systems?

As problem size scales up with processor count, the parallel workload $N \cdot p$ dominates execution, making the fixed serial part $s$ negligible.

What is Strong Scaling?

Measuring how execution time decreases as more processors are added while holding total problem size constant.