CHAPTER 5.1 - High-Performance Computing & AI Assisted Scientific Discovery

High performance computing

HPC

HPC

Aggregating power to solve problems too large, too complex, too time-critical for workstation.

Parallel, interconnect, algorithms

*Not just faster laptop* - ________ systems + high-speed ____________ (InfiniBand 200 Gb/s) + optimized parallel __________

Simulation, data science

Convergence of computational science + data science - __________ generates data, ____ ______ learns

Floating point operations

FLOPs

FLOPS, scalability, throughput, time-to-solution

4 HPC Metrics (FSTT)

10M

Frontier 1 ExaFLOPS is equal to ___ laptops @ 100 GFLOPS

Architecture, algorithm

Parallelism requires ____________ + _________ co-design - 1000 cores useless if algorithm serial

Computational complexity, data scale, time constraints

3 pressures in HPC, why a laptop is not enough (CDT)

Computational complexity

pressure in hpc. *measures how the time and resources required by an algorithm grow as the problem size increases*. In HPC, this helps us estimate:

• How long a computation will take

• How much memory it will need

• Whether parallelization can significantly speed it up

Big O Notation

time complexity is usually expressed using what notation?

O(1)

Growth rate in big O notation. Constant

O(log n)

Growth rate in big O notation. Very slow growth

O(n)

Growth rate in big O notation. Linear

O(n log n)

Growth rate in big O notation. Efficient for large data

O(n2)

Growth rate in big O notation. Grows quickly

O(n3)

Growth rate in big O notation. Very expensive

O(2n)

Growth rate in big O notation. Explosive growth

Intensive

Many scientific problems are inherently computationally _________:

• Climate simulation

• Molecular dynamics

• Computational fluid dynamics (CFD)

• Machine learning training

Exascale

Even with ________ computers, poor algorithmic complexity can make problems infeasible

Data scale

pressure in hpc. refers to the volume of data that must be generated, stored, transferred, and analyzed during computation. As scientific simulations and AI models grow, the challenge is often not just computation, but managing massive amounts of data

MB

To scale. 106 bytes

GB

To scale. 109 bytes

TB

To scale. 1012 bytes

PB

To scale. 1015 bytes

EB

To scale. 1018 bytes

Storage, data movement, analysis

as data grows, what 3 bottlenecks appear? (SDA)

Storage

type of bottleneck. Where do you keep petabytes of data? HPC systems use parallel file systems such as: Lustre, GPFS (Spectrum Scale), BeeGFS

Lustre, GPFS, BeeGFS

HPC systems use what 3 parallel file systems? (LGB)

Data Movement

type of bottleneck. Moving data is often slower and more expensive than computing on it. Example: • Computation may take seconds. • Transferring data may take minutes.

Computation may take seconds, Transferring data may take minutes

2 examples of data movement bottleneck (CT)

Analysis

type of bottleneck. Scientists often cannot manually examine all generated data. Machine learning and data analytics help: • Identify patterns • Detect anomalies • Extract insights automatically

Identify patterns, Detect anomalies, Extract insights automatically

Machine learning and data analytics, within the context of bottlenecks, help with what 3 actions? (IDE)

Time constraint

pressure in hpc. is the maximum acceptable time to complete a computation and deliver a useful result. In many HPC applications, getting the answer fast enough is as important as getting the correct answer

fast enough

In many HPC applications, getting the answer ____ ______ is as important as getting the correct answer

Time to solution

A key HPC metric which measures How long it takes to obtain the final answer for a given problem.

Time constraint, HPC

A ____ __________ defines the deadline for obtaining results. ___ exists largely to meet these deadlines by solving computationally and data-intensive problems within a practical time-to-solution.

Node, rack, cluster, interconnect, storage

5 parts of the HPC architecture, smallest to biggest (NRCIS)

Node, rack, cluster, interconnect, storage

____ computes, ____ groups nodes, _______ groups racks, ____________ enables communication, _______ holds and serves data. Together, these components enable an HPC system to scale from a few servers to thousands of nodes capable of performing petaflop- and exaflop scale computations

Compute Nodes

part of HPC architecture. The main building blocks of an HPC system.

Each node contains:

• Multiple CPUs (multi-core processors)

• Often one or more GPUs/accelerators

• Main memory (RAM)

*Thousands* of such nodes may exist in a supercomputer.

Thousands

_________ of compute nodes may exist in a rack in a supercomputer.

Rack

part of HPC architecture. houses multiple nodes.

Benefits:

• Organized deployment

• Shared power and cooling

• Easier maintenance

A rack may contain *dozens* of compute nodes.

Dozens

A rack may contain ______ of compute nodes.

Cluster

part of HPC architecture. Is a collection of racks working together as one HPC system.

Features:

• Thousands of CPUs/GPUs

• Managed by a scheduler (Slurm, PBS)

• Appears as a single computing resource to users

Interconnect

part of HPC architecture. The communication fabric connecting all nodes and racks.

Examples:

• InfiniBand (200 Gb/s, 400 Gb/s and beyond)

• HPE Slingshot

• High-speed Ethernet

Responsibilities:

• Node-to-node communication

• MPI message passing

• Fast data transfer

A slow interconnect can severely limit scalability

Scalability

A slow interconnect can severely limit ___________

Node-to-node communication, MPI message passing, Fast data transfer

3 Responsibilities of Interconnect in the HPC architecture (NMF)

Storage

part of HPC architecture. Shared storage accessible by the entire cluster.

Examples:

• Lustre

• GPFS (Spectrum Scale)

• BeeGFS

Purpose:

• Store input datasets

• Store simulation outputs

• Support parallel I/O from thousands of nodes

Store input datasets, Store simulation outputs, Support parallel I/O from thousands of nodes

3 purposes of the Storage in the HPC architecture (SSS)

Lustre

is a *high-performance parallel distributed file system* designed for large-scale HPC clusters and supercomputers. Unlike a normal file system on a single computer, ______ allows thousands of compute nodes to *access the same data simultaneously* at very high speeds

Task, data

2 types of parallelism (TD)

Task Parallelism

Different tasks concurrently - e.g., climate atmosphere + ocean + land models run together with exchange - needs load balancing

Data Parallelism

Same operation on different data - e.g., compute final grades for 10,000 BS CS students embarrassingly parallel, 10k independent

Amdahl's Law

S equals 1 / ((1-P)+P/N) - P parallel fraction, N cores. If 10% serial, max 10x speedup even infinite cores

Amdahl's Law

predicts the maximum speedup achievable by parallelizing a program. It states: The performance improvement from adding more processors is limited by the portion of the program that must run sequentially.

Amdahl's Law

states that the maximum speedup of a parallel program is limited by its sequential portion. Even with an unlimited number of processors, the non-parallelizable part creates a fundamental limit on performance improvement

Amdahl's Law

When problem size stays the same, some part is always serial cannot be parallelized. That serial part limits speedup, no matter how many cores you add.

Gustafson's Law

When you have more cores, you run a bigger problem, not same problem faster. Serial part stays same absolute size but becomes tiny compared to bigger parallel work, so scaling looks almost perfect

Amdahl's Law

Assumes the workload remains fixed. If part of a program is sequential, it limits the maximum speedup.

Gustafson's Law

Assumes that when more processors become available, scientists solve larger problems instead of keeping the workload fixed

Limitation, scalability

Amdahl emphasizes ___________, while Gustafson emphasizes ___________

MPI, OpenMP, CUDA, Hybrid

4 parallel programming models (MOCH)

MPI

type of parallel programming model. Distributed Memory — Between Nodes

OpenMP

type of parallel programming model. Shared Memory — Inside One Node

CUDA

type of parallel programming model. GPU — Thousands of Tiny Cores

Hybrid

type of parallel programming model. MPI + OpenMP + CUDA — Real HPC (COARE Model)

Message Passing Interface

MPI

MPI

type of parallel programming model. is the standard programming model used in HPC to allow multiple processes running on different nodes to communicate with each other. In simple terms: ___ is the language that compute nodes use to exchange data and coordinate work in a cluster

Open Multi Processing

OpenMP

OpenMP

type of parallel programming model. is an API used for shared-memory parallel programming on multicore CPUs. It allows a program to use multiple CPU cores within a single computer (or node) by creating multiple threads

Compute Unified Device Architecture

CUDA

CUDA

type of parallel programming model. is NVIDIA's parallel computing platform and programming model that allows developers to use NVIDIA GPUs for general-purpose computing, not just graphics. ____ lets programmers use thousands of GPU cores to perform calculations much faster than a CPU for highly parallel workloads.

Hybrid

type of parallel programming model. Combines multiple parallel programming technologies to fully utilize modern HPC systems. Most supercomputers use: • MPI + OpenMP + CUDA Because parallelism exists at multiple levels.

Distributed memory, Message passing, Cluster-level parallelism

3 roles of MPI (DMC)

Shared-memory parallelism, Uses multiple CPU cores

2 roles of OpenMP (SU)

Accelerates compute-intensive tasks, AI, simulations, matrix operations

2 roles of CUDA (AA)

Communication overhead, load imbalance, memory wall

3 major reasons why real HPC never reaches ideal performance (CLM)

Communication Overhead

major reason why HPC never reaches ideal performance. Processors/nodes must exchange data. Time spent Sending, Receiving messages and Synchronizing data is time not spent computing. Example: In MPI, nodes constantly communicate through InfiniBand.

Load Imbalance

major reason why HPC never reaches ideal performance. Work is not distributed equally. 1 Core finishes quickly and waits. The slowest processor determines when the whole job finishes.

Memory Wall

major reason why HPC never reaches ideal performance. CPUs are much faster than memory. Even if you have many cores, they can sit idle waiting for data. Computation speed grows faster than memory speed

Plateau

Real HPC performance never reaches ideal performance because processors must communicate, workloads are not perfectly balanced, and CPUs often wait for memory accesses. These overheads increase with scale and cause performance to _______

Overhead

Time spent talking, waiting, or fetching instead of computing.

Strong, weak

2 types of scaling (SW)

Strong Scaling

type of scaling. 1 worker → 1 hole → 10 hours | 10 workers → 1 hole → 1 hour. Same job, faster completion

Weak Scaling

type of scaling. 1 worker → 1 hole | 10 workers → 10 holes. Each worker still does one hole. Total work grows with workers

Faster, efficiently

We know an HPC system is better when it can perform computational tasks ______ and more ___________ than a conventional computer or previous system.

Speedup

key performance measure. How many times faster the HPC system completes the same task.

Execution Time

key performance measure. How much the processing time decreases.

Efficiency

key performance measure. How effectively the available CPU/GPU resources are utilized.

Throughput

key performance measure. How many tasks or calculations can be completed within a given period.

Scalability

key performance measure. How well performance improves when additional processors or computing resources are added.