Parallel Processing (HPC)

Serial Computing

• Single CPU

• Executes instructions one at a time, in a sequence

Parallel Computing

• Multiple CPUs

• Executes multiple instructions simultaneously

SISD is:

One cpu, one instruction

SIMD is:

One CPU, multiple instructions

• executes one at a time

MISD is:

Multiple CPUS/cores, only one instruction

• not practical

MIMD is:

Multiple CPUs/core, multiple instructions

• all the cores do their own thing

Example of SISD

Uniprocessors

Example of SIMD

Vector and array processors

Example of MISD

Not commercially implemented

Example of MIMD

SMPs, clusters, & NUMA systems

Characteristics of SMP (Symmetric multiprocessor)

• 2 or more similar processors

• Processors share same memory & I/O facilitates

• All processors share access to I/O devices

• All processors perform same functions

• All of them are controlled by one operating system

Reasons for the use of HPC

• Save time and/or money

• Solve larger problems

• Provide concurrency

• Use of non-local resources

What are the 3 types of HPC memory architecture

• Shared memory

• Distributed memory

• Hybrid distributed-shared memory

Advantage & disadvantage of shared memory

• Advantage: everyone has access

• Disadvantage: if one has an error, that everything else will have an error

What is shared memory

multiple processors can operate independently, but share same memory

What is distributed memory

Every CPU has it’s own dedicated RAM

Advantage & Disadvantage of distributed memory

• Advantage: the amount of CPUs determine how much RAM is needed & it’s faster

• Disadvantage: someone has to set it up

What is hybrid distributed-shared memory

CPU & GPU shared memory

Advantage & Disadvantage of hybrid distributed-shared memory

• Advantage: increased scalability

• Disadvantage: increased programming complexity

What are the 4 types of HPC processor architecture

• SISD

• SIMD

• MISD
  MIMD

Multiprogramming is:

Only one process can run at a time

Multiprocessing is:

More than one process can run at a time

Multithreading is:

Multiple threads running at a time