Concurrent Programming

From Scratch

Concurrent Programming

A form of computing in which several computations are executed during overlapping time periods, instead of with one completing before the next starts.

Parallel Computing

A type of computation in which many calculations or processes are carried out simultaneously

Concurrent vs. Parallel

Concurrent computing is more concerned with handling
difficulties in concurrent execution, while parallel computing is more
concerned with achieving improved run-times.

Concurrency

Multiple execution flows accessing a shared resource at the same time

Parallelism

Using multiple processing resources at once to solve a problem faster

Distributed Computing

computer systems whose inter-communicating components are
located on different networked computers

Single Instruction, Single Data (SISD)

A simple sequential machine, that executes one instruction at a time, operating on single data item

Single Instruction, Multiple Data (SIMD)

A machine where each instruction is applied on a collection of items. GPUs follow this design at their lowest level

Multiple Instructions, Single Data (MISD)

May seem like an oddity, but useful for fault tolerance

Multiple Instructions, Multiple Data (MIMD)

The most versatile machine category. Multi core machines follow this paradigm, including GPUs

Shared Memory

machines involved multiple CPUs sharing a single memory.

Master-Worker

Some processors have specific functions

Symmetric Multiprocessor (SMP)

All CPUs are identical

Distributed Memory

Machines are made up of separate machines that communicate through message passing