Lesson 15.1: Multiprocessing – Thread-Level Parallelism to Warehouse-Scale

Power and Silicon

power wall, ILP wall: _ _ _ costs grew faster than performance, importance of multiprocessing (1)

High End Servers

growing interests in _ _ _, importance of multiprocessing (2)

Data Intensive Applications

growth of _ _ _ , internet, importance of multiprocessing (3)

TLP

an architectural paradigm where a computer system executes multiple independent execution streams (threads) concurrently

Thread Level Parallelism

meaning of TLP

TLP

relies on explicitly separate threads defined by software or the operating systems

TLP

primarily implemented using the MIMD model

Multiple Instruction, Multiple Data

meaning of MIMD

MIMD Model

multiple autonomous processor concurrently execute different instructions on completely different pieces of data

Independently

MIMD processors operate entirely _

At least n threads

in order to take advantage of an MIMD multiprocessor with n processors, we must have _ _ _ _ or processors to executes

Shared memory and Non Shared Memory

two types of MIMD

MPP, Clusters

under non-shared memory

Uniform memory access, non uniform memory access

under shared memory

SMP

small number of cores

SMP

shared single memory with uniform memory (UMA) latency

Symmetric Multiprocessors

meaning of SMP

DSM

memory distributed among processors

DSM

non-uniform memory access/latency (NUMA)

DSM

processors connected via directed (switched) and non-direct (multi-hop) interconnection networks

Distributed Shared Memory

meaning of DSM

MPP

consists of nodes with each having its own processor, memory and I/O subsystem

Massively Parallel Processing

meaning of MPP

Global

_ address space shared by all, SM-MIMD (1)

Implicitly Assigned

data is _ _ to the address space, SM-MIMD (2)

Reading, Writing

cooperate by _ _ same shared variable, SM-MIMD (3)

BUS

communication through _ , SM-MIMD (4)

Global Address Space

concept of cache coherency applies due to shared _ _ _ , SM-MIMD (5)

Private Physical Address

_ _ _ space for each processor, DM-MIMD (1)

Explicitly Assigned

data must be _ _ to the private address space, DM-MIMD (2)

Message Passing

communication/synchronization via network by _ _ , DM-MIMD (3)

Does not apply

concept of cache coherency _ _ _ because no global address space, DM-MIMD (4)

Lack of Scalability

Pros of SM-MIMD (1)

Expense

Pros of SM-MIMD (2)

Data Sharing, Fast and Uniform

Cons of SM-MIMD (1)

Global Address, User Friendly

Cons of SM-MIMD (2)

Memory is Scalable

Pros of DM-MIMD (1)

Rapidly Access

Pros of DM-MIMD (2)

Cost Effectiveness

Pros of DM-MIMD (3)

Non Uniform Memory

Cons of DM-MIMD (1)

Programmer is Responsible

Cons of DM-MIMD (2)