Lecture 1: Parallel Computing Architectures

Lecture 1 introduces the foundations and principles of parallel computing, including Flynn's taxonomy, memory organization, and heterogeneous architectures [1-4].

Key points covered in the lecture include:

• The need for parallel computing is driven by the demand for efficient performance across various applications and the increasing complexity of computational problems [1, 5].

• Flynn's taxonomy categorizes computer architectures based on instruction and data flows, resulting in four classifications: SISD (Single Instruction, Single Data), SIMD (Single Instruction, Multiple Data), MISD (Multiple Instructions, Single Data), and MIMD (Multiple Instructions, Multiple Data) [2, 6, 7].

• Memory organization in parallel architectures involves shared memory systems (UMA, NUMA, NoRMA, COMA) and distributed memory systems (MPP, Clusters of Workstations) [4, 8-11]. Shared memory systems can struggle with synchronizing access to shared resources, while distributed memory systems may face challenges related to large message transfers [12].

• Heterogeneous architectures incorporate diverse processors like CPUs and GPUs, enabling parallel processing and efficient task handling [13-15]. CPUs and GPUs can communicate through high-speed buses or shared memory [16].