Specialized Architectures and Parallel Computing Study Guide

Introduction to Specialized Architectures

• Technological Demands: Rapid advancements in technology necessitate faster and more efficient computing systems. Specialized architectures are designed to address these specific demands.
• The Shift from General-Purpose CPUs: As traditional scaling (Moore's Law) slows, building general-purpose CPUs that attempt to handle all workloads perfectly is no longer efficient.
• The Von Neumann Bottleneck: Specialized architectures bypass the traditional "Von Neumann bottleneck" by placing logic and targeted memory closer together.
• Definition: Specialized CPU architectures are tailored processors designed for specific workloads rather than general-purpose computing.
• Mechanism of Efficiency: By integrating dedicated hardware accelerators or utilizing non-traditional instruction set architectures (ISAs), these processors maximize efficiency, lower power consumption, and boost performance for targeted tasks.
• Targeted Tasks: Key application areas include artificial intelligence (AI), edge computing, and signal processing.

Moore’s Law and its Modern Limitations

• Origin and Assertions: Originally an observation by Gordon E. Moore in 1965, Moore's Law asserts that the number of transistors on a microchip doubles approximately every two years.
• Economic and Performance Implications: This progression leads to more powerful and cost-effective computing power. Computers and the machines that run on them become smaller, faster, and cheaper over time as processes become more efficient and components shrink.
• Physical Limits and Predicted End: Many experts believe Moore's Law will end sometime in the 2020s. As components continue to shrink, they reach physical limits; it is unlikely that transistors smaller than atoms can be printed.
  • Current Estimation: Depending on the element, there is only approximately $1.5\,nm$ of space left to print on.

Why Moore’s Law Does Not Apply to AI

While Moore's Law focuses on transistor density in CPUs, AI performance scales differently through three primary pillars:

1. Massive Parallelization: Modern AI chips feature thousands of cores working simultaneously. This enables the parallelization of matrix computations, allowing neural networks to process millions of parameters in parallel rather than sequentially. Speed improvements scale with the number of cores.
2. Specialized Architectures (GPUs, TPUs, ASICs):
   • GPUs: Offer thousands of cores for parallel processing.
   • TPUs (Tensor Processing Units): Specifically designed for tensor operations in neural networks.
   • Custom ASICs: Dedicated chips like Nvidia's H100 achieve up to $1000\times$ better performance-per-watt compared to general-purpose CPUs for AI tasks.
3. Algorithmic and Systems Innovations:
   • Architectures: Transformer architecture and mixture-of-experts (MoE).
   • Techniques: Quantization techniques and memory management optimizations.
   • Impact: These innovations provide efficiency gains independent of hardware, reducing computational requirements for state-of-the-art models by orders of magnitude compared to naive implementations.

The Control Unit (CU) in Computer Architecture

• Core Role: The control unit is a crucial component of the Central Processing Unit (CPU). It is responsible for coordinating and directing the execution of instructions. It acts as the "brain" of the CPU.
• Primary Functions:
  • Interprets and decodes instructions fetched from memory.
  • Controls the flow of data between different CPU components.
  • Manages the overall operation of the processor.
  • Determines the sequence of operations and ensures proper coordination among components.
• Necessity: Without a control unit, the CPU would be unable to execute instructions or perform any meaningful tasks.

Types of Control Units: Hardwired Control

• Definition: Hardwired control consists of a combinational circuit that outputs desired controls for decoding and encoding functions.
• Instruction Decoding: The instruction loaded in the Instruction Register (IR) is decoded by the instruction decoder. If the IR is an 8-bit register, the decoder generates $2^8$ ($256$) lines.
• Implementation Goals: The major goals are to minimize circuit cost and achieve greater efficiency in operational speed.
• Internal Hardware Structure:
  • Composed of two decoders, a sequence counter (SC), and logic gates.
  • Instruction Register (IR): Stores instructions fetched from memory. It contains the operation code (Opcode), the I bit, and bits $0$ through $11$.
  • Opcode Decoding: A $3 \times 8$ decoder is used for the Opcode in bits $12$ to $14$. Outputs are represented as $D_0$ through $D_7$.
  • I Bit: The operation code of bit $15$ is transferred to a flip-flop denoted by the symbol $I$.
  • Sequence Counter (SC): Has the ability to count from $0$ to $15$ in binary.
• Benefits:
  • Speed: Faster than microprogrammed control units because it uses combinational circuits.
  • Latency: The delay depends only on the number of gates.
  • Optimization: Can be optimized for the fastest mode of operation.
  • Memory: Does not require control memory.
• Limitations:
  • Complexity: Design complexity increases as more control signals are required (requiring more encoders and decoders).
  • Inflexibility: Modifications are very difficult as they require physical rearranging of wires.
  • Scalability: Adding new features is complex.
  • Maintenance: Difficult to test and correct mistakes in the original design.
  • Cost: It is considered expensive due to design overhead.

Types of Control Units: Microprogrammed Control

• Definition: A control unit whose binary control values are saved as words in memory.
• Mechanism: A controller implements instructions by constructing a collection of signals at each system clock beat. Each output signal generates one micro-operation (e.g., register transfer).
• Microinstructions:
  • Sets of control signals are saved in internal "control" memory.
  • Each bit in a microinstruction is linked to one control signal.
  • Active Bit: If the bit is set, the signal is active.
  • Inactive Bit: If the bit is cleared, the signal is inactive.
  • The CU of a microprogram-controlled computer is essentially a "computer inside a computer."
• Register Roles:
  • Control Memory Address Register: Specifies the address of the microinstruction.
  • Control Data Register: Holds the microinstruction read from memory.
• Operation Sequence:
  1. The CPU breaks an instruction down into sequential micro-operations.
  2. Control signals saved in ROM execute the instructions on the data direction.
  3. The address of the next microinstruction is generated (either sequential or via a branch).
  4. Steps are repeated until all microinstructions for the set are executed.
• Advantages:
  • Systematic design process.
  • Easier to debug and change; more flexible.
  • Allows complex functions to be carried out easily.
  • Less error-prone and inexpensive to design.
  • Control functions are implemented in software (firmware) rather than physical hardware.
• Disadvantages:
  • Slower Execution: Each instruction requires multiple microinstructions and additional fetch cycles from control memory.
  • Memory Overhead: Requires additional memory (ROM) to store the microprogram.
  • Complexity: Writing and debugging the microprogram itself can be time-consuming.
  • Inefficiency: Less efficient for simple instructions compared to hardwired units.

Comparison: Hardwired vs. Microprogrammed Control

RISC Machine Components and Instruction Set

Reduced Instruction Set Computer (RISC) machines rely on several crucial components working in unison:

• Key Components:
  • Hardwired Control Unit: Coordinates operations using combinational logic.
  • Data Path: Performs arithmetic/logical operations; contains the ALU and registers for temporary storage.
  • Instruction Cache: Small, high-speed memory for frequently used instructions; reduces fetch time.
  • Data Cache: High-speed memory for frequently accessed data; stores data closer to the CPU.
  • Main Memory: Primary storage for instructions and data; larger but slower than caches.
• RISC Characteristics:
  • Simple and Streamlined: A small set of basic instructions that can be combined for complex tasks.
  • Basic Operations: Focuses on essential arithmetic, logical, and data movement functions.
  • Single Clock Cycle Execution: Each instruction typically completes in one cycle for high performance.

RISC Instruction Tables

Arithmetic Operations

Logical Operations

Load/Store Operations

Code Example: Simple RISC Program

.data
num1:   .word 5       # Declare num1 = 5
num2:   .word 3       # Declare num2 = 3
result: .word 0       # Declare result = 0

.text
main:
    lw t0, num1       # Load num1 into register t0
    lw t1, num2       # Load num2 into register t1
    add t2, t0, t1    # Add t0 and t1, store in t2
    sw t2, result     # Store t2 into memory at 'result'
    li a7, 10         # Load system call for termination
    ecall             # End program

Knowledge Check: Load-Store Architecture

• Question: In a typical RISC processor, why is the load-store architecture used?
• Answer: (B) To simplify the instruction set by separating memory access from computation.
• Rationale:
  • RISC forbids arithmetic operations directly on memory. All ALU operations must work on registers.
  • This design simplifies pipelining due to fixed-length instructions and reduces hardware complexity.
  • It aligns with the philosophy of simple, single-cycle instructions.

Parallel Architectures

• Definition: Parallel computing architecture involves the simultaneous execution of multiple computational tasks to enhance performance.
• Components: Processors, memory hierarchy, interconnects, and software stack.
• CPU Parallelism:
  • Multi-core CPUs: Multiple processing cores on a single chip; each core executes instructions independently.
  • Multi-threaded CPUs: Supports simultaneous execution of multiple threads within a single core, overlapping tasks to increase throughput.
• GPU Parallelism:
  • Stream Processors: Numerous shader cores optimized for data-parallel operations (graphics, machine learning).
  • CUDA Cores: Specialized processing units in NVIDIA GPUs for the CUDA parallel computing platform.
• APU (Accelerated Processing Unit): Integrates both CPU and GPU cores on a single chip. CPU handles general-purpose tasks; GPU handles parallel computations and graphics.

Taxonomy of Parallel Architectures (Flynn's Taxonomy & Others)

• SIMD (Single Instruction, Multiple Data): Multiple processors carry out identical instructions, but each uses its unique data set. A single control unit supervises all processing components.
• MISD (Multiple Instruction, Single Data): Multiple processors share the same input data but use different algorithms/instructions. Output of one becomes input for the next. This architecture is rare.
• MIMD (Multiple Instruction, Multiple Data): Multiple processors independently accept their own instruction streams and data streams. Highly adaptable but requires complex algorithms.
• SPMD (Single Program, Multiple Data): A subset of MIMD where each processor carries out the same program but on different data. Common in distributed memory systems using message passing (send/receive routines).
• SMP (Symmetric Multiprocessing): Multiple identical processors share the same physical memory managed by a single OS instance.
  • Cache Coherency: Protocols ensure updates in one CPU's cache are reflected across all others.
• Clusters: A network of linked computers working as a single system.
  • Open Cluster: Each computer has its own IP address.
  • Closed Cluster: Nodes are hidden behind a gateway node for higher security.

Challenges in Parallelism

• Amdahl’s Law: Adding more processors does not yield proportional speed gains if a task has a sequential component.
  • Formula: $S(N) = \frac{1}{(1 - P) + \frac{P}{N}}$
  • Where $S(N)$ is speedup, $P$ is the parallelizable proportion, and $1-P$ is the sequential portion.
  • The sequential portion dictates the maximum achievable speedup regardless of how many processors are added.
• Load Balancing: Distributing traffic/tasks among servers to prevent overloading one unit. Can be hardware-based (appliances) or software-based (cloud services).
• Race Conditions: Occur when program correctness depends on execution timing.
  • Solution: Synchronization mechanisms like mutexes (mutual exclusion) or semaphores.
  • Risk: Excessive synchronization can serialize execution and kill performance.
• Pipeline Stalls: Occur when a stage cannot proceed.
  • Causes: Data dependencies, resource contention, and branching/control flow decisions.
• Granularity:
  • Fine-grained: Program is broken into many small tasks; suitable for shared memory.
  • Coarse-grained: Program is split into larger tasks; suitable for distributed systems but harder to load balance.

Applications and Use Cases

• Smartphones: iPhone 14 uses a 6-core CPU and 5-core GPU to run 17 trillion tasks per second.
• Supercomputers: The American Summit supercomputer processes $200$ quadrillion operations per second for physics and environmental modeling.
• Blockchain: Uses parallel computing to validate transactions simultaneously on multiple nodes, increasing throughput and scalability.
• Laptops: Multi-core processors (e.g., Intel Core i5) allow real-time video editing and 3D graphics.
• Internet of Things (IoT): Parallel computing is crucial to sift through massive sensor data in real-time for power plants or traffic systems.
• Artificial Intelligence (AI): Crucial for training ML models (facial recognition, NLP) by performing operations simultaneously to reduce training time.

Questions & Discussion

• Define Bit-level parallelism: The increase in processor word size (how many bits processed at once).
• Define Instruction-level parallelism (ILP): Executing multiple instructions from a single program simultaneously (e.g., pipelining).
• Define Task parallelism: Distributing different tasks across multiple processors on the same data set.
• Define Superword-level parallelism: A form of SIMD parallelism where instructions operate on "superwords" (vectors) of data.