Parrallel and Distributed Computing

Parallel and Distributed Computing

  • Definition: Computational methods to execute multiple tasks simultaneously or across different systems.
Fault Tolerance Review
  • Benefits of Fault Tolerance:
    • Reduces system downtime.
    • Increases reliability and availability.
    • Ensures operational continuity during failures.
  • Network Fault Tolerance:
    • Implement redundancy (multiple pathways).
    • Use of error detection mechanisms to identify and recover from failures.
  • Vulnerabilities:
    • Points of failure in hardware (e.g., server outages).
    • Network limitations may introduce bottlenecks.
Traditional Computing
  • Process: A single processor handles tasks sequentially.
  • Limitations: Speed constrained by processor speed and overheating issues.
  • Structure:
    • Inputs -> Processor -> Outputs
Parallel Computing
  • Definition: Utilizing multiple processors to run tasks simultaneously.
  • Benefits:
    • Increased performance by dividing tasks.
    • Allows multiple commands to be executed at the same time.
  • Challenges:
    • Must program processors for task division.
    • Possible latency when tasks are interdependent.
    • Programming complexity increases, suggesting room for bugs.
  • Memory: Processors share a common memory space for data exchange.
Types of Parallel Processing
  1. SIMD (Single-Instruction, Multiple-Data):
    • Executes the same single instruction on different data.
    • Example: Adding 10 to each number in a list across processors.
  2. MIMD (Multiple-Instruction, Multiple-Data):
    • Executes different instructions on different data.
    • Example: Different processors handling user inputs, animations, and image fetching simultaneously.
Distributed Computing
  • Definition: Involves multiple computers working together across a network.
  • Characteristics:
    • Each computer has its own local memory.
    • Tasks are distributed, and communication is done through message passing.
  • Advantages:
    • Provides fault tolerance; if one node fails, others can continue functioning.
    • Ideal for handling large datasets or computations (e.g., cloud services).
Concurrency
  • Definition: Method of handling multiple tasks at once, potentially not simultaneously.
  • Difference from Parallelism: Concurrency overlaps tasks, while parallelism executes them at the same time.
  • Examples:
    • Concurrency: CPU switching between tasks like downloading and typing.
    • Parallelism: Multi-core processors processing different parts of a task at the same time.
Applications in Big Data
  • Importance: Large datasets require parallel/distributed processing for efficiency.
  • Challenges: Single systems struggle with processing vast amounts of data, motivating the use of parallel systems.
Summary Comparison: Parallel vs. Distributed Computing
  • Parallel Computing:
    • Executes tasks simultaneously within a single or closely connected system.
    • Uses shared memory, enabling fast communication.
    • Example: Multi-core CPU processes tasks.
  • Distributed Computing:
    • Tasks spread across multiple machines connected by a network.
    • Each machine has its own local memory, leading to potential latency.
    • Example: Search engines distributing data across several servers.
Internet vs. World Wide Web
  • Internet: A global network of interconnected devices.
  • World Wide Web: A system of accessible pages and websites on the Internet.
  • Analogy: The Internet as highways, and the web as destinations along those highways.
The World Wide Web**
  • Created by: Sir Timothy Berners-Lee.
  • Components:
    • URL: Uniform Resource Locator used to find web resources.
    • HTTP/HTTPS: Protocols for web communication.
    • HTML: The markup language for creating web pages.
The Impact of the Web
  • Initially designed for real-time access to scientific data.
  • Evolved into a broader innovation for diverse applications.
  • Reflects the unpredictable and wide-ranging impact of technology beyond intended purposes.
Additional Terms
  • Webpage: An individual document on the web, often written in HTML.
  • Link: A reference that connects one webpage to another.
  • Website: A collection of related web pages.
  • Web Browser: Software that displays web content, interpreting HTML tags to render pages correctly.