Parallel and Distributed Computing Concepts

This set of flashcards covers key concepts, definitions, and architectural models in parallel and distributed computing, facilitating review and understanding for exam preparation.

What is parallel computing?

The simultaneous use of multiple compute resources to solve a computational problem by breaking it into discrete parts that can be solved concurrently.

Why do we use parallel computing?

To save time and/or money by shortening the completion time of tasks, solving larger/more complex problems, providing concurrency, and making better use of underlying parallel hardware.

What defines distributed computing?

Distributed computing leverages multiple independent computers, often geographically dispersed, to collaborate on a single large task.

Describe Client-Server Architecture.

A model where clients request services or resources from centralized servers, common in web applications.

What is Peer-to-Peer architecture?

A network where nodes act both as clients and servers, sharing resources directly without centralized coordination.

What is cluster computing?

Connecting multiple computers (nodes) within a single location to work together as a unified system, often used for scientific research and high-performance computing.

What is data parallelism?

Performing the same operation concurrently on multiple pieces of data, used in applications like image processing.

What is task parallelism?

Executing different tasks simultaneously, where tasks may be independent or dependent on each other.

Explain Flynn's taxonomy.

A classification of parallel computers based on instruction streams and data streams, with categories such as SISD, SIMD, MISD, and MIMD.

What is Amdahl's Law?

A formula used to find the maximum improvement of a system when only part of the system is improved, focusing on fixed-size problems.

Explain Gustafson's Law.

A concept stating that the problem size can increase with the number of processors, allowing for scalability.

What are strong scaling and weak scaling?

Strong scaling maintains a fixed problem size while adding processors; weak scaling keeps the problem size per processor fixed as more processors are added.

Define shared memory architectures.

Architectures like SMP and NUMA, where multiple processors access the same memory space and may have varying access times.

What is Cache-Only Memory Architecture (COMA)?

A shared memory architecture where local memories act as caches rather than main memory.

What is distributed memory?

A system where each processor has its own private memory, requiring explicit communication for data exchange.

What is OpenMP?

A shared-memory parallel programming model used for parallelizing applications on multi-core processors.

Define CUDA.

A parallel computing platform developed by NVIDIA for general-purpose computing using NVIDIA GPUs.

What is MPI?

A standardized interface for message-passing in distributed computing environments.

What challenges arise in distributed systems concerning synchronization?

Challenges include network latency, deadlocks, and ensuring data integrity and task coordination among distributed nodes.

What is fault tolerance?

The capability to continue operating despite failures in one or more system components, crucial for maintaining reliability and data integrity.

Describe the CAP theorem.

A principle stating that distributed systems can provide at most two of the following three guarantees: Consistency, Availability, and Partition tolerance.

What is eventual consistency?

A consistency model that allows temporary inconsistencies between nodes, with the guarantee that they will eventually converge to the same state.

What is a distributed ledger?

A database that is shared, synchronized, and replicated across multiple sites or institutions, enabling consensus on transaction validity.

What are key considerations when designing a distributed inventory management system?

Scalability, fault tolerance, consistency/availability trade-offs, network considerations, and security/privacy.