Parallel processing, multi core systems and GPUs

Parallel processing

Parallel processing

When a computer carries out multiple computations simultaneously to solve a given problem. One approach is SIMD: where the same operation on multiple data pieces simultaneously ( commonly by GPUs). Or by MIMD: different instructions concurrently on different data pieces.

How can parallel processing be achieved through multiple processors in the same computer or distributed or multiple cores in a CPU or GPU?

• through the SIMD approach on GPUs, where same operation is carried out on multiple data pieces

• through MIMD approach, different instructions are executed concurrently on different data pieces.

• Parallel processing within a single CPU by utilizing its cores.

• Utilizing multiple CPUs, each with their own multiple cores, for higher degrees of parallelization.

What are the benefits of a multi-core system in terms of parallel processing and running multiple programs at the same time?

• Improved parallel processing capability by carrying out multiple computations simultaneously.

• Faster execution of tasks that can be parallelized.

• Ability to run multiple programs concurrently without significant performance degradation.

Benefits of parallel processing

• increased speed

  • Tasks can be completed faster by distributing the workload across multiple processors or cores.

• improved performance

  • allows for better utilisation of resources

• scalability

  • systems can scale up by adding more processors or cores to handle larger workloads

• concurrency

  • enables the execution of multiple tasks simultaneously, enhancing multitasking capabilities

• cost-effective

  • can be more cost effective than relying on a single, high-powered processor.

Limitations of parallel processing

• Dependency Issues: Some tasks may have dependencies that prevent them from being parallelized effectively, limiting potential speedup.

• Resource Contentions: Shared resources such as memory or I/O devices may become bottlenecks or points of contention, reducing performance gains.

• Scalability Limits: There may be practical limits to the scalability of parallel systems, beyond which adding more processors or cores does not result in significant performance improvement.

• Programming Complexity: Parallel programming requires specialized skills and tools, making it more challenging for developers compared to sequential programming.

What is SIMD

An array processor has several Arithmetic Logic Units (ALUs). This allows several elements of an array to be processed at the same time rather than in sequence. With an array processor, a single instruction is issued by a control unit and that instruction is applied to a number of data sets at the same time.

An array processor is a Single Instruction Multiple Data computer or SIMD

Limitations of array processing

This architecture relies on the fact that the data sets are all acting on a single instruction. However, if these data sets somehow rely on each other then you cannot apply parallel processing.

For example, if data A has to be processed before data B then you cannot do A and B simultaneously.

This dependency is what makes parallel processing difficult to implement. And it is why sequential machines (i.e. non parallel) are still extremely common.

GPUs

A graphics processing unit (GPU) is a device which unlike CPUs has lots of independent processors which work in parallel making it very efficient at completing repetitive tasks such as image processing and machine learning. GPUs are a type of co-processor (a secondary processor designed to supplement the activities of the primary processor).

Limitations of multi-core processing (MIMD- Multiple Instruction Multiple Data)

This type of processing is dependent on being able to cut a problem down into chunks that can be processed independently. But not many problems can be broken down in this way.

Programmers also have to write specific code to make use of the multi-core CPU. This is actually quite difficult and even now most applications running on a multi-core CPU will not be making full use of all the cores.

GPUs and their uses

A graphics processing unit (GPU) is specifically designed to perform the calculations associated with displaying graphics. GPUs:

✚ have instruction sets specifically designed for the sorts of calculations required in graphics processing

✚ have the ability to process these pieces of data in parallel, referred to as single instruction multiple data (SIMD)

✚ can either be placed on a graphics card with access to their own dedicated memory …

✚ … or can be embedded within a CPU.

✚ GPUs require additional software to convert CPU functions to GPU functions. It should be remembered that GPUs cannot themselves run programs. A computer still needs a CPU to run a program, but if there is a GPU, the CPU can pass on certain processing of data (traditionally the rendering of graphics) to it.