Parallel processing is currently in demand

Parallel and distributed computing


What is (are) the difference between parallel and distributed computing? When it comes to scalability and efficiency, solutions for computation in machine clusters are often used. Sometimes this is called parallel processing or distributed processing.

In a way, the computation always seems to be parallel because things are happening at the same time. But is distributed computation simply related to the use of more than one machine, or are there other specifics that distinguish these two types of processing? Wouldn't it be redundant to say that a calculation is distributed in parallel AND ?

Reply:


Simply put, "parallel" means running on different resources (CPUs) at the same time, while "distributed" means running on different computers, which creates network-related problems.

Parallel computing with, for example, OpenMP is not distributed, while parallel computing with message passing is often distributed.

To be in a "distributed but not parallel" environment would mean that resources are underutilized. This is rarely found, but conceptually possible.


The terms "parallel computing" and "distributed computing" overlap strongly, but they can be further differentiated. In fact, you already did this in your question, later asking for "parallel processing" and "distributed processing".

One could consider "distributed computing" to be the more general term that includes "distributed processing" as well as, for example, "distributed storage". The commonly used term "distributed" usually refers to some type of messaging over a network between physically separate computers.

The term "parallel computing" is also further defined, e.g. B. by explicitly differentiating between the terms "parallel" and "simultaneously", the first term denoting data parallelism and the latter denoting task parallelism, although this is hardly really strict and binding specifications.

You could say it that way

  • "Distributed processing" usually (although not necessarily) means that it is also "parallel processing".
  • "Distributed Computing" is more general and also covers aspects that are not related to parallelism
  • and obviously "parallel computing" / "parallel processing" does not mean that it is "distributed"

The answers presented so far are very nice, but I was also expecting a particular difference between parallel and distributed processing: the code that was executed. With parallel processes, the executed code is the same regardless of the degree of parallelism (instruction, data, task). You write one single code that is executed by different threads / processors, e.g. B. when calculating matrix products or when generating permutations.

On the other hand, distributed computing involves the execution of different algorithms / programs at the same time on different processors (from one or more machines). Such calculations are later combined to intermediate / final results using the available data communication / synchronization means (shared memory, network). In addition, distributed computing is very attractive for big data processing because it takes advantage of disk parallelism (usually the bottleneck in large databases).

Finally, the level of parallelism can be viewed more as a constraint on synchronization. For example, in the GPGPU that is Single-Instruction-Multiple-Data (SIMD), parallelism occurs by having different inputs for a single instruction exist, where each pair (data_i, command) is being executed by a different thread. This is the restriction that, in the case of deviating branches, many unnecessary calculations have to be discarded until the threads converge again. However, when it comes to CPU threads, they often diverge. However, synchronization structures can be used to enable specific sections of code to run concurrently.

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from.

By continuing, you consent to our use of cookies and other tracking technologies and affirm you're at least 16 years old or have consent from a parent or guardian.

You can read details in our Cookie policy and Privacy policy.