5.3 Parallel and Distributed Computing

Parallel vs. Sequential Computing

Sequential computing solves a problem with one processor executing one instruction at a time. Parallel computing uses multiple processors simultaneously to work on different parts of a problem at the same time, which can significantly speed up the process of solving problems compared to sequential approaches.

Distributed Computing

Distributed computing divides a task across multiple devices/computers connected by a network, each completing part of the task independently. This allows large problems (like search engines or SETI@home) to be solved faster and enables redundancy — if one computer fails, others continue working, increasing reliability.

Measuring Efficiency Gains

Splitting a task among multiple processors/computers can reduce the time needed to complete it, but the speedup is not always proportional to the number of processors due to overhead like communication and coordination between processors. Some problems are easier to parallelize than others, depending on whether subtasks depend on each other's results.

Key terms

Sequential computing

Solving a problem using a single processor executing one instruction at a time.

Parallel computing

Using multiple processors simultaneously to solve different parts of a problem faster.

Distributed computing

Dividing a task across multiple networked computers that work independently on parts of the task.

Overhead in parallel computing

The extra time/resources needed for coordination and communication among processors, which can reduce speedup.

Benefit of distributed systems

Increased reliability and redundancy — if one computer fails, others can continue the task.

Try an AP-style question

Why do scientists use distributed computing for climate models?

1. Climate data is very small
2. It makes the models less accurate
3. One computer is always faster
4. The calculations are too big for one computer