4.3 Parallel and Distributed Computing

Parallel vs. Distributed Computing

Parallel computing uses multiple processors within a single machine to work on tasks simultaneously, speeding up processing. Distributed computing uses multiple independent computers connected by a network to work together on a task. Both approaches divide work to solve problems faster than a single sequential processor could alone.

Speedup and Its Limits

Dividing a task among multiple processors can produce speedup — the task completes faster than on one processor. However, speedup is rarely perfectly proportional to the number of processors added, because of overhead (communication, coordination) and tasks that can't be divided evenly among processors.

Sequential vs. Parallelizable Tasks

A sequential computing solution processes one instruction at a time in order. Some problems are easily parallelizable — broken into independent subtasks solved simultaneously — while others aren't, because later steps depend on results from earlier ones. Identifying which parts of a problem can run independently is key to effective parallel/distributed solutions.

Key terms

Parallel computing

Using multiple processors within a single machine to work on parts of a task simultaneously.

Distributed computing

Using multiple independent, networked computers that work together to complete a task.

Speedup

The improvement in processing time achieved by dividing a task among multiple processors.

Sequential computing solution

A solution where instructions are processed one at a time, in order, without parallel execution.

Parallelizable problem

A problem that can be divided into independent subtasks solved simultaneously by different processors.

Try an AP-style question

What is the fastest parallel time with two processors?

1. 45 seconds
2. 40 seconds
3. 90 seconds
4. 50 seconds