Multiprocessing runs work in several processes, each normally with its own virtual address space. Separate address spaces help contain accidental memory changes and failures. CPU-bound work can run across multiple cores when enough independent work is available...
Asynchronous programming is a technique for achieving concurrency without forcing one operation to block progress while it waits for another to finish. Instead of requiring each task to complete before the next can make progress, asynchronous code can suspend waiting work and continue with other rea...
The Message Passing Interface (MPI) is a standardized, portable interface for message passing across a wide range of parallel-computing systems. It provides library routines for building parallel applications whose processes communicate by explicitly sending and receiving data. The MPI standard defi...
Parallel computing uses multiple hardware execution resources to make progress on more than one operation at a time. Those resources may exist inside one CPU core, across several cores or processor sockets, on a GPU, or across multiple networked machines. The useful performance gain depends not only...
Designing a parallel program means restructuring a computation so that useful work can proceed concurrently without making coordination costs dominate. A good design does more than create many tasks: it exposes enough parallelism, keeps processors busy, limits communication and synchronization, pres...
Multithreading lets several threads make progress within one process. Threads share its address space and resources such as open files, while each thread has its own stack, registers, and execution position...
Let's start with the terms used throughout these notes. Some sound similar but describe different things: how tasks are ordered, whether they run at the same time, and what happens while they wait. These ideas apply across programming languages; later notes explain how individual languages implement...
Evaluating a parallel program means more than asking whether it runs faster with more processors. Good performance analysis looks at how much faster it becomes, how efficiently it uses additional resources, where time is lost, and whether the behavior remains stable as the workload or machine size g...
GPUs (Graphics Processing Units) are throughput-oriented processors designed to apply similar operations across large numbers of data elements in parallel. They were originally built for graphics workloads, where millions of pixels or vertices must be processed, but the same architecture is also eff...