Asynchronous programming is a technique for achieving concurrency without forcing one operation to block progress while it waits for another to finish. Instead of requiring each task to complete before the next can make progress, asynchronous code can suspend waiting work and continue with other rea...
The Message Passing Interface (MPI) is a standardized, portable interface for message passing across a wide range of parallel-computing systems. It provides library routines for building parallel applications whose processes communicate by explicitly sending and receiving data. The MPI standard defi...
Parallel computing uses multiple hardware execution resources to make progress on more than one operation at a time. Those resources may exist inside one CPU core, across several cores or processor sockets, on a GPU, or across multiple networked machines. The useful performance gain depends not only...
Designing a parallel program means restructuring a computation so that useful work can proceed concurrently without making coordination costs dominate. A good design does more than create many tasks: it exposes enough parallelism, keeps processors busy, limits communication and synchronization, pres...
Multithreading is the use of multiple threads within a process so that work can proceed concurrently. A thread is a unit of execution that the operating system can schedule. Threads in the same process share an address space and many process resources, while each thread has its own execution state, ...
Multiprocessing runs work across multiple processes. Each process normally has its own virtual address space, so processes are more isolated from one another than threads, which typically share a process's memory. This isolation reduces accidental shared-state interference and improves fault isolati...
Let's start with the terms used throughout these notes. Some sound similar but describe different things: how tasks are ordered, whether they run at the same time, and what happens while they wait. These ideas apply across programming languages; later notes explain how individual languages implement...
Evaluating a parallel program means more than asking whether it runs faster with more processors. Good performance analysis looks at how much faster it becomes, how efficiently it uses additional resources, where time is lost, and whether the behavior remains stable as the workload or machine size g...
GPUs (Graphics Processing Units) are throughput-oriented processors designed to apply similar operations across large numbers of data elements in parallel. They were originally built for graphics workloads, where millions of pixels or vertices must be processed, but the same architecture is also eff...