Parallel And Concurrent Programming

Asynchronous Programming

EN

Asynchronous programming is a technique for achieving concurrency without forcing one operation to block progress while it waits for another to finish. Instead of requiring each task to complete before the next can make progress, asynchronous code can suspend waiting work and continue with other rea...

Mpi

EN

The Message Passing Interface (MPI) is a standardized, portable interface for message passing across a wide range of parallel-computing systems. It provides library routines for building parallel applications whose processes communicate by explicitly sending and receiving data. The MPI standard defi...

Multithreading

EN

Multithreading lets several threads make progress within one process. Threads share its address space and resources such as open files, while each thread has its own stack, registers, and execution position...

Multiprocessing

EN

Multiprocessing runs work in several processes, each normally with its own virtual address space. Separate address spaces help contain accidental memory changes and failures. CPU-bound work can run across multiple cores when enough independent work is available...

Basic Terminology

EN

Let's start with the terms used throughout these notes. Some sound similar but describe different things: how tasks are ordered, whether they run at the same time, and what happens while they wait. These ideas apply across programming languages; later notes explain how individual languages implement...

Gpu Programming

EN

The CUDA and OpenCL listings in this chapter are platform API and kernel mechanics, separate from the C++20 application examples. CUDA kernels need a CUDA toolchain, and OpenCL examples need its runtime and headers. Raw device allocation, transfer, and launch calls are shown to explain the GPU execu...

Hardware

EN

Parallel computing uses multiple hardware execution resources to make progress on more than one operation at a time. Those resources may exist inside one CPU core, across several cores or processor sockets, on a GPU, or across multiple networked machines. The useful performance gain depends not only...

Designing Parallel Programs

EN

Designing a parallel program means restructuring a computation so that useful work can proceed concurrently without making coordination costs dominate. A good design does more than create many tasks: it exposes enough parallelism, keeps processors busy, limits communication and synchronization, pres...

Evaluating Performance

EN

Evaluating a parallel program means more than asking whether it runs faster with more processors. Good performance analysis looks at how much faster it becomes, how efficiently it uses additional resources, where time is lost, and whether the behavior remains stable as the workload or machine size g...