Designing a parallel program means turning one large computation into smaller units of work that can run at the same time. A good design does more than create many tasks: it keeps processors busy, limits communication, preserves data locality, and maps work onto hardware in a way that reduces waitin...
GPUs (Graphics Processing Units) are throughput-oriented processors designed to run the same kind of operation across many data elements at once. They were originally built for graphics workloads, where millions of pixels or vertices must be processed in parallel, but the same architecture is also e...
Asynchronous programming is a technique used to achieve concurrency, where tasks can be executed independently without waiting for other tasks to finish. It allows for nonblocking behavior, in contrast to synchronous execution that waits for one task to complete before starting the next task...
Multithreading refers to the capability of a CPU, or a single core within a multi-core processor, to execute multiple threads concurrently. A thread is the smallest unit of processing that can be scheduled by an operating system. In a multithreaded environment, a program, or process, can perform mul...
Multiprocessing involves running multiple processes simultaneously. Each process has its own memory space, making them more isolated from each other compared to threads, which share the same memory. This isolation means that multiprocessing can be more robust and less prone to errors from shared sta...
The Message Passing Interface (MPI) is a standardized and portable message-passing system designed to function on a wide variety of parallel computing architectures. It provides a set of library routines that can be called from programming languages like C, C++, and Fortran to write parallel applica...
Let's start by defining some helpful terms and emphasizing the distinctions between related concepts. In general those concepts are universal and may be applied to any programming language. The differences between the languages will be discussed in greater detail later, when we attempt to explain th...
Parallel computing is the process of breaking a task into smaller parts that can be processed simultaneously by multiple processors. These notes explore the different ways of achieving parallelism in hardware and their impact on parallel computing performance...
Evaluating the performance of parallel computing systems is crucial for understanding their efficiency and identifying potential bottlenecks. Here are some metrics and concepts for evaluating performance...