Figure 3.
A comparison of the old (task-to-task) and new (node-to-node) communication schemes for the simplified case of two computing nodes with two cores each. In the old scheme, each task had its own export buffer, which contained the data to be exported sorted by the MPI task number of the destination. Using pair-wise MPI communication, the data are exchanged and ends up in the task-specific import buffer. In the new scheme, one export and one import buffer per node are allocated in the shared memory. Data exchanged within the node can be directly written into the import buffer. In contrast, all messages to the other node can be bundled in one, reducing the number of MPI calls and allowing better use of the entire network bandwidth through larger MPI message sizes.