关于MPI_Send与MPI_Recv语义

Posted 2020-07-16 mazinkaiser1991

这几天在研究MPI的实现问题，由于只是移植其中的几个函数，所以不需要将每个层次的MPI都进行移植。

首先来看看MPI_Send的标准通信模式，以下翻译自MPI: A Message-Passing Interface Standard Version 3.0（当前最新版本为3.1），3.4 Communication Modes。

下载地址：http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf

3.4 Communication Modes 通信模式

</pre><pre name="code" class="plain">The send call described in Section 3.2.1 is blocking: it does not return until the message data and envelope have been safely stored away so that the sender is free to modify
the send buffer. The message might be copied directly into the matching receive buffer, or it might be copied into a temporary system buffer.

3.2.1节中描述的send（MPI_Send）函数为阻塞模式：直到数据信息与数据信封已经被安全的保存，被缓存的发送数据缓冲区就可以被任意更改了。信息可能直接被发送到了匹配的接收缓冲区中，或者他被拷贝到了一个中间系统缓冲区中。

这里要一点我个人的理解，其实MPI_Send的语义仅要求在保证发送缓冲区可以被重新使用后，MPI_Send函数就可以返回，这就是所谓的阻塞发送。至于阻塞多久，MPI标准中并没有明确的规定，可以是先将待发送信息缓冲到某个系统中的缓冲区中，也可以阻塞直到全部信息发送到了接收缓冲区中。这与具体实现相关。来简单看一个实验。源代码如下：

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include "mpi.h"

const int MAX_STRING = 100;

int main()
{
	char greeting[MAX_STRING];
	int comm_sz;
	int my_rank;

	MPI_Init(NULL,NULL);
	MPI_Comm_size(MPI_COMM_WORLD,&comm_sz);
	MPI_Comm_rank(MPI_COMM_WORLD,&my_rank);

	if(my_rank==1){
		sprintf(greeting,"Greeting from process %d of %d!",my_rank,comm_sz);
		sleep(10);
		MPI_Send(greeting,strlen(greeting)+1,MPI_CHAR,0,0,MPI_COMM_WORLD);
		printf("rank %d has returned from MPI_Send\n",my_rank);
	}else if(my_rank==2){
		sprintf(greeting,"Greeting from process %d of %d!",my_rank,comm_sz);
		MPI_Send(greeting,strlen(greeting)+1,MPI_CHAR,0,0,MPI_COMM_WORLD);
		printf("rank %d has returned from MPI_Send\n",my_rank);
	}else if(my_rank==3){
		sprintf(greeting,"Greeting from process %d of %d!",my_rank,comm_sz);
		MPI_Send(greeting,strlen(greeting)+1,MPI_CHAR,0,0,MPI_COMM_WORLD);
		printf("rank %d has returned from MPI_Send\n",my_rank);
	}else if(my_rank==0){
		MPI_Recv(greeting,MAX_STRING,MPI_CHAR,1,0,MPI_COMM_WORLD,MPI_STATUS_IGNORE);
		printf("%s\n",greeting);
		MPI_Recv(greeting,MAX_STRING,MPI_CHAR,2,0,MPI_COMM_WORLD,MPI_STATUS_IGNORE);
		printf("%s\n",greeting);
		MPI_Recv(greeting,MAX_STRING,MPI_CHAR,3,0,MPI_COMM_WORLD,MPI_STATUS_IGNORE);
		printf("%s\n",greeting);
	}

	MPI_Finalize();
	return 0;
}

运行结果如下：

rank 2 has returned from MPI_Send
rank 3 has returned from MPI_Send（运行至此处要等待一些时间）
rank 1 has returned from MPI_Send
Greeting from process 1 of 4!
Greeting from process 2 of 4!
Greeting from process 3 of 4!

通过上面的实验我们可以验证，在MPICH实现中，MPI_Send将数据首先发送到了某个缓冲区后便返回，rank1由于被挂起，所以没有执行MPI_Send函数。再来看接收端，由于应被接收的信息还没有发送到接收段，所以会一直阻塞，此时其他发送端的信息已经发送至某个缓冲区中（这个缓冲区可能位于系统中，也可能位于接收端，以上都是我个人的猜想）。

接着看原文

Message buffering decouples the send and receive operations. A blocking send can complete as soon as the message was buffered, even if no matching receive has been 
executed by the receiver. On the other hand, message buffering can be expensive, as it entails additional memory-to-memory copying, and it requires the allocation of 
memory for buffering. MPI offers the choice of several communication modes that allow one to control the choice of the communication protocol.

消息缓冲机制使得发送与接收机制解耦合。阻塞发送可以在消息被缓冲后立刻返回，即使接收端还没有开始运行接收程序。另一方面，消息缓冲可能是代价高昂的，因为它需要额外的“内存到内存”的拷贝操作，同时它还需要申请用于缓冲的存储区。MPI 提供了多种不同的消息模式，允许用户选择特定的通信协议。

The send call described in Section 3.2.1 uses the standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may
buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI 
may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been posted, and the 
data has been moved to the receiver.

在3.2.1中描述的发送函数使用标准通信模式。在这一模式中，由MPI来决定是否对将要发送的信息进行缓冲。MPI 也许会缓冲将要发送的消息。在这种情况下，发送函数可能在相对应的接收函数被调用前返回。另一方面缓冲空间可能是无法使用的，或者MPI也许选择不缓冲即将发送的消息，基于性能方面的考虑。在这种情况下，发送函数将不会返回，直到一个对应的接收函数被调用，并且数据已经被移动到接收者。

以上这一段总结起来就是一句话：在标准模式下，消息可能会被缓冲，也可能不被缓冲。如果消息被缓冲，则send函数可以立即返回；若没有被缓冲，则send函数直到接收函数接收数据后才能返回。

Thus, a send in standard mode can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted. The standard mode 
send is non-local: successful completion of the send operation may depend on the occurrence of a matching receive.

所以，标准模式下的send函数可能开始被调用，无论相对应的接收函数是否被调用。发送函数可能在对应的接收函数被调用前返回。标准模式的发送是非本地的：发送操作的成功完成可能会依赖于对应的接收函数的的调用（非缓冲模式下依赖于接收函数的调用）。

Rationale. The reluctance of MPI to mandate whether standard sends are buffering or not stems from the desire to achieve portable programs. Since any system will run
out of buffer resources as message sizes are increased, and some implementations may want to provide little buffering, MPI takes the position that correct (and therefore,
portable) programs do not rely on system buffering in standard mode. Buffering may improve the performance of a correct program, but it doesn’t affect the result of the
program. If the user wishes to guarantee a certain amount of buffering, the user-provided buffer system of Section 3.6 should be used, along with the buffered-mode
send. (End of rationale.)

原理的阐述（应该是这么翻译吧）。MPI 标准没有强制要求使用缓冲区，以使程序可移植性更好。随着消息尺寸的增加，任何系统都将用尽缓冲区资源，同时某些系统可能会提供较少的缓冲区，在标准模式下 MPI 不依赖与消息缓冲（这一句翻译的不是很好）。缓冲可能会提升正确程序的性能，但是它不影响程序的结果。如果用户希望保证一定数量的缓冲区，在3.6节中描述的用户提供的缓冲系统应该被使用，这一缓冲区也可被缓冲模式的发送所使用。

There are three additional communication modes.
A buffered mode send operation can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted. However, unlike
the standard send, this operation is local, and its completion does not depend on the occurrence of a matching receive. Thus, if a send is executed and no matching receive is
posted, then MPI must buffer the outgoing message, so as to allow the send call to complete. An error will occur if there is insufficient buffer space. The amount of available 
buffer space is controlled by the user — see Section 3.6. Buffer allocation by the user may be required for the buffered mode to be effective.

除标准模式外，还有三种通信模式

缓冲模式的发送操作可以开始无论一个对应的接收程序是否被调用。同时他也可能返回在一个对应的接收函数被调用前。然而，与标准发送不同的是，这一操作是本地的，并且它的完成不依赖于对应的接收函数的发生。所以，如果一个发送函数被调用并且对应的接收函数没有启动，那么MPI一定要缓冲正在发送的消息，以便于发送函数可以完成。一个错误将被抛出如果没有足够的缓冲空间。可获得的缓冲空间是由用户直接控制的，具体内容请见3.6节。由用户进行的缓冲空间申请可能需要使能缓冲模式。

A send that uses the synchronous mode can be started whether or not a matching receive was posted. However, the send will complete successfully only if a matching 
receive is posted, and the receive operation has started to receive the message sent by the synchronous send. Thus, the completion of a synchronous send not only 
indicates that the send buffer can be reused, but it also indicates that the receiver has reached a certain point in its execution, namely that it has started executing the 
matching receive. If both sends and receives are blocking operations then the use of the synchronous mode provides synchronous communication semantics: 
a communication does not complete at either end before both processes rendezvous at the communication. A send executed in this mode is non-local.

使用同步模式的发送函数可能被调用无论对应的接收函数是否被调用。然而，仅在一个对应的接收函数被调用，并且开始接收由同步发送函数发送的消息后发送函数才成功返回。所以，一个同步发送的完成不仅意味着发送缓冲区可以被重用，同时也意味着接收函数已经达到它运行过程中的某个确定点，意味着对应的接收函数已经开始执行。如果接收与发送都是阻塞操作，那么同步模式的使用将会使用阻塞通信语义：在两个进程在通信过程中相遇之前，通信双方均不会返回。在这种模式下的发送操作是非本地。

A send that uses the ready communication mode may be started only if the matching receive is already posted. Otherwise, the operation is erroneous and its outcome is 
undefined. On some systems, this allows the removal of a hand-shake operation that is otherwise required and results in improved performance. The completion of the 
send operation does not depend on the status of a matching receive, and merely indicates that the send buffer can be reused. A send operation that uses the ready mode 
has the same semantics as a standard send operation, or a synchronous send operation; it is merely that the sender provides additional information to the system 
(namely that a matching receive is already posted), that can save some overhead. In a correct program, therefore, a ready send could be replaced by a standard send 
with no effect on the behavior of the program other than performance.

使用就绪模式的发送函数仅在对应的接收函数被调用后才可能被启动。否则的话，这一操作是错误并且它的结果是未定义的。在某些系统中，这种模式允许握手操作的移除并且能够带来性能上的提升。发送操作的完成不依赖于对应的接收函数的状态，并且仅意味着发送缓冲区可以被重用。使用就绪模式的发送操作具有与标准模式相同的语义，或与同步模式具有相同的语义；仅由发送者提供额外信息给系统（意味着一个对应的发送操作已经被调用），这一做法可以减少一些开销。在一个正确的程序中，所以，一个就绪发送可以被标准发送所代替，无论是程序的行为还是性能都不会产生影响。

Three additional send functions are provided for the three additional communication modes. The communication mode is indicated by a one letter prefix: B for buffered, 
S for synchronous, and R for ready.

以上三种额外的发送函数是由三种额外的通信模式所提供的。通信模式是由一个字母的前缀所表示的：B表示缓冲，S表示同步，R表示就绪。

接下来看看接收操作：

There is only one receive operation, but it matches any of the send modes. The receive operation described in the last section is blocking: it returns only after the receive 
buffer contains the newly received message. A receive can complete before the matching send has completed (of course, it can complete only after the matching send 
has started).

仅有一种接收操作模式，但是它可以与任意一种发送模式相匹配。在上一节中描述的接收操作是阻塞的：仅在接收缓冲区包含有消息时才返回。一个接收函数可以在对应的发送函数完成前返回（当然，它仅可以在对应的发送函数启动后才可以返回）。

Advice to implementors. Since a synchronous send cannot complete before a matching receive is posted, one will not normally buffer messages sent by such an operation.
It is recommended to choose buffering over blocking the sender, whenever possible, for standard sends. The programmer can signal his or her preference for blocking the
sender until a matching receive occurs by using the synchronous send mode.

给实现者的建议。由于一个同步发送操作无法完成，在对应的接收函数被调用前，因此由那种操作发送消息通常不会被缓冲。建议选择缓冲而不是阻塞发送者，无论何时可能的话，对于标准发送。编程人员可以提示他或她对于发送函数阻塞的偏好知道一个对应的接收函数相匹配通过使用同步发送模式（这一句翻译的不太对）。

A possible communication protocol for the various communication modes is outlined below.
ready send: The message is sent as soon as possible.
synchronous send: The sender sends a request-to-send message. The receiver stores this request. When a matching receive is posted, the receiver sends back a 
permission-to-send message, and the sender now sends the message.
standard send: First protocol may be used for short messages, and second protocol for long messages.
buffered send: The sender copies the message into a buffer and then sends it with a nonblocking send (using the same protocol as for standard send).
Additional control messages might be needed for flow control and error recovery. Of course, there are many other possible protocols.
Ready send can be implemented as a standard send. In this case there will be no performance advantage (or disadvantage) for the use of ready send.
A standard send can be implemented as a synchronous send. In such a case, no data buffering is needed. However, users may expect some buffering.
In a multithreaded environment, the execution of a blocking communication should block only the executing thread, allowing the thread scheduler to de-schedule this
thread and schedule another thread for execution. (End of advice to implementors.)

对于多种不同通信模式，一个可能的通信协议列举如下：

就绪发送：消息尽可能快的发送。

同步发送：发送者发送一个“要求发送”消息。接收者保存这一消息。当一个匹配的接收函数被调用时，接收者返回一个准许发送的消息，此时发送者正式开始发送消息。

标准发送：第一种协议可能用于短消息，第二种协议用于长消息。

缓冲发送：发送者拷贝消息进入缓冲区然后以非阻塞方式发送（使用与标准发送相同的协议）。

为了流控与错误恢复，额外的控制信息可能被需要。当然，还可能有许多其他的协议。

就绪发送可以实现为标准发送。在这种情况下，将不会任何性能提升（或下降）对于就绪发送的使用。

一个标准发送可以实现为一个同步发送。在这种情况下，不需要数据缓冲。然而，用户可能期望一些缓冲。

在多线程环境下，阻塞通信的执行应该仅阻塞正在执行的线程，允许线程调度器重调度这一线程（进入阻塞状态）并且调度另一进程开始执行。

3.5 Semantics of Point-to-Point Communication
A valid MPI implementation guarantees certain general properties of point-to-point communication, which are described in this section.

3.5 点到点通信语义

一个有效的MPI实现保证点到点通信的某些特定性质，具体内容请见下文。

Order 
Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot 
receive the second message if the first one is still pending. If a receiver posts two receives in succession,and both match the same message, then the second receive 
operation cannot be satisfied by this message, if the first one is still pending. This requirement facilitates matching of sends to receives. It guarantees that message-passing 
code is deterministic, if processes are single-threaded and the wildcard MPI_ANY_SOURCE is not used in receives. (Some of the calls described later, such as MPI_CANCEL 
or MPI_WAITANY, are additional sources of nondeterminism.)

消息序列

消息是不可超越的：如果一个发送者连续发送两条消息到相同的目的节点，同时与相同的接收函数相对应，那么这一操作不能先接收到第二条消息如果第一条消息仍处于挂起状态。如果接收端连续调用接收函数，并且均与相同的消息对应，那么第二个接收操作无法接收这一消息，如果第一条消息还处于挂起状态。这一必要条件保证了发送与接收的匹配。它保证了消息传送代码是确定性的，如果进程是单线程的，并且通配符MPI_ANY_SOURCE也没有被接收函数所使用。（某些之后讨论的函数调用，例如MPI_CANCEL或者MPI_WAITANY，是非确定性的额外来源。）

If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multithreaded, then the
semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. The operations are logically concurrent, 
even if one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically 
concurrent receive two successively sent messages, then the two messages can match the two receives in either order.

如果一个进程是单线程执行的，那么由这个线程执行的任意两个通信都是按顺序的。另一方面，如果进程是多线程的，那么由两个不同线程间执行的发送操作的相对顺序是未定义的。这些操作是逻辑上并发的，甚至如果其中一个物理上先于另一个。在这种情况下，这两条消息的接收可以是任意顺序的。类似的，如果两个接收操作逻辑上并发的接收两条连续消息，那么这两条消息可以以任意顺序（其实就是两种情况，AB或者BA）对应两个接收函数。

Progress 
If a pair of matching send and receives have been initiated on two processes, then at least one of these two operations will complete, independently of other actions 
in the system: the send operation will complete, unless the receive is satisfied by another message, and completes; the receive operation will complete, unless the message 
sent is consumed by another matching receive that was posted at the same destination process.

进一步

如果一组对应的发送与接收函数分别在两个不同的进程上初始化，那么至少这两个操作中的一个将会完成，与系统中的其他操作无关：发送操作将会完成，除非接收操作被另一个消息所满足，并返回；接收操作将会返回，除非在相同目的进程中启动的对应接收函数消耗了已发送的消息。

这一段我也没有弄懂它的实际意思，标准中给出了一个例子，大家可以参考着看看。

3.7 Nonblocking Communication
One can improve performance on many systems by overlapping communication and computation. This is especially true on systems where communication can be executed 
autonomously by an intelligent communication controller. Light-weight threads are one mechanism for achieving such overlap. An alternative mechanism that often 
leads to better performance is to use nonblocking communication. A nonblocking send start call initiates the send operation, but does not complete it. The send start 
call can return before the message was copied out of the send buffer. A separate send complete call is needed to complete the communication, i.e., to verify that the data 
has been copied out of the send buffer. With suitable hardware, the transfer of data out of the sender memory may proceed concurrently with computations done at 
the sender after the send was initiated and before it completed. Similarly, a nonblocking receive start call initiates the receive operation, but does not complete it. The call can 
return before a message is stored into the receive buffer. A separate receive complete call is needed to complete the receive operation and verify that the data has been 
received into the receive buffer. With suitable hardware, the transfer of data into the receiver memory may proceed concurrently with computations done after the receive 
was initiated and before it completed. The use of nonblocking receives may also avoid system buffering and memory-to-memory copying, as information is provided early
on the location of the receive buffer.

3.7 非阻塞式通信

程序可以提升性能通过覆盖通信与计算。这种方法在通信可以被智能通信控制器自动执行的系统中效果尤为明显。轻量级线程是达到这一覆盖的一种机制。另一种同样可以达到较好性能的可选方法是使用非阻塞式通信。一个非阻塞式发送开始函数初始化发送操作，但是并不完成它。发送开始函数可以返回在消息被拷贝进入发送缓冲区之前。一个单独的发送完成函数被用于完成通信操作，例如，确认数据已经被拷贝进入发送缓冲区。如果具有合适的硬件，从发送端内存发送数据的操作可能与在发送端完成的计算操作相并发，在发送函数被初始化并且在它完成之前（这句话的意思是说，在发送初始化函数返回前，消息已经被发出）。类似的，一个非阻塞式接收开始函数初始化接收操作，但是并不完成它。这个函数可以返回在消息被存储进入接收缓冲区之前。一个单独的接收完成函数被用于完成接收操作并且确认数据已经被存储进入接收缓冲区。如果具有合适的硬件，进入接收端内存数据区的接收操作将与计算并发执行，在接收函数被初始化之前并且在接收初始化函数返回前。非阻塞式接收函数的使用还可能能够避免系统缓冲操作与内存到内存的拷贝，由于信息早已经被提供到了接收缓冲区所在的节点中。

以上这段话概括起来就是四个函数：

1. 发送初始化函数，仅负责初始化发送环境，不负责具体的发送工作。
2. 发送完成函数，仅负责确认发送工作是否完成，这一函数返回后原来的用户发送缓冲区就可以被重新使用了。同样不负责具体的发送工作，发送工作是由通信控制器完成。
3. 接收初始化函数，仅负责接收初始化工作（具体工作内容不明）。
4. 接收完成函数，负责确认消息是否已经被拷贝进入用户消息缓冲区。

Nonblocking send start calls can use the same four modes as blocking sends: standard, buffered, synchronous and ready. These carry the same meaning. 
Sends of all modes, ready excepted, can be started whether a matching receive has been posted or not; a nonblocking ready send can be started only if a matching receive 
is posted. In all cases, the send start call is local: it returns immediately, irrespective of the status of other processes. If the call causes some system resource to be exhausted, 
then it will fail and return an error code. Quality implementations of MPI should ensure that this happens only in “pathological” cases. That is, an MPI implementation 
should be able to support a large number of pending nonblocking operations.

非阻塞式初始化函数可以使用与阻塞式发送相同的四种模式：标准，缓冲，同步与就绪。上述四种模式意义相同。所有四种模式的发送操作，就绪模式除外，可以被启动无论对应的接收函数是否被开始调用；一个非阻塞式就绪发送可以被开始调用仅在一个对应的接收函数被调用后开始执行。在所有四种情况中，发送初始化函数是本地的：它立刻返回，无论其他进程的状态。如果函数调用导致系统资源被耗尽，那么它将会失败并且返回一个错误码。高质量的MPI实现应该保证这种情况只在“病态”的情况发生。那么，MPI 实现应该能够支持大量挂起的非阻塞操作。

The send-complete call returns when data has been copied out of the send buffer. It may carry additional meaning, depending on the send mode.
If the send mode is synchronous, then the send can complete only if a matching receive has started. That is, a receive has been posted, and has been matched with the send. 
In this case, the send-complete call is non-local. Note that a synchronous, nonblocking send may complete, if matched by a nonblocking receive, before the receive complete 
call occurs. (It can complete as soon as the sender “knows” the transfer will complete, but before the receiver “knows” the transfer will complete.)
If the send mode is buffered then the message must be buffered if there is no pending receive. In this case, the send-complete call is local, and must succeed irrespective of 
the status of a matching receive.
If the send mode is standard then the send-complete call may return before a matching receive is posted, if the message is buffered. On the other hand, the 
receive-complete may not complete until a matching receive is posted, and the message was copied into the receive buffer.
Nonblocking sends can be matched with blocking receives, and vice-versa.

发送完成函数在数据已经被全部拷贝出发送缓冲区后返回。它也许会附带额外的信息，根据发送模式。

如果发送模式是同步模式，那么发送可以完成仅在一个对应的接收端被启动后。也即，一个接收函数已经被启动，并且已经与发送函数相匹配。在这种情况下，发送完成函数是非本地的。注意，一个同步的，非阻塞式发送也许会完成，如果一个对应的非阻塞接收函数，在接收完成函数发生前。（非阻塞式发送可以完成只要发送者“知道”传送将要完成，但是先于接收者“知道”数据传送将要完成。）

如果发送模式是缓冲模式那么消息一定会被缓冲如果此时没有未决的接收函数。在这种情况下，发送完成函数是本地的，并且一定会成功不管相对应的接收函数的状态。

如果消息被缓冲，那么标准发送模式下的发送完成函数可能会返回在对应接收函数返回前。另一方面，接收完成函数（这个地方貌似错了，应该是发送完成函数）可能无法完成直到一个对应的接收函数被启动，并且消息已经被拷贝进入接收缓冲区。

非阻塞发送可以被阻塞式发送相匹配，反过来情况相同。

3.7.2 Communication Initiation
We use the same naming conventions as for blocking communication: a prefix of B, S, or R is used for buffered, synchronous or ready mode. In addition a prefix of I 
(for immediate)indicates that the call is nonblocking.

3.7.2 通信初始化

我们使用与阻塞式通信相同的命名规则：一个前缀B、S、R分别代表缓冲、同步与就绪模式。另一个前缀I（用于代表立刻）表示这一函数调用是非阻塞式的。

Start a nonblocking receive.
These calls allocate a communication request object and associate it with the request handle (the argument request). The request can be used later to query the status of the
communication or wait for its completion.
A nonblocking send call indicates that the system may start copying data out of the send buffer. The sender should not modify any part of the send buffer after a nonblocking
send operation is called, until the send completes.
A nonblocking receive call indicates that the system may start writing data into the receive buffer. The receiver should not access any part of the receive buffer after a 
nonblocking receive operation is called, until the receive completes.

初始化一个非阻塞式的接收

这些函数申请了一个通信请求对象并且与请求句柄相联系（参数request）。这一请求对象稍后可用于查询通信的状态或者等待它的完成。

一个非阻塞式发送函数意味着系统可能开始从发送缓冲区中拷贝数据。发送端不应该修改发送缓冲区的任何一部分在非阻塞式发送操作被调用后，直到发送操作完成。

一个非阻塞式接收函数意味着系统可能开始向接收缓冲区中写入数据。接收端不应该使用接收缓冲区的任何一部分在非阻塞式接收操作被调用后，直到接收操作完成。

3.7.3 Communication Completion
The functions MPI_WAIT and MPI_TEST are used to complete a nonblocking communication. The completion of a send operation indicates that the sender is now free to 
update the locations in the send buffer (the send operation itself leaves the content of the send buffer unchanged). It does not indicate that the message has been received, 
rather, it may have been buffered by the communication subsystem. However, if a synchronous mode send was used, the completion of the send operation indicates that 
a matching receive was initiated,and that the message will eventually be received by this matching receive.

 3.7.3 通信完成

MPI_WAIT与MPI_TEST 函数用于完成非阻塞式通信操作。一个发送操作的完成意味着发送端现在可任意的更新发送缓冲区（发送操作本身保持发送缓冲区上下文保持不变）。这部意味着消息应该已经被接收，相反，它可能已经被通信子系统所缓冲。而后，如果一个同步模式的发送被使用，发送操作的完成意味着一个对应的接收函数被初始化，并且消息将最终被这个对应的接收函数所接收。

The completion of a receive operation indicates that the receive buffer contains the received message, the receiver is now free to access it, and that the status object is set. It
does not indicate that the matching send operation has completed (but indicates, of course, that the send was initiated).

一个接收操作的完成意味着接收缓冲区包含有被接收函数，接收端现在可以随意使用它，并且状态对象被设置。这不意味着对应的发送操作已经完成（但意味着，当然，发送操作已经被初始化）。

We shall use the following terminology: A null handle is a handle with value MPI_REQUEST_NULL . A persistent request and the handle to it are inactive if the request is 
not associated with any ongoing communication (see Section 3.9). A handle is active if it is neither null nor inactive. An empty status is a status which is set to 
return tag = MPI_ANY_TAG , source = MPI_ANY_SOURCE , error = MPI_SUCCESS , and is also internally configured so that calls to MPI_GET_COUNT, MPI_GET_ELEMENTS, and
MPI_GET_ELEMENTS_X return count = 0 and MPI_TEST_CANCELLED returns false . We set a status variable to empty when the value returned by it is not significant. Status is set
in this way so as to prevent errors due to accesses of stale information.

我们应该使用以下术语：一个空的句柄的值为MPI_REQUEST_NULL。请求对象与指向它的句柄是非活动的如果请求对象没有与任何正在进行的通信相关联。一个句柄是活跃的如果它不是空的或者非活跃的（通过这句话的描述，感觉句柄就三种状态“活跃”、“不活跃”、“空”）。一个空的状态对象被设置为tag = MPI_ANY_TAG , source = MPI_ANY_SOURCE , error = MPI_SUCCESS，同时被内部设置以便于MPI_GET_COUNT, MPI_GET_ELEMENTS与MPI_GET_ELEMENTS_X函数返回0，MPI_TEST_CANCELLED函数返回错误。我们设置一个状态变量为空当由它（应该指这几个函数：MPI_GET_COUNT、MPI_GET_ELEMENTS、MPI_GET_ELEMENTS_X、MPI_TEST_CANCELLED）返回的值是无效的。以这种方式设置状态对象是为了避免由于使用失效信息而产生的错误。