Hi All,
at the EuroPVM/MPI08 conference I was invited to comment on the proposal for
RMA (sorry for the delay). I was presenting a paper on synchronization issues
of the MPI-2 RMA standard. Therefore, my comments will focus on
synchronization in the MPI-3 RMA proposal.
I was especially looking at the Remote Completion parts. If I understand this
right, there are two mechanism to control Remote Completion:
1) The Remote Completion attribute requires remote completion before the
operation can complete ("When Remote Completion attribute is set ...,
completion guarantees that the operation has been completed on the
Target...").
2) The MPI_RMA_Complete call explicitly forces remote completion of an
operation. The proposal says "When MPI_RMA_Complete returns, all previously
issued RMA operations are completed at the specified targets" Does this make
the MPI_RMA_Complete-call a blocking call? Is there a way of non-blocking
completion checks similar to MPI_Test for send/recv?
If I'm wrong about the Remote Completion, you may stop reading here ;-).
Otherwise, the following may be a contribution for improvement:
I found out that it can be beneficial for one-sided communication to have
separate API-calls for completion and NOTIFICATION. What do I mean by
NOTIFICATION? For example, if an application knows its "last" operation (e.g.
of an iteration) on the target memory, it can tell the target about that fact
by a NOTIFICATION. This allows remote completion to start as early as
possible. While it does not force the completion (local or remote) at this
point. The operation is still non-blocking. Thus, the lib or network can
decide when to perform the operation and the completion until a blocking
fence or completion call is issued.
Concerning the interface for such a notification, it is more portable to
include the NOTIFICATION into the PUT/GET/ACC-calls. Some networks perform
better when sending a separate synchronisation message (With InfiniBand it is
better to write a small message for synchronization than make use of RDMA
with notification.). Other networks perform better if the synchronization can
be included in the header of a message. In the latter case, a separate call
for notification would require the library to either send an extra message or
defer the data transfer (this prevents/reduces overlap of computation and
communication which is one of the intention of non-blocking communication).
Thus, an additional attribute could be a nice way to signal a "last" message.
It would allow the implementation to decide whether to inline the
notification or to send an extra message.
If an application does not know the last operation, a separate call like the
MPI_RMA_Complete call comes into place (may make sense to have a non-blocking
version as well).
This would also enable the possibility to overlap the "synchronization
message" with computation (depending on the application) as it can be done
with data messages. However, this may be just a tiny bit of improvement,
since the synchronisation message is rather short.
Benefits: Performance improvements
Drawbacks: Some more complexity in the API (attribute + non-blocking
notification call)
I'm not sure if implementation issues are considered here. If yes:
Unfortunately, this NOTIFICATION signal depends on ordering of messages (for
the last message). The notification has to be delivered as the last message
regardless of a separate message or integration in the data message.
However, this is a "problem" of the implementation on top of non-ordering or
multi-path networks. I did not dig into details on this yet, but
MPI_RMA_Completion and MPI_RMA_Fence as well as MPI-2 MPI_Win_complete/fence
suffer a similar fate for completion in these environments.
Is there something I miss?
Best regards
Lars
PS:
For more details on this, the name of the paper was:
"Synchronization issues in the MPI-2 one-sided communication API."
Schneidenbach, B�hme, Schnor
