For those who were not able to read the pptx slides here the ppt version.
From: Erez Haba
Sent: Tuesday, February 26, 2008 9:02 AM
To: 'mpi-buff_access_at_[hidden]'
Subject: MPI Send Buffer Access - discussion
Hello gents,
This email thread is about removing the send buffer access restrictions from the MPI standard. I've attached the original proposal (below) and the slides I presented in the Jan forum meeting.
Let me know if you think that we need to add or remove content.
I think that this proposal is good for the March forum meeting with few things we need to complete.
TODO list:
- Post this proposal on the wiki pages
- Address any cons not addressed in this proposal. Do you know of any?
- Add Examples from real scientific applications that would benefit from this proposal.
If you have any example please bring it forward.
- Please review the proposal against the MPI spec to verify if any additional modifications to the spec are required.
Please let me know if you think we need a conf call for this proposal.
Thanks,
.Erez
From: owner-mpi-21_at_[hidden] [mailto:owner-mpi-21_at_[hidden]] On Behalf Of Erez Haba
Sent: Monday, December 17, 2007 10:14 AM
To: mpi-21_at_[hidden]
Subject: [mpi-21] Proposal EH1: Send buffer access
This is the proposal to remove the access restriction on the user send buffers. I think it make sense to apply this to the 2.1 document however I'm okay with this being discussed for 2.2.
Thanks,
.Erez
Background:
MPI 1.1 standard prohibits users from accessing for read their send buffer until the send operation completes. Be it access in the same thread in case of an async send operation or by another thread in the case of blocking send operation. The rational in the MPI 1.1 standard was to enable the performance for DMA engine that is not cache-coherent with the main processor.
Proposal:
Remove the access restriction on the send buffers.
Rational:
This restriction is counter intuitive for many programmers. It leads to programs that are not compliant with the spec, unaware of this limitation. This limitation prohibits a common usage of the async send API's, and or the blocking sends.
For example the following code sequence is not compliant with the spec.
void* my_buffer;
...
MPI_Isend(my_buffer, 100, MPI_CHAR, 3, 0, MPI_COMM_WORLD, &request1);
MPI_Isend(my_buffer, 100, MPI_CHAR, 4, 0, MPI_COMM_WORLD, &request2);
The email thread discussing the cache-coherency machines limitation concluded that it's a non issue for these machines. The application should resolve the multi-threaded non cache-coherency issues anyhow for sending data.
Cons:
This change will render implementations that modify the user buffer in-place (like byte-swap) invalid.
Proposed changes to the MPI document:
mpi1-report.pdf:
Section 3.4 Page 27 Line 17
Change:
The send call described in Section 3.2.1 is blocking: it does not return until the message
data and envelope have been safely stored away so that the sender is free to access and
overwrite the send buffer.
To: (change "access and overwrite" to "modify")
The send call described in Section 3.2.1 is blocking: it does not return until the message
data and envelope have been safely stored away so that the sender is free to
modify the send buffer.
mpi1-report.pdf:
Section 3.4 Page 29 Line 48
Change:
In a multi-threaded implementation of MPI, the system may de-schedule a thread that
is blocked on a send or receive operation, and schedule another thread for execution in the
same address space. In such a case it is the user's responsibility not to access or modify a
communication buffer until the communication completes. Otherwise, the outcome of the
computation is undefined.
To: (remove "access or")
In a multi-threaded implementation of MPI, the system may de-schedule a thread that
is blocked on a send or receive operation, and schedule another thread for execution in the
same address space. In such a case it is the user's responsibility not to modify a
communication buffer until the communication completes. Otherwise, the outcome of the
computation is undefined.
mpi1-report.pdf:
Section 3.4 Page 30 Lines 4-9
Remove:
Rationale. We prohibit read accesses to a send buffer while it is being used, even
though the send operation is not supposed to alter the content of this buffer. This
may seem more stringent than necessary, but the additional restriction causes little
loss of functionality and allows better performance on some systems - consider the
case where data transfer is done by a DMA engine that is not cache-coherent with the
main processor. (End of rationale.)
mpi1-report.pdf:
Section 3.9 Page 58 Line 11
Change:
If the request is for a send with ready mode, then a matching receive should be posted
before the call is made. The communication buffer should not be accessed after the call,
and until the operation completes.
To: (change "accessed" to "modified")
If the request is for a send with ready mode, then a matching receive should be posted
before the call is made. The communication buffer should not be modified after the call,
and until the operation completes.
mpi1-report.pdf:
Section 4.1 Page 94 Line 12
Change:
Collective routine calls can (but are not required to) return as soon as their participation
in the collective communication is complete. The completion of a call indicates that the
caller is now free to access locations in the communication buffer.
To: (change "access" to "modify")
Collective routine calls can (but are not required to) return as soon as their participation
in the collective communication is complete. The completion of a call indicates that the
caller is now free to modify locations in the communication buffer.
mpi2-report.pdf:
Section 6.3 Page 112 Line 20-31
Remove:
Rationale. The rule above is more lenient than for message passing, where we do
not allow two concurrent sends, with overlapping send buffers. Here, we allow two
concurrent puts with overlapping send buffers. The reasons for this relaxation are
1. Users do not like that restriction, which is not very natural (it prohibits concurrent
reads).
2. Weakening the rule does not prevent efficient implementation, as far as we know.
3. Weakening the rule is important for performance of RMA: we want to associate
one synchronization call with as many RMA operations is possible. If puts from
overlapping buffers cannot be concurrent, then we need to needlessly add synchronization
points in the code.
(End of rationale.)
*
