Hello gents, This email thread is about removing the send buffer access restrictions from the MPI standard. I've attached the original proposal (below) and the slides I presented in the Jan forum meeting. Let me know if you think that we need to add or remove content. I think that this proposal is good for the March forum meeting with few things we need to complete. TODO list: - Post this proposal on the wiki pages - Address any cons not addressed in this proposal. Do you know of any? - Add Examples from real scientific applications that would benefit from this proposal. If you have any example please bring it forward. - Please review the proposal against the MPI spec to verify if any additional modifications to the spec are required. Please let me know if you think we need a conf call for this proposal. Thanks, .Erez From: owner-mpi-21_at_[hidden] [mailto:owner-mpi-21_at_[hidden]] On Behalf Of Erez Haba Sent: Monday, December 17, 2007 10:14 AM To: mpi-21_at_[hidden] Subject: [mpi-21] Proposal EH1: Send buffer access This is the proposal to remove the access restriction on the user send buffers. I think it make sense to apply this to the 2.1 document however I'm okay with this being discussed for 2.2. Thanks, .Erez Background: MPI 1.1 standard prohibits users from accessing for read their send buffer until the send operation completes. Be it access in the same thread in case of an async send operation or by another thread in the case of blocking send operation. The rational in the MPI 1.1 standard was to enable the performance for DMA engine that is not cache-coherent with the main processor. Proposal: Remove the access restriction on the send buffers. Rational: This restriction is counter intuitive for many programmers. It leads to programs that are not compliant with the spec, unaware of this limitation. This limitation prohibits a common usage of the async send API's, and or the blocking sends. For example the following code sequence is not compliant with the spec. void* my_buffer; ... MPI_Isend(my_buffer, 100, MPI_CHAR, 3, 0, MPI_COMM_WORLD, &request1); MPI_Isend(my_buffer, 100, MPI_CHAR, 4, 0, MPI_COMM_WORLD, &request2); The email thread discussing the cache-coherency machines limitation concluded that it's a non issue for these machines. The application should resolve the multi-threaded non cache-coherency issues anyhow for sending data. Cons: This change will render implementations that modify the user buffer in-place (like byte-swap) invalid. Proposed changes to the MPI document: mpi1-report.pdf: Section 3.4 Page 27 Line 17 Change: The send call described in Section 3.2.1 is blocking: it does not return until the message data and envelope have been safely stored away so that the sender is free to access and overwrite the send buffer. To: (change "access and overwrite" to "modify") The send call described in Section 3.2.1 is blocking: it does not return until the message data and envelope have been safely stored away so that the sender is free to modify the send buffer. mpi1-report.pdf: Section 3.4 Page 29 Line 48 Change: In a multi-threaded implementation of MPI, the system may de-schedule a thread that is blocked on a send or receive operation, and schedule another thread for execution in the same address space. In such a case it is the user's responsibility not to access or modify a communication buffer until the communication completes. Otherwise, the outcome of the computation is undefined. To: (remove "access or") In a multi-threaded implementation of MPI, the system may de-schedule a thread that is blocked on a send or receive operation, and schedule another thread for execution in the same address space. In such a case it is the user's responsibility not to modify a communication buffer until the communication completes. Otherwise, the outcome of the computation is undefined. mpi1-report.pdf: Section 3.4 Page 30 Lines 4-9 Remove: Rationale. We prohibit read accesses to a send buffer while it is being used, even though the send operation is not supposed to alter the content of this buffer. This may seem more stringent than necessary, but the additional restriction causes little loss of functionality and allows better performance on some systems - consider the case where data transfer is done by a DMA engine that is not cache-coherent with the main processor. (End of rationale.) mpi1-report.pdf: Section 3.9 Page 58 Line 11 Change: If the request is for a send with ready mode, then a matching receive should be posted before the call is made. The communication buffer should not be accessed after the call, and until the operation completes. To: (change "accessed" to "modified") If the request is for a send with ready mode, then a matching receive should be posted before the call is made. The communication buffer should not be modified after the call, and until the operation completes. mpi1-report.pdf: Section 4.1 Page 94 Line 12 Change: Collective routine calls can (but are not required to) return as soon as their participation in the collective communication is complete. The completion of a call indicates that the caller is now free to access locations in the communication buffer. To: (change "access" to "modify") Collective routine calls can (but are not required to) return as soon as their participation in the collective communication is complete. The completion of a call indicates that the caller is now free to modify locations in the communication buffer. mpi2-report.pdf: Section 6.3 Page 112 Line 20-31 Remove: Rationale. The rule above is more lenient than for message passing, where we do not allow two concurrent sends, with overlapping send buffers. Here, we allow two concurrent puts with overlapping send buffers. The reasons for this relaxation are 1. Users do not like that restriction, which is not very natural (it prohibits concurrent reads). 2. Weakening the rule does not prevent efficient implementation, as far as we know. 3. Weakening the rule is important for performance of RMA: we want to associate one synchronization call with as many RMA operations is possible. If puts from overlapping buffers cannot be concurrent, then we need to needlessly add synchronization points in the code. (End of rationale.) *