July 2008 - Mpi-22 - mailman3.cels.anl.gov

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 14 Jul '08

14 Jul '08

On Jul 14, 2008, at 5:50 PM, Erez Haba wrote: > Issue #1: > Advice to user quote: > > Once a request is freed by a call to MPI_REQUEST_FREE, it is not > possible to check for the successful completion of the associated > communication with calls to MPI_WAIT or MPI_TEST. Also, if an error > occurs subsequently during the communication, an error code cannot > be returned to the user  such an error must be treated as fatal. > > This is the only place in the MPI standard that mandates an error to > be FATAL, regardless of the user settings. This is truly > unrecoverable because the user can not associate the error with the > failed send and cannot recover after MPI_Request_free was called. > This poses a problem for a fault-tolerance implementation as it must > handle this failure without the ability to notify the user for the > specific error for the lack of context. I'm not sure I agree with this premise. If you need this functionality, then you shouldn't be using MPI_REQUEST_FREE. Logically speaking, if you want to associate a specific error with a specific communication request, then you must have something to tie the error *to*. In this case, it's a request -- but the application has explicitly stated that it no longer cares about the request. Therefore: if you care about the request, don't free it. > Issue #2: > Advice to user quote: > > Questions arise as to how one knows when the operations have > completed when using [snip] > causes the send to fail. I don't quite understand examples 1 and 2 (how would they cause segv's in the TCP stack). It is permissible to (pseudocode): while (bytes_to_send > 0) { rc = write(fd, buffer, bytes_to_send); if (rc > 0) { buffer += rc; bytes_to_send -= rc; } else { ...error... } } free(buffer); regardless of what the receiver does. I'm not a kernel guy; does updating TCP sequence numbers also interact with the payload buffer? FWIW: I can see the RMA interconnect example much easier. You can imagine a scenario where a sender successfully sends and the receiver successfully receives, but the hardware ACK from the receiver gets lost. The receiver then sends an MPI message back to the sender, but the sender is still in the middle of a retransmit timeout (while waiting for the hardware ACK that was lost). In this case, the user app may free the buffer too soon, resulting in a segv (or some other lion, tiger, or bear) when the sending hardware tries to retransmit. Don't get me wrong; I'm not a fan of MPI_REQUEST_FREE either. :-) > Proposed Solution > 3 proposals from the least restrictive to the most restrictive: > > Solution #1: > Remove the advice to user to reuse the buffer once a reply has > arrived. There is no safe way to reuse the buffer (free), overwrite > is somewhat safer. > > Solution #2: > Remove the advice to user altogether, disallow the usage pattern of > freeing active requests. Only inactive requests are allowed to be > freed. (i.e., not started). > > Solution #3: > Deprecate MPI_Request_free. Users can always use MPI_Wait to > complete the request. > > Recommendation: > Use solution #2, as users still need to free requests if they are > not used; e.g., the app called MPI_Send_init but never got the start > that request; hence the request still need to be freed. I'm not an apps guy, but I thought there were real world apps out there that use MPI_REQUEST_FREE. So #2 would break real apps -- but I have no idea how many. Perhaps an alternative solution would be: Solution #4: Deprecate MPI_REQUEST_FREE so that it can actually be removed someday. State that requests that are freed can never know when it is safe to free the corresponding buffer (thus making MPI_REQUEST_FREE so unattractive that its use tapers off, and also making it perfectly permissible for an MPI implementation to segv if a user frees a buffer associated with a REQUEST_FREE'd request --- such as the scenarios described above -- because that would be an erroneous program :-) ). -- Jeff Squyres Cisco Systems

1 0

MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 14 Jul '08

14 Jul '08

The set of issues related to MPI_Request_free have come up in the last MPI forum meeting. I've added an MPI 2.2 proposal for resolving those issues. See: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/MPI_Request_free Please comment. Thanks, .Erez MPI_Request_free issues Author: Erez Haba Description: The MPI_Request_free mechanism was provided for reasons of performance and convenience on the sending side. However there are number of issues with its definitions and the advice to use text that conflicts with many implementations. Issue #1: Advice to user quote: "Once a request is freed by a call to MPI_REQUEST_FREE, it is not possible to check for the successful completion of the associated communication with calls to MPI_WAIT or MPI_TEST. Also, if an error occurs subsequently during the communication, an error code cannot be returned to the user - such an error must be treated as fatal." This is the only place in the MPI standard that mandates an error to be FATAL, regardless of the user settings. This is truly unrecoverable because the user can not associate the error with the failed send and cannot recover after MPI_Request_free was called. This poses a problem for a fault-tolerance implementation as it must handle this failure without the ability to notify the user for the specific error for the lack of context. Issue #2: Advice to user quote: "Questions arise as to how one knows when the operations have completed when using MPI_REQUEST_FREE. Depending on the program logic, there may be other ways in which the program knows that certain operations have completed and this makes usage of MPI_REQUEST_FREE practical. For example, an active send request could be freed when the logic of the program is such that the receiver sends a reply to the message sent - the arrival of the reply informs the sender that the send has completed and the send buffer can be reused." The suggestion to reuse (free) the buffer once a reply arrived seems straight forward, however it is a naïve one which might lead to access violation at best or worse data corruption. When zero copy is being use the local interconnect resource manager might still be using the send buffer even though a reply message was received. One alternative to enable that programming paradigm is always to copy the buffer (or the tail of the buffer) when using MPI_Isend. This will prevent any misuse of the user buffer. Consider the following examples: Example 1: TCP interconnect Rank 0 on node A sends message x to rank 1 on node B using the following sequence, MPI_Isend(buffer) MPI_Request_free Upon receive message x, rank 1 sends message y back to rank 0; when rank 0 receive message y it frees the buffer with the following sequence MPI_Recv free(buffer) This would result in access violation (seg fault) as the TCP stack still tries to touch buffer after it was freed. This would happen because although node B sends back message y, it did not piggyback the TCP acknowledgment seq numbers back with message x. As a result message y was consumed by the application and buffer was freed. Hence if the TCP stack on node A tries to resend buffer tail resulting in access violation. (note that node A sent message x using zcopy) Example 2: TCP interconnect (2 connections) To make it easier to understand think about the above same problem, but now there are two TCP connections each to only deliver messages in one direction and TCP acknowledgment in the other. This setting decouples the reply message from the TCP acknowledgment and makes the previous example easier to competence. Example 3: RMA interconnect (using RMA write) In this case rank 0 on node A issue its MPI_Isend using RDMA Write. The receiver is polling on memory detecting that the write complete and sends a reply back. The reply message bypasses the hardware acknowledgment that the write was successfully complete. Rank 0 processes it and the app frees the memory which disrupts the DMA on node A and causes the send to fail. Proposed Solution 3 proposals from the least restrictive to the most restrictive: Solution #1: Remove the advice to user to reuse the buffer once a reply has arrived. There is no safe way to reuse the buffer (free), overwrite is somewhat safer. Solution #2: Remove the advice to user altogether, disallow the usage pattern of freeing active requests. Only inactive requests are allowed to be freed. (i.e., not started). Solution #3: Deprecate MPI_Request_free. Users can always use MPI_Wait to complete the request. Recommendation: Use solution #2, as users still need to free requests if they are not used; e.g., the app called MPI_Send_init but never got the start that request; hence the request still need to be freed. History Calling MPI_Request_free was added for convenience (the user does not have to remember to free the request later) and performance (freeing the request is not on the critical path when the reply message arrives and require processing). Impact on Implementations (assuming solution 2) High quality implementations would detect a call to free an active request and return an error. Impact on Applications / Users Applications must refrain freeing the request before it's complete. Therefore, requiring the application to track the request handle to completion. (this potentially breaks existing applicaitons) Alternative Solutions Entry for the Change Log Using MPI_Request_free to free an active request has been disallowed. *

1 0

New proposal: Remove MPI::Fint
by unknown＠example.com 11 Jul '08

11 Jul '08

The MPI::Fint type is included in a table in A.1.1 in MPI-2.1 (and 2.0), but it looks like its inclusion is a mistake. The issue is pretty simple; see the following for details: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/MPI%3A%3AFint -- Jeff Squyres Cisco Systems

1 0

MPI Forum wiki
by unknown＠example.com 08 Jul '08

08 Jul '08

Policy change: all anonymous edits on the MPI Forum wiki have been disabled (the server got spam-bombed). You must register for an account and then login to be able to edit wiki pages, tickets, etc. For uninteresting reasons, the wiki we're using is not capable of registering users, so you have to use a secondary system to register for an account: https://svn.mpi-forum.org/wiki/start Once you get a username/password on that system, you can use that username/password to logon to the real MPI Forum wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki (this information is on the front page of the wiki as well) ***NOTE 1: It is *STRONGLY* encouraged that you visit the "Preferences" link in the top-right of the MPI Forum Wiki to set your e-mail address. ***NOTE 2: We copied all existing MPI Forum Subversion usernames/ passwords to the wiki system. Hence, if you were involved in the MPI-2.1 editing effort and have a Subversion account, you do not need to re-register; you can simply use your existing Subversion username and password to login to the wiki. Finally, note that Subversion and the wiki are two separate systems; changing your password on one of them will not be reflected on the other. Many thanks to Indiana University for donating system administrator time and server resources to get all this stuff setup. -- Jeff Squyres Cisco Systems

1 0

2 new proposals
by unknown＠example.com 07 Jul '08

07 Jul '08

I have submitted two new 2.2 proposals with the help of George Bosilca: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/TYPE_GET_CONTENTS https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/RepeatedNeighborGra… These are issues that came up as side-effects of MPI-2.1; I brought both of them up at the Menlo Park Forum meeting. -- Jeff Squyres Cisco Systems

1 0

MPI 2.2 Activities
by unknown＠example.com 03 Jul '08

03 Jul '08

Now that MPI 2.1 is done, MPI 2.2 is kicking into high gear. We will be using the 2.2 wiki ( http://svn.mpi-forum.org/trac/mpi-forum-web/ wiki/MpiTwoTwoWikiPage ) to discuss and prepare items; please see the (amended) presentation from this week's MPI Forum meeting for more details about the process. Also see the link "Web page of MPI 2.2 errata items" ; if you submitted an item to either the old MPI errata list or to the 2.1 process that was deferred to 2.2, please check and see if it is still valid and appropriate for MPI 2.2, and if so, prepare a wiki page for it (we may use tickets instead of wiki pages - the difference should be small to anyone preparing an entry). It the entry isn't relevant to 2.2, please let me know so that I can remove it from the pending items. Thanks! Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign *

1 0