July 2008 - mpi-forum - mailman3.cels.anl.gov

MPI 2.2 proposal: resolving MPI_Request_free issues
by Erez Haba 14 Jul '08

14 Jul '08

The set of issues related to MPI_Request_free have come up in the last MPI forum meeting. I've added an MPI 2.2 proposal for resolving those issues. See: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/MPI_Request_free Please comment. Thanks, .Erez MPI_Request_free issues Author: Erez Haba Description: The MPI_Request_free mechanism was provided for reasons of performance and convenience on the sending side. However there are number of issues with its definitions and the advice to use text that conflicts with many implementations. Issue #1: Advice to user quote: "Once a request is freed by a call to MPI_REQUEST_FREE, it is not possible to check for the successful completion of the associated communication with calls to MPI_WAIT or MPI_TEST. Also, if an error occurs subsequently during the communication, an error code cannot be returned to the user - such an error must be treated as fatal." This is the only place in the MPI standard that mandates an error to be FATAL, regardless of the user settings. This is truly unrecoverable because the user can not associate the error with the failed send and cannot recover after MPI_Request_free was called. This poses a problem for a fault-tolerance implementation as it must handle this failure without the ability to notify the user for the specific error for the lack of context. Issue #2: Advice to user quote: "Questions arise as to how one knows when the operations have completed when using MPI_REQUEST_FREE. Depending on the program logic, there may be other ways in which the program knows that certain operations have completed and this makes usage of MPI_REQUEST_FREE practical. For example, an active send request could be freed when the logic of the program is such that the receiver sends a reply to the message sent - the arrival of the reply informs the sender that the send has completed and the send buffer can be reused." The suggestion to reuse (free) the buffer once a reply arrived seems straight forward, however it is a naïve one which might lead to access violation at best or worse data corruption. When zero copy is being use the local interconnect resource manager might still be using the send buffer even though a reply message was received. One alternative to enable that programming paradigm is always to copy the buffer (or the tail of the buffer) when using MPI_Isend. This will prevent any misuse of the user buffer. Consider the following examples: Example 1: TCP interconnect Rank 0 on node A sends message x to rank 1 on node B using the following sequence, MPI_Isend(buffer) MPI_Request_free Upon receive message x, rank 1 sends message y back to rank 0; when rank 0 receive message y it frees the buffer with the following sequence MPI_Recv free(buffer) This would result in access violation (seg fault) as the TCP stack still tries to touch buffer after it was freed. This would happen because although node B sends back message y, it did not piggyback the TCP acknowledgment seq numbers back with message x. As a result message y was consumed by the application and buffer was freed. Hence if the TCP stack on node A tries to resend buffer tail resulting in access violation. (note that node A sent message x using zcopy) Example 2: TCP interconnect (2 connections) To make it easier to understand think about the above same problem, but now there are two TCP connections each to only deliver messages in one direction and TCP acknowledgment in the other. This setting decouples the reply message from the TCP acknowledgment and makes the previous example easier to competence. Example 3: RMA interconnect (using RMA write) In this case rank 0 on node A issue its MPI_Isend using RDMA Write. The receiver is polling on memory detecting that the write complete and sends a reply back. The reply message bypasses the hardware acknowledgment that the write was successfully complete. Rank 0 processes it and the app frees the memory which disrupts the DMA on node A and causes the send to fail. Proposed Solution 3 proposals from the least restrictive to the most restrictive: Solution #1: Remove the advice to user to reuse the buffer once a reply has arrived. There is no safe way to reuse the buffer (free), overwrite is somewhat safer. Solution #2: Remove the advice to user altogether, disallow the usage pattern of freeing active requests. Only inactive requests are allowed to be freed. (i.e., not started). Solution #3: Deprecate MPI_Request_free. Users can always use MPI_Wait to complete the request. Recommendation: Use solution #2, as users still need to free requests if they are not used; e.g., the app called MPI_Send_init but never got the start that request; hence the request still need to be freed. History Calling MPI_Request_free was added for convenience (the user does not have to remember to free the request later) and performance (freeing the request is not on the critical path when the reply message arrives and require processing). Impact on Implementations (assuming solution 2) High quality implementations would detect a call to free an active request and return an error. Impact on Applications / Users Applications must refrain freeing the request before it's complete. Therefore, requiring the application to track the request handle to completion. (this potentially breaks existing applicaitons) Alternative Solutions Entry for the Change Log Using MPI_Request_free to free an active request has been disallowed.

1 0

October MPI Forum meeting
by Richard Graham 11 Jul '08

11 Jul '08

Just to let people know that this meeting will be back at the airport location, if we can get a meeting rooms. The MS facility that we all liked so much is not available this time. Rich

1 0

MPI Forum wiki
by Jeff Squyres 08 Jul '08

08 Jul '08

Policy change: all anonymous edits on the MPI Forum wiki have been disabled (the server got spam-bombed). You must register for an account and then login to be able to edit wiki pages, tickets, etc. For uninteresting reasons, the wiki we're using is not capable of registering users, so you have to use a secondary system to register for an account: https://svn.mpi-forum.org/wiki/start Once you get a username/password on that system, you can use that username/password to logon to the real MPI Forum wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki (this information is on the front page of the wiki as well) ***NOTE 1: It is *STRONGLY* encouraged that you visit the "Preferences" link in the top-right of the MPI Forum Wiki to set your e-mail address. ***NOTE 2: We copied all existing MPI Forum Subversion usernames/ passwords to the wiki system. Hence, if you were involved in the MPI-2.1 editing effort and have a Subversion account, you do not need to re-register; you can simply use your existing Subversion username and password to login to the wiki. Finally, note that Subversion and the wiki are two separate systems; changing your password on one of them will not be reflected on the other. Many thanks to Indiana University for donating system administrator time and server resources to get all this stuff setup. -- Jeff Squyres Cisco Systems

1 0

An error in mpi program
by wg zhang 07 Jul '08

07 Jul '08

Hi,guys, I am not sure am I supposed to ask for help here or not. But I do need your guys' help to give me some hint solving this problem. I have a parallel fortran code which is written by us. It seems like working well but actually the output from 1 node to 5 nodes are the same(the results I think are correct).When I am using more than 6 nodes, the output is different,which I think is wrong. For example,the following lines are the output from 1(or 2) node calculation and 6 nodes calculation. Look at the data. The atomic positions are the same. But the order of atoms are different. Why dose this happen? Why the results are different with 5 nodes and 6 nodes? atom type atomic position 1 Al 0.0000 0.0000 -1.8869 2 O -1.6539 0.1249 -1.0842 3 O 0.7188 -1.4947 -1.0842 4 O 0.9351 1.3699 -1.0842 5 Al 2.3727 -1.3699 -0.2816 6 Al -2.3727 1.3699 0.2816 7 O 1.6539 -0.1249 1.0842 8 O -0.7188 1.4947 1.0842 9 O -0.9351 -1.3699 1.0842 10 Al 0.0000 0.0000 1.8869 1 Al 0.0000 0.0000 -1.8869 2 O -1.6539 0.1249 -1.0842 3 O 0.7188 -1.4947 -1.0842 4 Al 0.9351 1.3699 -1.0842 5 O 2.3727 -1.3699 -0.2816 6 O -2.3727 1.3699 0.2816 7 O 1.6539 -0.1249 1.0842 8 Al -0.7188 1.4947 1.0842 9 Al -0.9351 -1.3699 1.0842 10 Al 0.0000 0.0000 1.8869 Your help is really appreciated, Wayne

1 0

Re: [Mpi-forum] PLEASE READ: Dublin meeting
by Hubert Ritzdorf 07 Jul '08

07 Jul '08

Hi, it worked for me. I (*) downloaded the booking form of the Montrose Hotel, (*) requested a stay from 2nd till 5th September and (*) mentioned in the form (special request) that I want to attend the MPI Forum which is held together with the PVM/MPI conference. (*) sent it back and got the confirmation today. Hubert Richard Graham wrote: > I am told that the Montrose is the best choice. It is very close to > where we > will be meeting. > > Rich > > > On 7/3/08 12:15 PM, "Hubert Ritzdorf" <ritzdorf(a)it.neclab.eu> wrote: > > Hi Jeff, > > I can try to register to one of the Hotels (as one of those > who doesn't attend the PVM/MPI meeting). Should > I register to a specific hotel for the MPI-Forum > meeting ? > > Hubert > > Richard Graham wrote: > > Jeff, > > I will let you know as soon as I hear back about the hotel. What I > > did, since I have > > registered for Euro PVM/MPI is just use the conference rooming form > > to ask for > > a room at the Montrose for my entire stay, and it worked just fine. > > However, not > > sure about those not attending the conference. As soon as I have > > details (have been > > working on this for several months) I will let all know. > > > > Rich > > > > > > On 6/25/08 6:35 PM, "Jeff Brown" <jeffb(a)lanl.gov> wrote: > > > > can't remember if I responded to you on this ... > > > > Anyway, I'm planning to attend and would need a room checking > in Sept > > 2, out Sept 6 (4 nights). > > > > Do you have a hotel yet? > > > > Jeff > > > > At 02:11 PM 6/16/2008, you wrote: > > >rlgraham(a)ornl.gov > > > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > > ------------------------------------------------------------------------ > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > ------------------------------------------------------------------------ > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >

1 0

mpiexec
by andrea lucaroni 07 Jul '08

07 Jul '08

Hi all I download from http://www-unix.mcs.anl.gov/mpi/mpich1/download.htm mpich.tar.gz for Unix. I have gunzip ... tar ... ./configure make No find mpiexec for run the job in my cluster. Can you help me? Thank Andrea

1 0

Re: [Mpi-forum] mpi-forum Digest, Vol 6, Issue 2
by Quincey Koziol 03 Jul '08

03 Jul '08

On Jul 3, 2008, at 11:00 AM, mpi-forum-request(a)lists.mpi-forum.org wrote: > Date: Wed, 2 Jul 2008 17:49:58 -0700 > From: Jeff Squyres <Jsquyres(a)cisco.com> > Subject: [Mpi-forum] Feedback on upcoming meetings > To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> > Message-ID: <3046492B-2272-4411-B42E-113472A57A32(a)cisco.com> > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > > From the logistics meetings today, the feedback from those in the > room (which wasn't many, unfortunately) is as follows: > > - Count of those who need hotels for the Oct Chicago meeting (I think > Rich will need to ping the list to get more accurate numbers): > - Sunday: 3 > - Monday: 13 > - Tuesday: 13 > - Wednesday: 0 I'll need a room on Monday & Tuesday. Thanks, Quincey

1 0

Re: [Mpi-forum] PLEASE READ: Dublin meeting
by Richard Graham 03 Jul '08

03 Jul '08

I am told that the Montrose is the best choice. It is very close to where we will be meeting. Rich On 7/3/08 12:15 PM, "Hubert Ritzdorf" <ritzdorf(a)it.neclab.eu> wrote: > Hi Jeff, > > I can try to register to one of the Hotels (as one of those > who doesn't attend the PVM/MPI meeting). Should > I register to a specific hotel for the MPI-Forum > meeting ? > > Hubert > > Richard Graham wrote: >> > Jeff, >> > I will let you know as soon as I hear back about the hotel. What I >> > did, since I have >> > registered for Euro PVM/MPI is just use the conference rooming form >> > to ask for >> > a room at the Montrose for my entire stay, and it worked just fine. >> > However, not >> > sure about those not attending the conference. As soon as I have >> > details (have been >> > working on this for several months) I will let all know. >> > >> > Rich >> > >> > >> > On 6/25/08 6:35 PM, "Jeff Brown" <jeffb(a)lanl.gov> wrote: >> > >> > can't remember if I responded to you on this ... >> > >> > Anyway, I'm planning to attend and would need a room checking in Sept >> > 2, out Sept 6 (4 nights). >> > >> > Do you have a hotel yet? >> > >> > Jeff >> > >> > At 02:11 PM 6/16/2008, you wrote: >>> > >rlgraham(a)ornl.gov >> > >> > >> > _______________________________________________ >> > mpi-forum mailing list >> > mpi-forum(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >> > >> > >> > ------------------------------------------------------------------------ >> > >> > _______________________________________________ >> > mpi-forum mailing list >> > mpi-forum(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >> > > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] PLEASE READ: Dublin meeting
by Hubert Ritzdorf 03 Jul '08

03 Jul '08

Hi Jeff, I can try to register to one of the Hotels (as one of those who doesn't attend the PVM/MPI meeting). Should I register to a specific hotel for the MPI-Forum meeting ? Hubert Richard Graham wrote: > Jeff, > I will let you know as soon as I hear back about the hotel. What I > did, since I have > registered for Euro PVM/MPI is just use the conference rooming form > to ask for > a room at the Montrose for my entire stay, and it worked just fine. > However, not > sure about those not attending the conference. As soon as I have > details (have been > working on this for several months) I will let all know. > > Rich > > > On 6/25/08 6:35 PM, "Jeff Brown" <jeffb(a)lanl.gov> wrote: > > can't remember if I responded to you on this ... > > Anyway, I'm planning to attend and would need a room checking in Sept > 2, out Sept 6 (4 nights). > > Do you have a hotel yet? > > Jeff > > At 02:11 PM 6/16/2008, you wrote: > >rlgraham(a)ornl.gov > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > ------------------------------------------------------------------------ > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >

1 0

Re: [Mpi-forum] Feedback on upcoming meetings
by Bronis R. de Supinski 02 Jul '08

02 Jul '08

Rich: Re: > From the logistics meetings today, the feedback from those in the > room (which wasn't many, unfortunately) is as follows: > > - Count of those who need hotels for the Oct Chicago meeting (I think > Rich will need to ping the list to get more accurate numbers): > - Sunday: 3 > - Monday: 13 > - Tuesday: 13 > - Wednesday: 0 I will want a room on Monday and Tuesday for certain and probably for Sunday. > - June 2009 meeting (conflict with SC09 program committee meeting): > suggest 8-10 June OK. > - November 2009 meeting: suggest W-F the week before SC, 11-13 Nov This is a really poor suggestion. Most people are really busy the week before SC getting ready for it. The last thing you want is to be out of town during it. It would be better not to try to hold a forum meeting jointly with SC (I thought we all agreed on that position this year). Better choices would be Oct 26-28 or Nov 2-4. Even better would be to delay until Dec 7-9. Bronis > - Data on availability of flights from Europe: > - December 08 meeting: flights to San Fran and Chicago are available > - February 09 meeting: flights to San Fran and Portland are available > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >

1 0