February 2009 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] MPI non-blocking transfer
by Craig Rasmussen 10 Feb '09

10 Feb '09

More guidance from J3 Fortran: MPI 2.1 and MPI 2.1+ solution: 1. Put the transfer buffer, buf in a module (or a common block). The compiler must assume that MPI_Wait (or any other procedure) may use the module and write to buf. So code motion is inhibited and buf can't be stored in registers. 2. Declare buf to be volatile. Potential solution(s) for MPI 3.0: Provide a new attribute to the MPI_Wait procedure, let's call it the MPI_NONBLOCKING_ATTRIBUTE. The question is, what should the semantics be? I believe they are: 1. Inhibit code motion around the procedure. 2. Flush all registers before the call. 3. Reload registers after the call. The MPI_NONBLOCKING_ATTRIBUTE will only be in affect at the call site of MPI_Wait. The advantage of an attribute on the procedure is that it can occur in the interface declaration only and users code won't have to change. The is an advantage over BLOCK VOLATILE :: buf call MPI_Wait() END BLOCK In addition Nick believes that VOLATILE can't be used within a block construct in this way. If a new procedure attribute is acceptable, please let me know and I'll submit a J3 paper for consideration this week. Cheers, Craig On Feb 10, 2009, at 3:40 PM, Craig Rasmussen wrote: > > On Feb 9, 2009, at 4:06 PM, Aleksandar Donev wrote: > >> On Monday 09 February 2009 14:57, Rolf Rabenseifner wrote: >>> 2.3 The existing solution must continue to work, i.e., >>> all existing and correct MPI applications must continue >>> to work. >> There is no "existing solution", unless you count VOLATILE, which is >> only standard as of Fortran 2003. As you say, > > I would like to reiterate, the MPI Forum has already received > guidance from the J3 Fortran committee regarding this issue. The > guidance is to use the volatile attribute. I can dig up and resend > the official J3 document if you want. I'll also discuss it further > this afternoon and tomorrow. But I'm not sure the guidance will > change. Somehow the user must give the compiler specific > instruction about the usage of the buffer within the Fortran language. > > You don't really want Fortran to treat buffers like C does as that > will slow down all of your Fortran program. > > Cheers, > Craig > > >> >>> 2.4 It is not my goal to automatically correct existing wrong >>> MPI applications, i.e., applications without the DD trick >> the DD thing is a trick (I call them hacks), and actually does not >> solve >> the full problem, e.g., Nick has pointed out that problems can >> occur at >> the site of the call to MPI_Isend, *not* just the site of the wait >> (but >> the cases when it fails are marginal and likely do not happen often >> enough to notice). >> Best, >> Aleks >> -- >> Aleksandar Donev, Ph.D. >> Lawrence Postdoctoral Fellow @ Lawrence Livermore National Laboratory >> High Performance Computational Materials Science and Chemistry >> E-mail: donev1(a)llnl.gov >> Phone: (925) 424-6816 Fax: (925) 423-0785 >> Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 >> Web: http://cherrypit.princeton.edu/donev > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by N.M. Maclaren 10 Feb '09

10 Feb '09

On Feb 10 2009, Craig Rasmussen wrote: > >I would like to reiterate, the MPI Forum has already received guidance >from the J3 Fortran committee regarding this issue. The guidance is >to use the volatile attribute. I can dig up and resend the official >J3 document if you want. I'll also discuss it further this afternoon >and tomorrow. But I'm not sure the guidance will change. Somehow the >user must give the compiler specific instruction about the usage of >the buffer within the Fortran language. That is not what was said at the ISO/J3 meeting in Tokyo, and I can assure you that many of us would have voted against such guidance. The Fortran VOLATILE attribute is unsuitable for MPI non-blocking transfer buffers, on the grounds of serious inefficiency and incompatibility with good programming practice. But there is a worse problem. If you think about it, allowing an object to be VOLATILE in some scopes and not in others makes no sense whatsoever - if the object can change unpredictably anywhere, it will obviously change everywhere. I submitted a request for an interpretation on this one, but it seems to have got buried, so I need to retry. >You don't really want Fortran to treat buffers like C does as that >will slow down all of your Fortran program. Please don't be ridiculous. There is not a simple choice between VOLATILE and 'C semantics' - as has been pointed out at length, the precise semantic match to MPI non-blocking transfer buffers is the ASYNCHRONOUS attribute. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Craig Rasmussen 10 Feb '09

10 Feb '09

On Feb 9, 2009, at 4:06 PM, Aleksandar Donev wrote: > On Monday 09 February 2009 14:57, Rolf Rabenseifner wrote: >> 2.3 The existing solution must continue to work, i.e., >> all existing and correct MPI applications must continue >> to work. > There is no "existing solution", unless you count VOLATILE, which is > only standard as of Fortran 2003. As you say, I would like to reiterate, the MPI Forum has already received guidance from the J3 Fortran committee regarding this issue. The guidance is to use the volatile attribute. I can dig up and resend the official J3 document if you want. I'll also discuss it further this afternoon and tomorrow. But I'm not sure the guidance will change. Somehow the user must give the compiler specific instruction about the usage of the buffer within the Fortran language. You don't really want Fortran to treat buffers like C does as that will slow down all of your Fortran program. Cheers, Craig > >> 2.4 It is not my goal to automatically correct existing wrong >> MPI applications, i.e., applications without the DD trick > the DD thing is a trick (I call them hacks), and actually does not > solve > the full problem, e.g., Nick has pointed out that problems can occur > at > the site of the call to MPI_Isend, *not* just the site of the wait > (but > the cases when it fails are marginal and likely do not happen often > enough to notice). > Best, > Aleks > -- > Aleksandar Donev, Ph.D. > Lawrence Postdoctoral Fellow @ Lawrence Livermore National Laboratory > High Performance Computational Materials Science and Chemistry > E-mail: donev1(a)llnl.gov > Phone: (925) 424-6816 Fax: (925) 423-0785 > Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 > Web: http://cherrypit.princeton.edu/donev

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Aleksandar Donev 10 Feb '09

10 Feb '09

Hi, > This makes a lotta difference after the recent decision of the MPI > Forum to allow read access to a buffer engaged by the MPI_Isend > operation. This effectively prohibits in-place buffer modification by > the MPI that was described in the original example. I was describing modifications (optimizations) by the compiler, not the MPI library. To the compiler MPI is just another external library and so unless it knows buffer is special it cannot know not to do "in-place buffer modifications". For the other cases where this kind of thing can happen in Fortran, namely asynchronous I/O and co-arrays, the compiler has been warned by the presence of an attribute. My very strong opinion: The non-blocking issue *cannot* be resolved by the MPI Forum in independence, that is, by changing the interface of Wait, or whatever. It requires some cooperation from the Fortran committee. > BY the way, what do Fortran gurus think of this new MPI-2.2 feature? As a user I would certainly prefer to be able to read a buffer. In Fortran, for asynchronous I/O, we do allow the program to reference, but not define, a variable that is being written asynchronously. I would expect the same for MPI. But, if it has a high implementation/efficiency cost, I guess that is more important :-) Best, Aleks -- Aleksandar Donev, Ph.D. Lawrence Postdoctoral Fellow @ Lawrence Livermore National Laboratory High Performance Computational Materials Science and Chemistry E-mail: donev1(a)llnl.gov Phone: (925) 424-6816 Fax: (925) 423-0785 Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 Web: http://cherrypit.princeton.edu/donev

1 0

Re: [MPI3 Fortran] MPI Fortran meeting next week?
by Craig Rasmussen 10 Feb '09

10 Feb '09

As you know by now I couldn't come because I'm working out details of MPI changes in Fortran with the Fortran committee. I should be able to attend the next meeting. Craig On Feb 6, 2009, at 3:29 AM, Jeff Squyres wrote: > Craig -- > > I see the Fortran WG listed on the agenda for next week at the San > Jose meeting (http://meetings.mpi-forum.org/secretary/2009/02/agenda.php > ). Will there actually be a Fortran WG meeting next week? > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Supalov, Alexander 10 Feb '09

10 Feb '09

Thanks. This makes a lotta difference after the recent decision of the MPI Forum to allow read access to a buffer engaged by the MPI_Isend operation. This effectively prohibits in-place buffer modification by the MPI that was described in the original example. BY the way, what do Fortran gurus think of this new MPI-2.2 feature? -----Original Message----- From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Aleksandar Donev Sent: Tuesday, February 10, 2009 5:16 PM To: MPI-3 Fortran working group Cc: crasmussen(a)lanl.gov; wgropp(a)uiuc.edu Subject: Re: [MPI3 Fortran] MPI non-blocking transfer Hi, If you are worried about the difference between send and recv, just replace my example with: > call MPI_Irecv(buffer,...) > ... > temp=buffer(1) > buffer(1)=temp > ... > call MPI_Wait() > call DD(buffer) Aleks _______________________________________________ mpi3-fortran mailing list mpi3-fortran(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Aleksandar Donev 10 Feb '09

10 Feb '09

Hi, If you are worried about the difference between send and recv, just replace my example with: > call MPI_Irecv(buffer,...) > ... > temp=buffer(1) > buffer(1)=temp > ... > call MPI_Wait() > call DD(buffer) Aleks

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Supalov, Alexander 10 Feb '09

10 Feb '09

Hi, Thanks. Can you clarify why MPI would be putting anything into the buffer engaged in the MPI_Isend? As far as I understand, the "silly" example mentioned below would cause a problem only in this case. Imagine that MPI_Isend does not touch the buffer at all. Will anything change in this scenario? Best regards. Alexander -----Original Message----- From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Aleksandar Donev Sent: Tuesday, February 10, 2009 7:06 AM To: Rolf Rabenseifner Cc: crasmussen(a)lanl.gov; wgropp(a)uiuc.edu; mpi3-fortran(a)lists.mpi-forum.org Subject: Re: [MPI3 Fortran] MPI non-blocking transfer Hi, > I do not understand, why the DD trick should not work together > in the cases of > MPI_Isend + MPI_Wait + DD (to prohibit overwriting before MPI_Wait) I do not wish to rehash all the previous Fortran committee discussions, but, let me give the simplest, albeit contrived, example. The Fortran standard is designed so as to permit the compiler great freedom in optimization, for example, by giving the compiler the freedom to assume that a variable does not get modified behind its back. The permitted "optimizations" (code transformations) include, among other things, perfectly useless/harmless code motion: call MPI_Isend(buffer,...) ... call DD(buffer) the compiler *may* transform into intermediate code that does: call MPI_Isend(buffer,...) ... temp=buffer(1) buffer(1)=temp ... call DD(buffer) which will be broken since during the silly temp motion buffer(1) may change and then be overwritten. This is a silly example, but the point here is that there exist transformations that are rare, but allowed, and can cause parts of buffer to be copied to temporary storage and then back, overwritting what MPI is putting there. The only way to *properly* prevent that is to tell the compiler what is going on, rather than trying to *trick* it. Best, Aleks -- Aleksandar Donev, Ph.D. Lawrence Postdoctoral Fellow @ LLNL High Performance Computational Materials Science and Chemistry E-mail: donev1(a)llnl.gov Phone: (925) 424-6816 Fax: (925) 423-0785 Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 Web: http://cherrypit.princeton.edu/donev _______________________________________________ mpi3-fortran mailing list mpi3-fortran(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Aleksandar Donev 10 Feb '09

10 Feb '09

Hi, > I do not understand, why the DD trick should not work together > in the cases of > MPI_Isend + MPI_Wait + DD (to prohibit overwriting before MPI_Wait) I do not wish to rehash all the previous Fortran committee discussions, but, let me give the simplest, albeit contrived, example. The Fortran standard is designed so as to permit the compiler great freedom in optimization, for example, by giving the compiler the freedom to assume that a variable does not get modified behind its back. The permitted "optimizations" (code transformations) include, among other things, perfectly useless/harmless code motion: call MPI_Isend(buffer,...) ... call DD(buffer) the compiler *may* transform into intermediate code that does: call MPI_Isend(buffer,...) ... temp=buffer(1) buffer(1)=temp ... call DD(buffer) which will be broken since during the silly temp motion buffer(1) may change and then be overwritten. This is a silly example, but the point here is that there exist transformations that are rare, but allowed, and can cause parts of buffer to be copied to temporary storage and then back, overwritting what MPI is putting there. The only way to *properly* prevent that is to tell the compiler what is going on, rather than trying to *trick* it. Best, Aleks -- Aleksandar Donev, Ph.D. Lawrence Postdoctoral Fellow @ LLNL High Performance Computational Materials Science and Chemistry E-mail: donev1(a)llnl.gov Phone: (925) 424-6816 Fax: (925) 423-0785 Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 Web: http://cherrypit.princeton.edu/donev

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Craig Rasmussen 09 Feb '09

09 Feb '09

I have been able to set aside this week for helping out with this. I apologize for not being able to keep up with the discussion, but my funding has moved elsewhere (off from Fortran tools). I'm going through email now (and this evening) and will be able to comment more coherently tomorrow. I have a paper that I just submitted (after discussions with the relevant committee chairman) to the Fortran committee for discussion this week. I'll forward this to the MPI group later today to get comments. Cheers, Craig On Feb 9, 2009, at 4:06 PM, Aleksandar Donev wrote: > On Monday 09 February 2009 14:57, Rolf Rabenseifner wrote: >> 2.3 The existing solution must continue to work, i.e., >> all existing and correct MPI applications must continue >> to work. > There is no "existing solution", unless you count VOLATILE, which is > only standard as of Fortran 2003. As you say, >> 2.4 It is not my goal to automatically correct existing wrong >> MPI applications, i.e., applications without the DD trick > the DD thing is a trick (I call them hacks), and actually does not > solve > the full problem, e.g., Nick has pointed out that problems can occur > at > the site of the call to MPI_Isend, *not* just the site of the wait > (but > the cases when it fails are marginal and likely do not happen often > enough to notice). > Best, > Aleks > -- > Aleksandar Donev, Ph.D. > Lawrence Postdoctoral Fellow @ Lawrence Livermore National Laboratory > High Performance Computational Materials Science and Chemistry > E-mail: donev1(a)llnl.gov > Phone: (925) 424-6816 Fax: (925) 423-0785 > Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 > Web: http://cherrypit.princeton.edu/donev

1 0