February 2009 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] MPI buffer arguments - void and call-by-reference
by N.M. Maclaren 15 Feb '09

15 Feb '09

>The only one of those that ABSOLUTELY blocks copy-in/copy-out and the >more-or-less equivalent optimisations is VOLATILE. ASYNCHRONOUS does >the least blocking, and is therefore the most efficient. Ugh. I have thought of a case where even VOLATILE doesn't block copy-in/copy-out, but it's irrelevant to this debate. I am also pretty sure that I can construct an example where TARGET isn't enough to make MPI non-blocking transfers safe, but would need to check that with the standard very carefully. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] MPI buffer arguments - void and call-by-reference
by N.M. Maclaren 15 Feb '09

15 Feb '09

On Feb 14 2009, Bill Long wrote: >Aleksandar Donev wrote: >> On Monday 09 February 2009 14:59, Rolf Rabenseifner wrote: >> >>> Fortran compilers are allowed to choose >>> call-by-incopy-and-outcopy instead of call-by-reference. >>> >> There are already at least two cases where Fortran compilers are not >> allowed to choose. These are when both the actual and the dummy have >> the attribute: TARGET, VOLATILE, and ASYNCHRONOUS. Perhaps adding yet > >Just to be clear, the "and" above should be "or". You do not need to >specify all of these options - one will do. Um. While that is true, to a first approximation, it isn't really. The reason is that all three of those have subtly different constraints on when a compiler must not use copy-in/copy-out and other optimisations. For example, consider: SUBROUTINE Fred (array, index) INTEGER, INTENT(INOUT), TARGET :: array(:) INTEGER, INTENT(INOUT), POINTER :: index(:) PRINT *, SUM(array) CALL Joe(index) PRINT *, SUM(array) CONTAINS SUBROUTINE Joe (arg1, arg2) INTEGER, INTENT(INOUT), TARGET :: arg1(:) INTEGER, INTENT(INOUT), POINTER :: arg2(:) arg2 = arg2+arg1 END SUBROUTINE Joe END SUBROUTINE Fred The compiler must recalculate the sum and must not use copy-in/copy-out, in case index points to array. But, if array were ASYNCHRONOUS or a MPI non-blocking transfer buffer (and not TARGET), it could do either or both, because it knows that there is no WAIT statement or external call in Joe, and so array can neither start to be nor cease being a pending affector at any time during the call to Fred. The only one of those that ABSOLUTELY blocks copy-in/copy-out and the more-or-less equivalent optimisations is VOLATILE. ASYNCHRONOUS does the least blocking, and is therefore the most efficient. >The ultimate problem comes because a pointer (inside the MPI chambers) >has been associated with the buffer and actions on the buffer are being >taken through that pointer. Sorry, NO! You have omitted the word 'asynchronously', and that is critical! The WHOLE point of MPI non-blocking transfers is that they are asynchronous. > The TARGET attribute identifies a variable >as being (possibly) the target of a (Fortran) pointer. Given the >similarity of the situations, it would be reasonable to require that a >buffer have the TARGET attribute if it is used as the memory for a >non-blocking MPI data transfer routine. For the above reasons, it won't help. I really don't understand why there are these objections to ASYNCHRONOUS, and would appreciate it if anyone who does understand could send me clear, explicit statements of what they object to. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] MPI buffer arguments - void and call-by-reference
by Bill Long 14 Feb '09

14 Feb '09

Aleksandar Donev wrote: > On Monday 09 February 2009 14:59, Rolf Rabenseifner wrote: > > >> Fortran compilers are allowed to choose >> call-by-incopy-and-outcopy instead of call-by-reference. >> > There are already at least two cases where Fortran compilers are not > allowed to choose. These are when both the actual and the dummy have > the attribute: TARGET, VOLATILE, and ASYNCHRONOUS. Perhaps adding yet > Just to be clear, the "and" above should be "or". You do not need to specify all of these options - one will do. The ultimate problem comes because a pointer (inside the MPI chambers) has been associated with the buffer and actions on the buffer are being taken through that pointer. The TARGET attribute identifies a variable as being (possibly) the target of a (Fortran) pointer. Given the similarity of the situations, it would be reasonable to require that a buffer have the TARGET attribute if it is used as the memory for a non-blocking MPI data transfer routine. Cheers, Bill > another attribute where the compiler does not have freedom is useful, > but I am yet to be convinced. > > So first I want to ask: Other than with non-blocking transfer (which was > the subject of a separate message), what is the problem with copy > in/out? If it is a performance issue, than that is within the realm of > compiler optimization: As you said, the compiler has the choice and it > will usually choose the faster (often compilers include a run time > check to see if the actual is contiguous). > > >> Meaning: Same as what currently done, i.e., call-by-reference >> on the caller side, if call-by-reference is possible (i.e., >> not for non-contiguous subarrays) >> > What if call-by-reference is not possible? > How is this different from what is currently there and why is it useful > enough over what is already there to warrant inclusion in the Fortran > standard? > > Thanks, > Aleks > > -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by N.M. Maclaren 12 Feb '09

12 Feb '09

On Feb 11 2009, Aleksandar Donev wrote: >On Wednesday 11 February 2009 09:08, Craig Rasmussen wrote: > >> Ignoring the precise syntax for now, the important point is whether >> an attribute on a procedure will be sufficient. > >And that question has been answered with "no" in discussions: the data >(buffer) itself must be marked somehow, not just the Send/Recv/Wait >call site. ... Yes. An attribute of the transfer buffer is both necessary and sufficient. A property of a procedure is neither sufficient nor even helpful. That is PRECISELY why the ASYNCHRONOUS attribute is an attribute of objects! Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Aleksandar Donev 11 Feb '09

11 Feb '09

On Wednesday 11 February 2009 09:08, Craig Rasmussen wrote: > Ignoring the precise syntax for now, the important point is whether > an ï¿½ attribute on a procedure will be sufficient. And that question has been answered with "no" in discussions: the data (buffer) itself must be marked somehow, not just the Send/Recv/Wait call site. I understand there was lots of traffic and you missed a lot of the emails, but that is no reason to restart circular arguments. Thanks, Aleks -- Aleksandar Donev, Ph.D. Lawrence Postdoctoral Fellow @ Lawrence Livermore National Laboratory High Performance Computational Materials Science and Chemistry E-mail: donev1(a)llnl.gov Phone: (925) 424-6816 Fax: (925) 423-0785 Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 Web: http://cherrypit.princeton.edu/donev

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Craig Rasmussen 11 Feb '09

11 Feb '09

Sure. I'm attaching 08-185r1.txt. Please note this document is a bit dated in that there is now more support from the J3 and WG5 committees for doing something beyond asking programmers to put the VOLATILE attribute on variables (you've seen email from Nick, for example). Also the TYPE(*) syntax is moving forward. Cheers, Craig On Feb 10, 2009, at 6:42 PM, Rolf Rabenseifner wrote: > Craig, > It would be nice, if you can send it to me again. > > Thank you > Rolf > > On Tue, 10 Feb 2009 15:40:32 -0700 > Craig Rasmussen <crasmussen(a)lanl.gov> wrote: >> On Feb 9, 2009, at 4:06 PM, Aleksandar Donev wrote: >>> On Monday 09 February 2009 14:57, Rolf Rabenseifner wrote: >>>> 2.3 The existing solution must continue to work, i.e., >>>> all existing and correct MPI applications must continue >>>> to work. >>> There is no "existing solution", unless you count VOLATILE, which is >>> only standard as of Fortran 2003. As you say, >> I would like to reiterate, the MPI Forum has already received >> guidance from the J3 Fortran committee regarding this issue. The >> guidance is to use the volatile attribute. I can dig up and >> resend the official J3 document if you want. I'll also discuss it >> further this afternoon and tomorrow. But I'm not sure the >> guidance will change. Somehow the user must give the compiler >> specific instruction about the usage of the buffer within the >> Fortran language. >> You don't really want Fortran to treat buffers like C does as that >> will slow down all of your Fortran program. >> Cheers, >> Craig >>> >>>> 2.4 It is not my goal to automatically correct existing wrong >>>> MPI applications, i.e., applications without the DD trick >>> the DD thing is a trick (I call them hacks), and actually does >>> not solve >>> the full problem, e.g., Nick has pointed out that problems can >>> occur at >>> the site of the call to MPI_Isend, *not* just the site of the >>> wait (but >>> the cases when it fails are marginal and likely do not happen often >>> enough to notice). >>> Best, >>> Aleks >>> -- >>> Aleksandar Donev, Ph.D. >>> Lawrence Postdoctoral Fellow @ Lawrence Livermore National >>> Laboratory >>> High Performance Computational Materials Science and Chemistry >>> E-mail: donev1(a)llnl.gov >>> Phone: (925) 424-6816 Fax: (925) 423-0785 >>> Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 >>> Web: http://cherrypit.princeton.edu/donev > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Craig Rasmussen 11 Feb '09

11 Feb '09

On Feb 10, 2009, at 6:15 PM, Craig Rasmussen wrote: > > On Feb 10, 2009, at 4:48 PM, N.M. Maclaren wrote: > >>> You don't really want Fortran to treat buffers like C does as >>> that will slow down all of your Fortran program. >> >> Please don't be ridiculous. There is not a simple choice between >> VOLATILE >> and 'C semantics' - as has been pointed out at length, the precise >> semantic >> match to MPI non-blocking transfer buffers is the ASYNCHRONOUS >> attribute. >> > Ignoring the precise syntax for now, the important point is whether an attribute on a procedure will be sufficient. Cheers, Craig

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Supalov, Alexander 11 Feb '09

11 Feb '09

Dear Craig, Thanks. Re 1, how far this "no motion" area will spread? Or more specifically, what kind of code motion will it block? Best regards. Alexander -----Original Message----- From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Craig Rasmussen Sent: Wednesday, February 11, 2009 1:55 AM To: MPI-3 Fortran working group Cc: William Gropp Subject: Re: [MPI3 Fortran] MPI non-blocking transfer More guidance from J3 Fortran: MPI 2.1 and MPI 2.1+ solution: 1. Put the transfer buffer, buf in a module (or a common block). The compiler must assume that MPI_Wait (or any other procedure) may use the module and write to buf. So code motion is inhibited and buf can't be stored in registers. 2. Declare buf to be volatile. Potential solution(s) for MPI 3.0: Provide a new attribute to the MPI_Wait procedure, let's call it the MPI_NONBLOCKING_ATTRIBUTE. The question is, what should the semantics be? I believe they are: 1. Inhibit code motion around the procedure. 2. Flush all registers before the call. 3. Reload registers after the call. The MPI_NONBLOCKING_ATTRIBUTE will only be in affect at the call site of MPI_Wait. The advantage of an attribute on the procedure is that it can occur in the interface declaration only and users code won't have to change. The is an advantage over BLOCK VOLATILE :: buf call MPI_Wait() END BLOCK In addition Nick believes that VOLATILE can't be used within a block construct in this way. If a new procedure attribute is acceptable, please let me know and I'll submit a J3 paper for consideration this week. Cheers, Craig On Feb 10, 2009, at 3:40 PM, Craig Rasmussen wrote: > > On Feb 9, 2009, at 4:06 PM, Aleksandar Donev wrote: > >> On Monday 09 February 2009 14:57, Rolf Rabenseifner wrote: >>> 2.3 The existing solution must continue to work, i.e., >>> all existing and correct MPI applications must continue >>> to work. >> There is no "existing solution", unless you count VOLATILE, which is >> only standard as of Fortran 2003. As you say, > > I would like to reiterate, the MPI Forum has already received > guidance from the J3 Fortran committee regarding this issue. The > guidance is to use the volatile attribute. I can dig up and resend > the official J3 document if you want. I'll also discuss it further > this afternoon and tomorrow. But I'm not sure the guidance will > change. Somehow the user must give the compiler specific > instruction about the usage of the buffer within the Fortran language. > > You don't really want Fortran to treat buffers like C does as that > will slow down all of your Fortran program. > > Cheers, > Craig > > >> >>> 2.4 It is not my goal to automatically correct existing wrong >>> MPI applications, i.e., applications without the DD trick >> the DD thing is a trick (I call them hacks), and actually does not >> solve >> the full problem, e.g., Nick has pointed out that problems can >> occur at >> the site of the call to MPI_Isend, *not* just the site of the wait >> (but >> the cases when it fails are marginal and likely do not happen often >> enough to notice). >> Best, >> Aleks >> -- >> Aleksandar Donev, Ph.D. >> Lawrence Postdoctoral Fellow @ Lawrence Livermore National Laboratory >> High Performance Computational Materials Science and Chemistry >> E-mail: donev1(a)llnl.gov >> Phone: (925) 424-6816 Fax: (925) 423-0785 >> Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 >> Web: http://cherrypit.princeton.edu/donev > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran _______________________________________________ mpi3-fortran mailing list mpi3-fortran(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Craig Rasmussen 10 Feb '09

10 Feb '09

On Feb 10, 2009, at 4:48 PM, N.M. Maclaren wrote: >> You don't really want Fortran to treat buffers like C does as that >> will slow down all of your Fortran program. > > Please don't be ridiculous. There is not a simple choice between > VOLATILE > and 'C semantics' - as has been pointed out at length, the precise > semantic > match to MPI non-blocking transfer buffers is the ASYNCHRONOUS > attribute. > The intent was to remind our C friends that the reason that Fortran seems so "odd" in comparison to C is that it can make assumptions that improve performance. Because C can't really know which of its variables may be aliased, I believe its true that it has to flush all registers at every call site. Fortran doesn't. Cheers, Craig

1 0

Re: [MPI3 Fortran] MPI non-blocking transfer
by Craig Rasmussen 10 Feb '09

10 Feb '09

More guidance from J3 Fortran: MPI 2.1 and MPI 2.1+ solution: 1. Put the transfer buffer, buf in a module (or a common block). The compiler must assume that MPI_Wait (or any other procedure) may use the module and write to buf. So code motion is inhibited and buf can't be stored in registers. 2. Declare buf to be volatile. Potential solution(s) for MPI 3.0: Provide a new attribute to the MPI_Wait procedure, let's call it the MPI_NONBLOCKING_ATTRIBUTE. The question is, what should the semantics be? I believe they are: 1. Inhibit code motion around the procedure. 2. Flush all registers before the call. 3. Reload registers after the call. The MPI_NONBLOCKING_ATTRIBUTE will only be in affect at the call site of MPI_Wait. The advantage of an attribute on the procedure is that it can occur in the interface declaration only and users code won't have to change. The is an advantage over BLOCK VOLATILE :: buf call MPI_Wait() END BLOCK In addition Nick believes that VOLATILE can't be used within a block construct in this way. If a new procedure attribute is acceptable, please let me know and I'll submit a J3 paper for consideration this week. Cheers, Craig On Feb 10, 2009, at 3:40 PM, Craig Rasmussen wrote: > > On Feb 9, 2009, at 4:06 PM, Aleksandar Donev wrote: > >> On Monday 09 February 2009 14:57, Rolf Rabenseifner wrote: >>> 2.3 The existing solution must continue to work, i.e., >>> all existing and correct MPI applications must continue >>> to work. >> There is no "existing solution", unless you count VOLATILE, which is >> only standard as of Fortran 2003. As you say, > > I would like to reiterate, the MPI Forum has already received > guidance from the J3 Fortran committee regarding this issue. The > guidance is to use the volatile attribute. I can dig up and resend > the official J3 document if you want. I'll also discuss it further > this afternoon and tomorrow. But I'm not sure the guidance will > change. Somehow the user must give the compiler specific > instruction about the usage of the buffer within the Fortran language. > > You don't really want Fortran to treat buffers like C does as that > will slow down all of your Fortran program. > > Cheers, > Craig > > >> >>> 2.4 It is not my goal to automatically correct existing wrong >>> MPI applications, i.e., applications without the DD trick >> the DD thing is a trick (I call them hacks), and actually does not >> solve >> the full problem, e.g., Nick has pointed out that problems can >> occur at >> the site of the call to MPI_Isend, *not* just the site of the wait >> (but >> the cases when it fails are marginal and likely do not happen often >> enough to notice). >> Best, >> Aleks >> -- >> Aleksandar Donev, Ph.D. >> Lawrence Postdoctoral Fellow @ Lawrence Livermore National Laboratory >> High Performance Computational Materials Science and Chemistry >> E-mail: donev1(a)llnl.gov >> Phone: (925) 424-6816 Fax: (925) 423-0785 >> Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 >> Web: http://cherrypit.princeton.edu/donev > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

1 0