January 2009 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3909) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Bill Long 23 Jan '09

23 Jan '09

N.M. Maclaren wrote: > On Jan 23 2009, Bill Long wrote: >> >> Since a WAIT statement could occur in a subprogram (at least ones the >> compiler does not know about, such as ones not intrinsic or inlined) >> then it is part of the required implementation that operations on a >> variable with the asynchronous attribute not be moved across such a >> call. Included in the class of calls that block code motion would be >> those to the MPI routines. Having the asynchronous attribute in the >> caller scope effectively solves the code motion problem. > > Agreed. > >> Having it on the corresponding dummy probably solves it as well. > > What corresponding dummy? There may not BE one! If there is one, it should work. However, you are right that the general solution has to rely on having the asynchronous attribute in the caller. (Which probably means a code modification for the user.) > >> Jim pointed out that asynchronous may not be that useful in the end. >> I think this is an important point. As noted above, asynchronous can >> solve the code motion problem. It is not clear what you mean by >> "solve" in regard to the other main problem - copy-in/copy-out of >> actual arguments. What the asynchronous attribute will do is cause >> a program that would otherwise have to include copy-in/copy-out code >> at the call to the MPI_Ixxx routine to, instead, get a hard error at >> compile time based on a constraint violation. It does not make the >> program "work", which is probably somewhere on the user's agenda. > > That is correct. However, to make arbitrary broken programs work is > beyond any mortal's ability! The best that we can hope for is to > minimise > the amount of rewriting needed. > >> To get a working program, there are two possible solutions that I see: >> >> 1) Require that the user always supply a contiguous actual argument >> corresponding to the buffer dummy (in which case it seems to make >> more sense for THAT to be the requirement on the user in the first >> place), or >> >> 2) Provide a Fortran interface than can handle non-contiguous >> sections by having assumed-shape corresponding dummies. This >> solution off-loads the bulk of the work to the MPI folks. But is does >> seem more robust. > > Neither of those solutions will work. Sorry. > > Consider the situation where procedure Fred allocates a buffer, and then > passes it down through two call chains: > Fred->Joe->Bert->Alf->MPI_Isend and > Fred->Pete->Nick->Greg->MPI_Wait. The buffer is then used in Greg. The > requirement is that neither of the call chains performs a > copy-in/copy-out > at any stage. It is perfectly legal to pass a non-contiguous section to > an assumed size dummy (which is then contiguous); it doesn't matter > WHAT the > interface to MPI_Isend is - that assumed size array is a copy of the > buffer > that was allocated in Fred. > > The point is that the above problem has ALREADY been addressed for > Fortran > asynchronous I/O. If the politics are such that some vendors will refuse > to accept that solution for genuinely asynchronous transfers, then we > have > a big problem. But, make no mistake, it's political and not technical. > The problem is solved for asynchronous I/O in Fortran by exactly the same means. The solution relies on C1239 and C1240 [299:12-19] in 09-007. If you violate either of these constraints, your program will be rejected by the compiler and you have to fix it. Fixing it in Fortran - making the dummy assumed-shape - is easy. That's equivalent to the second option above. The alternative is to avoid the "If..." part at the beginning of the constraints, which is the first option above. I don't see any other options. If we plan to base a solution on asynchronous (or volatile) for MPI, then the solution is exactly the same as for asynchronous I/O in Fortran. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3909) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Van Snyder 23 Jan '09

23 Jan '09

On Fri, 2009-01-23 at 11:39 -0800, Bill Long wrote: > > Van Snyder wrote: > > On Fri, 2009-01-23 at 01:56 -0800, N.M. Maclaren wrote: > > > >> In theory, but not in practice. Not merely is the identifier an integer > >> value (and may be the result of arbitrarily complicated extraction), the > >> WAIT statement may be inside an external procedure. > >> > > > > That's what the ASYNCHRONOUS attribute for arguments is supposed to > > address. > > > > > > Since a WAIT statement could occur in a subprogram (at least ones the > compiler does not know about, such as ones not intrinsic or inlined) > then it is part of the required implementation that operations on a > variable with the asynchronous attribute not be moved across such a > call. Included in the class of calls that block code motion would be > those to the MPI routines. Having the asynchronous attribute in the > caller scope effectively solves the code motion problem. Having it on > the corresponding dummy probably solves it as well. > Jim pointed out that asynchronous may not be that useful in the end. I > think this is an important point. As noted above, asynchronous can > solve the code motion problem. It is not clear what you mean by "solve" > in regard to the other main problem - copy-in/copy-out of actual > arguments. What the asynchronous attribute will do is cause a program > that would otherwise have to include copy-in/copy-out code at the call > to the MPI_Ixxx routine to, instead, get a hard error at compile time > based on a constraint violation. It does not make the program "work", > which is probably somewhere on the user's agenda. It sounds like this is a problem for asynchronous I/O, regardless of whether it's done by I/O and WAIT statements hiding in external procedures, or by MPI hiding in external procedures. If ASYNCHRONOUS is defective, we need to repair it, without getting distracted by worrying whether it's spelled MPI_ISEND or ASYNCHRONOUS='YES' in an external procedure into which the processor can't see. > To get a working > program, there are two possible solutions that I see: > > 1) Require that the user always supply a contiguous actual argument > corresponding to the buffer dummy (in which case it seems to make more > sense for THAT to be the requirement on the user in the first place), or > > 2) Provide a Fortran interface than can handle non-contiguous sections > by having assumed-shape corresponding dummies. This solution off-loads > the bulk of the work to the MPI folks. But is does seem more robust. > > It would be useful to hear feedback from the MPI group on these > alternatives.

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3909) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by N.M. Maclaren 23 Jan '09

23 Jan '09

On Jan 23 2009, Bill Long wrote: > >Since a WAIT statement could occur in a subprogram (at least ones the >compiler does not know about, such as ones not intrinsic or inlined) >then it is part of the required implementation that operations on a >variable with the asynchronous attribute not be moved across such a >call. Included in the class of calls that block code motion would be >those to the MPI routines. Having the asynchronous attribute in the >caller scope effectively solves the code motion problem. Agreed. > Having it on >the corresponding dummy probably solves it as well. What corresponding dummy? There may not BE one! >Jim pointed out that asynchronous may not be that useful in the end. I >think this is an important point. As noted above, asynchronous can >solve the code motion problem. It is not clear what you mean by "solve" >in regard to the other main problem - copy-in/copy-out of actual >arguments. What the asynchronous attribute will do is cause a program >that would otherwise have to include copy-in/copy-out code at the call >to the MPI_Ixxx routine to, instead, get a hard error at compile time >based on a constraint violation. It does not make the program "work", >which is probably somewhere on the user's agenda. That is correct. However, to make arbitrary broken programs work is beyond any mortal's ability! The best that we can hope for is to minimise the amount of rewriting needed. > To get a working >program, there are two possible solutions that I see: > >1) Require that the user always supply a contiguous actual argument >corresponding to the buffer dummy (in which case it seems to make more >sense for THAT to be the requirement on the user in the first place), or > >2) Provide a Fortran interface than can handle non-contiguous sections >by having assumed-shape corresponding dummies. This solution off-loads >the bulk of the work to the MPI folks. But is does seem more robust. Neither of those solutions will work. Sorry. Consider the situation where procedure Fred allocates a buffer, and then passes it down through two call chains: Fred->Joe->Bert->Alf->MPI_Isend and Fred->Pete->Nick->Greg->MPI_Wait. The buffer is then used in Greg. The requirement is that neither of the call chains performs a copy-in/copy-out at any stage. It is perfectly legal to pass a non-contiguous section to an assumed size dummy (which is then contiguous); it doesn't matter WHAT the interface to MPI_Isend is - that assumed size array is a copy of the buffer that was allocated in Fred. The point is that the above problem has ALREADY been addressed for Fortran asynchronous I/O. If the politics are such that some vendors will refuse to accept that solution for genuinely asynchronous transfers, then we have a big problem. But, make no mistake, it's political and not technical. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3909) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Bill Long 23 Jan '09

23 Jan '09

Van Snyder wrote: > On Fri, 2009-01-23 at 01:56 -0800, N.M. Maclaren wrote: > >> In theory, but not in practice. Not merely is the identifier an integer >> value (and may be the result of arbitrarily complicated extraction), the >> WAIT statement may be inside an external procedure. >> > > That's what the ASYNCHRONOUS attribute for arguments is supposed to > address. > > Since a WAIT statement could occur in a subprogram (at least ones the compiler does not know about, such as ones not intrinsic or inlined) then it is part of the required implementation that operations on a variable with the asynchronous attribute not be moved across such a call. Included in the class of calls that block code motion would be those to the MPI routines. Having the asynchronous attribute in the caller scope effectively solves the code motion problem. Having it on the corresponding dummy probably solves it as well. Jim pointed out that asynchronous may not be that useful in the end. I think this is an important point. As noted above, asynchronous can solve the code motion problem. It is not clear what you mean by "solve" in regard to the other main problem - copy-in/copy-out of actual arguments. What the asynchronous attribute will do is cause a program that would otherwise have to include copy-in/copy-out code at the call to the MPI_Ixxx routine to, instead, get a hard error at compile time based on a constraint violation. It does not make the program "work", which is probably somewhere on the user's agenda. To get a working program, there are two possible solutions that I see: 1) Require that the user always supply a contiguous actual argument corresponding to the buffer dummy (in which case it seems to make more sense for THAT to be the requirement on the user in the first place), or 2) Provide a Fortran interface than can handle non-contiguous sections by having assumed-shape corresponding dummies. This solution off-loads the bulk of the work to the MPI folks. But is does seem more robust. It would be useful to hear feedback from the MPI group on these alternatives. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] [ukfortran] (SC22WG5.3890) (j3.2006) [MPI3 Fortran] MPI non-blocking transfers
by Malcolm Cohen 22 Jan '09

22 Jan '09

Bill Long wrote: > The one difference (from the compiler's point of view) is that the > volatility of a variable with the asynchronous attribute can be > "turned off" when the compiler sees a global WAIT. There is no such thing as a global WAIT. WAIT requires a file-unit-number - see C938 "A <file-unit-number> shall be specified in a <wait-spec-list>". Unless the compiler can see all the references to a variable (something that is sometimes the case but rather frequently not!) it will not know what unit the i/o is being performed on (and if it can see the references and knows the unit, it probably knows the ID= and can track that too). As has been pointed out, there are many differences between ASYNCHRONOUS and VOLATILE (that the "volatility" can be turned off as soon as the compiler reaches any calculation involving the variable is a much bigger difference than the WAIT case), but this is not one of them. Cheers, -- .....................Malcolm Cohen, Nihon NAG, Tokyo.

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3892) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Van Snyder 22 Jan '09

22 Jan '09

On Thu, 2009-01-22 at 11:21 -0800, Bill Long wrote: > This suffers from the same problem as Van's extra argument proposal. > The buffer may not be visible in the scoping unit of the call to > MPI_wait. In addition, it may not be clear which buffer is relevant > to a particular call. Nice idea, but I don't think it is practical. If the extra argument is a pointer that is TKR compatible with the buffer, the compiler has no choice but not to fiddle with anything that's TKR compatible and has the TARGET or POINTER attribute, or has such a subobject. If the buffer isn't a pointer and isn't a target and isn't accessible in the scoping unit of the MPI_WAIT call, the compiler can't possibly do anything nefarious to it. What other conjectures do you have that might cause the extra argument approach not to work?

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3891) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Van Snyder 22 Jan '09

22 Jan '09

On Thu, 2009-01-22 at 10:59 -0800, Bill Long wrote: > When we're talking about a Fortran's asynchronous > attribute, it seems reasonable to relate it to Fortran's concept of > I/O. When a Fortran program invokes an external procedure with an actual argument that has the ASYNCHRONOUS attribute, and all you have for the external procedure is a .o file, how does the compiler know whether the procedure does Fortran I/O, or something else?

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3886) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Van Snyder 22 Jan '09

22 Jan '09

On Thu, 2009-01-22 at 08:11 -0800, Bill Long wrote: > > >> One simple step toward solving the problem is to write an extra > >> interface layer that includes the argument, which then calls the real > >> MPI wait routine, not passing that argument. Then declaring the buffer > >> and the dummy argument of the wait routine interface layer to be > >> ASYNCHRONOUS ought to work, according to the rules we already have in > >> place for Fortran asynchronous I/O. > >> > > The basic problem with adding a buffer argument to a variant of the > MPI_wait routines is that the buffer variable may not be accessible in > the scoping unit of the call. This seems like a fatal flaw with this > approach. If it's not accessible in the scoping unit of the call, then the compiler can't possibly fiddle with it, and in particular can't possibly move references to it across the call.

1 0

Re: [MPI3 Fortran] [ukfortran] (SC22WG5.3898) (j3.2006) MPI non-blocking transfers
by N.M. Maclaren 22 Jan '09

22 Jan '09

On Jan 22 2009, Aleksandar Donev wrote: >On Thursday 22 January 2009 11:40, Bill Long wrote: > >> The only safe thing for the >> compiler to do is treat array as volatile throughout the subprogram >> containing these two calls. > >No! For ASYNCHRONOUS, there is a restriction on the *programmer* that if >array is referenced/defined, it is not currently a pending I/O >affector. This is the same as for coarrays, which may make things >clearer: If a coarray is defined/referenced, the compiler can fully >optimize without worrying about other images, *in-between* >image-control statements. The same logic applies for >ASYNCHRONOUS---optimize unless you see a WAIT. Note however that some >of the optimizations Nick mentioned should also be disabled for >coarrays. Notably, a piece of a coarray that is *not* >referenced/defined in a segment *may* indeed be referenced/defined by >other images. So the optimizer must not touch those pieces, even in >what would be harmless ways in a serial code. Yes, precisely. >For a classic (serial) optimizer, ASYNCHRONOUS and coarrayness and >threading and all of those things are essentially the same: Something >else, which is not included in the classic control/data flow analysis, >can store/load certain values, which are marked in some sense (via >CODIMENSION/ASYNCHRONOUS/VOLATILE attribute). Do you think all coarrays >are effectively VOLATILE??? If not, then the same applies to >ASYNCHRONOUS variables. Again, precisely. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] [ukfortran] (SC22WG5.3897) (j3.2006) MPI non-blocking transfers
by N.M. Maclaren 22 Jan '09

22 Jan '09

On Jan 22 2009, Bill Long wrote: >Aleksandar Donev wrote: >> >> ! complex calculations using array to calculate values: >> call mpi_isend(array,...) >> ... ! array not modified here >> call mpi_wait(...) >> ! more calculations involving array reading the values >> >> Surely it is ok, desirable, or even necessary to optimize the >> calculations??? > >This example shows the problem well. There is no way for the compiler >to tell that the mpi_wait call is related to the isend that is using >array as its buffer. It might be for a different isend (and hence >different buffer) started somewhere else. The only safe thing for the >compiler to do is treat array as volatile throughout the subprogram >containing these two calls. Which is just what happens if array is >declared volatile. It's irrelevant whether you spell the attribute >asynchronous or volatile. The effect is the same. Yes, it shows the problem well - but the problem is that you don't seem to understand either the Fortran ASYCHRONOUS attribute or MPI non-blocking semantics. The compiler can optimise those calculations without concern, just as it can if you replace the MPI transfer by Fortran asynchronous I/O. If the MPI_Wait is unrelated to the MPI_Isend, then the user has made an error and the behaviour is UNDEFINED. That is EXACTLY the same situation as when you have illegal argument aliasing - no respectable optimising compiler refuses to optimise argument accesses for that reason. >Reread Nick's earlier message. The optimization pitfalls extend past >just argument passing. Sigh. Yes, they do. But my previous message was describing why the ASYNCHRONOUS attribute was essential, and omitting ANY attribute was not going to work (irrespective of how many procedure flags you use). Once you have that attribute, the problems that I described there have ALREADY been dealt with. There are only three things that an optimising compiler CAN'T do with a variable with the ASYNCHRONOUS attribute that it can do with one without any attribute: 1) Move it (including copy-in/copy-out), if it might be pending. 2) Move accesses across a construct (usually a procedure call) that might change it to or from pending state. 3) Copy and restore parts of it that are not accessed by the code the user wrote, if it might be pending. Compared with what it can't do to ones with the VOLATILE attribute, that's nothing. >> I understand no compiler actually implements asynchronous so >> no implementor has actually pinned down the list of what optimizations >> are "not allowed", > >I'm sure at least some compilers do implement asynchronous. Apparently Intel does, and it isn't exactly new technology, anyway! Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0