January 2009 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3891) [ukfortran] MPI non-blocking transfers
by Aleksandar Donev 22 Jan '09

22 Jan '09

On Thursday 22 January 2009 11:40, Bill Long wrote: > The only safe thing for the > compiler to do is treat array as volatile throughout the subprogram > containing these two calls. No! For ASYNCHRONOUS, there is a restriction on the *programmer* that if array is referenced/defined, it is not currently a pending I/O affector. This is the same as for coarrays, which may make things clearer: If a coarray is defined/referenced, the compiler can fully optimize without worrying about other images, *in-between* image-control statements. The same logic applies for ASYNCHRONOUS---optimize unless you see a WAIT. Note however that some of the optimizations Nick mentioned should also be disabled for coarrays. Notably, a piece of a coarray that is *not* referenced/defined in a segment *may* indeed be referenced/defined by other images. So the optimizer must not touch those pieces, even in what would be harmless ways in a serial code. For a classic (serial) optimizer, ASYNCHRONOUS and coarrayness and threading and all of those things are essentially the same: Something else, which is not included in the classic control/data flow analysis, can store/load certain values, which are marked in some sense (via CODIMENSION/ASYNCHRONOUS/VOLATILE attribute). Do you think all coarrays are effectively VOLATILE??? If not, then the same applies to ASYNCHRONOUS variables. Aleks

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3891) [ukfortran] MPI non-blocking transfers
by Bill Long 22 Jan '09

22 Jan '09

Aleksandar Donev wrote: > On Thursday 22 January 2009 10:59, Bill Long wrote: > > >> Only for statements that might involve references or definitions of a >> volatile variable. The rest of the code can be optimized normally. >> Which is exactly what we need here. >> > No. Take: > > ! complex calculations using array to calculate values: > call mpi_isend(array,...) > ... ! array not modified here > call mpi_wait(...) > ! more calculations involving array reading the values > > Surely it is ok, desirable, or even necessary to optimize the > calculations??? > This example shows the problem well. There is no way for the compiler to tell that the mpi_wait call is related to the isend that is using array as its buffer. It might be for a different isend (and hence different buffer) started somewhere else. The only safe thing for the compiler to do is treat array as volatile throughout the subprogram containing these two calls. Which is just what happens if array is declared volatile. It's irrelevant whether you spell the attribute asynchronous or volatile. The effect is the same. > >>> ASYNCHRONOUS only disables >>> code motion accross waits and certain argument associations (such >>> as more care with copy in/out). The rest of the code can be >>> optimized as usual. >>> >> If that's the case, then there is a serious problem with ASYNCHRONOUS >> > But these are covered by the "disables...certain argument associations" > part of my sentence. In particular, depending on whether the > actual/dummy are (both) asynchronous, some transformations (e.g., copy > in/out for the hell of it, copy in/out of more than the piece of the > actual, code motion accross the argument call, etc.) should be > disabled. Reread Nick's earlier message. The optimization pitfalls extend past just argument passing. > I understand no compiler actually implements asynchronous so > no implementor has actually pinned down the list of what optimizations > are "not allowed", I'm sure at least some compilers do implement asynchronous. > but I can assure you, the list is much smaller than > for volatile, which, disables even the most basic optimization, such as > using registers during a calculation loop. > Again, I think Nick's examples suggest that the compiler needs to be more careful with asynchronous than you might think. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3886) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by N.M. Maclaren 22 Jan '09

22 Jan '09

On Jan 22 2009, Bill Long wrote: > >Except that the WAIT statement does have at least some semantics that >the compiler understands. Which raises an interesting point. VERY dubiously. The identifier is an integer value, and provides no direct indication of the affector. > Nick has >pointed out several places where it is theoretically possible for the >compiler optimizer to cause problems because the "invisible hand" of the >MPI runtime currently "owns" part of the program's memory space and >either expects it to remain static (sends) or modifies it (receives). That is correct. Just like Fortran asynchronous I/O. >Fortran already has an attribute designed exactly for this situation - >VOLATILE. VOLATILE and ASYNCHRONOUS normally appear together in the >standard since they are so similar in effect. The one difference (from >the compiler's point of view) is that the volatility of a variable with >the asynchronous attribute can be "turned off" when the compiler sees a >global WAIT. If we modify the ASYNCHRONOUS attribute to be used to >solve the issues with MPI calls, then WAIT is no longer special, and >VOLATILE and ASYNCHRONOUS become functionally identical. That is completely wrong, as Jim and Aleks have said. The one thing that they have in common is that the compiler mustn't move them. VOLATILE states that the variable may be change at any time, and the compiler had better generate code that is robust against that, which is incompatible with almost all optimisation. ASYNCHRONOUS says that either the variable is not special (and hence may be optimised as normal) or it is 'in use' and the program may not access it. All that the compiler is forbidden from doing is from moving code from where the variable might be in the first state into where it might be in the second - and those changes can happen ONLY in subroutine calls and special I/O statements. > (The other is that it incorrectly suggests that MPI calls >have something to do with I/O which is not the case (except for the >actual MPI I/O calls).) As Aleks has told you, that is wrong. Stream I/O to a FIFO is a form of I/O, and MPI point-to-point transfers and simply a form of that. That has been accepted, and all forms of such message passing referred to as I/O, since before any of us started computing. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3892) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Bill Long 22 Jan '09

22 Jan '09

Jim Xia wrote: > > > Except that the WAIT statement does have at least some semantics that > > the compiler understands. Which raises an interesting point. Nick has > > pointed out several places where it is theoretically possible for the > > compiler optimizer to cause problems because the "invisible hand" of > the > > MPI runtime currently "owns" part of the program's memory space and > > either expects it to remain static (sends) or modifies it (receives). > > > The "invisible hand" can be expressed by a visible hint if the > association between the two entities are explicitly established in > language. For example something like ALIAS_ASSOCIATED(buffer, > MPI_WAIT) hints an aliasing (or asynchronous) relationship between > buffer and MPI_WAIT, then the compiler (more precisely the optimizer) > may have enough information not to perform the code motion. I think > it is also more direct and obvious than asynchronous/wait. This suffers from the same problem as Van's extra argument proposal. The buffer may not be visible in the scoping unit of the call to MPI_wait. In addition, it may not be clear which buffer is relevant to a particular call. Nice idea, but I don't think it is practical. > > > > Fortran already has an attribute designed exactly for this situation - > > VOLATILE. VOLATILE and ASYNCHRONOUS normally appear together in the > > standard since they are so similar in effect. > > > I disagree here. VOLATILE and ASYNCHRONOUS has near zero similarity > in semantics. Volatile tells compilers the variable may be accessed > by another processor so don't trust register/cache, and always goes to > memory for any reference. I can't imagine any relation to > asynchronous except that both may disable optimization one way or > another. In the second sentence above, replace "Volatile" -> "Asynchronous" and "another process" -> "a concurrently executing thread in the I/O library", and it still makes sense. As your last sentence says, the similarity is in disabling optimizations, and that is the topic here. > > > The one difference (from > > the compiler's point of view) is that the volatility of a variable with > > the asynchronous attribute can be "turned off" when the compiler sees a > > global WAIT. > > > What is the volatility of an asynchronous variable? That its value can get changed by an invisible process. > > > If we modify the ASYNCHRONOUS attribute to be used to > > solve the issues with MPI calls, then WAIT is no longer special, and > > VOLATILE and ASYNCHRONOUS become functionally identical. > > I don't think they will. They have different semantics. > > > That's one of > > the undesirable side effect that makes me dislike the idea of modifying > > ASYNCHRONOUS. (The other is that it incorrectly suggests that MPI > calls > > have something to do with I/O which is not the case (except for the > > actual MPI I/O calls).) If we use a different name, like > > ASYNC_EXTERNAL, is there any difference between that and VOLATILE? I > > suspect not. In which case, the obvious solution is to just use > > VOLATILE. (Which was the conclusion of the previous round of > > discussions on this topic.) VOLATILE had a couple of very attractive > > features. It requires zero changes in either the Fortran standard > or in > > any compiler implementations. Thus, it is an immediately available > > solution for the MPI group. VOLATILE might be strong enough to solve > > the code-motion problem as well. > > I doubt volatile can effectively "turns off" code motion since it does > not have that semantics. You might be right about that, since the buffer is not part of the MPI_wait call. The subroutine prefix option should work, though. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3891) [ukfortran] MPI non-blocking transfers
by Aleksandar Donev 22 Jan '09

22 Jan '09

On Thursday 22 January 2009 10:59, Bill Long wrote: > Only for statements that might involve references or definitions of a > volatile variable. ï¿½The rest of the code can be optimized normally. ï¿½ > Which is exactly what we need here. No. Take: ! complex calculations using array to calculate values: call mpi_isend(array,...) ... ! array not modified here call mpi_wait(...) ! more calculations involving array reading the values Surely it is ok, desirable, or even necessary to optimize the calculations??? > > ASYNCHRONOUS only disables > > code motion accross waits and certain argument associations (such > > as more care with copy in/out). The rest of the code can be > > optimized as usual. > > If that's the case, then there is a serious problem with ASYNCHRONOUS But these are covered by the "disables...certain argument associations" part of my sentence. In particular, depending on whether the actual/dummy are (both) asynchronous, some transformations (e.g., copy in/out for the hell of it, copy in/out of more than the piece of the actual, code motion accross the argument call, etc.) should be disabled. I understand no compiler actually implements asynchronous so no implementor has actually pinned down the list of what optimizations are "not allowed", but I can assure you, the list is much smaller than for volatile, which, disables even the most basic optimization, such as using registers during a calculation loop. Best, Aleks -- Aleksandar Donev, Ph.D. Lawrence Postdoctoral Fellow @ Lawrence Livermore National Laboratory High Performance Computational Materials Science and Chemistry E-mail: donev1(a)llnl.gov Phone: (925) 424-6816 Fax: (925) 423-0785 Address: P.O.Box 808, L-367, Livermore, CA 94551-9900 Web: http://cherrypit.princeton.edu/donev

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3891) [ukfortran] MPI non-blocking transfers
by Bill Long 22 Jan '09

22 Jan '09

Aleksandar Donev wrote: > On Thursday 22 January 2009 09:10, Bill Long wrote: > > >> If we use a different name, like >> ASYNC_EXTERNAL, is there any difference between that and VOLATILE? I >> suspect not. >> > Yes, I explained this a long time ago, many times, but we go in circles > again. > For one thing, VOLATILE has some ill-defined semantics. > Vendors seem to know what to do. The VOLATILE concept is utterly trivial to implement. > For two, VOLATILE disables all optimization. Only for statements that might involve references or definitions of a volatile variable. The rest of the code can be optimized normally. Which is exactly what we need here. > ASYNCHRONOUS only disables > code motion accross waits and certain argument associations (such as > more care with copy in/out). The rest of the code can be optimized as > usual. If that's the case, then there is a serious problem with ASYNCHRONOUS, as illustrated by Nick's examples. In most places in a code, VOLATILE and ASYNCHRONOUS have exactly the same effect on optimization. In fact, the simplest implementation of ASYNCHRONOUS that is semantically safe is to make is synonym for VOLATILE. > Surely this is important for a high-performance library like > MPI. > The high performance part of MPI occurs inside the library routines. That's unaffected by the declaration of the buffer outside. > >> The other is that it incorrectly suggests that MPI calls >> have something to do with I/O which is not the case (except for the >> actual MPI I/O calls >> > And I already explained this a long time ago, and even quoted you the > Wikipedia definition of I/O. But if you care not to listen, that is > fine, let's continue going in circles. > I forgot that I was addressing more than the Fortran list. I meant that MPI has nothing to do with the Fortran concept of I/O (not Wikipedia's). When we're talking about a Fortran's asynchronous attribute, it seems reasonable to relate it to Fortran's concept of I/O. Cheers, Bill > Best, > Aleks > > _______________________________________________ > J3 mailing list > J3(a)j3-fortran.org > http://j3-fortran.org/mailman/listinfo/j3 > -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3886) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Aleksandar Donev 22 Jan '09

22 Jan '09

On Thursday 22 January 2009 09:10, Bill Long wrote: > If we use a different name, like > ASYNC_EXTERNAL, is there any difference between that and VOLATILE? ï¿½I > suspect not. Yes, I explained this a long time ago, many times, but we go in circles again. For one thing, VOLATILE has some ill-defined semantics. For two, VOLATILE disables all optimization. ASYNCHRONOUS only disables code motion accross waits and certain argument associations (such as more care with copy in/out). The rest of the code can be optimized as usual. Surely this is important for a high-performance library like MPI. > The other is that it incorrectly suggests that MPI calls > have something to do with I/O which is not the case (except for the > actual MPI I/O calls And I already explained this a long time ago, and even quoted you the Wikipedia definition of I/O. But if you care not to listen, that is fine, let's continue going in circles. Best, Aleks

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3886) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Bill Long 22 Jan '09

22 Jan '09

N.M. Maclaren wrote: > On Jan 22 2009, Van Snyder wrote: > >> Assuming the problem is caused by the buffer not appearing as an actual >> argument, it's not a defect in Fortran but rather in the design of MPI. >> > > If that were the whole problem, it would be. It would also be a defect in > the design for the Fortran WAIT statement. > > Except that the WAIT statement does have at least some semantics that the compiler understands. Which raises an interesting point. Nick has pointed out several places where it is theoretically possible for the compiler optimizer to cause problems because the "invisible hand" of the MPI runtime currently "owns" part of the program's memory space and either expects it to remain static (sends) or modifies it (receives). Fortran already has an attribute designed exactly for this situation - VOLATILE. VOLATILE and ASYNCHRONOUS normally appear together in the standard since they are so similar in effect. The one difference (from the compiler's point of view) is that the volatility of a variable with the asynchronous attribute can be "turned off" when the compiler sees a global WAIT. If we modify the ASYNCHRONOUS attribute to be used to solve the issues with MPI calls, then WAIT is no longer special, and VOLATILE and ASYNCHRONOUS become functionally identical. That's one of the undesirable side effect that makes me dislike the idea of modifying ASYNCHRONOUS. (The other is that it incorrectly suggests that MPI calls have something to do with I/O which is not the case (except for the actual MPI I/O calls).) If we use a different name, like ASYNC_EXTERNAL, is there any difference between that and VOLATILE? I suspect not. In which case, the obvious solution is to just use VOLATILE. (Which was the conclusion of the previous round of discussions on this topic.) VOLATILE had a couple of very attractive features. It requires zero changes in either the Fortran standard or in any compiler implementations. Thus, it is an immediately available solution for the MPI group. VOLATILE might be strong enough to solve the code-motion problem as well. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3886) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Jeff Squyres 22 Jan '09

22 Jan '09

On Jan 22, 2009, at 11:11 AM, Bill Long wrote: >>> One simple step toward solving the problem is to write an extra >>> interface layer that includes the argument, which then calls the >>> real >>> MPI wait routine, not passing that argument. Then declaring the >>> buffer >>> and the dummy argument of the wait routine interface layer to be >>> ASYNCHRONOUS ought to work, according to the rules we already have >>> in >>> place for Fortran asynchronous I/O. > > The basic problem with adding a buffer argument to a variant of the > MPI_wait routines is that the buffer variable may not be accessible > in the scoping unit of the call. This seems like a fatal flaw with > this approach. It's also a problem for the array variants of MPI_TEST and MPI_WAIT (that take variable-length arrays of requests). Also remember that MPI_TEST may or may not complete the request; just calling MPI_TEST (or any of its array variants) does not guarantee that the buffer is "owned" by the application again. -- Jeff Squyres Cisco Systems

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3886) [ukfortran] [MPI3 Fortran] MPI non-blocking transfers
by Bill Long 22 Jan '09

22 Jan '09

Adding the MPI list back into the reply. Please use "reply all" for this discussion.... N.M. Maclaren wrote: > On Jan 22 2009, Van Snyder wrote: > > >> One simple step toward solving the problem is to write an extra >> interface layer that includes the argument, which then calls the real >> MPI wait routine, not passing that argument. Then declaring the buffer >> and the dummy argument of the wait routine interface layer to be >> ASYNCHRONOUS ought to work, according to the rules we already have in >> place for Fortran asynchronous I/O. >> The basic problem with adding a buffer argument to a variant of the MPI_wait routines is that the buffer variable may not be accessible in the scoping unit of the call. This seems like a fatal flaw with this approach. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0