John, this is already handled by all hints about code movement and register optimization in the old MPI-2.2 and in more detail in the upcoming MPI-3.0. To summarize, there are several options: - TARGET buf - Using a call to a dummy routine MPI_F_SYNC_REG(buf) immediately after MPI_Wait - Storing buf as a module variable or in a common block - (VOALTILE buf, not recommended) 'ASYNCHRONOUS buf' is explicitly not an option, because a Fortran compiler - that implements Fortran asynchronous input/output with blocking I/O routines - is allowed to ignore the ASYNCHRONOUS attribute. Only compilers with really asynchronous Fortran I/O have to also internally imply the TARGET attribute when the ASYNCHRONOUS attribute is given in the application. Is my analysis correct?

You do not need to be a member of J3. Would you like to join interop-tr? Most of the internals of the TR, I do not understand. As I see, the list moderator can agree to deliver the few mails from me also to the other members of the interop-tr. It isn't a problem for me to put you, Bill, Nick, Craig, and Reinhold on the CC.

Best regards Rolf ----- Original Message -----

From: "John Reid" <[email protected]> To: "Rolf Rabenseifner" <[email protected]> Cc: "MPI-3 Fortran working group" <[email protected]>, [email protected] Sent: Thursday, April 14, 2011 4:33:56 PM Subject: Re: [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846 Rolf Rabenseifner wrote:

...

[I expect that the mail is not delivered by [email protected] because I'm not a member of J3]

You do not need to be a member of J3. Would you like to join interop-tr?

...

After all the discussion, it really looks best to ignore the existence of the ASYNCHRONOUS attribute when looking at the co-existence of Fortran nonblocking and MPI.

Although the idea of needing no text in our TR is welcome, I do not agree with your conclusion. Consider the code

REAL :: buf(100,100) : ! Code that involves buf CALL MPI_Irecv(buf,...req,...) : ! Code that does not involve buf CALL MPI_Wait(req,...) : ! Code that involves buf

Since there is no mention of buf in the call of MPI_Wait, the compiler is free to perform code motion involving it. For example, it might load buf into a GPU during the calculations that do not involve buf, ready for those that do.

Asynchronous communication is so like asynchronous I/O that I think we need to have the same rules. If we can somehow write this into the normative text of the TR, the effect of adding ASYNCHRONOUS to the declaration of buf in the above example would be that the compiler will not be allowed to make the harmful change that I have just outlined.

Cheers,

John.

...

Based on the discussion, I plan to add the following text into the draft for MPI-3.0 and based on Option 1 in a previous email:

------------------ [Text about an application with parts of an array involved in a nonblocking MPI operation and other parts involved in computation, and a compiler optimization with temporarily copying variables into a local memory and afterward back to the original memory]

Note that this type of compiler optimization can not be prevented when buf is declared with the ASYNCHRONOUS Fortran attribute, and it should not be prevented by declaring buf as VOLATILE because: - A Fortran asynchronous input/output operation on a part of an array variable implies that the access to the whole array (the so-called pending I/O storage sequence affector, see Section 9.6.2.5 in [Fortran 2008]) is restricted as if the whole array is involved in the pending I/O (see Section 9.6.4.1, paragraphs 5 and 6 in [Fortran 2008]). Therefore, the ASYNCHRONOUS cannot be used if parts of the array is involved in a asynchronous operation and other parts are used for computation. - In principle, the Fortran standard restricts the scope of the ASYNCHRONOUS attribute only to Fortran asynchronous input and output operations. - A Fortran compiler may implement the Fortran asynchronous input/output operations as blocking I/O. In this case, the compiler can fully ignore the ASYNCHRONOUS attribute. - The VOLATILE implies that all accesses to any storage unit (word) of buf must be directly done in the main memory exactly in the sequence defined by the application program. The attribute VOLATILE prevents every register or cache optimization. - Therefore, VOLATILE may cause a huge performance degradation.

Instead of solving the problem, it is better to prevent the problem, i.e., when overlapping communication and computation, the communication (or nonblocking IO) and the computation should be executed on different sets of variables. In this case, the temporary memory modifications are done only on the variables used in the computation and cannot have any side effect on the data used in the nonblocking MPI operations. ---------------

The first three items in the list suggest that it is a bad idea to enlarge the scope of ASYNCHRONOUS to nonblocking MPI.

Best regards Rolf

----- Original Message -----

...

From: "N.M. Maclaren" <[email protected]> To: "John Reid" <[email protected]>, [email protected] Cc: [email protected], "Rolf Rabenseifner" <[email protected]> Sent: Thursday, April 14, 2011 11:33:32 AM Subject: Re: [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846 On Apr 14 2011, John Reid wrote:

...

If the comment in the middle is changed to

! asynchronous I/O on ptr variable

the program would be non-conforming because A would be an affector without the ASYNCHRONOUS attribute.

Perhaps we need normative text that says (somehow) that asynchronous communication is regarded as if it were asynchronous i/o. In which case my reading of the standard is wrong. I have just looked again, and think that either I probably was or the wording is more confusing than it should be.

I don't think that Fortran intends to forbid:

PROGRAM Main REAL :: array(100) CALL Fred(array(:50),array(51:)) END PROGRAM Main

SUBROUTINE Fred (buffer, scratch) REAL, ASYNCHRONOUS :: buffer(:) REAL :: scratch(:) CALL MPI_Irecv(buffer,...) CALL DGEMM(scratch,...) CALL MPI_Wait(...) END SUBROUTINE Fred

As you say, the same problem can be shown using Fortran asynchronous I/O.

However, this issue is not limited to ASYNCHRONOUS and affects variables that have the TARGET attribute, too. I assert that it is conceptually identical to the one where TARGET is used, MPI_Irecv sets up a pointer and DGEMM uses that pointer. I.e. it's a question of exactly when association stops.

Regards, Nick Maclaren.

-- Scanned by iCritical.

-- Dr. Rolf Rabenseifner . . . . . . . . . .. email [email protected] High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)