Dear member of the WG5 and J3 Fortran standardization committee, First, we want to thank you for your effort in defining the TR 29113 as an important key to solve the problems between MPI and Fortran. This is a comment from the MPI Forum on your WG5 letter ballot on N1845 in ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1846.txt and http://www.nag.co.uk/sc22wg5/. After reading your N1845 working draft from March 3, 2011, we detected one major gap: there is no standardized possibility of using nonblocking features of MPI in Fortran programs to overlap computation with communication. Specifically, Fortran MPI applications will be prohibited from initiating any form of asynchronous communication and then continuing to perform local computations while MPI advances the communication "in the background." This is a key issue for us because nonblocking communication is a critical technique for deadlock-free communication and high performance performance. An easy use case to describe is one that is popular in current high performance computing environments: when the MPI library utilizes hardware offload technologies (such as RDMA) with a communication co-processor (such as a NIC). The problem is, in principle, independent of MPI. It also occurs with libc/POSIX asynchronous IO (aio), and is therefore a topic that should be solved by the "TR on further interoperability of Fortran with C." We noticed that there is already a decision about this issue: to extend the meaning of the ASYNCHRONOUS attribute rather than to define a new attribute. http://www.j3-fortran.org/doc/year/09/09-235r2.txt contains the minutes from the J3 meeting, May 4-8, 2009. On this topic, the following vote was recorded: Paper 09-231 "Answer to MPI Forum regarding MPI asynchronous operations" [Rasmussen] discussed how to prevent code-motion and copy-in/out: SV: extend ASYNCHRONOUS - invent new attribute - undefined: 8-2-3 First, we would like to propose how to solve the problem, then we will present a short example to illustrate the specific problem. We recommend the following change of the first two paragraphs in this section of the Fortran 2008 standard: 5.3.4 ASYNCHRONOUS attribute 1 An entity with the ASYNCHRONOUS attribute is a variable that may be subject to asynchronous input/output or other asynchronous access by means other than Fortran. Asynchronous input/output can be any asynchronous access to the variable or parts of it, potentially within the scope of this standard by Fortran asynchronous I/O or possibly via methods outside the scope of this standard, such as the libc/POSIX asynchronous IO (aio) or nonblocking message passing, one-sided communication, or nonblocking parallel I/O as part of the Message Passing Interface (MPI) standard. 2 The base object of a variable shall have the ASYNCHRONOUS attribute in a scoping unit if - the variable appears in an executable statement or specification expression in that scoping unit and - any statement of the scoping unit is executed while the variable is a pending I/O storage sequence affector (9.6.2.5) or any other pending asynchronous access by means other than Fortran. This should be treated only as a wording proposal. We provide the above text only as a suggestion; J3/WG5 are certainly free to adjust it as they feel necessary. The key idea is that we need ASYNCHRONOUS to also include potential memory modifications from agents outside the scope of the Fortran standard. The following snipit is a correct code that allows the MPI library to internally use DMA, communications offload (e.g., to a NIC), or other method to asynchronously communicate parts of the array while the user application/main CPU simultaneously reads and writes to ***another*** part of the same array. Specifically, this code snipit is is free of race conditions. USE mpi_f08 REAL, ASYNCHRONOUS :: buf(100,100) ! communicating a boundary of the array: CALL MPI_Irecv(buf(1,1:100),...req,...) DO j=1,100 DO i=2,100 buf(i,j)=.... ! work only on the inner area END DO END DO CALL MPI_Wait(req,...) It is important that the compiler is ***not*** allowed to translate this program (by using temporary memory modifications) into something like this: USE mpi_f08 REAL, ASYNCHRONOUS :: buf(100,100), buf_1dim(10000) EQUIVALENCE (buf(1,1), buf_1dim(1)) CALL MPI_Irecv(buf(1,1:100),...req,...) tmp(1:100)=buf(1,1:100) ! saving the boundary DO k=1,10000 buf_1dim(k)=... ! work on the whole array END DO buf(1,1:100)=tmp(1:100) ! restoring the boundary CALL MPI_Wait(req,...) While the MPI library receives buf(1,1:100) "in the background", the "work" part overwrites this part of the array as part of a numerical optimization to achieve one long loop instead of a 2-loop-nesting. With the current status of Fortran 2008 + TR 29113 (Version N1845), a compiler can ignore the ASYNCHRONOUS attribute. For example, a compiler may choose to not implement Fortran asynchronous I/O in an asynchronous way, meaning that there is no need for the compiler to make any restrictions on optimization. To be clear: with the current Fortran 2008, this optimization is still allowed. With our proposal, this optimization will be prohibited. We want to mention that the use of VOLATILE would solve the problem, but at a high performance price because it would also prohibit any optimization of the "work" part (e.g., automatic cache optimization, instruction reordering, etc.). Especially since Fortran is used with MPI in high-performance, numerically intensive applications, we feel that VOLATILE cannot be recommended as a solution. In the era of multi-core hardware and vector instructions on each CPU, nonblocking communication must be allowed to overlap with efficient numerical code written in Fortran in order to achieve scalable, highly-performance applications. Your vote in J3 gave us the confidence that you can solve this problem within your TR 29113. In short, if we were members of WG5, we would answer the ballot with "Yes, but I recommend the following changes..." with the proposal described above. We appreciate the fact that you have asked for external comments on N1845. We hope that our comments above are sufficient information, and also hope that this letter conveys the critical importance with which we consider this issue. Many thanks for your time.