April 2011 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846
by Rolf Rabenseifner 14 Apr '11

14 Apr '11

----- Original Message ----- > From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> > >From: "Rolf Rabenseifner" Nick, Let's look at CALL MPI_Irecv(buf, rq) CALL MPI_Wait(rq) xnew=buf and the new value for buf may be received within the MPI_Wait call. > >To summarize, there are several options: > > - TARGET buf > > Why should that work? Fortran compilers are required to take > notice of it only when also operating with POINTER variables. And > there is nothing stopping them from moving such data dynamically, > if they update all associated pointers. That's old technology, > after all. With TARGET buf CALL MPI_Irecv(buf, rq) CALL MPI_Wait(rq) xnew=buf With MPI_Irecv(buf, rq), the implementation of MPI_Irecv can have made a pointer (stored in or behind rq). Later on MPI_Wait(rq), the Fortran compiler has to expect that buf is accessed through this pointer. Therefore, the compiler must not move "xnew=buf" before the MPI_Wait statement. When I understand correctly, then "TAGET buf" means only Fortran pointers and the Fortran system has not detected such a pointer. Therefore it may be possible that "xnew=buf" is still moved before the MPI_Wait statement. Correct? I see several options to solve the problem: Option Target-1: The TR 29113 enhances the meaning of pointer also to C pointers in the framework of the TARGET attribute. Option Target-2: MPI-3.0 requires that "TARGET buf" works. An MPI implementation can implement this by calling a Fortran helper within MPI_Irecv that stores such a pointer visibly for the Fortran run-time environment and that frees this pointer in MPI_Wait. An MPI implementation can implement this also by requiring the quality of Target-1 from the compiler. Option Target-3: We give up with this solution-path for this code-movement optimization problem. Are there other options? > > - Using a call to a dummy routine MPI_F_SYNC_REG(buf) > > immediately after MPI_Wait > > That won't work, either, because it doesn't stop copy-in/copy-out > and other such techniques. With CALL MPI_Irecv(buf, rq) CALL MPI_Wait(rq) CALL MPI_F_SYNC_REG(buf) xnew=buf The content of MPI_F_SYNC_REG is outside the scope of the Fortran optimization. I do not see that the compiler is allowed to move "xnew=buf" before the MPI_Wait statement because "MPI_F_SYNC_REG(buf)" may have modified the content of buf. Therefore the problem is solved. Correct? If not, please can you show a detailed code, how such a compiled code could work and in xnew is at the end ***not*** the value that was stored in buf by MPI_Wait. > > - Storing buf as a module variable or in a common block > > That won't work, either, for the above reason. With COMMON /xxx/ buf CALL MPI_Irecv(buf, rq) CALL MPI_Wait(rq) xnew=buf The compiler has to expect that MPI_Wait has access to the COMMON block /xxx/ and therefore the compiler is allowed to move "xnew=buf" before the MPI_Wait statement. Therefore the problem is solved. Correct? If not, please can you show a detailed code, how such a compiled code could work and in xnew is at the end ***not*** the value that was stored in buf by MPI_Wait. > >> - (VOALTILE buf, not recommended) > > Why should that work any better than ASYNCHRONOUS? All it says (in > both > C and Fortran, incidentally) is "All bets are off - consult your > vendor > documentation for what might happen." ASYNCHRONOUS has the right > semantics. With VOLATILE buf CALL MPI_Irecv(buf, rq) CALL MPI_Wait(rq) xnew=buf By definition of volatile, the compiler has to guarantee that all accesses to buf are done through the memory and in the sequence of the application. It must expect that any library routine call and any other thread my write some data into buf; outside the scope of the Fortran statements. Therefore the compiler must not move a statement with buf across any library call. Therefore, "xnew=buf" cannot be moved before the MPI_Wait statement. Therefore the problem is solved. Correct? Or have I misunderstood the wording of VOLATILE: Fortran 2008, 5.3.19: "The VOLATILE attribute specifies that an object may be referenced, defined, or become undefined, by means not specified by the program." Our MPI_Wait is such a "by means not specified by the program." Fortran 2008, NOTE 5.25: "The Fortran processor should use the most recent definition of a volatile object when a value is required. Likewise, it should make the most recent Fortran definition available. It is the programmer's responsibility to manage any interaction with non-Fortran processes." "recent" means, that the sequence must not be modified. Otherwise "xnew=buf" does not access the recent definition of buf. Correct? > >'ASYNCHRONOUS buf' is explicitly not an option, because > >a Fortran compiler > >- that implements Fortran asynchronous input/output > > with blocking I/O routines > >- is allowed to ignore the ASYNCHRONOUS attribute. > > If you are being that legalistic, the problem is insoluble in all of > C, C++ and Fortran. Sorry, but that IS what the standards say! Only when I'm wrong with all of my opinions showed above. > >Only compilers with really asynchronous Fortran I/O > >have to also internally imply the TARGET attribute when > >the ASYNCHRONOUS attribute is given > > Er, no, they don't. The attributes are different and have subtly > different properties. > > >> You do not need to be a member of J3. Would you like to join > >> interop-tr? > > > >Most of the internals of the TR, I do not understand. > >As I see, the list moderator can agree to deliver the few > >mails from me also to the other members of the interop-tr. > >It isn't a problem for me to put you, Bill, Nick, Craig, > >and Reinhold on the CC. > > The problem is when we reply. Unless I leave you in the list > explicitly, > you won't see the reply. And duplicating people has other problems. If I should be on the list, then it is okay for me. > Regards, > Nick Maclaren. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Derived-data, a new/late comment on Fortran WG5 ballot N1846
by Rasmussen, Craig E 14 Apr '11

14 Apr '11

I have questions for Rolf that I'm going to take offline from the Fortran standards groups. My opinion is the use of SEQUENCE is an implementation issue and should not be explicitly part of the MPI-3 API. I'm willing to be convinced otherwise, but by using BIND(C), the C and Fortran compilers will have to align structures in the same way so I think SEQUENCE just gets in the way and is not useful. -craig ________________________________________ From: Rolf Rabenseifner [rabenseifner(a)hlrs.de] Sent: Thursday, April 14, 2011 8:22 AM To: John Reid; interop-tr(a)j3-fortran.org Cc: Reinhold Bader; Rasmussen, Craig E; Bill Long; N.M. Maclaren; Jeff Squyres; MPI-3 Fortran working group Subject: Derived-data, a new/late comment on Fortran WG5 ballot N1846 Dear John, dear members of the WG5 and J3 Fortran committees, I'm sorry that with this additional comment on problems regarding derived types, I missed the deadline of the N1846 ballot for the N1845 draft of the TR 29113 on extensions on Fortran-C interoperability. It looks like that there are two problems with derived types and the Fortran-MPI-compatibility with a background in the Fortran-C-interoperability. Problem A: Sequence derived types in the MPI routine interfaces, which are declared with BIND(C); this problem occurs with the new MPI handles and MPI statuses. Problem B: Derived type, SEQUENCE derived type, and BIND(C) derived type as actual user buffers for TYPE(*),DIMENSION(..) dummy arguments in BIND(C) MPI routine interfaces. Problem C: Based on problem B we have the question whether the new TYPE(*),DIMENSION(..) dummy arguments in BIND(C) MPI routine interfaces really allow all actual arguments that were possible with implicit interfaces or with explicit interfaces in combination with pragmas like !DEC$ ATTRIBUTES NO_ARG_CHECK :: buf and !$PRAGMA IGNORE_TKR buf At least for problems A and B, the solution may be "simple" by allowing SEQUENCE derived types as element of a BIND(C) derived type and in BIND(C) subroutine dummy arguments. Why we need this is shown in very detail in the following description of Problems A and B. A still open question is Problem C. To Problem A: ------------- For better type-safety and compile-time argument-checking, the MPI Forum wants to use named types for MPI handles and MPI status. In MPI-1.1 up to MPI-2.2, we use INTEGER and INTEGER, DIMENSION(MPI_STATUS_SIZE) for this. With the new module mpi_f08 in MPI-3.0, we want to use MODULE mpi_f08 IMPLICIT NONE TYPE :: MPI_Comm SEQUENCE INTEGER :: MPI_VAL END TYPE MPI_Comm TYPE :: MPI_Status SEQUENCE INTEGER :: MPI_SOURCE INTEGER :: MPI_TAG INTEGER :: MPI_ERROR INTEGER :: MPI_intern_highsize INTEGER :: MPI_intern_lowsize END TYPE MPI_Status END MODULE mpi_f08 Within existing user applications, it is important, that the new handles and status variables can be used everywhere, where the old-style INTEGER handles and INTEGER-array statuses are already used. This definition can be used in - user-defined arrays of handles or statuses - handles and statuses as part of a user-defined common block - handles and statuses as part of a user-defined derived type - handles and statuses as part of a user-defined sequence derived type In my attached the tests Bu1.f90, Bu2.f90 and Bu3.f90, this was checked by Reinhold Bader for many compilers. If an application already uses C-Fortran interoperability, it may be that such handles and statuses are also declared with a BIND(C) derived type. The test Bu4.f90 showed that this works with ifort and pgi, but is rejected by gfortran, nagfor, xlf, crayftn, and pathscale. In the tests Au1, Au2, Au3, Au4, the SEQUENCE attribute was removed from the definition of MPI handles and statuses; nearly all compiler reject nearly all use-cases. In the tests Cu1, Cu2, Cu3, Cu4, the SEQUENCE was substituted by BIND(C). This is rejected by all compilers for the case that a handle or status is part of a user-defined sequence derived type (see Cu3). Another problem is the implementation of the MPI routines. We want to be able to use wrappers written in C, i.e., to use BIND(C) for the definition of MPI routines. The test Bc1 calls an MPI routine that has a pure Fortran implementation, while Bc2 calls an MPI routine that is implemented with BIND(C). Bc1 of course works with all compilers, while Bc2 is rejected by gfortran, nagfor, xlf and pathscale. Same rejections occur if the handles are defined without SEQUENCE (see Ac2). Of course, in case C, handles are done with BIND(C), all compiler accept the MPI routines (see Cc2), but this does not really help, because the application use-case Cu3 was rejected by all compilers. How can this problem be solved? Sequence derived types are similar to common blocks and as long as normal Fortran types (INTEGER, REAL) are allowed with BIND(C), why is a sequence of such types is not allowed with BIND(C)? Which additions would be necessary to solve this problem within the TR 29113? To Problem B: ------------- As an example, here an MPI interface SUBROUTINE MPI_xxx(buf,...) BIND(C) TYPE(*), DIMENSION(..) :: buf END within the new MPI module and the following user code TYPE :: user_xxx SEQUENCE REAL, DIMENSION(100) :: x, y, z INTEGER, DIMEMSION(20) :: a, b END TYPE user_xxx TYPE(user_xxx) :: strct(5) CALL MPI_xxx(strct, ...) When I understood correctly, then strct is handed over through a dope vector with elem_len = sizeof(strct(1)). Without any SEQUENCE or BIND(C) for user_xxx, the compiler may optimize and modify the memory layout of the structure during the execution of the application, i.e., any address difference need not to be constant, e.g., LOC(strct(5)%b(1)) - LOC(strct(1)%x(1)) need not to be the same during the whole execution and within the application and the called MPI routine. Is this correct? To achieve a well-defined memory layout, the user has to use SEQUENCE or BIND(C) for user_xxx. Therefore in the past, the advice for MPI users was, to use the SEQUENCE attribute. Results from the compiler tests are: All compiler allow the usage of SEQUENCE and BIND(C) derived type as actual buffer arguments in implicit interfaces of MPI routines, see cases 1 and 2 in the tests to problem B. All compiler with a IGNORE_TKR directive allow SEQUENCE and BIND(C) derived type as actual buffer arguments in an explicit interface defined with INTERFACE, see cases 4 and 5. Normal derived types (that are neither SEQUENCE, nor BIND(C), nor defined within a module) cannot be used as such an actual buffer argument with the IBM xlf compiler, see cases 3 and 6. Same results have been obtained with explicit interfaces defined with CONTAINS, except for xlf, which does not allow the IGNORE_TKR directive outside of INTERFACE declarations. Do I understand correctly, that with N1845 of TR 29113, TYPE(*),DIMENSION(..) works only with BIND(C) derived types, but - not with normal derived types, and - not with sequence derived types? Is there a possibility to extend the TR that also sequence derived type can be handed over to TYPE(*), DIMENSION(..) dummy arguments? Problem C: ---------- Do we have other actual arguments that could be used in implicit interfaces or in explicit interfaces with these special pragmas, and that are still forbidden for TYPE(*),DIMENSION(..) dummy arguments? Best regards and my apologies that we detected the problem not earlier, Rolf --------------------------------------------------------- Test and file names for Problem A: ---------------------------------- Implementing new MPI handles and status with ... A = Fortran derived type B = Fortran sequence derived type C = Fortran BIND(C) derived type Usage of new MPI handles and status in the application within ... u1 = a user defined common block u2 = a user defined derived type u3 = a user defined sequence derived type u4 = a user defined BIND(C) derived type Call an MPI routine that is defined c1 = without BIND(C) c2 = with BIND(C) Compiler: Result are: R - rejected by the compiler i - compiles but issues an information or warning y - compiles Test --> | A A A A A A | B B B B B B | C C C C C C | | u u u u c c | u u u u c c | u u u u c c | | 1 2 3 4 1 2 | 1 2 3 4 1 2 | 1 2 3 4 1 2 | ------------------+--------------+--------------+--------------+ ifort 6.1 | R y R y y y | y y y y y y | R y R y y y | ifort 11.1 | y y R y y y | y y y y y y | y y R y y y | ifort 12.0 | y y R y y y | y y y y y y | y y R y y y | ------------------+--------------+--------------+--------------+ gfortran 4.6 | R y R R y R | y y y R y R | i i R i i i | nagfor 5.2 | R i R R i R | y i i R i R | y i R i i i | ------------------+--------------+--------------+--------------+ xlf 13.1 | R y R R y R | y y y R y R | y y R y y y | crayftn 7.3 | R y R R y y | y y y R y y | y y R y y y | ------------------+--------------+--------------+--------------+ pgi 11.3 | y y R y y y | y y y y y y | y y R y y y | pathscale 3.3b | R i R R i R | i i i R i R | i i R i i i | ------------------+--------------+--------------+--------------+ Notes: "i" entries from the NAG compiler are due to defined but unused entities. "i" entries from the gfortran compiler are wrt INTEGER, ... potentially being non-interoperable. "i" entries from the Pathscale compiler are (irrelevant) link-time messages. Tar file: - MPI3_bind-C_sequence_tests.tar.gz (attached) ----------------------------------------------------------------- Test and file names for Problem B: ---------------------------------- Test and file names: derived_types_test_1_impl_integer.f90 derived_types_test_2_impl_sequ_bc.f90 derived_types_test_3_impl_derived.f90 derived_types_test_4_expl_integer.f90 derived_types_test_5_expl_sequ_bc.f90 derived_types_test_6_expl_derived.f90 derived_types_test_7_all.f90 derived_types_test_1+2_impl_int+seq+bc.f90 with impl = Implicitly defined MPI routine is called. expl = Two explicitly defined MPI routines are called - without BIND(C), - with BIND(C) in the subroutine declaration. The buf dummy argument is defined with a TKR ignore directive. integer = Actual argument is a derived type and it is passed through first (integer) element. Three different derived types are used: - normal, - with SEQUENCE attribute, - with BIND(C) attribute. sequ_bc = Actual argument is a derived type with - with SEQUENCE attribute, - with BIND(C) attribute. derived = Actual argument is a normal derived type. The last test "1+2" is a special test to show a compile-time error only detected by nagfor. This test compines the calls in ..._1_impl_integer.f90 and ..._2_impl_sequ_bc.f90. Compiler results may be: R - rejected by the compiler Rs - rejected by the compiler (routines and main separately compiled) i - compiles but issues an information or warning and runs correctly y - compiles and runs correctly ys - compiles and runs correctly (routines and main separately compiled) N - compiles, then runs with run time failure Test --> | ---impl--- ---expl--- all impl | int seq drv int seq drv int+seq | 1 2 3 4 5 6 7 1+2 ------------------+--------------------------------------- ifort 11.1 | y y y y y y y y ifort 12.0 | y y y y y y y y ------------------+--------------------------------------- gfortran 4.6 | y y y R R R R y nagfor 5.2 | ys ys ys Rs Rs Rs Rs Rs ------------------+--------------------------------------- xlf 13.1 | y y R y y R R y crayftn 7.3 | i i i y y y i i ------------------+--------------------------------------- pgi 11.3 | y y y y y y y y pathscale 3.3b | i i i y y y i i ------------------+--------------------------------------- Notes: - Compile-time information, warning and errors with the CALL statements gfortran + 1,all : Rank mismatch warning (rank-1 vs scalar lines for all invocations) gfortran + 2,3,all : Type mismatch warning gfortran + 4,all : Rank mismatch error (rank-1 vs scalar lines for all invocations), i.e., none of the TKR_IGNORE directives work gfortran + 5,6,all : Type mismatch error, i.e., none of the TKR_IGNORE directives work nagfor + 4,all : Rank mismatch error (rank-1 vs scalar lines for all invocations), i.e., none of the TKR_IGNORE directives work nagfor + 5,6,all : Type mismatch error, i.e., none of the TKR_IGNORE directives work nagfor + 1+2, all : Error: Inconsistency between several calls of implicitly defined routine: "Inconsistent data type INTEGER (previously USER_BUFFER_NORMAL_TYPE) for argument 3 in reference to FOO_IMPL", i.e., a combination of the tests in 1 and 2 or 3 may fail. crayftn+1,2,3,all,1+2 : Rank mismatch diagnosed, but runs correctly xlf + 3,6, all : Severe error: "A derived-type object that appears as an actual argument in an external procedure reference must be of a sequence type or a type with the BIND(C) attribute, unless it is use-associated or of a type that is use-associated or defined in the specification part of a module. pathscale+1-3,all,1+2 : Warns about rank mismatch, but runs correctly. - Compile-time information, warning and errors with the declaration statement of MPI_foo..., i.e., when compiling the MPI library: gfortran + 4,5,6,all : Warning with BIND(C) declarations: INTEGER may not be interoperable with C, related to line "SUBROUTINE MPI_foo_ex_c(c1,c2,buf,test1,test2) BIND(C)" nagfor + "routines" : Additional information that BIND(C) is an "extension" in line "SUBROUTINE MPI_foo_ex_c(c1,c2,buf,test1,test2) BIND(C)" - Other compile-time information, warning and errors: gfortran+1,2,4,5,all,1+2:Warning with BIND(C) declarations: INTEGER may not be interoperable with C, related to line "TYPE, BIND(C) :: user_buffer_bind_c_type" nagfor+1,2,4,5,all,1+2: Additional information that BIND(C) is an "extension" in line "TYPE, BIND(C) :: user_buffer_bind_c_type" - Runtime errors were never detected, i.e., in the cases with actual argument = buf%VAL1 (i.e., the first element of the derived type), all compilers used call by reference, i.e., the called implicit or explicit routines could correctly access also the second element of the derived type. Tar file: - MPI3_derived_types_test.tar.gz (attached) Acknowledgments: I want to thank Reinhold Bader (LRZ, Munich, Germany) who made all the compiler protocols. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846
by John Reid 14 Apr '11

14 Apr '11

Rolf Rabenseifner wrote: > John, > > this is already handled by all hints about code movement and > register optimization in the old MPI-2.2 and in more > detail in the upcoming MPI-3.0. > > To summarize, there are several options: > - TARGET buf > - Using a call to a dummy routine MPI_F_SYNC_REG(buf) > immediately after MPI_Wait > - Storing buf as a module variable or in a common block > - (VOALTILE buf, not recommended) > > 'ASYNCHRONOUS buf' is explicitly not an option, because > a Fortran compiler > - that implements Fortran asynchronous input/output > with blocking I/O routines > - is allowed to ignore the ASYNCHRONOUS attribute. > > Only compilers with really asynchronous Fortran I/O > have to also internally imply the TARGET attribute when > the ASYNCHRONOUS attribute is given in the application. > > Is my analysis correct? Yes, but they all seem clugey and error prone to me. Given that asynchronous communication and asynchronous I/O are so alike and we already have syntax for the latter, should we not apply it also to the former? Even the word is appropriate. Cheers, John. -- Scanned by iCritical.

1 0

Re: [MPI3 Fortran] [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846
by N.M. Maclaren 14 Apr '11

14 Apr '11

On Apr 14 2011, Rolf Rabenseifner wrote: > >this is already handled by all hints about code movement and >register optimization in the old MPI-2.2 and in more >detail in the upcoming MPI-3.0. > >To summarize, there are several options: > - TARGET buf > - Using a call to a dummy routine MPI_F_SYNC_REG(buf) > immediately after MPI_Wait > - Storing buf as a module variable or in a common block > - (VOALTILE buf, not recommended) > >'ASYNCHRONOUS buf' is explicitly not an option, because >a Fortran compiler >- that implements Fortran asynchronous input/output > with blocking I/O routines >- is allowed to ignore the ASYNCHRONOUS attribute. > >Only compilers with really asynchronous Fortran I/O >have to also internally imply the TARGET attribute when >the ASYNCHRONOUS attribute is given in the application. > >Is my analysis correct? Not even remotely, I am afraid. Here are some comments: >To summarize, there are several options: > - TARGET buf Why should that work? Fortran compilers are required to take notice of it only when also operating with POINTER variables. And there is nothing stopping them from moving such data dynamically, if they update all associated pointers. That's old technology, after all. > - Using a call to a dummy routine MPI_F_SYNC_REG(buf) > immediately after MPI_Wait That won't work, either, because it doesn't stop copy-in/copy-out and other such techniques. > - Storing buf as a module variable or in a common block That won't work, either, for the above reason. >> - (VOALTILE buf, not recommended) Why should that work any better than ASYNCHRONOUS? All it says (in both C and Fortran, incidentally) is "All bets are off - consult your vendor documentation for what might happen." ASYNCHRONOUS has the right semantics. >'ASYNCHRONOUS buf' is explicitly not an option, because >a Fortran compiler >- that implements Fortran asynchronous input/output > with blocking I/O routines >- is allowed to ignore the ASYNCHRONOUS attribute. If you are being that legalistic, the problem is insoluble in all of C, C++ and Fortran. Sorry, but that IS what the standards say! >Only compilers with really asynchronous Fortran I/O >have to also internally imply the TARGET attribute when >the ASYNCHRONOUS attribute is giv Er, no, they don't. The attributes are different and have subtly different properties. >> You do not need to be a member of J3. Would you like to join >> interop-tr? > >Most of the internals of the TR, I do not understand. >As I see, the list moderator can agree to deliver the few >mails from me also to the other members of the interop-tr. >It isn't a problem for me to put you, Bill, Nick, Craig, >and Reinhold on the CC. The problem is when we reply. Unless I leave you in the list explicitly, you won't see the reply. And duplicating people has other problems. Regards, Nick Maclaren.

1 0

Re: [MPI3 Fortran] [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846
by Dan Nagle 14 Apr '11

14 Apr '11

Hello, On Apr 14, 2011, at 11:26 , Rolf Rabenseifner wrote: > > Is my analysis correct? I believe your analysis is correct. I may be in the minority on the Fortran committees at this time. When the actual edits needed (why?) to make asynchronous suitable for arbitrary asynchronous libraries are adduced, we'll see what the Fortran committees then think. I have too many burn on my fingers to place a bet. -- Cheers! Dan Nagle

1 0

Re: [MPI3 Fortran] [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846
by Rolf Rabenseifner 14 Apr '11

14 Apr '11

John, this is already handled by all hints about code movement and register optimization in the old MPI-2.2 and in more detail in the upcoming MPI-3.0. To summarize, there are several options: - TARGET buf - Using a call to a dummy routine MPI_F_SYNC_REG(buf) immediately after MPI_Wait - Storing buf as a module variable or in a common block - (VOALTILE buf, not recommended) 'ASYNCHRONOUS buf' is explicitly not an option, because a Fortran compiler - that implements Fortran asynchronous input/output with blocking I/O routines - is allowed to ignore the ASYNCHRONOUS attribute. Only compilers with really asynchronous Fortran I/O have to also internally imply the TARGET attribute when the ASYNCHRONOUS attribute is given in the application. Is my analysis correct? > You do not need to be a member of J3. Would you like to join > interop-tr? Most of the internals of the TR, I do not understand. As I see, the list moderator can agree to deliver the few mails from me also to the other members of the interop-tr. It isn't a problem for me to put you, Bill, Nick, Craig, and Reinhold on the CC. Best regards Rolf ----- Original Message ----- > From: "John Reid" <John.Reid(a)stfc.ac.uk> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org>, interop-tr(a)j3-fortran.org > Sent: Thursday, April 14, 2011 4:33:56 PM > Subject: Re: [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846 > Rolf Rabenseifner wrote: > > [I expect that the mail is not delivered by > > interop-tr(a)j3-fortran.org > > because I'm not a member of J3] > > You do not need to be a member of J3. Would you like to join > interop-tr? > > > After all the discussion, it really looks best to > > ignore the existence of the ASYNCHRONOUS attribute > > when looking at the co-existence of Fortran nonblocking > > and MPI. > > Although the idea of needing no text in our TR is welcome, I do not > agree with > your conclusion. Consider the code > > REAL :: buf(100,100) > : ! Code that involves buf > CALL MPI_Irecv(buf,...req,...) > : ! Code that does not involve buf > CALL MPI_Wait(req,...) > : ! Code that involves buf > > Since there is no mention of buf in the call of MPI_Wait, the compiler > is free > to perform code motion involving it. For example, it might load buf > into a GPU > during the calculations that do not involve buf, ready for those that > do. > > Asynchronous communication is so like asynchronous I/O that I think we > need to > have the same rules. If we can somehow write this into the normative > text of the > TR, the effect of adding ASYNCHRONOUS to the declaration of buf in the > above > example would be that the compiler will not be allowed to make the > harmful > change that I have just outlined. > > Cheers, > > John. > > > > > > Based on the discussion, I plan to add the following text > > into the draft for MPI-3.0 and based on Option 1 in a > > previous email: > > > > ------------------ > > [Text about an application with parts of an array > > involved in a nonblocking MPI operation and other > > parts involved in computation, and a compiler optimization > > with temporarily copying variables into a local memory > > and afterward back to the original memory] > > > > Note that this type of compiler optimization can > > not be prevented when buf is declared with the > > ASYNCHRONOUS Fortran attribute, and it should not > > be prevented by declaring buf as VOLATILE because: > > - A Fortran asynchronous input/output operation on a part > > of an array variable implies that the access to the whole > > array (the so-called pending I/O storage sequence affector, > > see Section 9.6.2.5 in [Fortran 2008]) is restricted > > as if the whole array is involved in the pending I/O > > (see Section 9.6.4.1, paragraphs 5 and 6 in [Fortran 2008]). > > Therefore, the ASYNCHRONOUS cannot be used if parts of the > > array is involved in a asynchronous operation and other > > parts are used for computation. > > - In principle, the Fortran standard restricts the scope > > of the ASYNCHRONOUS attribute only to Fortran asynchronous > > input and output operations. > > - A Fortran compiler may implement the Fortran asynchronous > > input/output operations as blocking I/O. In this case, > > the compiler can fully ignore the ASYNCHRONOUS attribute. > > - The VOLATILE implies that all accesses to any storage unit (word) > > of buf must be directly done in the main memory exactly in > > the sequence defined by the application program. The attribute > > VOLATILE prevents every register or cache optimization. > > - Therefore, VOLATILE may cause a huge performance degradation. > > > > Instead of solving the problem, it is better to prevent the problem, > > i.e., when overlapping communication and computation, the > > communication > > (or nonblocking IO) and the computation should be executed on > > different > > sets of variables. In this case, the temporary memory modifications > > are done only on the variables used in the computation and cannot > > have any side effect on the data used in the nonblocking MPI > > operations. > > --------------- > > > > The first three items in the list suggest that it is a bad idea > > to enlarge the scope of ASYNCHRONOUS to nonblocking MPI. > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> > >> To: "John Reid" <John.Reid(a)stfc.ac.uk>, interop-tr(a)j3-fortran.org > >> Cc: interop-tr(a)j3-fortran.org, "Rolf Rabenseifner" > >> <rabenseifner(a)hlrs.de> > >> Sent: Thursday, April 14, 2011 11:33:32 AM > >> Subject: Re: [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot > >> N1846 > >> On Apr 14 2011, John Reid wrote: > >>> If the comment in the middle is changed to > >>> > >>> ! asynchronous I/O on ptr variable > >>> > >>> the program would be non-conforming because A would be an affector > >>> without the ASYNCHRONOUS attribute. > >>> > >>> Perhaps we need normative text that says (somehow) that > >>> asynchronous > >>> communication is regarded as if it were asynchronous i/o. > >> In which case my reading of the standard is wrong. I have just > >> looked > >> again, and think that either I probably was or the wording is more > >> confusing than it should be. > >> > >> I don't think that Fortran intends to forbid: > >> > >> PROGRAM Main > >> REAL :: array(100) > >> CALL Fred(array(:50),array(51:)) > >> END PROGRAM Main > >> > >> SUBROUTINE Fred (buffer, scratch) > >> REAL, ASYNCHRONOUS :: buffer(:) > >> REAL :: scratch(:) > >> CALL MPI_Irecv(buffer,...) > >> CALL DGEMM(scratch,...) > >> CALL MPI_Wait(...) > >> END SUBROUTINE Fred > >> > >> As you say, the same problem can be shown using Fortran > >> asynchronous > >> I/O. > >> > >> However, this issue is not limited to ASYNCHRONOUS and affects > >> variables > >> that have the TARGET attribute, too. I assert that it is > >> conceptually > >> identical to the one where TARGET is used, MPI_Irecv sets up a > >> pointer > >> and DGEMM uses that pointer. I.e. it's a question of exactly when > >> association stops. > >> > >> > >> Regards, > >> Nick Maclaren. > > > > -- > Scanned by iCritical. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846
by John Reid 14 Apr '11

14 Apr '11

Rolf Rabenseifner wrote: > [I expect that the mail is not delivered by interop-tr(a)j3-fortran.org > because I'm not a member of J3] You do not need to be a member of J3. Would you like to join interop-tr? > After all the discussion, it really looks best to > ignore the existence of the ASYNCHRONOUS attribute > when looking at the co-existence of Fortran nonblocking > and MPI. Although the idea of needing no text in our TR is welcome, I do not agree with your conclusion. Consider the code REAL :: buf(100,100) : ! Code that involves buf CALL MPI_Irecv(buf,...req,...) : ! Code that does not involve buf CALL MPI_Wait(req,...) : ! Code that involves buf Since there is no mention of buf in the call of MPI_Wait, the compiler is free to perform code motion involving it. For example, it might load buf into a GPU during the calculations that do not involve buf, ready for those that do. Asynchronous communication is so like asynchronous I/O that I think we need to have the same rules. If we can somehow write this into the normative text of the TR, the effect of adding ASYNCHRONOUS to the declaration of buf in the above example would be that the compiler will not be allowed to make the harmful change that I have just outlined. Cheers, John. > > Based on the discussion, I plan to add the following text > into the draft for MPI-3.0 and based on Option 1 in a > previous email: > > ------------------ > [Text about an application with parts of an array > involved in a nonblocking MPI operation and other > parts involved in computation, and a compiler optimization > with temporarily copying variables into a local memory > and afterward back to the original memory] > > Note that this type of compiler optimization can > not be prevented when buf is declared with the > ASYNCHRONOUS Fortran attribute, and it should not > be prevented by declaring buf as VOLATILE because: > - A Fortran asynchronous input/output operation on a part > of an array variable implies that the access to the whole > array (the so-called pending I/O storage sequence affector, > see Section 9.6.2.5 in [Fortran 2008]) is restricted > as if the whole array is involved in the pending I/O > (see Section 9.6.4.1, paragraphs 5 and 6 in [Fortran 2008]). > Therefore, the ASYNCHRONOUS cannot be used if parts of the > array is involved in a asynchronous operation and other > parts are used for computation. > - In principle, the Fortran standard restricts the scope > of the ASYNCHRONOUS attribute only to Fortran asynchronous > input and output operations. > - A Fortran compiler may implement the Fortran asynchronous > input/output operations as blocking I/O. In this case, > the compiler can fully ignore the ASYNCHRONOUS attribute. > - The VOLATILE implies that all accesses to any storage unit (word) > of buf must be directly done in the main memory exactly in > the sequence defined by the application program. The attribute > VOLATILE prevents every register or cache optimization. > - Therefore, VOLATILE may cause a huge performance degradation. > > Instead of solving the problem, it is better to prevent the problem, > i.e., when overlapping communication and computation, the communication > (or nonblocking IO) and the computation should be executed on different > sets of variables. In this case, the temporary memory modifications > are done only on the variables used in the computation and cannot > have any side effect on the data used in the nonblocking MPI operations. > --------------- > > The first three items in the list suggest that it is a bad idea > to enlarge the scope of ASYNCHRONOUS to nonblocking MPI. > > Best regards > Rolf > > ----- Original Message ----- >> From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> >> To: "John Reid" <John.Reid(a)stfc.ac.uk>, interop-tr(a)j3-fortran.org >> Cc: interop-tr(a)j3-fortran.org, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >> Sent: Thursday, April 14, 2011 11:33:32 AM >> Subject: Re: [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846 >> On Apr 14 2011, John Reid wrote: >>> If the comment in the middle is changed to >>> >>> ! asynchronous I/O on ptr variable >>> >>> the program would be non-conforming because A would be an affector >>> without the ASYNCHRONOUS attribute. >>> >>> Perhaps we need normative text that says (somehow) that asynchronous >>> communication is regarded as if it were asynchronous i/o. >> In which case my reading of the standard is wrong. I have just looked >> again, and think that either I probably was or the wording is more >> confusing than it should be. >> >> I don't think that Fortran intends to forbid: >> >> PROGRAM Main >> REAL :: array(100) >> CALL Fred(array(:50),array(51:)) >> END PROGRAM Main >> >> SUBROUTINE Fred (buffer, scratch) >> REAL, ASYNCHRONOUS :: buffer(:) >> REAL :: scratch(:) >> CALL MPI_Irecv(buffer,...) >> CALL DGEMM(scratch,...) >> CALL MPI_Wait(...) >> END SUBROUTINE Fred >> >> As you say, the same problem can be shown using Fortran asynchronous >> I/O. >> >> However, this issue is not limited to ASYNCHRONOUS and affects >> variables >> that have the TARGET attribute, too. I assert that it is conceptually >> identical to the one where TARGET is used, MPI_Irecv sets up a pointer >> and DGEMM uses that pointer. I.e. it's a question of exactly when >> association stops. >> >> >> Regards, >> Nick Maclaren. > -- Scanned by iCritical.

1 0

[MPI3 Fortran] Derived-data, a new/late comment on Fortran WG5 ballot N1846
by Rolf Rabenseifner 14 Apr '11

14 Apr '11

Dear John, dear members of the WG5 and J3 Fortran committees, I'm sorry that with this additional comment on problems regarding derived types, I missed the deadline of the N1846 ballot for the N1845 draft of the TR 29113 on extensions on Fortran-C interoperability. It looks like that there are two problems with derived types and the Fortran-MPI-compatibility with a background in the Fortran-C-interoperability. Problem A: Sequence derived types in the MPI routine interfaces, which are declared with BIND(C); this problem occurs with the new MPI handles and MPI statuses. Problem B: Derived type, SEQUENCE derived type, and BIND(C) derived type as actual user buffers for TYPE(*),DIMENSION(..) dummy arguments in BIND(C) MPI routine interfaces. Problem C: Based on problem B we have the question whether the new TYPE(*),DIMENSION(..) dummy arguments in BIND(C) MPI routine interfaces really allow all actual arguments that were possible with implicit interfaces or with explicit interfaces in combination with pragmas like !DEC$ ATTRIBUTES NO_ARG_CHECK :: buf and !$PRAGMA IGNORE_TKR buf At least for problems A and B, the solution may be "simple" by allowing SEQUENCE derived types as element of a BIND(C) derived type and in BIND(C) subroutine dummy arguments. Why we need this is shown in very detail in the following description of Problems A and B. A still open question is Problem C. To Problem A: ------------- For better type-safety and compile-time argument-checking, the MPI Forum wants to use named types for MPI handles and MPI status. In MPI-1.1 up to MPI-2.2, we use INTEGER and INTEGER, DIMENSION(MPI_STATUS_SIZE) for this. With the new module mpi_f08 in MPI-3.0, we want to use MODULE mpi_f08 IMPLICIT NONE TYPE :: MPI_Comm SEQUENCE INTEGER :: MPI_VAL END TYPE MPI_Comm TYPE :: MPI_Status SEQUENCE INTEGER :: MPI_SOURCE INTEGER :: MPI_TAG INTEGER :: MPI_ERROR INTEGER :: MPI_intern_highsize INTEGER :: MPI_intern_lowsize END TYPE MPI_Status END MODULE mpi_f08 Within existing user applications, it is important, that the new handles and status variables can be used everywhere, where the old-style INTEGER handles and INTEGER-array statuses are already used. This definition can be used in - user-defined arrays of handles or statuses - handles and statuses as part of a user-defined common block - handles and statuses as part of a user-defined derived type - handles and statuses as part of a user-defined sequence derived type In my attached the tests Bu1.f90, Bu2.f90 and Bu3.f90, this was checked by Reinhold Bader for many compilers. If an application already uses C-Fortran interoperability, it may be that such handles and statuses are also declared with a BIND(C) derived type. The test Bu4.f90 showed that this works with ifort and pgi, but is rejected by gfortran, nagfor, xlf, crayftn, and pathscale. In the tests Au1, Au2, Au3, Au4, the SEQUENCE attribute was removed from the definition of MPI handles and statuses; nearly all compiler reject nearly all use-cases. In the tests Cu1, Cu2, Cu3, Cu4, the SEQUENCE was substituted by BIND(C). This is rejected by all compilers for the case that a handle or status is part of a user-defined sequence derived type (see Cu3). Another problem is the implementation of the MPI routines. We want to be able to use wrappers written in C, i.e., to use BIND(C) for the definition of MPI routines. The test Bc1 calls an MPI routine that has a pure Fortran implementation, while Bc2 calls an MPI routine that is implemented with BIND(C). Bc1 of course works with all compilers, while Bc2 is rejected by gfortran, nagfor, xlf and pathscale. Same rejections occur if the handles are defined without SEQUENCE (see Ac2). Of course, in case C, handles are done with BIND(C), all compiler accept the MPI routines (see Cc2), but this does not really help, because the application use-case Cu3 was rejected by all compilers. How can this problem be solved? Sequence derived types are similar to common blocks and as long as normal Fortran types (INTEGER, REAL) are allowed with BIND(C), why is a sequence of such types is not allowed with BIND(C)? Which additions would be necessary to solve this problem within the TR 29113? To Problem B: ------------- As an example, here an MPI interface SUBROUTINE MPI_xxx(buf,...) BIND(C) TYPE(*), DIMENSION(..) :: buf END within the new MPI module and the following user code TYPE :: user_xxx SEQUENCE REAL, DIMENSION(100) :: x, y, z INTEGER, DIMEMSION(20) :: a, b END TYPE user_xxx TYPE(user_xxx) :: strct(5) CALL MPI_xxx(strct, ...) When I understood correctly, then strct is handed over through a dope vector with elem_len = sizeof(strct(1)). Without any SEQUENCE or BIND(C) for user_xxx, the compiler may optimize and modify the memory layout of the structure during the execution of the application, i.e., any address difference need not to be constant, e.g., LOC(strct(5)%b(1)) - LOC(strct(1)%x(1)) need not to be the same during the whole execution and within the application and the called MPI routine. Is this correct? To achieve a well-defined memory layout, the user has to use SEQUENCE or BIND(C) for user_xxx. Therefore in the past, the advice for MPI users was, to use the SEQUENCE attribute. Results from the compiler tests are: All compiler allow the usage of SEQUENCE and BIND(C) derived type as actual buffer arguments in implicit interfaces of MPI routines, see cases 1 and 2 in the tests to problem B. All compiler with a IGNORE_TKR directive allow SEQUENCE and BIND(C) derived type as actual buffer arguments in an explicit interface defined with INTERFACE, see cases 4 and 5. Normal derived types (that are neither SEQUENCE, nor BIND(C), nor defined within a module) cannot be used as such an actual buffer argument with the IBM xlf compiler, see cases 3 and 6. Same results have been obtained with explicit interfaces defined with CONTAINS, except for xlf, which does not allow the IGNORE_TKR directive outside of INTERFACE declarations. Do I understand correctly, that with N1845 of TR 29113, TYPE(*),DIMENSION(..) works only with BIND(C) derived types, but - not with normal derived types, and - not with sequence derived types? Is there a possibility to extend the TR that also sequence derived type can be handed over to TYPE(*), DIMENSION(..) dummy arguments? Problem C: ---------- Do we have other actual arguments that could be used in implicit interfaces or in explicit interfaces with these special pragmas, and that are still forbidden for TYPE(*),DIMENSION(..) dummy arguments? Best regards and my apologies that we detected the problem not earlier, Rolf --------------------------------------------------------- Test and file names for Problem A: ---------------------------------- Implementing new MPI handles and status with ... A = Fortran derived type B = Fortran sequence derived type C = Fortran BIND(C) derived type Usage of new MPI handles and status in the application within ... u1 = a user defined common block u2 = a user defined derived type u3 = a user defined sequence derived type u4 = a user defined BIND(C) derived type Call an MPI routine that is defined c1 = without BIND(C) c2 = with BIND(C) Compiler: Result are: R - rejected by the compiler i - compiles but issues an information or warning y - compiles Test --> | A A A A A A | B B B B B B | C C C C C C | | u u u u c c | u u u u c c | u u u u c c | | 1 2 3 4 1 2 | 1 2 3 4 1 2 | 1 2 3 4 1 2 | ------------------+--------------+--------------+--------------+ ifort 6.1 | R y R y y y | y y y y y y | R y R y y y | ifort 11.1 | y y R y y y | y y y y y y | y y R y y y | ifort 12.0 | y y R y y y | y y y y y y | y y R y y y | ------------------+--------------+--------------+--------------+ gfortran 4.6 | R y R R y R | y y y R y R | i i R i i i | nagfor 5.2 | R i R R i R | y i i R i R | y i R i i i | ------------------+--------------+--------------+--------------+ xlf 13.1 | R y R R y R | y y y R y R | y y R y y y | crayftn 7.3 | R y R R y y | y y y R y y | y y R y y y | ------------------+--------------+--------------+--------------+ pgi 11.3 | y y R y y y | y y y y y y | y y R y y y | pathscale 3.3b | R i R R i R | i i i R i R | i i R i i i | ------------------+--------------+--------------+--------------+ Notes: "i" entries from the NAG compiler are due to defined but unused entities. "i" entries from the gfortran compiler are wrt INTEGER, ... potentially being non-interoperable. "i" entries from the Pathscale compiler are (irrelevant) link-time messages. Tar file: - MPI3_bind-C_sequence_tests.tar.gz (attached) ----------------------------------------------------------------- Test and file names for Problem B: ---------------------------------- Test and file names: derived_types_test_1_impl_integer.f90 derived_types_test_2_impl_sequ_bc.f90 derived_types_test_3_impl_derived.f90 derived_types_test_4_expl_integer.f90 derived_types_test_5_expl_sequ_bc.f90 derived_types_test_6_expl_derived.f90 derived_types_test_7_all.f90 derived_types_test_1+2_impl_int+seq+bc.f90 with impl = Implicitly defined MPI routine is called. expl = Two explicitly defined MPI routines are called - without BIND(C), - with BIND(C) in the subroutine declaration. The buf dummy argument is defined with a TKR ignore directive. integer = Actual argument is a derived type and it is passed through first (integer) element. Three different derived types are used: - normal, - with SEQUENCE attribute, - with BIND(C) attribute. sequ_bc = Actual argument is a derived type with - with SEQUENCE attribute, - with BIND(C) attribute. derived = Actual argument is a normal derived type. The last test "1+2" is a special test to show a compile-time error only detected by nagfor. This test compines the calls in ..._1_impl_integer.f90 and ..._2_impl_sequ_bc.f90. Compiler results may be: R - rejected by the compiler Rs - rejected by the compiler (routines and main separately compiled) i - compiles but issues an information or warning and runs correctly y - compiles and runs correctly ys - compiles and runs correctly (routines and main separately compiled) N - compiles, then runs with run time failure Test --> | ---impl--- ---expl--- all impl | int seq drv int seq drv int+seq | 1 2 3 4 5 6 7 1+2 ------------------+--------------------------------------- ifort 11.1 | y y y y y y y y ifort 12.0 | y y y y y y y y ------------------+--------------------------------------- gfortran 4.6 | y y y R R R R y nagfor 5.2 | ys ys ys Rs Rs Rs Rs Rs ------------------+--------------------------------------- xlf 13.1 | y y R y y R R y crayftn 7.3 | i i i y y y i i ------------------+--------------------------------------- pgi 11.3 | y y y y y y y y pathscale 3.3b | i i i y y y i i ------------------+--------------------------------------- Notes: - Compile-time information, warning and errors with the CALL statements gfortran + 1,all : Rank mismatch warning (rank-1 vs scalar lines for all invocations) gfortran + 2,3,all : Type mismatch warning gfortran + 4,all : Rank mismatch error (rank-1 vs scalar lines for all invocations), i.e., none of the TKR_IGNORE directives work gfortran + 5,6,all : Type mismatch error, i.e., none of the TKR_IGNORE directives work nagfor + 4,all : Rank mismatch error (rank-1 vs scalar lines for all invocations), i.e., none of the TKR_IGNORE directives work nagfor + 5,6,all : Type mismatch error, i.e., none of the TKR_IGNORE directives work nagfor + 1+2, all : Error: Inconsistency between several calls of implicitly defined routine: "Inconsistent data type INTEGER (previously USER_BUFFER_NORMAL_TYPE) for argument 3 in reference to FOO_IMPL", i.e., a combination of the tests in 1 and 2 or 3 may fail. crayftn+1,2,3,all,1+2 : Rank mismatch diagnosed, but runs correctly xlf + 3,6, all : Severe error: "A derived-type object that appears as an actual argument in an external procedure reference must be of a sequence type or a type with the BIND(C) attribute, unless it is use-associated or of a type that is use-associated or defined in the specification part of a module. pathscale+1-3,all,1+2 : Warns about rank mismatch, but runs correctly. - Compile-time information, warning and errors with the declaration statement of MPI_foo..., i.e., when compiling the MPI library: gfortran + 4,5,6,all : Warning with BIND(C) declarations: INTEGER may not be interoperable with C, related to line "SUBROUTINE MPI_foo_ex_c(c1,c2,buf,test1,test2) BIND(C)" nagfor + "routines" : Additional information that BIND(C) is an "extension" in line "SUBROUTINE MPI_foo_ex_c(c1,c2,buf,test1,test2) BIND(C)" - Other compile-time information, warning and errors: gfortran+1,2,4,5,all,1+2:Warning with BIND(C) declarations: INTEGER may not be interoperable with C, related to line "TYPE, BIND(C) :: user_buffer_bind_c_type" nagfor+1,2,4,5,all,1+2: Additional information that BIND(C) is an "extension" in line "TYPE, BIND(C) :: user_buffer_bind_c_type" - Runtime errors were never detected, i.e., in the cases with actual argument = buf%VAL1 (i.e., the first element of the derived type), all compilers used call by reference, i.e., the called implicit or explicit routines could correctly access also the second element of the derived type. Tar file: - MPI3_derived_types_test.tar.gz (attached) Acknowledgments: I want to thank Reinhold Bader (LRZ, Munich, Germany) who made all the compiler protocols. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846
by Rolf Rabenseifner 14 Apr '11

14 Apr '11

[I expect that the mail is not delivered by interop-tr(a)j3-fortran.org because I'm not a member of J3] After all the discussion, it really looks best to ignore the existence of the ASYNCHRONOUS attribute when looking at the co-existence of Fortran nonblocking and MPI. Based on the discussion, I plan to add the following text into the draft for MPI-3.0 and based on Option 1 in a previous email: ------------------ [Text about an application with parts of an array involved in a nonblocking MPI operation and other parts involved in computation, and a compiler optimization with temporarily copying variables into a local memory and afterward back to the original memory] Note that this type of compiler optimization can not be prevented when buf is declared with the ASYNCHRONOUS Fortran attribute, and it should not be prevented by declaring buf as VOLATILE because: - A Fortran asynchronous input/output operation on a part of an array variable implies that the access to the whole array (the so-called pending I/O storage sequence affector, see Section 9.6.2.5 in [Fortran 2008]) is restricted as if the whole array is involved in the pending I/O (see Section 9.6.4.1, paragraphs 5 and 6 in [Fortran 2008]). Therefore, the ASYNCHRONOUS cannot be used if parts of the array is involved in a asynchronous operation and other parts are used for computation. - In principle, the Fortran standard restricts the scope of the ASYNCHRONOUS attribute only to Fortran asynchronous input and output operations. - A Fortran compiler may implement the Fortran asynchronous input/output operations as blocking I/O. In this case, the compiler can fully ignore the ASYNCHRONOUS attribute. - The VOLATILE implies that all accesses to any storage unit (word) of buf must be directly done in the main memory exactly in the sequence defined by the application program. The attribute VOLATILE prevents every register or cache optimization. - Therefore, VOLATILE may cause a huge performance degradation. Instead of solving the problem, it is better to prevent the problem, i.e., when overlapping communication and computation, the communication (or nonblocking IO) and the computation should be executed on different sets of variables. In this case, the temporary memory modifications are done only on the variables used in the computation and cannot have any side effect on the data used in the nonblocking MPI operations. --------------- The first three items in the list suggest that it is a bad idea to enlarge the scope of ASYNCHRONOUS to nonblocking MPI. Best regards Rolf ----- Original Message ----- > From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> > To: "John Reid" <John.Reid(a)stfc.ac.uk>, interop-tr(a)j3-fortran.org > Cc: interop-tr(a)j3-fortran.org, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Sent: Thursday, April 14, 2011 11:33:32 AM > Subject: Re: [Interop-tr] [Mpi-forum] Comment on Fortran WG5 ballot N1846 > On Apr 14 2011, John Reid wrote: > > > >If the comment in the middle is changed to > > > > ! asynchronous I/O on ptr variable > > > > the program would be non-conforming because A would be an affector > > without the ASYNCHRONOUS attribute. > > > >Perhaps we need normative text that says (somehow) that asynchronous > >communication is regarded as if it were asynchronous i/o. > > In which case my reading of the standard is wrong. I have just looked > again, and think that either I probably was or the wording is more > confusing than it should be. > > I don't think that Fortran intends to forbid: > > PROGRAM Main > REAL :: array(100) > CALL Fred(array(:50),array(51:)) > END PROGRAM Main > > SUBROUTINE Fred (buffer, scratch) > REAL, ASYNCHRONOUS :: buffer(:) > REAL :: scratch(:) > CALL MPI_Irecv(buffer,...) > CALL DGEMM(scratch,...) > CALL MPI_Wait(...) > END SUBROUTINE Fred > > As you say, the same problem can be shown using Fortran asynchronous > I/O. > > However, this issue is not limited to ASYNCHRONOUS and affects > variables > that have the TARGET attribute, too. I assert that it is conceptually > identical to the one where TARGET is used, MPI_Irecv sets up a pointer > and DGEMM uses that pointer. I.e. it's a question of exactly when > association stops. > > > Regards, > Nick Maclaren. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Nonblocking MPI and Fortran temporary memory modifications
by N.M. Maclaren 04 Apr '11

04 Apr '11

On Mar 27 2011, Rolf Rabenseifner wrote: > >Reinhold showed me that the ASYNCHRONOUS attribute may not help >because a compiler may implement asynchronous Fortran I/O >with blocking I/O and therefore may ignore all ASYNCHRONOUS >attributes. That is a defect in the method of language specification used by most ISO standards. When we discussed it, the majority agreed that a Fortran compiler was obviously at liberty to implement asynchronous synchronously either at the initiation or completion, but should not ignore the attribute because the asynchronicity might be in C code. I agree that it isn't clearly non-conforming, but it is clearly not according to the intent. >Pages 24-31 in the pdf (pages 549-556 in the original document) >show our knowledge about the nonblocking and datatype (MPI_BOTTOM) problems >in the combination of Fortran and MPI. MPI_BOTTOM is truly horrible, and is unfixable, whether in Fortran or C. Yes, it can be done using C_PTR, but it is still undefined behaviour in both languages. >Unfortunately in the tabï¿½le on page 550, we have to modify all entries >about ASYNCHRONOUS into "NOT solved". Not really. We have agreed on the direction - with, if I recall, only one dissenter. >Do you see further problems with nonblocking and "MPI_BOTTOM", >that are not mentioned in this section and that should be. I will try to take a look, but my advice is not to use MPI_BOTTOM in portable code. I may be able to help with extra constraints. Regards, Nick.

1 0