October 2011 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Jeff Squyres 07 Oct '11

07 Oct '11

I guess this looks ok -- there's some grammar changes I'd make, though. One thing to be careful of: > * Advice to users. > * Structures combining different basic datatypes should be defined so that there will > * be no gaps based on alignment rules, and if used as an array of structures, then > * also without such an alignment-gap at the end of the structure. > * In MPI communication, the content of such gaps would not be communicated. ^^^ This is not necessarily true. A smart enough MPI implementation may realize that it's faster to treat a small gap as a contiguous datatype and just ignore the extra data, especially if it's in a homogeneous environment, and copy-in/copy-out is used anyway. On Oct 6, 2011, at 7:55 PM, Rolf Rabenseifner wrote: > Dear all, > > here is my proposal to solve the alignment problems. > Please check carefully. > > It is touching only the Fortran side. > C problems with alignments are not really touched, but should be small. > > ---------------------------------------------------------------------- > > (Lines with * are new, lines with : are modified, lines with x are removed) > > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf > p86:45-47 (=MPI-2.2 p78:45-47) read > > If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon > is the least non-negative increment needed to round extent(Typemap) to the next multiple > of max_i k_i. > The complete definition of extent is given on page 107. > > but should read > > If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon > is the least non-negative increment needed to round extent(Typemap) to the next multiple > of max_i k_i. > * In Fortran, whether the alignments k_i are computed according to the alignments > * used by the compiler in common blocks, SEQUENCE derived types, or BIND(C) derived types, > * is implementation-dependent. > : The complete definition of extent is given in Section 4.1.6 on page 106. > > ________ > > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf > p87:7-11 (=MPI-2.2 p79:7-11) read > > Rationale. The definition of extent is motivated by the assumption that the amount > of padding added at the end of each structure in an array of structures is the least > needed to fulfill alignment constraints. More explicit control of the extent is provided > in Section 4.1.6. Such explicit control is needed in cases where the assumption does > not hold, for example, where union types are used. > (End of rationale.) > > but should read > > Rationale. The definition of extent is motivated by the assumption that the amount > of padding added at the end of each structure in an array of structures is the least > needed to fulfill alignment constraints. More explicit control of the extent is provided > in Section 4.1.6. Such explicit control is needed in cases where the assumption does > not hold, for example, where union types are used. > * In Fortran, structures can be expressed with several language features, e.g., > * common blocks, SEQUENCE derived types, or BIND(C) derived types. The compiler may > * use different alignments, and therefore, it is recommended to use MPI_TYPE_CREATE_RESIZED > * for arrays of structures if an alignment may cause an alignment-gap at the end of > * a structure as described in Section 4.1.6 on page 106 and in Section 16.2.15 on page 580. > (End of rationale.) > > > ________ > > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf > p107:13-17 (=MPI-2.2 p96:42-45) read > > If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon > is the least non-negative increment needed to round extent(Typemap) to the next multiple > of max_i k_i. > The formal definitions given for the various datatype constructors apply now, > with the amended definition of extent. > > but should read > > If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon > is the least non-negative increment needed to round extent(Typemap) to the next multiple > of max_i k_i. > * In Fortran, whether the alignments k_i are computed according to the alignments > * used by the compiler in common blocks, SEQUENCE derived types, or BIND(C) derived types, > * is implementation-dependent. > The formal definitions given for the various datatype constructors apply now, > with the amended definition of extent. > > * Rationale. > * Before Fortran 2003, MPI_TYPE_CREATE_STRUCT could be applied to Fortran common blocks and > * SEQUENCE derived types. With Fortran 2003, this list was extended by BIND(C) derived types > * and MPI implementors have implemented the alignments k_i different, i.e., some based on > * the alignments used in SEQUENCE derived types, and others according to BIND(C) derived types. > * End of rationale. > > * Advice to implementors. > * In Fortran, it is generally recommended to use BIND(C) derived types instead of > * common blocks or SEQUENCE derived types. Therefore it is recommended to calculate the > * alignments k_i based on BIND(C) derived types. > * End of advoice to implementors. > > * Advice to users. > * Structures combining different basic datatypes should be defined so that there will > * be no gaps based on alignment rules, and if used as an array of structures, then > * also without such an alignment-gap at the end of the structure. > * In MPI communication, the content of such gaps would not be communicated. > * If alignment-gaps may occur, e.g., between an odd number of floats ore REAL before > * a double or DOUBLE PRECISION data, then such gaps should be added explicitely in > * the structure and also to the MPI derived datatype handle, because the communication > * of a contiguous derived datatype may be significantly faster than the communication > * of one that is non-contiguous due such alignment-gaps. > * > * Example: > * Instead of > * > * TYPE, BIND(C) :: my_data > * REAL, DIMENSION(3) :: x > * ! there may be a gap of the size of one REAL > * ! if the alignment of a DOUBLE PRECISION is > * ! two times the size of a REAL > * DOUBLE PRECISION: p > * END TYPE > * > * one should define > * > * TYPE, BIND(C) :: my_data > * REAL, DIMENSION(3) :: x > * REAL :: gap1 > * DOUBLE PRECISION: p > * END TYPE > * > * and also including gap1 in the matching MPI derived datatype. > __________ > > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf > p580:12,21,26,30,34,36: > > Add a forth structure entry: logical l after the double precision d > ________ > > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf > p581:11-29 read > > x TODO: The following text about the extent of derived types should be > x strongly checked by specialist on derived types!!! The correct variant should > x be chosen. > To use a derived type in an array requires a correct extent of the datatype handle to > take care of the alignment rules applied by the compiler. These alignment rules may imply > that there are gaps between the elements of a derived type, and also between the array > elements. > x The alignment rules in Section 4.1 on page 85 and Section 4.1.6 on page 106 > x apply only to > x VARIANT 1: SEQUENCE > x VARIANT 2: BIND(C) > x derived types. > The extent of an iteroperable derived type (i.e., defined with BIND(C)) and > a SEQUENCE derived type with the same content may be different because C and Fortran > may apply different alignment rules. > x VARIANT 1: > x Using the SEQUENCE attribute instead of BIND(C) in the declararion on mytype, one can > x directly use newtype to send the fooarr array. > x VARIANT 2: > x In the example, one can directly use newtype to send the fooarr array. The resized > x newarrtype datatype is only needed, if mytype is a SEQUENCE derived type. > > but should read > > To use a derived type in an array requires a correct extent of the datatype handle to > take care of the alignment rules applied by the compiler. These alignment rules may imply > that there are gaps between the elements of a derived type, and also between the array > elements. > The extent of an iteroperable derived type (i.e., defined with BIND(C)) and > a SEQUENCE derived type with the same content may be different because C and Fortran > may apply different alignment rules. > As mentioned in Section 4.1 on page 86 and Section 4.1.6 on page 106, > one should resize the datatype before using it for an array of structures fooarr, > if elements may have different alignments that may cause an alignment-gap > at the end of the structure. > _________________________ > > Best regards > Rolf > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rolf Rabenseifner 06 Oct '11

06 Oct '11

Dear all, here is my proposal to solve the alignment problems. Please check carefully. It is touching only the Fortran side. C problems with alignments are not really touched, but should be small. ---------------------------------------------------------------------- (Lines with * are new, lines with : are modified, lines with x are removed) mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf p86:45-47 (=MPI-2.2 p78:45-47) read If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon is the least non-negative increment needed to round extent(Typemap) to the next multiple of max_i k_i. The complete definition of extent is given on page 107. but should read If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon is the least non-negative increment needed to round extent(Typemap) to the next multiple of max_i k_i. * In Fortran, whether the alignments k_i are computed according to the alignments * used by the compiler in common blocks, SEQUENCE derived types, or BIND(C) derived types, * is implementation-dependent. : The complete definition of extent is given in Section 4.1.6 on page 106. ________ mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf p87:7-11 (=MPI-2.2 p79:7-11) read Rationale. The definition of extent is motivated by the assumption that the amount of padding added at the end of each structure in an array of structures is the least needed to fulfill alignment constraints. More explicit control of the extent is provided in Section 4.1.6. Such explicit control is needed in cases where the assumption does not hold, for example, where union types are used. (End of rationale.) but should read Rationale. The definition of extent is motivated by the assumption that the amount of padding added at the end of each structure in an array of structures is the least needed to fulfill alignment constraints. More explicit control of the extent is provided in Section 4.1.6. Such explicit control is needed in cases where the assumption does not hold, for example, where union types are used. * In Fortran, structures can be expressed with several language features, e.g., * common blocks, SEQUENCE derived types, or BIND(C) derived types. The compiler may * use different alignments, and therefore, it is recommended to use MPI_TYPE_CREATE_RESIZED * for arrays of structures if an alignment may cause an alignment-gap at the end of * a structure as described in Section 4.1.6 on page 106 and in Section 16.2.15 on page 580. (End of rationale.) ________ mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf p107:13-17 (=MPI-2.2 p96:42-45) read If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon is the least non-negative increment needed to round extent(Typemap) to the next multiple of max_i k_i. The formal definitions given for the various datatype constructors apply now, with the amended definition of extent. but should read If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon is the least non-negative increment needed to round extent(Typemap) to the next multiple of max_i k_i. * In Fortran, whether the alignments k_i are computed according to the alignments * used by the compiler in common blocks, SEQUENCE derived types, or BIND(C) derived types, * is implementation-dependent. The formal definitions given for the various datatype constructors apply now, with the amended definition of extent. * Rationale. * Before Fortran 2003, MPI_TYPE_CREATE_STRUCT could be applied to Fortran common blocks and * SEQUENCE derived types. With Fortran 2003, this list was extended by BIND(C) derived types * and MPI implementors have implemented the alignments k_i different, i.e., some based on * the alignments used in SEQUENCE derived types, and others according to BIND(C) derived types. * End of rationale. * Advice to implementors. * In Fortran, it is generally recommended to use BIND(C) derived types instead of * common blocks or SEQUENCE derived types. Therefore it is recommended to calculate the * alignments k_i based on BIND(C) derived types. * End of advoice to implementors. * Advice to users. * Structures combining different basic datatypes should be defined so that there will * be no gaps based on alignment rules, and if used as an array of structures, then * also without such an alignment-gap at the end of the structure. * In MPI communication, the content of such gaps would not be communicated. * If alignment-gaps may occur, e.g., between an odd number of floats ore REAL before * a double or DOUBLE PRECISION data, then such gaps should be added explicitely in * the structure and also to the MPI derived datatype handle, because the communication * of a contiguous derived datatype may be significantly faster than the communication * of one that is non-contiguous due such alignment-gaps. * * Example: * Instead of * * TYPE, BIND(C) :: my_data * REAL, DIMENSION(3) :: x * ! there may be a gap of the size of one REAL * ! if the alignment of a DOUBLE PRECISION is * ! two times the size of a REAL * DOUBLE PRECISION: p * END TYPE * * one should define * * TYPE, BIND(C) :: my_data * REAL, DIMENSION(3) :: x * REAL :: gap1 * DOUBLE PRECISION: p * END TYPE * * and also including gap1 in the matching MPI derived datatype. __________ mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf p580:12,21,26,30,34,36: Add a forth structure entry: logical l after the double precision d ________ mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf p581:11-29 read x TODO: The following text about the extent of derived types should be x strongly checked by specialist on derived types!!! The correct variant should x be chosen. To use a derived type in an array requires a correct extent of the datatype handle to take care of the alignment rules applied by the compiler. These alignment rules may imply that there are gaps between the elements of a derived type, and also between the array elements. x The alignment rules in Section 4.1 on page 85 and Section 4.1.6 on page 106 x apply only to x VARIANT 1: SEQUENCE x VARIANT 2: BIND(C) x derived types. The extent of an iteroperable derived type (i.e., defined with BIND(C)) and a SEQUENCE derived type with the same content may be different because C and Fortran may apply different alignment rules. x VARIANT 1: x Using the SEQUENCE attribute instead of BIND(C) in the declararion on mytype, one can x directly use newtype to send the fooarr array. x VARIANT 2: x In the example, one can directly use newtype to send the fooarr array. The resized x newarrtype datatype is only needed, if mytype is a SEQUENCE derived type. but should read To use a derived type in an array requires a correct extent of the datatype handle to take care of the alignment rules applied by the compiler. These alignment rules may imply that there are gaps between the elements of a derived type, and also between the array elements. The extent of an iteroperable derived type (i.e., defined with BIND(C)) and a SEQUENCE derived type with the same content may be different because C and Fortran may apply different alignment rules. As mentioned in Section 4.1 on page 86 and Section 4.1.6 on page 106, one should resize the datatype before using it for an array of structures fooarr, if elements may have different alignments that may cause an alignment-gap at the end of the structure. _________________________ Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] [Mpi3-tools] Questions on the F2008 profiling interface issues
by Rolf Rabenseifner 03 Oct '11

03 Oct '11

Dear all, This is an updated version of the text changes based on the results of the discussion in the tel-con today. Please check again. And do we need to have all these 13 groups? I expect that all MPI implementors implement the new 3.0 routines in the same way as 2.2, therefore I would like to combine these groups, i.e. to remove the groups - MPI_NEIGHBOR_ALLTOALL - MPI_IBCAST - MPI_IBARRIER ----- Original Message ----- > From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > To: "MPI3 Tools" <mpi3-tools(a)lists.mpi-forum.org>, "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> > Sent: Monday, October 3, 2011 12:18:09 PM > Subject: Re: [MPI3 Fortran] [Mpi3-tools] Questions on the F2008 profiling interface issues according to pages:lines in https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/229/mpi-repo… and the decision of the Santorini 2011 meeting, we should add the following text to accommodate the tools implementors. P559:20-21 reads equals .FALSE.. If but should read equals .FALSE., and After p559 the following text should be added: ---------------------------- To simplify the development of profiling libraries, the MPI routines are grouped together and it is required that if the peer routine of a group is available within an MPI library with one of its possible linker names then all of the routines in this group must be also provided according to the same linker name scheme, and if the peer routine is not available through a linker name scheme then all other routines have also not to be available through this scheme. Peer routines and their groups: - MPI_ALLOC_MEM MPI_ALLOC_MEM and MPI_WIN_ALLOCATE. - MPI_FREE_MEM Only this routine is in this group. - MPI_GET_ADDRESS MPI_GET_ADDRESS and MPI_ADDRESS. - MPI_SEND All routines with choice buffer arguments that are not declared as ASYNCHRONOUS within the mpi_f08 module and exist already in MPI-2.2. - MPI_NEIGHBOR_ALLTOALL All routines with choice buffer arguments that are not declared as ASYNCHRONOUS within the mpi_f08 module and are new in MPI-3.0. - MPI_ISEND All routines with choice buffer arguments that are declared as ASYNCHRONOUS within the mpi_f08 module and exist already in MPI-2.2. - MPI_IBCAST All routines with choice buffer arguments that are declared as ASYNCHRONOUS within the mpi_f08 module and are new in MPI-3.0. - MPI_OP_CREATE Only this routine is in this group. - MPI_REGISTER_DATAREP Only this routine is in this group. - MPI_COMM_KEYVAL_CREATE All other routines with callback function arguments. - MPI_COMM_DUP_FN All predefined callback routines. - MPI_COMM_RANK All other MPI routines that exist already in MPI-2.2. - MPI_IBARRIER All other MPI routines that are new in MPI-3.0. Additionally, four C preprocessor macros are available in mpi.h for each routine group. The name of the macros are the peer routine name written as in the list above and appended with one of the following suffixes and meanings: - _mpi_f08_linkernames_BIND_C The macro is set to 1 if the BIND(C) linker name with the linker suffix _f08 is available for all routines within this group (e.g., MPI_Send_f08), otherwise it is set to 0. - _mpi_f08_linkernames_Fortran The macro is set to 1 if the Fortran linker name with the linker suffix _f08 is available for all routines within this group (e.g., mpi_send_f08__), otherwise it is set to 0. - _mpi_linkernames_BIND_C The macro is set to 1 if the BIND(C) linker name with the linker suffix _f is available for all routines within this group (e.g., MPI_Send_f), otherwise it is set to 0. - _mpi_linkernames_Fortran The macro is set to 1 if the Fortran linker name without a linker suffix is available for all routines within this group (e.g., mpi_send__), otherwise it is set to 0. For example ... #define MPI_SEND_mpi_f08_linkernames_BIND_C 0 #define MPI_SEND_mpi_f08_linkernames_Fortran 1 #define MPI_SEND_mpi_linkernames_BIND_C 0 #define MPI_SEND_mpi_linkernames_Fortran 1 #define MPI_ISEND_mpi_f08_linkernames_BIND_C 1 #define MPI_ISEND_mpi_f08_linkernames_Fortran 1 #define MPI_ISEND_mpi_linkernames_BIND_C 1 #define MPI_ISEND_mpi_linkernames_Fortran 1 ... #define MPI_COMM_DUP_FN_mpi_f08_linkernames_BIND_C 1 #define MPI_COMM_DUP_FN_mpi_f08_linkernames_Fortran 0 #define MPI_COMM_DUP_FN_mpi_linkernames_BIND_C 0 #define MPI_COMM_DUP_FN_mpi_linkernames_Fortran 1 ... shows, that MPI_SEND, MPI_RECV, and all other routines in this group are only available through their Fortran linker names (e.g., mpi_send_f08__, mpi_send__, mpi_recv_f08__, mpi_recv__, ...), while MPI_ISEND, MPI_IRECV, ... are available with all four interfaces to support the current modules providing TR quality, but to support also application routines that are compiled with and older MPI library version with .._BIND_C set to 0 and only _Fortran set to 1. For the predefined callbacks, there is no choice, because the interfaces must fit to the callback function prototypes which are BIND(C) based for mpi_f08 and without BIND(C) for the mpi module and mpif.h. Advice to implementors. If all the following conditions are fulfilled (which is the case for most compilers) - the handles in the mpi_f08 module occupy one Fortran numerical storage unit (same as an INTEGER handle), and - the internal argument passing used to pass an actual ierror argument to a non optional ierror dummy argument is binary compatible to passing an actual ierror argument to an ierror dummy argument that is declared as OPTIONAL, and - the internal argument passing for ASYNCHRONOUS and non-ASYNCHRONOUS arguments is the same, and - the internal routine call mechanism is the same for the Fortran and the C compiler, and - the compiler does not provide TR 29113, then for most groups, the implementor may use the same internal routine implementations for all Fortran support methods with only different linker names. For TR 29113 quality, new routines are needed only for the routine groups of MPI_ISEND and MPI_IBCAST. End of advice to implementors. ---------------------------- Not directly relevant for the tools people, but giving the reason for differentiating several Choice-buffer routine groups, the following changes are also done: P552:14-18 read - Set the INTEGER compile-time constant MPI_SUBARRAYS_SUPPORTED to .TRUE. and declare choice buffers using the Fortran 2008 TR 29113 feature assumed-type and assumed-rank, i.e., TYPE(*), DIMENSION(..), if the underlying Fortran compiler supports it. With this, non-contiguous sub-arrays can be used as buffers in nonblocking routines. but should read - Set the INTEGER compile-time constant MPI_SUBARRAYS_SUPPORTED to .TRUE. and declare choice buffers using the Fortran 2008 TR 29113 feature assumed-type and assumed-rank, i.e., TYPE(*), DIMENSION(..) in all nonblocking, split collective and persistent communication routines, if the underlying Fortran compiler supports it. With this, non-contiguous sub-arrays can be used as buffers in nonblocking routines. Rationale. In all blocking routines, i.e., if the choice-buffer is not declared as ASYNCHRONOUS, the TR 29113 feature is not needed for the support of non-contiguous buffers because the compiler can pass the buffer by in-and-out-copy through a contiguous scratch array. End of rationale. P555:7-10 read - Set the INTEGER compile-time constant MPI_SUBARRAYS_SUPPORTED to .TRUE. if all choice buffer arguments are declared with TYPE(*), DIMENSION(..), otherwise set it to .FALSE.. With MPI_SUBARRAYS_SUPPORTED==.TRUE., non-contiguous subarrays can be used as buers in nonblocking routines. but should read - Set the INTEGER compile-time constant MPI_SUBARRAYS_SUPPORTED to .TRUE. if all choice buffer arguments in all nonblocking, split collective and persistent communication routines are declared with TYPE(*), DIMENSION(..), otherwise set it to .FALSE.. With MPI_SUBARRAYS_SUPPORTED==.TRUE., non-contiguous subarrays can be used as buers in nonblocking routines. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Tools meeting Monday Oct 3
by Rolf Rabenseifner 03 Oct '11

03 Oct '11

And the same for each peer rotine of each routine group > > # Symbols for mpif.h and use mpi > # define MPI_SEND_with_mpi_OLD > # > MPI_SEND > mpi_send > mpi_send_ > mpi_send__ > > > #define MPI_SEND_with_mpi_TR > # > MPI_Send_f # has choice buffer as array descriptor > > > # Symbols for use mpi_f08 (not using the new interop TR) > # define MPI_SEND_with_mpi_f08_OLD > # > MPI_SEND_f08 > mpi_send_f08 > mpi_send_f08_ > mpi_send_f08__ > > > # define MPI_SEND_with_mpi_f08_TR > # > MPI_Send_f08 # has choice buffer as array descriptor -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Tools meeting Monday Oct 3
by Rasmussen, Craig E 03 Oct '11

03 Oct '11

On Oct 3, 2011, at 9:26 AM, Rasmussen, Craig E wrote: We have potential problem going to C for character strings and optional arguments. # Symbols for mpif.h and use mpi # define MPI_SEND_NAMEMANGLING_TR 0 # MPI_SEND mpi_send mpi_send_ mpi_send__ #define MPI_SEND_NAMEMANGLINE_TR 1 MPI_Send_f # has choice buffer as array descriptor # Symbols for use mpi_f08 (not using the new interop TR) # define MPI_SEND_NAMEMANGLINE_TR 0 # MPI_SEND_f08 mpi_send_f08 mpi_send_f08_ mpi_send_f08__ # define MPI_SEND_NAMEMANGLING_TR 1 # MPI_Send_f08 # has choice buffer as array descriptor

1 0

Re: [MPI3 Fortran] [Mpi3-tools] Questions on the F2008 profiling interface issues
by Rasmussen, Craig E 03 Oct '11

03 Oct '11

I'm here and hopefully connected. Jeff suggested I send out a list of symbols the profiling interface needs to support. I'll do that in a few minutes. -craig On Oct 3, 2011, at 8:49 AM, Jeff Squyres wrote: > Craig already agreed to be here today (in about 10 mins). > > > > On Oct 2, 2011, at 5:21 PM, Martin Schulz wrote: > >> Hi Rolf, >> >> Sorry, this must have gone lost in the long email trail. All the meeting information is on the tools Wiki at: >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools >> >> I know the holiday is a problem, but I couldn't see an option on how to avoid this since moving it to next week would have been too late to provide feedback in time. It would be great if you could make it anyway. Otherwise, any feedback by email would be great as well, in particular on the grouping of _f08 functions we talked about during the last forum. >> >> Thanks, >> >> Martin >> >> >> On Oct 2, 2011, at 2:15 PM, Rolf Rabenseifner wrote: >> >>> >>>>> I think it >>>>> would be a great benefit to have people of the Fortran WG to join on >>>>> the call on Monday. I hope this is possible. >>>> >>>> Yes, that would be great - all are cc-ed. >>> >>> What must I do? >>> Can you provide date, time, number? >>> Oct. 3 is a holiday in Germany, therefore, I'm nut sure >>> that I can join. >>> >>> Rolf >>> >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) >> >> ________________________________________________________________________ >> Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm >> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >> >> >> > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ >

1 0

Re: [MPI3 Fortran] [Mpi3-tools] Questions on the F2008 profiling interface issues
by Jeff Squyres 03 Oct '11

03 Oct '11

Craig already agreed to be here today (in about 10 mins). On Oct 2, 2011, at 5:21 PM, Martin Schulz wrote: > Hi Rolf, > > Sorry, this must have gone lost in the long email trail. All the meeting information is on the tools Wiki at: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools > > I know the holiday is a problem, but I couldn't see an option on how to avoid this since moving it to next week would have been too late to provide feedback in time. It would be great if you could make it anyway. Otherwise, any feedback by email would be great as well, in particular on the grouping of _f08 functions we talked about during the last forum. > > Thanks, > > Martin > > > On Oct 2, 2011, at 2:15 PM, Rolf Rabenseifner wrote: > >> >>>> I think it >>>> would be a great benefit to have people of the Fortran WG to join on >>>> the call on Monday. I hope this is possible. >>> >>> Yes, that would be great - all are cc-ed. >> >> What must I do? >> Can you provide date, time, number? >> Oct. 3 is a holiday in Germany, therefore, I'm nut sure >> that I can join. >> >> Rolf >> >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [MPI3 Fortran] [Mpi3-tools] Questions on the F2008 profiling interface issues
by Rolf Rabenseifner 03 Oct '11

03 Oct '11

Dear all, according to pages:lines in https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/229/mpi-repo… and the decision of the Santorini 2011 meeting, we should add the following text to accommodate the tools implementors. P559:20-21 reads equals .FALSE.. If but should read equals .FALSE., and After p559 the following text should be added: ---------------------------- To simplify the development of profiling libraries, the MPI routines are grouped together and it is required that if the peer routine of a group is available within an MPI library with one of its possible linker names then all of the routines in this group must be also provided according to the same linker name scheme, and if the peer routine is not available through a linker name scheme then all other routines have also not to be available through this scheme. Peer routines and their groups: - MPI_ALLOC_MEM MPI_ALLOC_MEM and MPI_WIN_ALLOCATE. - MPI_FREE_MEM Only this routine is in this group. - MPI_GET_ADDRESS MPI_GET_ADDRESS and MPI_ADDRESS. - MPI_SEND All routines with choice buffer arguments that are not declared as ASYNCHRONOUS within the mpi_f08 module and exist already in MPI-2.2. - MPI_NEIGHBOR_ALLTOALL All routines with choice buffer arguments that are not declared as ASYNCHRONOUS within the mpi_f08 module and are new in MPI-3.0. - MPI_ISEND All routines with choice buffer arguments that are declared as ASYNCHRONOUS within the mpi_f08 module and exist already in MPI-2.2. - MPI_IBCAST All routines with choice buffer arguments that are declared as ASYNCHRONOUS within the mpi_f08 module and are new in MPI-3.0. - MPI_OP_CREATE Only this routine is in this group. - MPI_REGISTER_DATAREP Only this routine is in this group. - MPI_COMM_KEYVAL_CREATE All other routines with callback function arguments. - MPI_COMM_DUP_FN All predefined callback routines. - MPI_COMM_RANK All other MPI routines that exist already in MPI-2.2. - MPI_IBARRIER All other MPI routines that are new in MPI-3.0. Advice to implementors. If all the following conditions are fulfilled (which is the case for most compilers) - the handles in the mpi_f08 module occupy one Fortran numerical storage unit (same as an INTEGER handle), and - the internal argument passing used to pass an actual ierror argument to a non optional ierror dummy argument is binary compatible to passing an actual ierror argument to an ierror dummy argument that is declared as OPTIONAL, and - the internal argument passing for ASYNCHRONOUS and non-ASYNCHRONOUS arguments is the same, and - the internal routine call mechanism is the same for the Fortran and the C compiler, and - the compiler does not provide TR 29113, then for most groups, the implementor may use the same internal routine implementations for all Fortran support methods with only different linker names. For TR 29113 quality, new routines are needed only for the routine groups of MPI_ISEND and MPI_IBCAST. End of advice to implementors. ---------------------------- Not directly relevant for the tools people, but giving the reason for differentiating several Choice-buffer routine groups, the following changes are also done: P552:14-18 read - Set the INTEGER compile-time constant MPI_SUBARRAYS_SUPPORTED to .TRUE. and declare choice buffers using the Fortran 2008 TR 29113 feature assumed-type and assumed-rank, i.e., TYPE(*), DIMENSION(..), if the underlying Fortran compiler supports it. With this, non-contiguous sub-arrays can be used as buffers in nonblocking routines. but should read - Set the INTEGER compile-time constant MPI_SUBARRAYS_SUPPORTED to .TRUE. and declare choice buffers using the Fortran 2008 TR 29113 feature assumed-type and assumed-rank, i.e., TYPE(*), DIMENSION(..) in all nonblocking, split collective and persistent communication routines, if the underlying Fortran compiler supports it. With this, non-contiguous sub-arrays can be used as buffers in nonblocking routines. Rationale. In all blocking routines, i.e., if the choice-buffer is not declared as ASYNCHRONOUS, the TR 29113 feature is not needed for the support of non-contiguous buffers because the compiler can pass the buffer by in-and-out-copy through a contiguous scratch array. End of rationale. P555:7-10 read - Set the INTEGER compile-time constant MPI_SUBARRAYS_SUPPORTED to .TRUE. if all choice buffer arguments are declared with TYPE(*), DIMENSION(..), otherwise set it to .FALSE.. With MPI_SUBARRAYS_SUPPORTED==.TRUE., non-contiguous subarrays can be used as buers in nonblocking routines. but should read - Set the INTEGER compile-time constant MPI_SUBARRAYS_SUPPORTED to .TRUE. if all choice buffer arguments in all nonblocking, split collective and persistent communication routines are declared with TYPE(*), DIMENSION(..), otherwise set it to .FALSE.. With MPI_SUBARRAYS_SUPPORTED==.TRUE., non-contiguous subarrays can be used as buers in nonblocking routines. I'll try to be at the tel-con today. Best regards Rolf ----- Original Message ----- > From: "Martin Schulz" <schulzm(a)llnl.gov> > To: "Marc-Andre Hermanns" <m.a.hermanns(a)grs-sim.de> > Cc: "MPI3 Tools" <mpi3-tools(a)lists.mpi-forum.org>, "Martin Schulz" <schulz6(a)llnl.gov>, "Craig E Rasmussen" > <rasmussn(a)lanl.gov>, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Andreas > Knüpfer" <andreas.knuepfer(a)tu-dresden.de>, "Todd Gamblin" <tgamblin(a)llnl.gov>, "Tobias Hilbrich" > <tobias.hilbrich(a)tu-dresden.de>, "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> > Sent: Sunday, October 2, 2011 6:13:46 AM > Subject: Re: [Mpi3-tools] Questions on the F2008 profiling interface issues > Hi Marc-Andre, > > On Sep 30, 2011, at 6:27 AM, Marc-Andre Hermanns wrote: > > > Martin, > > > >> I was in Dresden the last few days (visiting the Vampir group, in > >> particular Andreas Knuepfer and Tobias Hilbrich who are responsible > >> for > >> their MPI wrappers) and we sat down to talk about the issues around > >> the > >> profiling interface in ticket #229. > >> > >> Here is a quick summary with a few questions that came up: > >> > >> * With the new Fortran symbol naming variants, there are 6 or 8 or > >> 10 > >> versions for every MPI subroutine. Tools could intercept all > >> versions by > >> providing own symbols for all versions. > > > > If it is clear which name the symbols have, additional symbol names > > should be easy to generate, so at this stage I think it doesn't > > matter > > much, whether we have 6, 8, or 10 different schemes to cover. Right? > > I agree - as long as we can generate a full list of names, we can > intercept them all. Adding wrappers shouldn't be a problem. > > > > >> Now the problem: Within the wrapper function a tool needs to call > >> the > >> correct PMPI call which usually is the same prefix (either '_f08', > >> '_f', '__' etc.) as the MPI call it is in. A mapping to matching C > >> functions (as most tools do it now) is not possible due to problems > >> with calling conventions for callback routines. > > > > As far as I know our wrappers call the C routines, because we jump > > through all sorts of hoops to get a proper Fortran to C conversion > > and > > back. So in this regard, we need to fix this. Interestingly this has > > not > > popped up yet. > > I agree - this is remarkable. > > > > > If we write Fortran wrappers in Fortran, how would a Fortran wrapper > > compiled with Fortran 77 have to call the C measurement code inside > > the > > wrapper? If the Fortran compiler implicitly adds underscores to the > > calls, do I have to provide a special Fortran interface to my > > measurement system then? > > That is a question for the Fortran experts. > > > > >> * Would the following solution work? All wrappers for Fortran are > >> written > >> in Fortran instead of C. > >> [...] > >> For every MPI function, there are 6 or 8 variants and all of them > >> need to be provided by a tool. This would probably mean that we > >> have > >> to compile wrappers with all three calling conventions (mpif.h, use > >> mpi_f and use mpi_f08) and then link them together? > > > > Don't we need to compile with all three calling conventions on all > > possible compilers on a given system? > > Also, here, I think the Fortran experts need to weigh in. > > > > > We currently provide all 4 name-mangling schemes with every install > > of > > Scalasca, because we had some issues with applications using two > > different Fortran compilers for different sub-modules. (This came up > > in > > a project some years back, and ever since then we provide all > > schemes in > > the same library and it seems to work.) > > > >> * As discussed at the forum, if we want to keep our tools in C > >> (which is > >> probably still the easier and better option), we need to know the > >> symbols > >> on the "p" side and we need a portable way to figure this out. We > >> talked > >> about the solution of two groups of routines - one for which we > >> have _f08 > >> symbols for one for which we don't. Rolf, how will you add this to > >> the > >> standard, in particular how will you form those groups. > > > > I started to get some doubts about this approach. One of the issues > > was > > that some choice-buffer routines had names longer than the proposed > > limit. So they need to be mangled differently, especially, their > > symbol > > needs to be shortened. And as I understood it, the current idea is > > to > > leave it to the MPI implementation to define the shortened name. > > I thought that was just for the argument checking in mpif.h and that > we decided to not support this. Are there more locations were the > names are too long? > > > > > It is one thing, finding out which suffix is used. Finding how > > function > > names are mangled without any form of restricting to possible > > schemes > > seems to be another ball game? > > > > Am I missing something? > > Yes, if we do have to shorten names, I agree - I think, though, we got > around the issue, or did we not? > > > > > I think, as it was pointed out in Santorini with the Fortran > > Bindings in > > general, we (or at least I) don't have enough Fortran expertise to > > think > > this through all to the end and consider all implications. I think > > it > > would be a great benefit to have people of the Fortran WG to join on > > the > > call on Monday. I hope this is possible. > > Yes, that would be great - all are cc-ed. > > Martin > > > > > > Cheers, > > Marc-Andre > > -- > > Marc-Andre Hermanns > > German Research School for > > Simulation Sciences GmbH > > c/o Laboratory for Parallel Programming > > 52056 Aachen | Germany > > > > Tel +49 241 80 99753 > > Fax +49 241 80 6 99753 > > Web www.grs-sim.de > > > > Members: Forschungszentrum Jülich GmbH | RWTH Aachen University > > Registered in the commercial register of the local court of > > Düren (Amtsgericht Düren) under registration number HRB 5268 > > Registered office: Jülich > > Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes > > > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm > CASC @ Lawrence Livermore National Laboratory, Livermore, USA -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] [Mpi3-tools] Questions on the F2008 profiling interface issues
by Martin Schulz 02 Oct '11

02 Oct '11

Hi Rolf, Sorry, this must have gone lost in the long email trail. All the meeting information is on the tools Wiki at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools I know the holiday is a problem, but I couldn't see an option on how to avoid this since moving it to next week would have been too late to provide feedback in time. It would be great if you could make it anyway. Otherwise, any feedback by email would be great as well, in particular on the grouping of _f08 functions we talked about during the last forum. Thanks, Martin On Oct 2, 2011, at 2:15 PM, Rolf Rabenseifner wrote: > >>> I think it >>> would be a great benefit to have people of the Fortran WG to join on >>> the call on Monday. I hope this is possible. >> >> Yes, that would be great - all are cc-ed. > > What must I do? > Can you provide date, time, number? > Oct. 3 is a holiday in Germany, therefore, I'm nut sure > that I can join. > > Rolf > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [MPI3 Fortran] [Mpi3-tools] Questions on the F2008 profiling interface issues
by Martin Schulz 01 Oct '11

01 Oct '11

Hi Marc-Andre, On Sep 30, 2011, at 6:27 AM, Marc-Andre Hermanns wrote: > Martin, > >> I was in Dresden the last few days (visiting the Vampir group, in >> particular Andreas Knuepfer and Tobias Hilbrich who are responsible for >> their MPI wrappers) and we sat down to talk about the issues around the >> profiling interface in ticket #229. >> >> Here is a quick summary with a few questions that came up: >> >> * With the new Fortran symbol naming variants, there are 6 or 8 or 10 >> versions for every MPI subroutine. Tools could intercept all versions by >> providing own symbols for all versions. > > If it is clear which name the symbols have, additional symbol names > should be easy to generate, so at this stage I think it doesn't matter > much, whether we have 6, 8, or 10 different schemes to cover. Right? I agree - as long as we can generate a full list of names, we can intercept them all. Adding wrappers shouldn't be a problem. > >> Now the problem: Within the wrapper function a tool needs to call the >> correct PMPI call which usually is the same prefix (either '_f08', >> '_f', '__' etc.) as the MPI call it is in. A mapping to matching C >> functions (as most tools do it now) is not possible due to problems >> with calling conventions for callback routines. > > As far as I know our wrappers call the C routines, because we jump > through all sorts of hoops to get a proper Fortran to C conversion and > back. So in this regard, we need to fix this. Interestingly this has not > popped up yet. I agree - this is remarkable. > > If we write Fortran wrappers in Fortran, how would a Fortran wrapper > compiled with Fortran 77 have to call the C measurement code inside the > wrapper? If the Fortran compiler implicitly adds underscores to the > calls, do I have to provide a special Fortran interface to my > measurement system then? That is a question for the Fortran experts. > >> * Would the following solution work? All wrappers for Fortran are written >> in Fortran instead of C. >> [...] >> For every MPI function, there are 6 or 8 variants and all of them >> need to be provided by a tool. This would probably mean that we have >> to compile wrappers with all three calling conventions (mpif.h, use >> mpi_f and use mpi_f08) and then link them together? > > Don't we need to compile with all three calling conventions on all > possible compilers on a given system? Also, here, I think the Fortran experts need to weigh in. > > We currently provide all 4 name-mangling schemes with every install of > Scalasca, because we had some issues with applications using two > different Fortran compilers for different sub-modules. (This came up in > a project some years back, and ever since then we provide all schemes in > the same library and it seems to work.) > >> * As discussed at the forum, if we want to keep our tools in C (which is >> probably still the easier and better option), we need to know the symbols >> on the "p" side and we need a portable way to figure this out. We talked >> about the solution of two groups of routines - one for which we have _f08 >> symbols for one for which we don't. Rolf, how will you add this to the >> standard, in particular how will you form those groups. > > I started to get some doubts about this approach. One of the issues was > that some choice-buffer routines had names longer than the proposed > limit. So they need to be mangled differently, especially, their symbol > needs to be shortened. And as I understood it, the current idea is to > leave it to the MPI implementation to define the shortened name. I thought that was just for the argument checking in mpif.h and that we decided to not support this. Are there more locations were the names are too long? > > It is one thing, finding out which suffix is used. Finding how function > names are mangled without any form of restricting to possible schemes > seems to be another ball game? > > Am I missing something? Yes, if we do have to shorten names, I agree - I think, though, we got around the issue, or did we not? > > I think, as it was pointed out in Santorini with the Fortran Bindings in > general, we (or at least I) don't have enough Fortran expertise to think > this through all to the end and consider all implications. I think it > would be a great benefit to have people of the Fortran WG to join on the > call on Monday. I hope this is possible. Yes, that would be great - all are cc-ed. Martin > > Cheers, > Marc-Andre > -- > Marc-Andre Hermanns > German Research School for > Simulation Sciences GmbH > c/o Laboratory for Parallel Programming > 52056 Aachen | Germany > > Tel +49 241 80 99753 > Fax +49 241 80 6 99753 > Web www.grs-sim.de > > Members: Forschungszentrum Jï¿½lich GmbH | RWTH Aachen University > Registered in the commercial register of the local court of > Dï¿½ren (Amtsgericht Dï¿½ren) under registration number HRB 5268 > Registered office: Jï¿½lich > Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes > ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0