September 2011 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Supalov, Alexander 27 Sep '11

27 Sep '11

PS. Please redistribute the test program that was contained in Rolf's original email. -----Original Message----- From: Supalov, Alexander Sent: Wednesday, September 28, 2011 12:21 AM To: MPI-3 Fortran working group Subject: RE: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION Importance: High Hi, I'm surprised to see no Intel MPI on the list below. Both Intel MPI and MS MPI are based on MPICH2. Please add me (for Intel MPI) to this kind of announcement from now on. We also make Intel compiler, so it makes a lot of sense to ask at the source rather than elsewhere. Best regards. Alexander -----Original Message----- From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Rolf Rabenseifner Sent: Tuesday, September 27, 2011 5:22 PM To: MPI-3 Fortran working group Subject: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION For your info: I informed the providers of independent MPI libraries with Fortran bindings about a serious problem with the definition of Fortran datatypes. My apologies if I missed some of the independent MPI libraries. In this case, please also run the program and please send the result to me. Best regards Rolf ----- Forwarded Message ----- From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> To: "Richard Treumann" <treumann(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji Sumimoto" <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Fab Tillier" <ftillier(a)microsoft.com> Sent: Tuesday, September 27, 2011 5:15:15 PM Subject: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION Dear all, as far as I know, you represent some MPI libraries that are not directly based on another one in the list: - mpich2: Rajeev - OpenMPI: Jeff - IBM: Rich Treumann - NEC: Hubert - Fujitsu: Shinji Sumimoto - Microsoft: Fab (Which independent library is missing on this list?) (Who of the list uses already mich2 or OpenMPI as bases or does not provide a Fortran binding, and can be therefore removed from this list?) The problem is simple and has at the end only three numbers: Aligments of: - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. with Intel) - DOUBLE PRECISION within a BIND(C) derived type = 8 - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 The details: Which is the alignment of Fortran DOUBLE PRECISION according to k_i on MPI-2.2, page 78 line 45 and page 96 line 42. Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 (Fortran 90) in Fortran SEQUENCE derived types (which have the same memory layout according to the SEQUENCE-rules), the MPI alignment and the Fortran alignment in such constructs should be the same, e.g., the Intel compiler has alignment 4 !!! This may be significantly different to C, where Intel uses alignment 8! With Fortran 2003, we got the Fortran BIND(C) derived types. Here, a Fortran DOUBLE PRECISION has with Intel's compiler the aligment 8. As far as I understand, mpich2 and Cray produce following results with an Intel compiler: Aligments of: - DOUBLE PRECISION within a SEQUENCE derived type = 4 - DOUBLE PRECISION within a BIND(C) derived type = 8 - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 This is definitely wrong compared to MPI-1.1 and MPI-2.0 but may be helpful for MPI-3.0 when we like to switch to a definition that is based on BIND(C). Question: What is the output of the attached test program when running with - your MPI library - and your compiler - and with Intel's compiler Please can you send me the output of such mpiruns? Thanks in advance and best regards Rolf PS: We need not to care about 8/8/8. Only 4/8/x causes problems: If all those have x=8 then I'll propose to adopt the MPI-3.0 to the reality. If we have some implementations with 4/8/8 and others with 4/8/4 then we have a serious problem because only one answer can be correct, and the question would be whether the historical answer (x=4) or the modern answer (x=8) should be chosen. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Supalov, Alexander 27 Sep '11

27 Sep '11

Hi, I'm surprised to see no Intel MPI on the list below. Both Intel MPI and MS MPI are based on MPICH2. Please add me (for Intel MPI) to this kind of announcement from now on. We also make Intel compiler, so it makes a lot of sense to ask at the source rather than elsewhere. Best regards. Alexander -----Original Message----- From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Rolf Rabenseifner Sent: Tuesday, September 27, 2011 5:22 PM To: MPI-3 Fortran working group Subject: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION For your info: I informed the providers of independent MPI libraries with Fortran bindings about a serious problem with the definition of Fortran datatypes. My apologies if I missed some of the independent MPI libraries. In this case, please also run the program and please send the result to me. Best regards Rolf ----- Forwarded Message ----- From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> To: "Richard Treumann" <treumann(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji Sumimoto" <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Fab Tillier" <ftillier(a)microsoft.com> Sent: Tuesday, September 27, 2011 5:15:15 PM Subject: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION Dear all, as far as I know, you represent some MPI libraries that are not directly based on another one in the list: - mpich2: Rajeev - OpenMPI: Jeff - IBM: Rich Treumann - NEC: Hubert - Fujitsu: Shinji Sumimoto - Microsoft: Fab (Which independent library is missing on this list?) (Who of the list uses already mich2 or OpenMPI as bases or does not provide a Fortran binding, and can be therefore removed from this list?) The problem is simple and has at the end only three numbers: Aligments of: - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. with Intel) - DOUBLE PRECISION within a BIND(C) derived type = 8 - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 The details: Which is the alignment of Fortran DOUBLE PRECISION according to k_i on MPI-2.2, page 78 line 45 and page 96 line 42. Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 (Fortran 90) in Fortran SEQUENCE derived types (which have the same memory layout according to the SEQUENCE-rules), the MPI alignment and the Fortran alignment in such constructs should be the same, e.g., the Intel compiler has alignment 4 !!! This may be significantly different to C, where Intel uses alignment 8! With Fortran 2003, we got the Fortran BIND(C) derived types. Here, a Fortran DOUBLE PRECISION has with Intel's compiler the aligment 8. As far as I understand, mpich2 and Cray produce following results with an Intel compiler: Aligments of: - DOUBLE PRECISION within a SEQUENCE derived type = 4 - DOUBLE PRECISION within a BIND(C) derived type = 8 - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 This is definitely wrong compared to MPI-1.1 and MPI-2.0 but may be helpful for MPI-3.0 when we like to switch to a definition that is based on BIND(C). Question: What is the output of the attached test program when running with - your MPI library - and your compiler - and with Intel's compiler Please can you send me the output of such mpiruns? Thanks in advance and best regards Rolf PS: We need not to care about 8/8/8. Only 4/8/x causes problems: If all those have x=8 then I'll propose to adopt the MPI-3.0 to the reality. If we have some implementations with 4/8/8 and others with 4/8/4 then we have a serious problem because only one answer can be correct, and the question would be whether the historical answer (x=4) or the modern answer (x=8) should be chosen. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Bill Long 27 Sep '11

27 Sep '11

Hi Rolf, Although you list below a "CrayMPI", for organizing the results, keep in mind that it is based on mpich2. On 9/27/11 4:40 PM, Rolf Rabenseifner wrote: > Here the summary of the feedback and a proposal > to solve this inconsistency. > > > A) The 4 / 8 / 4 group = alignment of MPI_DOUBLE_PRECISION is correct for (old-style) SEQUENCE derived types > > JeffSquyres_OpenMP_linux-x86-64-ifort.txt: Intel + OpenMPI on x86-64 Results: 4 / 8 / 4 > FabTillier_Intel+MicrosoftMPI_out.txt: Intel + MS-MPI on ? Results: 4 / 8 / 4 > > B) The 4 / 8 / 8 group = alignment of MPI_DOUBLE_PRECISION is correct for (modern) BIND(C) derived types > > Rajeev_mpich_out.ifort.x86-64.txt: Intel + mpich2 on x86-64 Results: 4 / 8 / 8 > Rolf_asama+cray_output.txt: Intel + NEC-MPI? on asama Results: 4 / 8 / 8 > BillLong_Cray_5compiler_out.txt: Intel + CrayMPI on Cray XE6 Results: 4 / 8 / 8 > BrianSmith-IBM_outlog-bgp-xl.txt: Intel + IBM-XL on BGP Results: 4 / 8 / 8 > BrianSmith-IBM_outlog-xlf.txt: xlf + IBM-XL on ? Results: 4 / 8 / 8 > > C) No problems because all DOUBLE PRECISION alignments are identical = 8 > > BillLong_Cray_5compiler_out.txt: PGI + CrayMPI on Cray XE6 Results: 8 / 8 / 8 > BillLong_Cray_5compiler_out.txt: Cray + CrayMPI on Cray XE6 Results: 8 / 8 / 8 > BillLong_Cray_5compiler_out.txt: gfortran + CrayMPI on Cray XE6 Results: 8 / 8 / 8 > BillLong_Cray_5compiler_out.txt: pathscale + CrayMPI on Cray XE6 Results: 8 / 8 / 8 > BrianSmith-IBM_outlog-gcc.txt: gfortran + IBM-XL on BGQ Results: 8 / 8 / 8 > JeffSquyres_OpenMP_linux-x86-64-gfortran.txt: gfortran + OpenMPI on x86-64 Results: 8 / 8 / 8 > JeffSquyres_OpenMP_osx-x86-84-gfortran.txt: gfortran + OpenMPI on osx-x86-64 Results: 8 / 8 / 8 > Rajeev_gnu_laptop_out.txt: gfortran + mpich2 on laptop Results: 8 / 8 / 8 > Rajeev_mpich_out.gfortran.x86-64.txt: gfortran + mpich2 on x86-64 Results: 8 / 8 / 8 > > D) No problems because all DOUBLE PRECISION alignments are identical = 4 > > Rajeev_mpich_out.gfortran.ia32.txt: gfortran + mpich2 on ia32 Results: 4 / 4 / 4 > Rajeev_mpich_out.ifort.ia32.txt: Intel + mpich2 on ia32 Results: 4 / 4 / 4 > > Proposal: > --------- > > The goal of a standard is the possibility to write portable code. > Therefore the answer should be standardized and not implementation-specific, > or in other words, not defining it is the worst solution. > > I expect only a few Fortran codes are using arrays of derived types > in Fortran. My experience leads me to think that codes use arrays of derived types more often than you expect. > The oldest library is mpich. > Therefore I would propose to standardize Fortran alignments are based > on BIND(C) which should be the same as in C. > This would seem to be the efficient solution. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101

1 0

[MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rolf Rabenseifner 27 Sep '11

27 Sep '11

Here the summary of the feedback and a proposal to solve this inconsistency. ----- Forwarded Message ----- From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> To: "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji Sumimoto" <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Howard Pritchard" <howardp(a)cray.com>, "Brian Smith" <smithbr(a)us.ibm.com>, "Charles J Archer" <archerc(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, "Fab Tillier" <ftillier(a)microsoft.com>, "Bill Long" <longb(a)cray.com>, "Bill Gropp" <wgropp(a)uiuc.edu>, "Richard Graham" <rlgraham(a)ornl.gov> Cc: "N.M. Maclaren" <nmm1(a)cam.ac.uk>, "Iain Bason" <iain.bason(a)oracle.com> Sent: Tuesday, September 27, 2011 11:38:45 PM Subject: Re: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION Dear all, thank you very much for the fast answers. The result is maximal bad, i.e., about 50% of the MPIs use SEQUENCE alignments and other about 50% use BIND(C) based calculation of the alignment. Here the results (also attached as summary_out.txt). Please read carefully my proposal after this summary sheet. In all protocols: ----------------- Size of one Fortran REAL = 4 Size of one Fortran DOUBLE PRECISION = 8 Results: -------- a / b / c = Alignments of: - DOUBLE PRECISION within a SEQUENCE derived type = a - DOUBLE PRECISION within a BIND(C) derived type = b - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = c A) The 4 / 8 / 4 group = alignment of MPI_DOUBLE_PRECISION is correct for (old-style) SEQUENCE derived types JeffSquyres_OpenMP_linux-x86-64-ifort.txt: Intel + OpenMPI on x86-64 Results: 4 / 8 / 4 FabTillier_Intel+MicrosoftMPI_out.txt: Intel + MS-MPI on ? Results: 4 / 8 / 4 B) The 4 / 8 / 8 group = alignment of MPI_DOUBLE_PRECISION is correct for (modern) BIND(C) derived types Rajeev_mpich_out.ifort.x86-64.txt: Intel + mpich2 on x86-64 Results: 4 / 8 / 8 Rolf_asama+cray_output.txt: Intel + NEC-MPI? on asama Results: 4 / 8 / 8 BillLong_Cray_5compiler_out.txt: Intel + CrayMPI on Cray XE6 Results: 4 / 8 / 8 BrianSmith-IBM_outlog-bgp-xl.txt: Intel + IBM-XL on BGP Results: 4 / 8 / 8 BrianSmith-IBM_outlog-xlf.txt: xlf + IBM-XL on ? Results: 4 / 8 / 8 C) No problems because all DOUBLE PRECISION alignments are identical = 8 BillLong_Cray_5compiler_out.txt: PGI + CrayMPI on Cray XE6 Results: 8 / 8 / 8 BillLong_Cray_5compiler_out.txt: Cray + CrayMPI on Cray XE6 Results: 8 / 8 / 8 BillLong_Cray_5compiler_out.txt: gfortran + CrayMPI on Cray XE6 Results: 8 / 8 / 8 BillLong_Cray_5compiler_out.txt: pathscale + CrayMPI on Cray XE6 Results: 8 / 8 / 8 BrianSmith-IBM_outlog-gcc.txt: gfortran + IBM-XL on BGQ Results: 8 / 8 / 8 JeffSquyres_OpenMP_linux-x86-64-gfortran.txt: gfortran + OpenMPI on x86-64 Results: 8 / 8 / 8 JeffSquyres_OpenMP_osx-x86-84-gfortran.txt: gfortran + OpenMPI on osx-x86-64 Results: 8 / 8 / 8 Rajeev_gnu_laptop_out.txt: gfortran + mpich2 on laptop Results: 8 / 8 / 8 Rajeev_mpich_out.gfortran.x86-64.txt: gfortran + mpich2 on x86-64 Results: 8 / 8 / 8 D) No problems because all DOUBLE PRECISION alignments are identical = 4 Rajeev_mpich_out.gfortran.ia32.txt: gfortran + mpich2 on ia32 Results: 4 / 4 / 4 Rajeev_mpich_out.ifort.ia32.txt: Intel + mpich2 on ia32 Results: 4 / 4 / 4 Proposal: --------- The goal of a standard is the possibility to write portable code. Therefore the answer should be standardized and not implementation-specific, or in other words, not defining it is the worst solution. I expect only a few Fortran codes are using arrays of derived types in Fortran. The oldest library is mpich. Therefore I would propose to standardize Fortran alignments are based on BIND(C) which should be the same as in C. This would imply that IBM-XL and mpich2 and derivatives need no change. But it also implies that the younger OpenMPI and Microsoft-MPI need a change. These implementation may add an environment switch to keep the old behavior if this switch is set, but I would expect that there is nearly no user who needs or want to use this switch. Fab and Jeff, could you live with this proposal? Best regards Rolf PS: Additional remarks: Yes, some compiler detect that DOUBLE PR. on 4 byte alignment is inefficient, but this is only an information. Other compilers detect, that we use DOUBLE PRECISION in BIND(C) derived types instead of using the C double through these new intrinsic C types in Fortran. This is also okay and all compiler work as expected, they use the C alignment. > Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 09:14:33 > > > In my output from Cray, the compilers > > - gnu > > - pgi > > - cray > > all use 8 byte double alignment in C and Fortran. > > ==> no problem with those compilers on the tested platform. > > Maybe we have differences between IA64 and 32 architectures > > for the same compiler. > > > > Important is to check MPI with those compilers that have > > Fortran 4 byte double precision alignment (e.g. Intel). > > > > Best regards > > Rolf > > > >> On Sep 27, 2011, at 10:21 AM, Rolf Rabenseifner wrote: > >> > >>> Dear all, (now with attachment) > >>> > >>> as far as I know, you represent some MPI libraries that are > >>> not directly based on another one in the list: > >>> - mpich2: Rajeev > >>> - OpenMPI: Jeff > >>> - IBM: Rich Treumann > >>> - NEC: Hubert > >>> - Fujitsu: Shinji Sumimoto > >>> - Microsoft: Fab > >>> (Which independent library is missing on this list?) > >>> (Who of the list uses already mich2 or OpenMPI as bases > >>> or does not provide a Fortran binding, > >>> and can be therefore removed from this list?) > >>> > >>> The problem is simple and has at the end only three numbers: > >>> > >>> Alignments of: > >>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. with > >>> Intel) > >>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > >>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 > >>> The details: > >>> Which is the alignment of Fortran DOUBLE PRECISION according to > >>> k_i on MPI-2.2, page 78 line 45 and page 96 line 42. > >>> > >>> Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 (Fortran > >>> 90) > >>> in Fortran SEQUENCE derived types (which have the same memory > >>> layout > >>> according to the SEQUENCE-rules), the MPI alignment and the > >>> Fortran > >>> alignment in such constructs should be the same, e.g., the Intel > >>> compiler has alignment 4 !!! > >>> > >>> This may be significantly different to C, where Intel uses > >>> alignment > >>> 8! > >>> > >>> With Fortran 2003, we got the Fortran BIND(C) derived types. > >>> Here, a Fortran DOUBLE PRECISION has with Intel's compiler > >>> the alignment 8. > >>> > >>> As far as I understand, mpich2 and Cray produce following results > >>> with > >>> an Intel compiler: > >>> > >>> Alignments of: > >>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 > >>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > >>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 > >>> > >>> This is definitely wrong compared to MPI-1.1 and MPI-2.0 > >>> but may be helpful for MPI-3.0 when we like to switch > >>> to a definition that is based on BIND(C). > >>> > >>> Question: > >>> What is the output of the attached test program when running > >>> with > >>> - your MPI library > >>> - and your compiler > >>> - and with Intel's compiler > >>> > >>> Please can you send me the output of such mpiruns? > >>> > >>> Thanks in advance and best regards > >>> Rolf > >>> > >>> PS: We need not to care about 8/8/8. > >>> Only 4/8/x causes problems: > >>> If all those have x=8 then I'll propose to > >>> adopt the MPI-3.0 to the reality. > >>> If we have some implementations with 4/8/8 and > >>> others with 4/8/4 then we have a serious problem > >>> because only one answer can be correct, > >>> and the question would be whether the historical > >>> answer (x=4) or the modern answer (x=8) > >>> should be chosen. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Iain Bason 27 Sep '11

27 Sep '11

On Sep 27, 2011, at 11:58 AM, N.M. Maclaren wrote: > On Sep 27 2011, Rolf Rabenseifner wrote: >> >> as far as I know, you represent some MPI libraries that are not directly based on another one in the list: >> - mpich2: Rajeev >> - OpenMPI: Jeff >> - IBM: Rich Treumann >> - NEC: Hubert >> - Fujitsu: Shinji Sumimoto - Microsoft: Fab (Which independent library is missing on this list?) > > When I used their systems and if I recall correctly, SGI and Sun did not > maintain from one of those bases - but that is some time ago, and I am > not even sure whether they still have a MPI. If you still have contacts, > it's worth checking. Sun/Oracle MPI is now based on OpenMPI. Iain

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by N.M. Maclaren 27 Sep '11

27 Sep '11

On Sep 27 2011, Rolf Rabenseifner wrote: > >as far as I know, you represent some MPI libraries that are >not directly based on another one in the list: > - mpich2: Rajeev > - OpenMPI: Jeff > - IBM: Rich Treumann > - NEC: Hubert > - Fujitsu: Shinji Sumimoto > - Microsoft: Fab >(Which independent library is missing on this list?) When I used their systems and if I recall correctly, SGI and Sun did not maintain from one of those bases - but that is some time ago, and I am not even sure whether they still have a MPI. If you still have contacts, it's worth checking. > Aligments of: > - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. with Intel) > - DOUBLE PRECISION within a BIND(C) derived type = 8 > - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 > >The details: >Which is the alignment of Fortran DOUBLE PRECISION according to >k_i on MPI-2.2, page 78 line 45 and page 96 line 42. Gug. That has been causing trouble since I started using Fortran (40+ years back), but what you describe has been the consensus for 20+ years. However, a few points: This is often option-dependent, most MPIs work with only some compiler options, and so it is important to check what the result is for those. Only a few code-generation option variations matter, of course. COMMON and SEQUENCE are not formally obsolescent, but are generally disapproved of, and so consistency with BIND(C) is relatively important. It's still a tricky question. Strictly, there are other possible variations, but they need such a borderline use of the standard that I agree they should be ignored. I doubt that anyone uses them in portable code. Regards, Nick Maclaren.

1 0

[MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rolf Rabenseifner 27 Sep '11

27 Sep '11

For your info: I informed the providers of independent MPI libraries with Fortran bindings about a serious problem with the definition of Fortran datatypes. My apologies if I missed some of the independent MPI libraries. In this case, please also run the program and please send the result to me. Best regards Rolf ----- Forwarded Message ----- From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> To: "Richard Treumann" <treumann(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji Sumimoto" <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Fab Tillier" <ftillier(a)microsoft.com> Sent: Tuesday, September 27, 2011 5:15:15 PM Subject: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION Dear all, as far as I know, you represent some MPI libraries that are not directly based on another one in the list: - mpich2: Rajeev - OpenMPI: Jeff - IBM: Rich Treumann - NEC: Hubert - Fujitsu: Shinji Sumimoto - Microsoft: Fab (Which independent library is missing on this list?) (Who of the list uses already mich2 or OpenMPI as bases or does not provide a Fortran binding, and can be therefore removed from this list?) The problem is simple and has at the end only three numbers: Aligments of: - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. with Intel) - DOUBLE PRECISION within a BIND(C) derived type = 8 - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 The details: Which is the alignment of Fortran DOUBLE PRECISION according to k_i on MPI-2.2, page 78 line 45 and page 96 line 42. Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 (Fortran 90) in Fortran SEQUENCE derived types (which have the same memory layout according to the SEQUENCE-rules), the MPI alignment and the Fortran alignment in such constructs should be the same, e.g., the Intel compiler has alignment 4 !!! This may be significantly different to C, where Intel uses alignment 8! With Fortran 2003, we got the Fortran BIND(C) derived types. Here, a Fortran DOUBLE PRECISION has with Intel's compiler the aligment 8. As far as I understand, mpich2 and Cray produce following results with an Intel compiler: Aligments of: - DOUBLE PRECISION within a SEQUENCE derived type = 4 - DOUBLE PRECISION within a BIND(C) derived type = 8 - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 This is definitely wrong compared to MPI-1.1 and MPI-2.0 but may be helpful for MPI-3.0 when we like to switch to a definition that is based on BIND(C). Question: What is the output of the attached test program when running with - your MPI library - and your compiler - and with Intel's compiler Please can you send me the output of such mpiruns? Thanks in advance and best regards Rolf PS: We need not to care about 8/8/8. Only 4/8/x causes problems: If all those have x=8 then I'll propose to adopt the MPI-3.0 to the reality. If we have some implementations with 4/8/8 and others with 4/8/4 then we have a serious problem because only one answer can be correct, and the question would be whether the historical answer (x=4) or the modern answer (x=8) should be chosen. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0