September 2011 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by N.M. Maclaren 28 Sep '11

28 Sep '11

On Sep 28 2011, Rolf Rabenseifner wrote: >> A very good question. Instead of referring to SEQUENCE, it should say >> that MPI_DOUBLE_PRECISION would use the numeric storage sequence rules >> (16.4.3.1 in Fortran 2003) for alignment. > >please can you explain, what "16.4.3.1 in Fortran 2003" tells >about alignments. >In SEQUENCE derived types and COMMON blocks, >when I understand this correctly, then DOUBLE PRECISION >must have also one numerical storage unit as alignment, >i.e., with REAL=4 bytes, DOUBLE PRECISION alignment must be >also 4 byte, i.e., exactly what the Intel compiler is doing >in a SEQUENCE derived type or COMMON block. That is my understanding, too, and I am pretty sure that I am correct. >The implication is that the compiler issues nice "warnings" about >misaligned data. Yes. >Do I understand correctly, that the other compilers with >REAL-size=4 byte and DOUBLE PRECISION size and alignment = 8 byte, >do it wrong, i.e.not according to Fortran 2003 or 2008? Not quite. If they align DOUBLE PRECISION at 8 bytes in SEQUENCE derived types or COMMON blocks, then my understanding is that they are incorrect. But they may do so anywhere else, such as in all other derived types, and it is permitted for DOUBLE PRECISION and REAL(KIND=C_DOUBLE) to be identical except for different alignments (indeed, that is very likely). My understanding is that, if DOUBLE PRECISION and REAL(KIND=C_DOUBLE) have different alignments, then KIND(0.0D0) must not equal C_DOUBLE. Even if they align at 8 bytes in other contexts, EQUIVALENCE can be used to generate 4-byte aligned DOUBLE PRECISION from an 8-byte aligned object, which is required to work. But EQUIVALENCE is horrible and strongly disapproved of. Are there any compilers that break those rules? Regards, Nick Maclaren.

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rolf Rabenseifner 28 Sep '11

28 Sep '11

Alexander and Charles, thank you for your lists. I updated the summary_out.txt . It has now three bad categories: A) 4 / 8 / 4 and B) 4 / 8 / 8 and C) 4 / 4 / 8 which is new and very strange because MPI's alignment has nothing to do with the compilers alignment. Alexander, > Why don't we leave this to the implementation and the compiler to > decide? The reason is simple: It is not possible to write portable code according to the rule in MPI-2.2 p79:7-9 "The definition of extent is motivated by the assumption that the amount of padding added at the end of each structure in an array of structures is the least needed to fulfill alignment constraints." The goal of this standardization is to be able to write portable application code. And the goal of this MPI sentence is, to be able to write such portable code without use of MPI_TYPE_CREATE_RESIZED. In C, this should work, in Fortran there are incompatibilities that should be resolved with MPI-3.0. Best regards Rolf ----- Original Message ----- > From: "Alexander Supalov" <alexander.supalov(a)intel.com> > To: "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> > Sent: Wednesday, September 28, 2011 5:48:09 PM > Subject: Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION > Hi, > > Why don't we leave this to the implementation and the compiler to > decide? Our testing results (see attachment) are all over the map on > the platforms we currently support. I'd not like to deal w/ any of > this specifically in the future product. What happens is platform > specific and looks acceptable to the users. And the standard example > may just show one possible resolution. Why bother? > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-fortran-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Rolf > Rabenseifner > Sent: Wednesday, September 28, 2011 9:14 AM > To: MPI-3 Fortran working group > Subject: Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries > with the alignment of MPI_DOUBLE_PRECISION > > Fab and all, > > your idea seems optimal, but ... (see later)!!!! > > In detail: > We have the problem that the Intel ***Fortran*** compiler > answers with two different values a=4 and b=8. > Both are Fortran values, a=4 for Fortran SEQUENCE derived > types and b=8 for Fortran BIND(C) derived types. > (Fortran non-sequence and non-bind(C) derived types > are not really supported by MPI). > The MPI Forum has to decide which value should be used > in the example on page 581 of > https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/229/mpi-repo… > i.e., whether Alternative 1 or 2 is the correct one. > > Your proposal (I name it Alternative 3) is that > the user has to choose the mapped C basic type. > This is a non-trivial mapping. > For MPI_INTEGER it is MPI_INT or MPI_LONG? > For MPI_INTEGER8 it is MPI_LONG_LONG? > What are we doing with the unnamed predefined > datatypes, produced by MPI_TYPE_CREATE_F90_COMPLEX / > _REAL / _INTEGER? > > But your proposal inspires me to propose an > Alternative 4.1 and 4.2: > > We need a routine that converts a given Fortran basic > named or unnamed datatype with > > 4.1) by default "SEQUENCE alignment" into an > unnamed predefined datatype with "BIND(C) alignment"; > this works together with Alternative 1 for page 581. > MPI_F2003_CONVERT_SEQ_TO_BIND_C(IN seq_datatype, OUT bind_c_datatype) > > 4.2) by default "BIND(C) alignment" into an > unnamed predefined datatype with "SEQUENCE alignment"; > this works together with Alternative 2 for page 581. > MPI_F2003_CONVERT_BIND_C_TO_SEQ(IN bind_c_datatype, OUT seq_datatype) > > Best regards > Rolf > > ----- Original Message ----- > > From: "Fab Tillier" <ftillier(a)microsoft.com> > > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff Squyres" > > <jsquyres(a)cisco.com>, "Shinji Sumimoto" > > <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" > > <hubert.ritzdorf(a)emea.nec.com>, "Howard Pritchard" > > <howardp(a)cray.com>, > > "Brian Smith" <smithbr(a)us.ibm.com>, "Charles J Archer" > > <archerc(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, > > "Bill Long" <longb(a)cray.com>, "Bill Gropp" <wgropp(a)uiuc.edu>, > > "Richard Graham" <rlgraham(a)ornl.gov> > > Cc: "N.M. Maclaren" <nmm1(a)cam.ac.uk>, "Iain Bason" > > <iain.bason(a)oracle.com> > > Sent: Wednesday, September 28, 2011 12:35:53 AM > > Subject: RE: Serious problem/bug in MPI libraries with the alignment > > of MPI_DOUBLE_PRECISION > > Hi Rolf, > > > > It seems that all the a == 4 results come from using the Intel > > Fortran > > compiler suite. Given that Microsoft doesn't ship a Fortran > > compiler, > > mandating a == 8 doesn't put us in a position to succeed. Therefore, > > I > > cannot support such a proposal. > > > > however, shouldn't a DOUBLE PRECISION BIND(C) derived type map to > > MPI_DOUBLE, not MPI_DOUBLE_PRECISION? > > > Alternative 3: > > Can't we simply document that MPI_DOUBLE_PRECISION (and any Fortran > > types that need similar treatment) apply only to SEQUENCE derived > > types, while BIND(C) derived types should use the C types? > > > > -Fab > > > > Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 14:38:45 > > > > > Dear all, > > > > > > thank you very much for the fast answers. > > > The result is maximal bad, > > > i.e., about 50% of the MPIs use SEQUENCE alignments and other > > > about 50% use BIND(C) based calculation of the alignment. > > > > > > Here the results (also attached as summary_out.txt). > > > Please read carefully my proposal after this summary sheet. > > > > > > In all protocols: > > > ----------------- > > > Size of one Fortran REAL = 4 > > > Size of one Fortran DOUBLE PRECISION = 8 > > > Results: > > > -------- > > > > > > a / b / c = Alignments of: > > > - DOUBLE PRECISION within a SEQUENCE derived type = a > > > - DOUBLE PRECISION within a BIND(C) derived type = b > > > - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = c > > > > > > A) The 4 / 8 / 4 group = alignment of MPI_DOUBLE_PRECISION is > > > correct for (old-style) SEQUENCE derived types > > > > > > JeffSquyres_OpenMP_linux-x86-64-ifort.txt: Intel + OpenMPI on > > > x86-64 > > > Results: 4 / 8 / 4 > > > FabTillier_Intel+MicrosoftMPI_out.txt: Intel + MS-MPI on ? > > > Results: 4 / 8 / 4 > > > > > > B) The 4 / 8 / 8 group = alignment of MPI_DOUBLE_PRECISION > > > is correct for (modern) BIND(C) derived types > > > > > > Rajeev_mpich_out.ifort.x86-64.txt: Intel + mpich2 on x86-64 > > > Results: 4 / 8 / 8 > > > Rolf_asama+cray_output.txt: Intel + NEC-MPI? on asama > > > Results: 4 / 8 / 8 > > > BillLong_Cray_5compiler_out.txt: Intel + CrayMPI on Cray XE6 > > > Results: 4 / 8 / 8 > > > BrianSmith-IBM_outlog-bgp-xl.txt: Intel + IBM-XL on BGP > > > Results: 4 / 8 / 8 > > > BrianSmith-IBM_outlog-xlf.txt: xlf + IBM-XL on ? > > > Results: 4 / 8 / 8 > > > > > > C) No problems because all DOUBLE PRECISION alignments > > > are identical = 8 > > > > > > BillLong_Cray_5compiler_out.txt: PGI + CrayMPI on Cray XE6 > > > Results: 8 / 8 / 8 > > > BillLong_Cray_5compiler_out.txt: Cray + CrayMPI on Cray XE6 > > > Results: 8 / 8 / 8 > > > BillLong_Cray_5compiler_out.txt: gfortran + CrayMPI on Cray XE6 > > > Results: 8 / 8 / 8 > > > BillLong_Cray_5compiler_out.txt: pathscale + CrayMPI on Cray XE6 > > > Results: 8 / 8 / 8 > > > BrianSmith-IBM_outlog-gcc.txt: gfortran + IBM-XL on BGQ > > > Results: 8 / 8 / 8 > > > JeffSquyres_OpenMP_linux-x86-64-gfortran.txt: gfortran + OpenMPI > > > on x86-64 > > > Results: 8 / 8 / 8 > > > JeffSquyres_OpenMP_osx-x86-84-gfortran.txt: gfortran + OpenMPI on > > > osx-x86-64 > > > Results: 8 / 8 / 8 > > > Rajeev_gnu_laptop_out.txt: gfortran + mpich2 on laptop > > > Results: 8 / 8 / 8 > > > Rajeev_mpich_out.gfortran.x86-64.txt: gfortran + mpich2 on x86-64 > > > Results: 8 / 8 / 8 > > > > > > D) No problems because all DOUBLE PRECISION alignments are > > > identical = 4 > > > > > > Rajeev_mpich_out.gfortran.ia32.txt: gfortran + mpich2 on ia32 > > > Results: 4 / 4 / 4 > > > Rajeev_mpich_out.ifort.ia32.txt: Intel + mpich2 on ia32 > > > Results: 4 / 4 / 4 > > > > > > Proposal: > > > --------- > > > > > > The goal of a standard is the possibility to write portable code. > > > Therefore the answer should be standardized and not > > > implementation- > > > specific, or in other words, not defining it is the worst > > > solution. > > > > > > I expect only a few Fortran codes are using arrays of derived > > > types > > > in Fortran. > > > The oldest library is mpich. > > > Therefore I would propose to standardize Fortran alignments are > > > based > > > on BIND(C) which should be the same as in C. > > > > > > This would imply that IBM-XL and mpich2 and derivatives need no > > > change. > > > But it also implies that the younger OpenMPI and Microsoft-MPI > > > need a change. These implementation may add an environment switch > > > to keep the old behavior if this switch is set, > > > but I would expect that there is nearly no user > > > who needs or want to use this switch. > > > > > > Fab and Jeff, could you live with this proposal? > > > > > > Best regards > > > Rolf > > > > > > PS: Additional remarks: > > > Yes, some compiler detect that DOUBLE PR. on 4 byte alignment > > > is > > > inefficient, but this is only an information. Other compilers > > > detect, that we use DOUBLE PRECISION in BIND(C) derived types > > > instead of using the C double through these new intrinsic C > > > types in > > > Fortran. This is also okay and all compiler work as expected, > > > they > > > use the C alignment. > > > > > > > > >> Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 09:14:33 > > >> > > >>> In my output from Cray, the compilers > > >>> - gnu > > >>> - pgi > > >>> - cray > > >>> all use 8 byte double alignment in C and Fortran. > > >>> ==> no problem with those compilers on the tested platform. > > >>> Maybe we have differences between IA64 and 32 architectures > > >>> for the same compiler. > > >>> > > >>> Important is to check MPI with those compilers that have > > >>> Fortran 4 byte double precision alignment (e.g. Intel). > > >>> > > >>> Best regards > > >>> Rolf > > >>> > > > > > >>>> On Sep 27, 2011, at 10:21 AM, Rolf Rabenseifner wrote: > > >>>> > > >>>>> Dear all, (now with attachment) > > >>>>> > > >>>>> as far as I know, you represent some MPI libraries that are > > >>>>> not directly based on another one in the list: > > >>>>> - mpich2: Rajeev > > >>>>> - OpenMPI: Jeff > > >>>>> - IBM: Rich Treumann > > >>>>> - NEC: Hubert > > >>>>> - Fujitsu: Shinji Sumimoto > > >>>>> - Microsoft: Fab > > >>>>> (Which independent library is missing on this list?) > > >>>>> (Who of the list uses already mich2 or OpenMPI as bases > > >>>>> or does not provide a Fortran binding, > > >>>>> and can be therefore removed from this list?) > > >>>>> > > >>>>> The problem is simple and has at the end only three numbers: > > >>>>> > > >>>>> Alignments of: > > >>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. > > >>>>> with > > >>>>> Intel) > > >>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > > >>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 > > >>>>> The details: > > >>>>> Which is the alignment of Fortran DOUBLE PRECISION according > > >>>>> to > > >>>>> k_i on MPI-2.2, page 78 line 45 and page 96 line 42. > > >>>>> > > >>>>> Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 > > >>>>> (Fortran > > >>>>> 90) > > >>>>> in Fortran SEQUENCE derived types (which have the same memory > > >>>>> layout > > >>>>> according to the SEQUENCE-rules), the MPI alignment and the > > >>>>> Fortran > > >>>>> alignment in such constructs should be the same, e.g., the > > >>>>> Intel > > >>>>> compiler has alignment 4 !!! > > >>>>> > > >>>>> This may be significantly different to C, where Intel uses > > >>>>> alignment > > >>>>> 8! > > >>>>> > > >>>>> With Fortran 2003, we got the Fortran BIND(C) derived types. > > >>>>> Here, a Fortran DOUBLE PRECISION has with Intel's compiler > > >>>>> the alignment 8. > > >>>>> > > >>>>> As far as I understand, mpich2 and Cray produce following > > >>>>> results > > >>>>> with > > >>>>> an Intel compiler: > > >>>>> > > >>>>> Alignments of: > > >>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 > > >>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > > >>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 > > >>>>> > > >>>>> This is definitely wrong compared to MPI-1.1 and MPI-2.0 > > >>>>> but may be helpful for MPI-3.0 when we like to switch > > >>>>> to a definition that is based on BIND(C). > > >>>>> > > >>>>> Question: > > >>>>> What is the output of the attached test program when running > > >>>>> with > > >>>>> - your MPI library > > >>>>> - and your compiler > > >>>>> - and with Intel's compiler > > >>>>> > > >>>>> Please can you send me the output of such mpiruns? > > >>>>> > > >>>>> Thanks in advance and best regards > > >>>>> Rolf > > >>>>> > > >>>>> PS: We need not to care about 8/8/8. > > >>>>> Only 4/8/x causes problems: > > >>>>> If all those have x=8 then I'll propose to > > >>>>> adopt the MPI-3.0 to the reality. > > >>>>> If we have some implementations with 4/8/8 and > > >>>>> others with 4/8/4 then we have a serious problem > > >>>>> because only one answer can be correct, > > >>>>> and the question would be whether the historical > > >>>>> answer (x=4) or the modern answer (x=8) > > >>>>> should be chosen. > > > > > > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > -------------------------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen, Deutschland > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 > Ust.-IdNr./VAT Registration No.: DE129385895 > Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rolf Rabenseifner 28 Sep '11

28 Sep '11

Fab, > > For MPI_INTEGER it is MPI_INT or MPI_LONG? > > I suppose it depends on the size of the Fortran integer. MPI_INTEGER > presumably matches the Fortran integer size, and this would map to one > of the MPI_INT{8,16,32,64}_T? Yes, this is a nice example that such application code is not portable! The goal of this standardization is, that the user can write portable code. Best regards Rolf ----- Original Message ----- > From: "Fab Tillier" <ftillier(a)microsoft.com> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "N.M. Maclaren" <nmm1(a)cam.ac.uk>, "Iain Bason" <iain.bason(a)oracle.com>, "Jeff Squyres" <jsquyres(a)cisco.com>, > "Shinji Sumimoto" <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Howard Pritchard" > <howardp(a)cray.com>, "Brian Smith" <smithbr(a)us.ibm.com>, "Charles J Archer" <archerc(a)us.ibm.com>, "Rajeev Thakur" > <thakur(a)mcs.anl.gov>, "Bill Long" <longb(a)cray.com>, "Bill Gropp" <wgropp(a)uiuc.edu>, "Richard Graham" > <rlgraham(a)ornl.gov>, "Alexander Supalov" <alexander.supalov(a)intel.com> > Sent: Wednesday, September 28, 2011 5:25:26 PM > Subject: RE: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION > Hi Rolf, > > Rolf Rabenseifner wrote on Wed, 28 Sep 2011 at 00:07:50 > > > I'll continue this discussion on > > MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> > > If you are not on this list, please join to this list. > > Please DO NOT reply to this mail. PLEASE REPLY on the mailing-list. > > > > Fab and all, > > > > your idea seems optimal, but ... (see later)!!!! > > > > In detail: We have the problem that the Intel ***Fortran*** compiler > > answers with two different values a=4 and b=8. both are Fortran > > values, > > a=4 for Fortran SEQUENCE derived types and b=8 for Fortran BIND(C) > > derived types. (Fortran non sequence and non bind(C) derived types > > are > > not supported). The MPI Forum has to decide which value should be > > used > > in the example on page 581 of > > https://svn.mpi-forum.org/trac/mpi-forum- > > web/attachment/ticket/229/mpi-report-F2008-2011-09-08- > > changeonlyplustickets.pdf i.e., whether Alternative 1 or 2 is the > > correct one. > > > > Your proposal is that the user has to choose the > > mapped C basic type. > > This is a non-trivial mapping. > > Is it? I would think it would be worth trying to define a mapping, > though my Fortran knowledge is limited so perhaps you know a-priori > that it is non-trivial, while to me it seems perfectly simple. > > > For MPI_INTEGER it is MPI_INT or MPI_LONG? > > I suppose it depends on the size of the Fortran integer. MPI_INTEGER > presumably matches the Fortran integer size, and this would map to one > of the MPI_INT{8,16,32,64}_T? > > I would shy away from mapping anything to MPI_LONG due to the > different sizes of long on different platforms. MPI_LONG should map to > the C long type. On Windows it is always the same as int (always > 32-bits), while on other platforms it tends to be the same as ssize_t. > > > For MPI_INTEGER8 it is MPI_LONG_LONG? > > MPI_INTEGER8 should map to MPI_INT64_t, shouldn't it? > > > What are we doing with the unnamed predefined > > datatypes, produced by MPI_TYPE_CREATE_F90_COMPLEX / > > _REAL / _INTEGER? > > These would all be SEQUENCE types for backward compatibility? Or were > these created as a precursor to BIND(C) support? I really don't know > how these are used... > > Would MPI_REAL map to MPI_FLOAT? > > > But your proposal (I name it Alternative 3) > > inspires me to propose an Alternative 4.1 and 4.2: > > > > We need a routine that converts a given Fortran basic > > named or unnamed datatype with > > > > 4.1) by default "SEQUENCE alignment" into an > > unnamed predefined datatype with "BIND(C) alignment"; > > this works together with Alternative 1 for page 581. > > MPI_F2003_CONVERT_SEQ_TO_BIND_C(IN seq_datatype, OUT > > bind_c_datatype) > > > > 4.2) by default "BIND(C) alignment" into an > > unnamed predefined datatype with "SEQUENCE alignment"; > > this works together with Alternative 2 for page 581. > > MPI_F2003_CONVERT_BIND_C_TO_SEQ(IN bind_c_datatype, OUT > > seq_datatype) > > But what do we do with the definition of MPI_DOUBLE_PRECISION, etc? Do > we document these as applying to SEQUENCE (or as Nick suggested, the > Fortran numeric storage rules)? > > Cheers, > -Fab > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "Fab Tillier" <ftillier(a)microsoft.com> To: "Rolf > >> Rabenseifner" > >> <rabenseifner(a)hlrs.de>, "Jeff Squyres" <jsquyres(a)cisco.com>, > >> "Shinji > >> Sumimoto" <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" > > <hubert.ritzdorf(a)emea.nec.com>, "Howard Pritchard" > > <howardp(a)cray.com>, > >> "Brian Smith" <smithbr(a)us.ibm.com>, "Charles J Archer" > >> <archerc(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, "Bill > >> Long" > >> <longb(a)cray.com>, "Bill Gropp" <wgropp(a)uiuc.edu>, "Richard Graham" > >> <rlgraham(a)ornl.gov> Cc: "N.M. Maclaren" <nmm1(a)cam.ac.uk>, "Iain > >> Bason" > >> <iain.bason(a)oracle.com> Sent: Wednesday, September 28, 2011 > >> 12:35:53 AM > >> Subject: RE: Serious problem/bug in MPI libraries with the > >> alignment of > >> MPI_DOUBLE_PRECISION Hi Rolf, > >> > >> It seems that all the a == 4 results come from using the Intel > >> Fortran > >> compiler suite. Given that Microsoft doesn't ship a Fortran > >> compiler, > >> mandating a == 8 doesn't put us in a position to succeed. > >> Therefore, I > >> cannot support such a proposal. > >> > >> however, shouldn't a DOUBLE PRECISION BIND(C) derived type map to > >> MPI_DOUBLE, not MPI_DOUBLE_PRECISION? > >> > >> Can't we simply document that MPI_DOUBLE_PRECISION (and any Fortran > >> types that need similar treatment) apply only to SEQUENCE derived > >> types, while BIND(C) derived types should use the C types? > >> > >> -Fab > >> > >> Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 14:38:45 > >> > >>> Dear all, > >>> > >>> thank you very much for the fast answers. > >>> The result is maximal bad, > >>> i.e., about 50% of the MPIs use SEQUENCE alignments and other > >>> about 50% use BIND(C) based calculation of the alignment. > >>> > >>> Here the results (also attached as summary_out.txt). > >>> Please read carefully my proposal after this summary sheet. > >>> > >>> In all protocols: > >>> ----------------- > >>> Size of one Fortran REAL = 4 > >>> Size of one Fortran DOUBLE PRECISION = 8 > >>> Results: > >>> -------- > >>> > >>> a / b / c = Alignments of: > >>> - DOUBLE PRECISION within a SEQUENCE derived type = a > >>> - DOUBLE PRECISION within a BIND(C) derived type = b > >>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = c > >>> A) The 4 / 8 / 4 group = alignment of MPI_DOUBLE_PRECISION is > >>> correct for > >>> (old-style) SEQUENCE derived types > >>> > >>> JeffSquyres_OpenMP_linux-x86-64-ifort.txt: Intel + OpenMPI on > >>> x86-64 > >>> Results: 4 / 8 / 4 > >>> FabTillier_Intel+MicrosoftMPI_out.txt: Intel + MS-MPI on ? > >>> Results: 4 / 8 / 4 > >>> > >>> B) The 4 / 8 / 8 group = alignment of MPI_DOUBLE_PRECISION is > >>> correct for > >>> (modern) BIND(C) derived types > >>> > >>> Rajeev_mpich_out.ifort.x86-64.txt: Intel + mpich2 on x86-64 > >>> Results: 4 / 8 / 8 > >>> Rolf_asama+cray_output.txt: Intel + NEC-MPI? on asama > >>> Results: 4 / 8 / 8 > >>> BillLong_Cray_5compiler_out.txt: Intel + CrayMPI on Cray XE6 > >>> Results: 4 / 8 / 8 > >>> BrianSmith-IBM_outlog-bgp-xl.txt: Intel + IBM-XL on BGP > >>> Results: 4 / 8 / 8 > >>> BrianSmith-IBM_outlog-xlf.txt: xlf + IBM-XL on ? Results: 4 / > >>> 8 / 8 > >>> > >>> C) No problems because all DOUBLE PRECISION alignments are > >>> identical > >>> = 8 > >>> > >>> BillLong_Cray_5compiler_out.txt: PGI + CrayMPI on Cray XE6 > >>> Results: 8 / 8 / 8 > >>> BillLong_Cray_5compiler_out.txt: Cray + CrayMPI on Cray XE6 > >>> Results: 8 / 8 / 8 > >>> BillLong_Cray_5compiler_out.txt: gfortran + CrayMPI on Cray XE6 > >>> Results: 8 / 8 / 8 > >>> BillLong_Cray_5compiler_out.txt: pathscale + CrayMPI on Cray XE6 > >>> Results: 8 / 8 / 8 > >>> BrianSmith-IBM_outlog-gcc.txt: gfortran + IBM-XL on BGQ > >>> Results: 8 / 8 / 8 > >>> JeffSquyres_OpenMP_linux-x86-64-gfortran.txt: gfortran + OpenMPI > >>> on > >>> x86-64 Results: 8 / 8 / 8 > >>> JeffSquyres_OpenMP_osx-x86-84-gfortran.txt: gfortran + OpenMPI on > >>> osx-x86-64 Results: 8 / 8 / 8 > >>> Rajeev_gnu_laptop_out.txt: gfortran + mpich2 on laptop > >>> Results: 8 / 8 / 8 > >>> Rajeev_mpich_out.gfortran.x86-64.txt: gfortran + mpich2 on x86-64 > >>> Results: 8 / 8 / 8 > >>> > >>> D) No problems because all DOUBLE PRECISION alignments are > >>> identical > >>> = 4 > >>> > >>> Rajeev_mpich_out.gfortran.ia32.txt: gfortran + mpich2 on ia32 > >>> Results: 4 / 4 / 4 > >>> Rajeev_mpich_out.ifort.ia32.txt: Intel + mpich2 on ia32 Results: > >>> 4 / 4 / 4 > >>> > >>> Proposal: > >>> --------- > >>> > >>> The goal of a standard is the possibility to write portable code. > >>> Therefore the answer should be standardized and not > >>> implementation- > >>> specific, or in other words, not defining it is the worst > >>> solution. > >>> > >>> I expect only a few Fortran codes are using arrays of derived > >>> types > >>> in Fortran. > >>> The oldest library is mpich. > >>> Therefore I would propose to standardize Fortran alignments are > >>> based > >>> on BIND(C) which should be the same as in C. > >>> > >>> This would imply that IBM-XL and mpich2 and derivatives need no > >>> change. > >>> But it also implies that the younger OpenMPI and Microsoft-MPI > >>> need a change. These implementation may add an environment switch > >>> to keep the old behavior if this switch is set, > >>> but I would expect that there is nearly no user > >>> who needs or want to use this switch. > >>> > >>> Fab and Jeff, could you live with this proposal? > >>> > >>> Best regards > >>> Rolf > >>> > >>> PS: Additional remarks: > >>> Yes, some compiler detect that DOUBLE PR. on 4 byte alignment > >>> is > >>> inefficient, but this is only an information. Other compilers > >>> detect, that we use DOUBLE PRECISION in BIND(C) derived types > >>> instead of using the C double through these new intrinsic C > >>> types in > >>> Fortran. This is also okay and all compiler work as expected, > >>> they > >>> use the C alignment. > >>> > >>>> Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 09:14:33 > >>>> > >>>>> In my output from Cray, the compilers > >>>>> - gnu > >>>>> - pgi > >>>>> - cray > >>>>> all use 8 byte double alignment in C and Fortran. > >>>>> ==> no problem with those compilers on the tested platform. > >>>>> Maybe we have differences between IA64 and 32 architectures > >>>>> for the same compiler. > >>>>> > >>>>> Important is to check MPI with those compilers that have > >>>>> Fortran 4 byte double precision alignment (e.g. Intel). > >>>>> > >>>>> Best regards > >>>>> Rolf > >>>>> > >>> > >>>>>> On Sep 27, 2011, at 10:21 AM, Rolf Rabenseifner wrote: > >>>>>> > >>>>>>> Dear all, (now with attachment) > >>>>>>> > >>>>>>> as far as I know, you represent some MPI libraries that are > >>>>>>> not directly based on another one in the list: > >>>>>>> - mpich2: Rajeev > >>>>>>> - OpenMPI: Jeff > >>>>>>> - IBM: Rich Treumann > >>>>>>> - NEC: Hubert > >>>>>>> - Fujitsu: Shinji Sumimoto > >>>>>>> - Microsoft: Fab > >>>>>>> (Which independent library is missing on this list?) > >>>>>>> (Who of the list uses already mich2 or OpenMPI as bases > >>>>>>> or does not provide a Fortran binding, > >>>>>>> and can be therefore removed from this list?) > >>>>>>> > >>>>>>> The problem is simple and has at the end only three numbers: > >>>>>>> > >>>>>>> Alignments of: > >>>>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. > >>>>>>> with > >>>>>>> Intel) > >>>>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > >>>>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 > >>>>>>> The details: > >>>>>>> Which is the alignment of Fortran DOUBLE PRECISION according > >>>>>>> to > >>>>>>> k_i on MPI-2.2, page 78 line 45 and page 96 line 42. > >>>>>>> > >>>>>>> Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 > >>>>>>> (Fortran > >>>>>>> 90) > >>>>>>> in Fortran SEQUENCE derived types (which have the same memory > >>>>>>> layout > >>>>>>> according to the SEQUENCE-rules), the MPI alignment and the > >>>>>>> Fortran > >>>>>>> alignment in such constructs should be the same, e.g., the > >>>>>>> Intel > >>>>>>> compiler has alignment 4 !!! > >>>>>>> > >>>>>>> This may be significantly different to C, where Intel uses > >>>>>>> alignment > >>>>>>> 8! > >>>>>>> > >>>>>>> With Fortran 2003, we got the Fortran BIND(C) derived types. > >>>>>>> Here, a Fortran DOUBLE PRECISION has with Intel's compiler > >>>>>>> the alignment 8. > >>>>>>> > >>>>>>> As far as I understand, mpich2 and Cray produce following > >>>>>>> results > >>>>>>> with > >>>>>>> an Intel compiler: > >>>>>>> > >>>>>>> Alignments of: > >>>>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 > >>>>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > >>>>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 > >>>>>>> > >>>>>>> This is definitely wrong compared to MPI-1.1 and MPI-2.0 > >>>>>>> but may be helpful for MPI-3.0 when we like to switch > >>>>>>> to a definition that is based on BIND(C). > >>>>>>> > >>>>>>> Question: > >>>>>>> What is the output of the attached test program when running > >>>>>>> with > >>>>>>> - your MPI library > >>>>>>> - and your compiler > >>>>>>> - and with Intel's compiler > >>>>>>> > >>>>>>> Please can you send me the output of such mpiruns? > >>>>>>> > >>>>>>> Thanks in advance and best regards > >>>>>>> Rolf > >>>>>>> > >>>>>>> PS: We need not to care about 8/8/8. > >>>>>>> Only 4/8/x causes problems: > >>>>>>> If all those have x=8 then I'll propose to > >>>>>>> adopt the MPI-3.0 to the reality. > >>>>>>> If we have some implementations with 4/8/8 and > >>>>>>> others with 4/8/4 then we have a serious problem > >>>>>>> because only one answer can be correct, > >>>>>>> and the question would be whether the historical > >>>>>>> answer (x=4) or the modern answer (x=8) > >>>>>>> should be chosen. > >>> > >>> > > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rolf Rabenseifner 28 Sep '11

28 Sep '11

> A very good question. Instead of referring to SEQUENCE, it should say > that MPI_DOUBLE_PRECISION would use the numeric storage sequence rules > (16.4.3.1 in Fortran 2003) for alignment. Nick, please can you explain, what "16.4.3.1 in Fortran 2003" tells about alignments. In SEQUENCE derived types and COMMON blocks, when I understand this correctly, then DOUBLE PRECISION must have also one numerical storage unit as alignment, i.e., with REAL=4 bytes, DOUBLE PRECISION alignment must be also 4 byte, i.e., exactly what the Intel compiler is doing in a SEQUENCE derived type or COMMON block. The implication is that the compiler issues nice "warnings" about misaligned data. Do I understand correctly, that the other compilers with REAL-size=4 byte and DOUBLE PRECISION size and alignment = 8 byte, do it wrong, i.e.not according to Fortran 2003 or 2008? Best regards Rolf ----- Original Message ----- > From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> > To: "Fab Tillier" <ftillier(a)microsoft.com> > Cc: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji Sumimoto" > <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Howard Pritchard" <howardp(a)cray.com>, > "Brian Smith" <smithbr(a)us.ibm.com>, "Charles J Archer" <archerc(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, > "Bill Long" <longb(a)cray.com>, "Bill Gropp" <wgropp(a)uiuc.edu>, "Richard Graham" <rlgraham(a)ornl.gov>, "Iain Bason" > <iain.bason(a)oracle.com> > Sent: Wednesday, September 28, 2011 10:26:37 AM > Subject: RE: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION > On Sep 27 2011, Fab Tillier wrote: > > > > It seems that all the a == 4 results come from using the Intel > > Fortran > > compiler suite. Given that Microsoft doesn't ship a Fortran > > compiler, > > mandating a == 8 doesn't put us in a position to succeed. Therefore, > > I > > cannot support such a proposal. > > I agree that Microsoft should not be forgotten, but it is not a major > platform for MPI use, and the tail should not wag the dog. We have the > ghastly example of C99 long long where some people claimed Microsoft > compatibility required an incompatible change to the standard; largely > as a result, the official C standard is no longer used as a portable > programming base by a good half of the C-using community. > > > however, shouldn't a DOUBLE PRECISION BIND(C) derived type map to > > MPI_DOUBLE, not MPI_DOUBLE_PRECISION? > > > > Can't we simply document that MPI_DOUBLE_PRECISION (and any Fortran > > types > > that need similar treatment) apply only to SEQUENCE derived types, > > while > > BIND(C) derived types should use the C types? > > A very good question. Instead of referring to SEQUENCE, it should say > that MPI_DOUBLE_PRECISION would use the numeric storage sequence rules > (16.4.3.1 in Fortran 2003) for alignment. > > > Slightly related to this, can't we PLEASE deprecate those horrible > MPI_INTEGER8 etc. types? They have been a portability disaster for > half a century, even among Fortran IV compilers (where they started)! > MPI has provided some perfectly good mechanisms to use Fortran's > selectable KINDs. The reason that it is related is that there is no > guarantee that a compiler will give them the same alignment as another > type even if they use the same storage, because they are non-standard > and therefore any specification is the compiler's choice. > > > Regards, > Nick Maclaren. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rasmussen, Craig E 28 Sep '11

28 Sep '11

Rolf, You know my position but I will restate it for everyone. I don't mind conversion routines that help out an old user base but I strongly feel that the use of sequence types should be discouraged. They belong in "we should strongly discourage the usage of category" along with common blocks. For example, I just got home and did some checking with my Intel compiler and get warnings like: mpi_alignment.f90(17): warning #6380: The structure length is not a multiple of its largest element; could create misalignments for arrays of this type. [SEQ_D_R] type :: seq_d_r When use sequence the programmer tells the compiler that she knows better than the compiler how to lay out types. This kind of hack has no place in a standard. By standardizing on BIND(C) derived types we tell the Fortran compiler to use the same layout of the C compiler and so we get maximal interoperability reduce the potential for problems. Regards, Craig On Sep 28, 2011, at 1:14 AM, Rolf Rabenseifner wrote: > Fab and all, > > your idea seems optimal, but ... (see later)!!!! > > In detail: > We have the problem that the Intel ***Fortran*** compiler > answers with two different values a=4 and b=8. > Both are Fortran values, a=4 for Fortran SEQUENCE derived > types and b=8 for Fortran BIND(C) derived types. > (Fortran non-sequence and non-bind(C) derived types > are not really supported by MPI). > The MPI Forum has to decide which value should be used > in the example on page 581 of > https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/229/mpi-repo… > i.e., whether Alternative 1 or 2 is the correct one. > > Your proposal (I name it Alternative 3) is that > the user has to choose the mapped C basic type. > This is a non-trivial mapping. > For MPI_INTEGER it is MPI_INT or MPI_LONG? > For MPI_INTEGER8 it is MPI_LONG_LONG? > What are we doing with the unnamed predefined > datatypes, produced by MPI_TYPE_CREATE_F90_COMPLEX / > _REAL / _INTEGER? > > But your proposal inspires me to propose an > Alternative 4.1 and 4.2: > > We need a routine that converts a given Fortran basic > named or unnamed datatype with > > 4.1) by default "SEQUENCE alignment" into an > unnamed predefined datatype with "BIND(C) alignment"; > this works together with Alternative 1 for page 581. > MPI_F2003_CONVERT_SEQ_TO_BIND_C(IN seq_datatype, OUT bind_c_datatype) > > 4.2) by default "BIND(C) alignment" into an > unnamed predefined datatype with "SEQUENCE alignment"; > this works together with Alternative 2 for page 581. > MPI_F2003_CONVERT_BIND_C_TO_SEQ(IN bind_c_datatype, OUT seq_datatype) > > Best regards > Rolf > > ----- Original Message ----- >> From: "Fab Tillier" <ftillier(a)microsoft.com> >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji Sumimoto" >> <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Howard Pritchard" <howardp(a)cray.com>, >> "Brian Smith" <smithbr(a)us.ibm.com>, "Charles J Archer" <archerc(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, >> "Bill Long" <longb(a)cray.com>, "Bill Gropp" <wgropp(a)uiuc.edu>, "Richard Graham" <rlgraham(a)ornl.gov> >> Cc: "N.M. Maclaren" <nmm1(a)cam.ac.uk>, "Iain Bason" <iain.bason(a)oracle.com> >> Sent: Wednesday, September 28, 2011 12:35:53 AM >> Subject: RE: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION >> Hi Rolf, >> >> It seems that all the a == 4 results come from using the Intel Fortran >> compiler suite. Given that Microsoft doesn't ship a Fortran compiler, >> mandating a == 8 doesn't put us in a position to succeed. Therefore, I >> cannot support such a proposal. >> >> however, shouldn't a DOUBLE PRECISION BIND(C) derived type map to >> MPI_DOUBLE, not MPI_DOUBLE_PRECISION? >> > Alternative 3: >> Can't we simply document that MPI_DOUBLE_PRECISION (and any Fortran >> types that need similar treatment) apply only to SEQUENCE derived >> types, while BIND(C) derived types should use the C types? >> >> -Fab >> >> Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 14:38:45 >> >>> Dear all, >>> >>> thank you very much for the fast answers. >>> The result is maximal bad, >>> i.e., about 50% of the MPIs use SEQUENCE alignments and other >>> about 50% use BIND(C) based calculation of the alignment. >>> >>> Here the results (also attached as summary_out.txt). >>> Please read carefully my proposal after this summary sheet. >>> >>> In all protocols: >>> ----------------- >>> Size of one Fortran REAL = 4 >>> Size of one Fortran DOUBLE PRECISION = 8 >>> Results: >>> -------- >>> >>> a / b / c = Alignments of: >>> - DOUBLE PRECISION within a SEQUENCE derived type = a >>> - DOUBLE PRECISION within a BIND(C) derived type = b >>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = c >>> >>> A) The 4 / 8 / 4 group = alignment of MPI_DOUBLE_PRECISION is >>> correct for (old-style) SEQUENCE derived types >>> >>> JeffSquyres_OpenMP_linux-x86-64-ifort.txt: Intel + OpenMPI on x86-64 >>> Results: 4 / 8 / 4 >>> FabTillier_Intel+MicrosoftMPI_out.txt: Intel + MS-MPI on ? >>> Results: 4 / 8 / 4 >>> >>> B) The 4 / 8 / 8 group = alignment of MPI_DOUBLE_PRECISION >>> is correct for (modern) BIND(C) derived types >>> >>> Rajeev_mpich_out.ifort.x86-64.txt: Intel + mpich2 on x86-64 >>> Results: 4 / 8 / 8 >>> Rolf_asama+cray_output.txt: Intel + NEC-MPI? on asama >>> Results: 4 / 8 / 8 >>> BillLong_Cray_5compiler_out.txt: Intel + CrayMPI on Cray XE6 >>> Results: 4 / 8 / 8 >>> BrianSmith-IBM_outlog-bgp-xl.txt: Intel + IBM-XL on BGP >>> Results: 4 / 8 / 8 >>> BrianSmith-IBM_outlog-xlf.txt: xlf + IBM-XL on ? >>> Results: 4 / 8 / 8 >>> >>> C) No problems because all DOUBLE PRECISION alignments >>> are identical = 8 >>> >>> BillLong_Cray_5compiler_out.txt: PGI + CrayMPI on Cray XE6 >>> Results: 8 / 8 / 8 >>> BillLong_Cray_5compiler_out.txt: Cray + CrayMPI on Cray XE6 >>> Results: 8 / 8 / 8 >>> BillLong_Cray_5compiler_out.txt: gfortran + CrayMPI on Cray XE6 >>> Results: 8 / 8 / 8 >>> BillLong_Cray_5compiler_out.txt: pathscale + CrayMPI on Cray XE6 >>> Results: 8 / 8 / 8 >>> BrianSmith-IBM_outlog-gcc.txt: gfortran + IBM-XL on BGQ >>> Results: 8 / 8 / 8 >>> JeffSquyres_OpenMP_linux-x86-64-gfortran.txt: gfortran + OpenMPI on x86-64 >>> Results: 8 / 8 / 8 >>> JeffSquyres_OpenMP_osx-x86-84-gfortran.txt: gfortran + OpenMPI on osx-x86-64 >>> Results: 8 / 8 / 8 >>> Rajeev_gnu_laptop_out.txt: gfortran + mpich2 on laptop >>> Results: 8 / 8 / 8 >>> Rajeev_mpich_out.gfortran.x86-64.txt: gfortran + mpich2 on x86-64 >>> Results: 8 / 8 / 8 >>> >>> D) No problems because all DOUBLE PRECISION alignments are >>> identical = 4 >>> >>> Rajeev_mpich_out.gfortran.ia32.txt: gfortran + mpich2 on ia32 >>> Results: 4 / 4 / 4 >>> Rajeev_mpich_out.ifort.ia32.txt: Intel + mpich2 on ia32 >>> Results: 4 / 4 / 4 >>> >>> Proposal: >>> --------- >>> >>> The goal of a standard is the possibility to write portable code. >>> Therefore the answer should be standardized and not implementation- >>> specific, or in other words, not defining it is the worst solution. >>> >>> I expect only a few Fortran codes are using arrays of derived types >>> in Fortran. >>> The oldest library is mpich. >>> Therefore I would propose to standardize Fortran alignments are >>> based >>> on BIND(C) which should be the same as in C. >>> >>> This would imply that IBM-XL and mpich2 and derivatives need no >>> change. >>> But it also implies that the younger OpenMPI and Microsoft-MPI >>> need a change. These implementation may add an environment switch >>> to keep the old behavior if this switch is set, >>> but I would expect that there is nearly no user >>> who needs or want to use this switch. >>> >>> Fab and Jeff, could you live with this proposal? >>> >>> Best regards >>> Rolf >>> >>> PS: Additional remarks: >>> Yes, some compiler detect that DOUBLE PR. on 4 byte alignment is >>> inefficient, but this is only an information. Other compilers >>> detect, that we use DOUBLE PRECISION in BIND(C) derived types >>> instead of using the C double through these new intrinsic C >>> types in >>> Fortran. This is also okay and all compiler work as expected, >>> they >>> use the C alignment. >>> >>> >>>> Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 09:14:33 >>>> >>>>> In my output from Cray, the compilers >>>>> - gnu >>>>> - pgi >>>>> - cray >>>>> all use 8 byte double alignment in C and Fortran. >>>>> ==> no problem with those compilers on the tested platform. >>>>> Maybe we have differences between IA64 and 32 architectures >>>>> for the same compiler. >>>>> >>>>> Important is to check MPI with those compilers that have >>>>> Fortran 4 byte double precision alignment (e.g. Intel). >>>>> >>>>> Best regards >>>>> Rolf >>>>> >>> >>>>>> On Sep 27, 2011, at 10:21 AM, Rolf Rabenseifner wrote: >>>>>> >>>>>>> Dear all, (now with attachment) >>>>>>> >>>>>>> as far as I know, you represent some MPI libraries that are >>>>>>> not directly based on another one in the list: >>>>>>> - mpich2: Rajeev >>>>>>> - OpenMPI: Jeff >>>>>>> - IBM: Rich Treumann >>>>>>> - NEC: Hubert >>>>>>> - Fujitsu: Shinji Sumimoto >>>>>>> - Microsoft: Fab >>>>>>> (Which independent library is missing on this list?) >>>>>>> (Who of the list uses already mich2 or OpenMPI as bases >>>>>>> or does not provide a Fortran binding, >>>>>>> and can be therefore removed from this list?) >>>>>>> >>>>>>> The problem is simple and has at the end only three numbers: >>>>>>> >>>>>>> Alignments of: >>>>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. >>>>>>> with >>>>>>> Intel) >>>>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 >>>>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 >>>>>>> The details: >>>>>>> Which is the alignment of Fortran DOUBLE PRECISION according to >>>>>>> k_i on MPI-2.2, page 78 line 45 and page 96 line 42. >>>>>>> >>>>>>> Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 >>>>>>> (Fortran >>>>>>> 90) >>>>>>> in Fortran SEQUENCE derived types (which have the same memory >>>>>>> layout >>>>>>> according to the SEQUENCE-rules), the MPI alignment and the >>>>>>> Fortran >>>>>>> alignment in such constructs should be the same, e.g., the Intel >>>>>>> compiler has alignment 4 !!! >>>>>>> >>>>>>> This may be significantly different to C, where Intel uses >>>>>>> alignment >>>>>>> 8! >>>>>>> >>>>>>> With Fortran 2003, we got the Fortran BIND(C) derived types. >>>>>>> Here, a Fortran DOUBLE PRECISION has with Intel's compiler >>>>>>> the alignment 8. >>>>>>> >>>>>>> As far as I understand, mpich2 and Cray produce following >>>>>>> results >>>>>>> with >>>>>>> an Intel compiler: >>>>>>> >>>>>>> Alignments of: >>>>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 >>>>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 >>>>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 >>>>>>> >>>>>>> This is definitely wrong compared to MPI-1.1 and MPI-2.0 >>>>>>> but may be helpful for MPI-3.0 when we like to switch >>>>>>> to a definition that is based on BIND(C). >>>>>>> >>>>>>> Question: >>>>>>> What is the output of the attached test program when running >>>>>>> with >>>>>>> - your MPI library >>>>>>> - and your compiler >>>>>>> - and with Intel's compiler >>>>>>> >>>>>>> Please can you send me the output of such mpiruns? >>>>>>> >>>>>>> Thanks in advance and best regards >>>>>>> Rolf >>>>>>> >>>>>>> PS: We need not to care about 8/8/8. >>>>>>> Only 4/8/x causes problems: >>>>>>> If all those have x=8 then I'll propose to >>>>>>> adopt the MPI-3.0 to the reality. >>>>>>> If we have some implementations with 4/8/8 and >>>>>>> others with 4/8/4 then we have a serious problem >>>>>>> because only one answer can be correct, >>>>>>> and the question would be whether the historical >>>>>>> answer (x=4) or the modern answer (x=8) >>>>>>> should be chosen. >>> >>> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Supalov, Alexander 28 Sep '11

28 Sep '11

Hi, Why don't we leave this to the implementation and the compiler to decide? Our testing results (see attachment) are all over the map on the platforms we currently support. I'd not like to deal w/ any of this specifically in the future product. What happens is platform specific and looks acceptable to the users. And the standard example may just show one possible resolution. Why bother? Best regards. Alexander -----Original Message----- From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Rolf Rabenseifner Sent: Wednesday, September 28, 2011 9:14 AM To: MPI-3 Fortran working group Subject: Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION Fab and all, your idea seems optimal, but ... (see later)!!!! In detail: We have the problem that the Intel ***Fortran*** compiler answers with two different values a=4 and b=8. Both are Fortran values, a=4 for Fortran SEQUENCE derived types and b=8 for Fortran BIND(C) derived types. (Fortran non-sequence and non-bind(C) derived types are not really supported by MPI). The MPI Forum has to decide which value should be used in the example on page 581 of https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/229/mpi-repo… i.e., whether Alternative 1 or 2 is the correct one. Your proposal (I name it Alternative 3) is that the user has to choose the mapped C basic type. This is a non-trivial mapping. For MPI_INTEGER it is MPI_INT or MPI_LONG? For MPI_INTEGER8 it is MPI_LONG_LONG? What are we doing with the unnamed predefined datatypes, produced by MPI_TYPE_CREATE_F90_COMPLEX / _REAL / _INTEGER? But your proposal inspires me to propose an Alternative 4.1 and 4.2: We need a routine that converts a given Fortran basic named or unnamed datatype with 4.1) by default "SEQUENCE alignment" into an unnamed predefined datatype with "BIND(C) alignment"; this works together with Alternative 1 for page 581. MPI_F2003_CONVERT_SEQ_TO_BIND_C(IN seq_datatype, OUT bind_c_datatype) 4.2) by default "BIND(C) alignment" into an unnamed predefined datatype with "SEQUENCE alignment"; this works together with Alternative 2 for page 581. MPI_F2003_CONVERT_BIND_C_TO_SEQ(IN bind_c_datatype, OUT seq_datatype) Best regards Rolf ----- Original Message ----- > From: "Fab Tillier" <ftillier(a)microsoft.com> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji Sumimoto" > <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Howard Pritchard" <howardp(a)cray.com>, > "Brian Smith" <smithbr(a)us.ibm.com>, "Charles J Archer" <archerc(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, > "Bill Long" <longb(a)cray.com>, "Bill Gropp" <wgropp(a)uiuc.edu>, "Richard Graham" <rlgraham(a)ornl.gov> > Cc: "N.M. Maclaren" <nmm1(a)cam.ac.uk>, "Iain Bason" <iain.bason(a)oracle.com> > Sent: Wednesday, September 28, 2011 12:35:53 AM > Subject: RE: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION > Hi Rolf, > > It seems that all the a == 4 results come from using the Intel Fortran > compiler suite. Given that Microsoft doesn't ship a Fortran compiler, > mandating a == 8 doesn't put us in a position to succeed. Therefore, I > cannot support such a proposal. > > however, shouldn't a DOUBLE PRECISION BIND(C) derived type map to > MPI_DOUBLE, not MPI_DOUBLE_PRECISION? > Alternative 3: > Can't we simply document that MPI_DOUBLE_PRECISION (and any Fortran > types that need similar treatment) apply only to SEQUENCE derived > types, while BIND(C) derived types should use the C types? > > -Fab > > Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 14:38:45 > > > Dear all, > > > > thank you very much for the fast answers. > > The result is maximal bad, > > i.e., about 50% of the MPIs use SEQUENCE alignments and other > > about 50% use BIND(C) based calculation of the alignment. > > > > Here the results (also attached as summary_out.txt). > > Please read carefully my proposal after this summary sheet. > > > > In all protocols: > > ----------------- > > Size of one Fortran REAL = 4 > > Size of one Fortran DOUBLE PRECISION = 8 > > Results: > > -------- > > > > a / b / c = Alignments of: > > - DOUBLE PRECISION within a SEQUENCE derived type = a > > - DOUBLE PRECISION within a BIND(C) derived type = b > > - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = c > > > > A) The 4 / 8 / 4 group = alignment of MPI_DOUBLE_PRECISION is > > correct for (old-style) SEQUENCE derived types > > > > JeffSquyres_OpenMP_linux-x86-64-ifort.txt: Intel + OpenMPI on x86-64 > > Results: 4 / 8 / 4 > > FabTillier_Intel+MicrosoftMPI_out.txt: Intel + MS-MPI on ? > > Results: 4 / 8 / 4 > > > > B) The 4 / 8 / 8 group = alignment of MPI_DOUBLE_PRECISION > > is correct for (modern) BIND(C) derived types > > > > Rajeev_mpich_out.ifort.x86-64.txt: Intel + mpich2 on x86-64 > > Results: 4 / 8 / 8 > > Rolf_asama+cray_output.txt: Intel + NEC-MPI? on asama > > Results: 4 / 8 / 8 > > BillLong_Cray_5compiler_out.txt: Intel + CrayMPI on Cray XE6 > > Results: 4 / 8 / 8 > > BrianSmith-IBM_outlog-bgp-xl.txt: Intel + IBM-XL on BGP > > Results: 4 / 8 / 8 > > BrianSmith-IBM_outlog-xlf.txt: xlf + IBM-XL on ? > > Results: 4 / 8 / 8 > > > > C) No problems because all DOUBLE PRECISION alignments > > are identical = 8 > > > > BillLong_Cray_5compiler_out.txt: PGI + CrayMPI on Cray XE6 > > Results: 8 / 8 / 8 > > BillLong_Cray_5compiler_out.txt: Cray + CrayMPI on Cray XE6 > > Results: 8 / 8 / 8 > > BillLong_Cray_5compiler_out.txt: gfortran + CrayMPI on Cray XE6 > > Results: 8 / 8 / 8 > > BillLong_Cray_5compiler_out.txt: pathscale + CrayMPI on Cray XE6 > > Results: 8 / 8 / 8 > > BrianSmith-IBM_outlog-gcc.txt: gfortran + IBM-XL on BGQ > > Results: 8 / 8 / 8 > > JeffSquyres_OpenMP_linux-x86-64-gfortran.txt: gfortran + OpenMPI on x86-64 > > Results: 8 / 8 / 8 > > JeffSquyres_OpenMP_osx-x86-84-gfortran.txt: gfortran + OpenMPI on osx-x86-64 > > Results: 8 / 8 / 8 > > Rajeev_gnu_laptop_out.txt: gfortran + mpich2 on laptop > > Results: 8 / 8 / 8 > > Rajeev_mpich_out.gfortran.x86-64.txt: gfortran + mpich2 on x86-64 > > Results: 8 / 8 / 8 > > > > D) No problems because all DOUBLE PRECISION alignments are > > identical = 4 > > > > Rajeev_mpich_out.gfortran.ia32.txt: gfortran + mpich2 on ia32 > > Results: 4 / 4 / 4 > > Rajeev_mpich_out.ifort.ia32.txt: Intel + mpich2 on ia32 > > Results: 4 / 4 / 4 > > > > Proposal: > > --------- > > > > The goal of a standard is the possibility to write portable code. > > Therefore the answer should be standardized and not implementation- > > specific, or in other words, not defining it is the worst solution. > > > > I expect only a few Fortran codes are using arrays of derived types > > in Fortran. > > The oldest library is mpich. > > Therefore I would propose to standardize Fortran alignments are > > based > > on BIND(C) which should be the same as in C. > > > > This would imply that IBM-XL and mpich2 and derivatives need no > > change. > > But it also implies that the younger OpenMPI and Microsoft-MPI > > need a change. These implementation may add an environment switch > > to keep the old behavior if this switch is set, > > but I would expect that there is nearly no user > > who needs or want to use this switch. > > > > Fab and Jeff, could you live with this proposal? > > > > Best regards > > Rolf > > > > PS: Additional remarks: > > Yes, some compiler detect that DOUBLE PR. on 4 byte alignment is > > inefficient, but this is only an information. Other compilers > > detect, that we use DOUBLE PRECISION in BIND(C) derived types > > instead of using the C double through these new intrinsic C > > types in > > Fortran. This is also okay and all compiler work as expected, > > they > > use the C alignment. > > > > > >> Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 09:14:33 > >> > >>> In my output from Cray, the compilers > >>> - gnu > >>> - pgi > >>> - cray > >>> all use 8 byte double alignment in C and Fortran. > >>> ==> no problem with those compilers on the tested platform. > >>> Maybe we have differences between IA64 and 32 architectures > >>> for the same compiler. > >>> > >>> Important is to check MPI with those compilers that have > >>> Fortran 4 byte double precision alignment (e.g. Intel). > >>> > >>> Best regards > >>> Rolf > >>> > > > >>>> On Sep 27, 2011, at 10:21 AM, Rolf Rabenseifner wrote: > >>>> > >>>>> Dear all, (now with attachment) > >>>>> > >>>>> as far as I know, you represent some MPI libraries that are > >>>>> not directly based on another one in the list: > >>>>> - mpich2: Rajeev > >>>>> - OpenMPI: Jeff > >>>>> - IBM: Rich Treumann > >>>>> - NEC: Hubert > >>>>> - Fujitsu: Shinji Sumimoto > >>>>> - Microsoft: Fab > >>>>> (Which independent library is missing on this list?) > >>>>> (Who of the list uses already mich2 or OpenMPI as bases > >>>>> or does not provide a Fortran binding, > >>>>> and can be therefore removed from this list?) > >>>>> > >>>>> The problem is simple and has at the end only three numbers: > >>>>> > >>>>> Alignments of: > >>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. > >>>>> with > >>>>> Intel) > >>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > >>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 > >>>>> The details: > >>>>> Which is the alignment of Fortran DOUBLE PRECISION according to > >>>>> k_i on MPI-2.2, page 78 line 45 and page 96 line 42. > >>>>> > >>>>> Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 > >>>>> (Fortran > >>>>> 90) > >>>>> in Fortran SEQUENCE derived types (which have the same memory > >>>>> layout > >>>>> according to the SEQUENCE-rules), the MPI alignment and the > >>>>> Fortran > >>>>> alignment in such constructs should be the same, e.g., the Intel > >>>>> compiler has alignment 4 !!! > >>>>> > >>>>> This may be significantly different to C, where Intel uses > >>>>> alignment > >>>>> 8! > >>>>> > >>>>> With Fortran 2003, we got the Fortran BIND(C) derived types. > >>>>> Here, a Fortran DOUBLE PRECISION has with Intel's compiler > >>>>> the alignment 8. > >>>>> > >>>>> As far as I understand, mpich2 and Cray produce following > >>>>> results > >>>>> with > >>>>> an Intel compiler: > >>>>> > >>>>> Alignments of: > >>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 > >>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > >>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 > >>>>> > >>>>> This is definitely wrong compared to MPI-1.1 and MPI-2.0 > >>>>> but may be helpful for MPI-3.0 when we like to switch > >>>>> to a definition that is based on BIND(C). > >>>>> > >>>>> Question: > >>>>> What is the output of the attached test program when running > >>>>> with > >>>>> - your MPI library > >>>>> - and your compiler > >>>>> - and with Intel's compiler > >>>>> > >>>>> Please can you send me the output of such mpiruns? > >>>>> > >>>>> Thanks in advance and best regards > >>>>> Rolf > >>>>> > >>>>> PS: We need not to care about 8/8/8. > >>>>> Only 4/8/x causes problems: > >>>>> If all those have x=8 then I'll propose to > >>>>> adopt the MPI-3.0 to the reality. > >>>>> If we have some implementations with 4/8/8 and > >>>>> others with 4/8/4 then we have a serious problem > >>>>> because only one answer can be correct, > >>>>> and the question would be whether the historical > >>>>> answer (x=4) or the modern answer (x=8) > >>>>> should be chosen. > > > > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) _______________________________________________ mpi3-fortran mailing list mpi3-fortran(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [MPI3 Fortran] Serious problem with alignment of MPI_DOUBLE_PRECISION
by N.M. Maclaren 28 Sep '11

28 Sep '11

On Sep 28 2011, William Gropp wrote: > >This entire problem comes from the natural but mistaken idea that the >MPI library should assign some default padding when creating a >datatype. This is a bad idea because the alignment rules can depend >on compiler switches or other compiler-specific choices; for example, >some compilers allow the user to select between always 8 byte >alignment, leading to structures that might have gaps but permit the >uniform use of faster load operations or 4 byte alignment, but at a >potential performance cost. In other words, even for a single >compiler, there need not be a unique choice of extent, and hence this >problem is insoluble. Yes. In fact, for Fortran derived types other than SEQUENCE and BIND(C) and for all C structs, there is no reason for the rules to be the same for every type even in a single file and compilation. The only cases where the rules must be the same for C are for compatible structs and ones with a common initial sequence. > Further, it is not necessary - the user can >either create the datatype with the appropriate extent (by using >MPI_UB) or resize it appropriately (with MPI_TYPE_CREATE_RESIZED since >MPI 2.0). Rather than continue to get this wrong, I recommend simply >telling the user not to assume that the MPI library can always divine >how the compiler is going to chose to pad a derived type or struct. >We can add an advice to users here that explains the problem and >recommends using the existing MPI features to handle this problem. >This is better coding and will affect few users. It assuredly avoids MPI getting tangled up in a particularly foul area of the languages more than is necessary, and would help to future-proof the specification! As far as users are concerned, my limited survey of open-source scientific codes found no examples that would be affected, and I have never seen a user code that would, either. However, that's merely my experience, and I don't claim that it is representative of all MPI communities. The issue of Fortran DOUBLE PRECISION versus C double in simple arrays remains, but Fab Tillier has described what the Fortran standard itself says about the matter. Regards, Nick Maclaren.

1 0

Re: [MPI3 Fortran] Serious problem with alignment of MPI_DOUBLE_PRECISION
by William Gropp 28 Sep '11

28 Sep '11

This entire problem comes from the natural but mistaken idea that the MPI library should assign some default padding when creating a datatype. This is a bad idea because the alignment rules can depend on compiler switches or other compiler-specific choices; for example, some compilers allow the user to select between always 8 byte alignment, leading to structures that might have gaps but permit the uniform use of faster load operations or 4 byte alignment, but at a potential performance cost. In other words, even for a single compiler, there need not be a unique choice of extent, and hence this problem is insoluble. Further, it is not necessary - the user can either create the datatype with the appropriate extent (by using MPI_UB) or resize it appropriately (with MPI_TYPE_CREATE_RESIZED since MPI 2.0). Rather than continue to get this wrong, I recommend simply telling the user not to assume that the MPI library can always divine how the compiler is going to chose to pad a derived type or struct. We can add an advice to users here that explains the problem and recommends using the existing MPI features to handle this problem. This is better coding and will affect few users. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rolf Rabenseifner 28 Sep '11

28 Sep '11

Fab and all, your idea seems optimal, but ... (see later)!!!! In detail: We have the problem that the Intel ***Fortran*** compiler answers with two different values a=4 and b=8. Both are Fortran values, a=4 for Fortran SEQUENCE derived types and b=8 for Fortran BIND(C) derived types. (Fortran non-sequence and non-bind(C) derived types are not really supported by MPI). The MPI Forum has to decide which value should be used in the example on page 581 of https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/229/mpi-repo… i.e., whether Alternative 1 or 2 is the correct one. Your proposal (I name it Alternative 3) is that the user has to choose the mapped C basic type. This is a non-trivial mapping. For MPI_INTEGER it is MPI_INT or MPI_LONG? For MPI_INTEGER8 it is MPI_LONG_LONG? What are we doing with the unnamed predefined datatypes, produced by MPI_TYPE_CREATE_F90_COMPLEX / _REAL / _INTEGER? But your proposal inspires me to propose an Alternative 4.1 and 4.2: We need a routine that converts a given Fortran basic named or unnamed datatype with 4.1) by default "SEQUENCE alignment" into an unnamed predefined datatype with "BIND(C) alignment"; this works together with Alternative 1 for page 581. MPI_F2003_CONVERT_SEQ_TO_BIND_C(IN seq_datatype, OUT bind_c_datatype) 4.2) by default "BIND(C) alignment" into an unnamed predefined datatype with "SEQUENCE alignment"; this works together with Alternative 2 for page 581. MPI_F2003_CONVERT_BIND_C_TO_SEQ(IN bind_c_datatype, OUT seq_datatype) Best regards Rolf ----- Original Message ----- > From: "Fab Tillier" <ftillier(a)microsoft.com> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji Sumimoto" > <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" <hubert.ritzdorf(a)emea.nec.com>, "Howard Pritchard" <howardp(a)cray.com>, > "Brian Smith" <smithbr(a)us.ibm.com>, "Charles J Archer" <archerc(a)us.ibm.com>, "Rajeev Thakur" <thakur(a)mcs.anl.gov>, > "Bill Long" <longb(a)cray.com>, "Bill Gropp" <wgropp(a)uiuc.edu>, "Richard Graham" <rlgraham(a)ornl.gov> > Cc: "N.M. Maclaren" <nmm1(a)cam.ac.uk>, "Iain Bason" <iain.bason(a)oracle.com> > Sent: Wednesday, September 28, 2011 12:35:53 AM > Subject: RE: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION > Hi Rolf, > > It seems that all the a == 4 results come from using the Intel Fortran > compiler suite. Given that Microsoft doesn't ship a Fortran compiler, > mandating a == 8 doesn't put us in a position to succeed. Therefore, I > cannot support such a proposal. > > however, shouldn't a DOUBLE PRECISION BIND(C) derived type map to > MPI_DOUBLE, not MPI_DOUBLE_PRECISION? > Alternative 3: > Can't we simply document that MPI_DOUBLE_PRECISION (and any Fortran > types that need similar treatment) apply only to SEQUENCE derived > types, while BIND(C) derived types should use the C types? > > -Fab > > Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 14:38:45 > > > Dear all, > > > > thank you very much for the fast answers. > > The result is maximal bad, > > i.e., about 50% of the MPIs use SEQUENCE alignments and other > > about 50% use BIND(C) based calculation of the alignment. > > > > Here the results (also attached as summary_out.txt). > > Please read carefully my proposal after this summary sheet. > > > > In all protocols: > > ----------------- > > Size of one Fortran REAL = 4 > > Size of one Fortran DOUBLE PRECISION = 8 > > Results: > > -------- > > > > a / b / c = Alignments of: > > - DOUBLE PRECISION within a SEQUENCE derived type = a > > - DOUBLE PRECISION within a BIND(C) derived type = b > > - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = c > > > > A) The 4 / 8 / 4 group = alignment of MPI_DOUBLE_PRECISION is > > correct for (old-style) SEQUENCE derived types > > > > JeffSquyres_OpenMP_linux-x86-64-ifort.txt: Intel + OpenMPI on x86-64 > > Results: 4 / 8 / 4 > > FabTillier_Intel+MicrosoftMPI_out.txt: Intel + MS-MPI on ? > > Results: 4 / 8 / 4 > > > > B) The 4 / 8 / 8 group = alignment of MPI_DOUBLE_PRECISION > > is correct for (modern) BIND(C) derived types > > > > Rajeev_mpich_out.ifort.x86-64.txt: Intel + mpich2 on x86-64 > > Results: 4 / 8 / 8 > > Rolf_asama+cray_output.txt: Intel + NEC-MPI? on asama > > Results: 4 / 8 / 8 > > BillLong_Cray_5compiler_out.txt: Intel + CrayMPI on Cray XE6 > > Results: 4 / 8 / 8 > > BrianSmith-IBM_outlog-bgp-xl.txt: Intel + IBM-XL on BGP > > Results: 4 / 8 / 8 > > BrianSmith-IBM_outlog-xlf.txt: xlf + IBM-XL on ? > > Results: 4 / 8 / 8 > > > > C) No problems because all DOUBLE PRECISION alignments > > are identical = 8 > > > > BillLong_Cray_5compiler_out.txt: PGI + CrayMPI on Cray XE6 > > Results: 8 / 8 / 8 > > BillLong_Cray_5compiler_out.txt: Cray + CrayMPI on Cray XE6 > > Results: 8 / 8 / 8 > > BillLong_Cray_5compiler_out.txt: gfortran + CrayMPI on Cray XE6 > > Results: 8 / 8 / 8 > > BillLong_Cray_5compiler_out.txt: pathscale + CrayMPI on Cray XE6 > > Results: 8 / 8 / 8 > > BrianSmith-IBM_outlog-gcc.txt: gfortran + IBM-XL on BGQ > > Results: 8 / 8 / 8 > > JeffSquyres_OpenMP_linux-x86-64-gfortran.txt: gfortran + OpenMPI on x86-64 > > Results: 8 / 8 / 8 > > JeffSquyres_OpenMP_osx-x86-84-gfortran.txt: gfortran + OpenMPI on osx-x86-64 > > Results: 8 / 8 / 8 > > Rajeev_gnu_laptop_out.txt: gfortran + mpich2 on laptop > > Results: 8 / 8 / 8 > > Rajeev_mpich_out.gfortran.x86-64.txt: gfortran + mpich2 on x86-64 > > Results: 8 / 8 / 8 > > > > D) No problems because all DOUBLE PRECISION alignments are > > identical = 4 > > > > Rajeev_mpich_out.gfortran.ia32.txt: gfortran + mpich2 on ia32 > > Results: 4 / 4 / 4 > > Rajeev_mpich_out.ifort.ia32.txt: Intel + mpich2 on ia32 > > Results: 4 / 4 / 4 > > > > Proposal: > > --------- > > > > The goal of a standard is the possibility to write portable code. > > Therefore the answer should be standardized and not implementation- > > specific, or in other words, not defining it is the worst solution. > > > > I expect only a few Fortran codes are using arrays of derived types > > in Fortran. > > The oldest library is mpich. > > Therefore I would propose to standardize Fortran alignments are > > based > > on BIND(C) which should be the same as in C. > > > > This would imply that IBM-XL and mpich2 and derivatives need no > > change. > > But it also implies that the younger OpenMPI and Microsoft-MPI > > need a change. These implementation may add an environment switch > > to keep the old behavior if this switch is set, > > but I would expect that there is nearly no user > > who needs or want to use this switch. > > > > Fab and Jeff, could you live with this proposal? > > > > Best regards > > Rolf > > > > PS: Additional remarks: > > Yes, some compiler detect that DOUBLE PR. on 4 byte alignment is > > inefficient, but this is only an information. Other compilers > > detect, that we use DOUBLE PRECISION in BIND(C) derived types > > instead of using the C double through these new intrinsic C > > types in > > Fortran. This is also okay and all compiler work as expected, > > they > > use the C alignment. > > > > > >> Rolf Rabenseifner wrote on Tue, 27 Sep 2011 at 09:14:33 > >> > >>> In my output from Cray, the compilers > >>> - gnu > >>> - pgi > >>> - cray > >>> all use 8 byte double alignment in C and Fortran. > >>> ==> no problem with those compilers on the tested platform. > >>> Maybe we have differences between IA64 and 32 architectures > >>> for the same compiler. > >>> > >>> Important is to check MPI with those compilers that have > >>> Fortran 4 byte double precision alignment (e.g. Intel). > >>> > >>> Best regards > >>> Rolf > >>> > > > >>>> On Sep 27, 2011, at 10:21 AM, Rolf Rabenseifner wrote: > >>>> > >>>>> Dear all, (now with attachment) > >>>>> > >>>>> as far as I know, you represent some MPI libraries that are > >>>>> not directly based on another one in the list: > >>>>> - mpich2: Rajeev > >>>>> - OpenMPI: Jeff > >>>>> - IBM: Rich Treumann > >>>>> - NEC: Hubert > >>>>> - Fujitsu: Shinji Sumimoto > >>>>> - Microsoft: Fab > >>>>> (Which independent library is missing on this list?) > >>>>> (Who of the list uses already mich2 or OpenMPI as bases > >>>>> or does not provide a Fortran binding, > >>>>> and can be therefore removed from this list?) > >>>>> > >>>>> The problem is simple and has at the end only three numbers: > >>>>> > >>>>> Alignments of: > >>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. > >>>>> with > >>>>> Intel) > >>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > >>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 > >>>>> The details: > >>>>> Which is the alignment of Fortran DOUBLE PRECISION according to > >>>>> k_i on MPI-2.2, page 78 line 45 and page 96 line 42. > >>>>> > >>>>> Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 > >>>>> (Fortran > >>>>> 90) > >>>>> in Fortran SEQUENCE derived types (which have the same memory > >>>>> layout > >>>>> according to the SEQUENCE-rules), the MPI alignment and the > >>>>> Fortran > >>>>> alignment in such constructs should be the same, e.g., the Intel > >>>>> compiler has alignment 4 !!! > >>>>> > >>>>> This may be significantly different to C, where Intel uses > >>>>> alignment > >>>>> 8! > >>>>> > >>>>> With Fortran 2003, we got the Fortran BIND(C) derived types. > >>>>> Here, a Fortran DOUBLE PRECISION has with Intel's compiler > >>>>> the alignment 8. > >>>>> > >>>>> As far as I understand, mpich2 and Cray produce following > >>>>> results > >>>>> with > >>>>> an Intel compiler: > >>>>> > >>>>> Alignments of: > >>>>> - DOUBLE PRECISION within a SEQUENCE derived type = 4 > >>>>> - DOUBLE PRECISION within a BIND(C) derived type = 8 > >>>>> - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 > >>>>> > >>>>> This is definitely wrong compared to MPI-1.1 and MPI-2.0 > >>>>> but may be helpful for MPI-3.0 when we like to switch > >>>>> to a definition that is based on BIND(C). > >>>>> > >>>>> Question: > >>>>> What is the output of the attached test program when running > >>>>> with > >>>>> - your MPI library > >>>>> - and your compiler > >>>>> - and with Intel's compiler > >>>>> > >>>>> Please can you send me the output of such mpiruns? > >>>>> > >>>>> Thanks in advance and best regards > >>>>> Rolf > >>>>> > >>>>> PS: We need not to care about 8/8/8. > >>>>> Only 4/8/x causes problems: > >>>>> If all those have x=8 then I'll propose to > >>>>> adopt the MPI-3.0 to the reality. > >>>>> If we have some implementations with 4/8/8 and > >>>>> others with 4/8/4 then we have a serious problem > >>>>> because only one answer can be correct, > >>>>> and the question would be whether the historical > >>>>> answer (x=4) or the modern answer (x=8) > >>>>> should be chosen. > > > > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
by Rolf Rabenseifner 28 Sep '11

28 Sep '11

For the case that somebody missed the email, here again the attachment mpi_alignment.f90 ----- Original Message ----- > From: "Alexander Supalov" <alexander.supalov(a)intel.com> > To: "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> > Sent: Wednesday, September 28, 2011 5:37:21 AM > > Subject: Re: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION > PS. Please redistribute the test program that was contained in Rolf's > original email. > > -----Original Message----- > From: Supalov, Alexander > Sent: Wednesday, September 28, 2011 12:21 AM > To: MPI-3 Fortran working group > Subject: RE: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries > with the alignment of MPI_DOUBLE_PRECISION > Importance: High > > Hi, > > I'm surprised to see no Intel MPI on the list below. Both Intel MPI > and MS MPI are based on MPICH2. Please add me (for Intel MPI) to this > kind of announcement from now on. We also make Intel compiler, so it > makes a lot of sense to ask at the source rather than elsewhere. > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-fortran-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Rolf > Rabenseifner > Sent: Tuesday, September 27, 2011 5:22 PM > To: MPI-3 Fortran working group > Subject: [MPI3 Fortran] Fwd: Serious problem/bug in MPI libraries with > the alignment of MPI_DOUBLE_PRECISION > > For your info: > I informed the providers of independent MPI libraries with Fortran > bindings about a serious problem with the definition of Fortran > datatypes. > > My apologies if I missed some of the independent MPI libraries. > > In this case, please also run the program and please send the result > to me. > > Best regards > Rolf > > ----- Forwarded Message ----- > From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > To: "Richard Treumann" <treumann(a)us.ibm.com>, "Rajeev Thakur" > <thakur(a)mcs.anl.gov>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Shinji > Sumimoto" <s-sumi(a)labs.fujitsu.com>, "Hubert Ritzdorf" > <hubert.ritzdorf(a)emea.nec.com>, "Fab Tillier" <ftillier(a)microsoft.com> > Sent: Tuesday, September 27, 2011 5:15:15 PM > Subject: Serious problem/bug in MPI libraries with the alignment of > MPI_DOUBLE_PRECISION > > Dear all, > > as far as I know, you represent some MPI libraries that are not > directly based on another one in the list: > - mpich2: Rajeev > - OpenMPI: Jeff > - IBM: Rich Treumann > - NEC: Hubert > - Fujitsu: Shinji Sumimoto > - Microsoft: Fab > (Which independent library is missing on this list?) (Who of the list > uses already mich2 or OpenMPI as bases or does not provide a Fortran > binding, and can be therefore removed from this list?) > > The problem is simple and has at the end only three numbers: > > Aligments of: > - DOUBLE PRECISION within a SEQUENCE derived type = 4 (e.g. with > Intel) > - DOUBLE PRECISION within a BIND(C) derived type = 8 > - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 4 or 8 > > The details: > Which is the alignment of Fortran DOUBLE PRECISION according to k_i on > MPI-2.2, page 78 line 45 and page 96 line 42. > > Since 1977 (Fortran 77) in Fortran COMMON blocks and 1990 (Fortran 90) > in Fortran SEQUENCE derived types (which have the same memory layout > according to the SEQUENCE-rules), the MPI alignment and the Fortran > alignment in such constructs should be the same, e.g., the Intel > compiler has alignment 4 !!! > > This may be significantly different to C, where Intel uses alignment > 8! > > With Fortran 2003, we got the Fortran BIND(C) derived types. > Here, a Fortran DOUBLE PRECISION has with Intel's compiler the > aligment 8. > > As far as I understand, mpich2 and Cray produce following results with > an Intel compiler: > > Aligments of: > - DOUBLE PRECISION within a SEQUENCE derived type = 4 > - DOUBLE PRECISION within a BIND(C) derived type = 8 > - MPI_DOUBLE_PRECISION (= k_i in MPI-2.2 p.78:45) = 8 > > This is definitely wrong compared to MPI-1.1 and MPI-2.0 but may be > helpful for MPI-3.0 when we like to switch to a definition that is > based on BIND(C). > > Question: > What is the output of the attached test program when running with > - your MPI library > - and your compiler > - and with Intel's compiler > > Please can you send me the output of such mpiruns? > > Thanks in advance and best regards > Rolf > > PS: We need not to care about 8/8/8. > Only 4/8/x causes problems: > If all those have x=8 then I'll propose to > adopt the MPI-3.0 to the reality. > If we have some implementations with 4/8/8 and > others with 4/8/4 then we have a serious problem > because only one answer can be correct, > and the question would be whether the historical > answer (x=4) or the modern answer (x=8) > should be chosen. > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > -------------------------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen, Deutschland > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 > Ust.-IdNr./VAT Registration No.: DE129385895 > Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0