December 2013 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] ABI
by Jeff Squyres (jsquyres) 05 Dec '13

05 Dec '13

I think there was one other key takeaway from the ABI WG: it would seem like a good public solution would be to develop an open source morph MPI layer that can talk to any MPI implementation on the back-end. A few people have tried this over the years with moderate success, but I don't think that anyone has fully implemented MPI3 and pushed it to the entire HPC community. So none of them have gotten too widely adopted. My $0.02 is that a public, free, open source Morph MPI layer with an active community behind it would still be a very good starting point for whatever could be achieved via some kind of binary compatibility. On Dec 5, 2013, at 12:42 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > "Supalov, Alexander" <alexander.supalov(a)intel.com> writes: >> I am afraid I cannot comment on particular snippets of other people's >> email 5-6 year after the events. If you wish, we can get this trail >> offline and go through the material with you. > > No worries. I'm happy to discuss further, though this list probably > isn't the best place. Are the interested parties subscribed to > discuss(a)mpich.org? > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] ABI
by Jed Brown 05 Dec '13

05 Dec '13

"Supalov, Alexander" <alexander.supalov(a)intel.com> writes: > I am afraid I cannot comment on particular snippets of other people's > email 5-6 year after the events. If you wish, we can get this trail > offline and go through the material with you. No worries. I'm happy to discuss further, though this list probably isn't the best place. Are the interested parties subscribed to discuss(a)mpich.org?

1 0

Re: [Mpi-forum] ABI
by Supalov, Alexander 05 Dec '13

05 Dec '13

I think that is about all that we had. Jeff Brown, who chaired the WG, as far as I can recall, might have more material, as probably do I in my private archive. Anyway, the point here is to re-merge that WG's work with the reignited discussion rather than to replace it. The situation then was that after the initial assessment, we could not detect enough enthusiasm on the part of the forum and implementations to get any closer than they were before. The ABI WG then went dormant. I hope that the situation changed somewhat now that we have a joint MPICH ABI initiative. I am afraid I cannot comment on particular snippets of other people's email 5-6 year after the events. If you wish, we can get this trail offline and go through the material with you. -----Original Message----- From: Jed Brown [mailto:[email protected]] Sent: Thursday, December 05, 2013 5:43 PM To: Supalov, Alexander; Main MPI Forum mailing list Subject: RE: [Mpi-forum] ABI "Supalov, Alexander" <alexander.supalov(a)intel.com> writes: > I hope so. Just click on the "Attachments" tab and look into the DOC files there. That just defines some terms and scope, but Analysis, Solution, and Issues are TBD. Then there are various emails, but they are mainly high-level motivation. The pointer vs. integer sizes thing came up in one email, where Jeff Squyres said > For an ABI, you need to standardize both the size and the value. It > may not be difficult to agree on the sizes: sizeof(void*)/64 bits > because you can treat it as an int or a pointer. But the value is > where the religious debates come in. For example: MPI_COMM_WORLD -- > it is "0" or is it a pointer resolved at compile/link time? which I think is a red herring since the value can be a link-time constant and an implementation is free to "optimize" internally. Is there more technical content somewhere? Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [Mpi-forum] ABI
by Jed Brown 05 Dec '13

05 Dec '13

"N.M. Maclaren" <nmm1(a)cam.ac.uk> writes: > Eh? I was saying that you shouldn't bet on pointers being treated the > same as integers in all common calling conventions. I am not familiar > with ALL current calling conventions - and remember that many ports > work from the distributed base of OpenMP and MPICH, so are not within > the platforms in the distributed source. Let's not let the perfect be the enemy of the practical. The simple fact is that there would be a huge benefit to users and packagers if just MPICH and Open MPI had a common ABI on x86/x86-64, and perhaps ARM in the coming years. I never said it made sense to add to the standard. > As I said, this most definitely used NOT to be the case until quite > recently, and I have no evidence that it has changed! Furthermore, > this is NOT purely a hardware matter, and several of the differences I > remember were entirely due to the compiler's choices. Calling convention is the compiler's choice, but if the arithmetic unit is the same for integers and pointers, compilers seem happy to treat them the same. > There is no such concept in the C standard. They are not initialization > constants (the relevant concept), as can be seen by failing to compile: > > typedef int MPI_Datatype; > extern const MPI_Datatype MPI_INT; > static MPI_Datatype weeble = MPI_INT; > > with gcc -std=c99 -pedantic -Wall -Wextra -O3 -c. You're right, I was thinking of pointers to arrays, which can be initialized this way typedef const int *MPI_Datatype; extern const int MPI_INT[1]; static MPI_Datatype weeble = MPI_INT; But this is indeed a different model because it requires a pointer dereference. Hmm. > Eh? I said that there were differences between MPI implementations, and > by that I meant their semantics. Different run-time semantics is a source-level "incompatibility" as well. Many projects deal with that just fine, so why can't the deal at run-time?

1 0

Re: [Mpi-forum] ABI
by Jed Brown 05 Dec '13

05 Dec '13

"Supalov, Alexander" <alexander.supalov(a)intel.com> writes: > I hope so. Just click on the "Attachments" tab and look into the DOC files there. That just defines some terms and scope, but Analysis, Solution, and Issues are TBD. Then there are various emails, but they are mainly high-level motivation. The pointer vs. integer sizes thing came up in one email, where Jeff Squyres said > For an ABI, you need to standardize both the size and the value. It > may not be difficult to agree on the sizes: sizeof(void*)/64 bits > because you can treat it as an int or a pointer. But the value is > where the religious debates come in. For example: MPI_COMM_WORLD -- > it is "0" or is it a pointer resolved at compile/link time? which I think is a red herring since the value can be a link-time constant and an implementation is free to "optimize" internally. Is there more technical content somewhere?

1 0

Re: [Mpi-forum] ABI
by N.M. Maclaren 05 Dec '13

05 Dec '13

On Dec 5 2013, Jed Brown wrote: >>> >>> Suppose MPICH used intptr_t for handles. I believe pointers are treated >>> the same as integers for all the common calling conventions. >> >> I wouldn't bet on it, especially not in the medium term. It's true on >> systems like the x86 (and its predecessor in dominance System/370), but >> I don't know all of the systems that either of those has been ported to. >> It certaainly used not to be the case. > >1. I never suggested having the Forum standardize an ABI. > >2. Why prevent something good just because some vendor some day might >build a CPU that handles pointers in a way that has not been seen on any >popular architecture in decades? Eh? I was saying that you shouldn't bet on pointers being treated the same as integers in all common calling conventions. I am not familiar with ALL current calling conventions - and remember that many ports work from the distributed base of OpenMP and MPICH, so are not within the platforms in the distributed source. As I said, this most definitely used NOT to be the case until quite recently, and I have no evidence that it has changed! Furthermore, this is NOT purely a hardware matter, and several of the differences I remember were entirely due to the compiler's choices. >> The killer there is whether they can be used in initialisations. That >> would break: >> >> static MPI_Datatype weeble = MPI_INT; > >Sorry, both MPICH and OpenMPI's mpi.h would contain: > > extern const MPI_Datatype MPI_INT; > > ... > >So these are link-time constants and the user can write > > static MPI_Datatype weeble = MPI_INT; There is no such concept in the C standard. They are not initialization constants (the relevant concept), as can be seen by failing to compile: typedef int MPI_Datatype; extern const MPI_Datatype MPI_INT; static MPI_Datatype weeble = MPI_INT; with gcc -std=c99 -pedantic -Wall -Wextra -O3 -c. >> Lastly, there are some very nasty differences that would have to be >> tackled, such as: >> >> Attributes (especially the I/O ones). I meant to propose some fixes >> for MPI 3 (the current specification is weird), but didn't do it. Sorry. >> But there are probably quite a lot of such corner-cases. > >Can you explain? Where does the standard require these to be >compile-time constants? Eh? I said that there were differences between MPI implementations, and by that I meant their semantics. >> Implementation assumptions. Not just signals and threading, but >> the handling of stdin/stdout/stderr, passive one-sided and more. > >What does this have to do with the *binary* interface? Because the purpose of binary interfaces is to allow the exchangeability of compiled code. Merely getting programs to link and ensure compatible data formats is not enough - people rather like their programs to actually work! I have been involved in a couple of ABI standardisation, and the semantic issues were much nastier than the simple external name, data type and calling convention ones. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] ABI
by Supalov, Alexander 05 Dec '13

05 Dec '13

I hope so. Just click on the "Attachments" tab and look into the DOC files there. -----Original Message----- From: Jed Brown [mailto:[email protected]] Sent: Thursday, December 05, 2013 5:21 PM To: Supalov, Alexander; Main MPI Forum mailing list Subject: Re: [Mpi-forum] ABI "Supalov, Alexander" <alexander.supalov(a)intel.com> writes: > We looked into this matter in substantial detail back in the times of > ABI WG (pre-MPI-2.1, I think). Details can be found in > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/AbiWikiPage#no1 . Is there much technical content beyond the spreadsheet enumerating differences, a lot of which don't matter (init-time constants)? There also seem to be a lot of items labeled as compile-time constants, for which I cannot find the place in the standard where that is required. Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [Mpi-forum] ABI
by Jed Brown 05 Dec '13

05 Dec '13

"Supalov, Alexander" <alexander.supalov(a)intel.com> writes: > We looked into this matter in substantial detail back in the times of > ABI WG (pre-MPI-2.1, I think). Details can be found in > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/AbiWikiPage#no1 . Is there much technical content beyond the spreadsheet enumerating differences, a lot of which don't matter (init-time constants)? There also seem to be a lot of items labeled as compile-time constants, for which I cannot find the place in the standard where that is required.

1 0

Re: [Mpi-forum] ABI
by Jed Brown 05 Dec '13

05 Dec '13

"N.M. Maclaren" <nmm1(a)cam.ac.uk> writes: >> Suppose MPICH used intptr_t for handles. I believe pointers are treated >> the same as integers for all the common calling conventions. > > I wouldn't bet on it, especially not in the medium term. It's true on > systems like the x86 (and its predecessor in dominance System/370), but > I don't know all of the systems that either of those has been ported to. > It certaainly used not to be the case. 1. I never suggested having the Forum standardize an ABI. 2. Why prevent something good just because some vendor some day might build a CPU that handles pointers in a way that has not been seen on any popular architecture in decades? > The killer there is whether they can be used in initialisations. That > would break: > > static MPI_Datatype weeble = MPI_INT; Sorry, both MPICH and OpenMPI's mpi.h would contain: extern const MPI_Datatype MPI_INT; MPICH's mpi.h would have typedef intptr_t MPI_Datatype; while OMPI's mpi.h would have typedef struct ompi_datatype_t *MPI_Datatype; In some source file going into libmpi.so, MPICH would have const MPI_Datatype MPI_INT = ((MPI_Datatype)0x4c000405); while OMPI would have const MPI_Datatype MPI_INT = ((MPI_Datatype)&ompi_mpi_int); So these are link-time constants and the user can write static MPI_Datatype weeble = MPI_INT; They cannot use MPI_INT as a label in a case statement, but the standard is explicit about that not being necessary (2.5.4 "Named Constants"). All named constants, with the exceptions noted below for Fortran, can be used in initialization expressions or assignments, but not necessarily in array declarations or as labels in C switch or Fortran select/case statements. This implies named constants to be link-time but not necessarily compile-time constants. [...] The constants that are required to be compile-time constants (and can thus be used for array length declarations and labels in C switch and Fortran case/select statements) are: MPI_MAX_PROCESSOR_NAME MPI_MAX_LIBRARY_VERSION_STRING MPI_MAX_ERROR_STRING MPI_MAX_DATAREP_STRING MPI_MAX_INFO_KEY MPI_MAX_INFO_VAL MPI_MAX_OBJECT_NAME MPI_MAX_PORT_NAME MPI_VERSION MPI_SUBVERSION MPI_STATUS_SIZE (Fortran only) MPI_ADDRESS_KIND (Fortran only) MPI_COUNT_KIND (Fortran only) MPI_INTEGER_KIND (Fortran only) MPI_OFFSET_KIND (Fortran only) MPI_SUBARRAYS_SUPPORTED (Fortran only) MPI_ASYNC_PROTECTS_NONBLOCKING (Fortran only) > Lastly, there are some very nasty differences that would have to be > tackled, such as: > > Attributes (especially the I/O ones). I meant to propose some fixes > for MPI 3 (the current specification is weird), but didn't do it. Sorry. > But there are probably quite a lot of such corner-cases. Can you explain? Where does the standard require these to be compile-time constants? > Implementation assumptions. Not just signals and threading, but > the handling of stdin/stdout/stderr, passive one-sided and more. What does this have to do with the *binary* interface?

1 0

Re: [Mpi-forum] ABI
by Supalov, Alexander 05 Dec '13

05 Dec '13

I can check this out. -----Original Message----- From: mpi-forum [mailto:[email protected]] On Behalf Of Jeff Hammond Sent: Thursday, December 05, 2013 2:24 PM To: Main MPI Forum mailing list Subject: Re: [Mpi-forum] ABI Any chance that Intel will issue a patent license for anyone to create a non-commercial, open-source implementation of http://www.google.com/patents/US7966624? Jeff On Thu, Dec 5, 2013 at 4:04 AM, Supalov, Alexander <alexander.supalov(a)intel.com> wrote: > Hi everybody, > > We looked into this matter in substantial detail back in the times of ABI WG (pre-MPI-2.1, I think). Details can be found in https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/AbiWikiPage#no1 . > > Best regards. > > Alexander > > -----Original Message----- > From: mpi-forum [mailto:[email protected]] On > Behalf Of N.M. Maclaren > Sent: Thursday, December 05, 2013 10:44 AM > To: Main MPI Forum mailing list > Subject: Re: [Mpi-forum] ABI > > On Dec 4 2013, Jed Brown wrote: >> >>However, the question of integer (MPICH) versus pointer (OMPI) handles >>is a long-standing impasse. > > Yes, but I don't think that it's the worst issue. > >>Suppose MPICH used intptr_t for handles. I believe pointers are >>treated the same as integers for all the common calling conventions. > > I wouldn't bet on it, especially not in the medium term. It's true on systems like the x86 (and its predecessor in dominance System/370), but I don't know all of the systems that either of those has been ported to. > It certaainly used not to be the case. > >> This >>leaves a question of the predefined constants, which are currently >>compile-time constants of some sort (either integers or addresses of >>privately-defined structs), but I'm not aware of text in the standard >>that requires them to be defined in a header. That is, can we have >> >> extern MPI_Datatype MPI_INT; >> >>in the header and leave it to the library to provide the value? > > The killer there is whether they can be used in initialisations. That would break: > > static MPI_Datatype weeble = MPI_INT; > > Lastly, there are some very nasty differences that would have to be tackled, such as: > > Attributes (especially the I/O ones). I meant to propose some fixes for MPI 3 (the current specification is weird), but didn't do it. Sorry. > But there are probably quite a lot of such corner-cases. > > Implementation assumptions. Not just signals and threading, but the handling of stdin/stdout/stderr, passive one-sided and more. > > This doesn't mean that it can't be done, merely that it isn't as easy as just agreeing the types and values. > > > Regards, > Nick Maclaren. > > > > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: > Feldkirchen bei Muenchen > Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas > Lusk > Registergericht: Muenchen HRB 47456 > Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. > (BLZ 502 109 00) 600119052 > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond jeff.science(a)gmail.com _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0