September 2012 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 21 Sep '12

21 Sep '12

On Fri, Sep 21, 2012 at 10:49 AM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > On the contrary - that is essential for any kind of sane specification > or implementation of MPI_Irecv_reduce, just as it is for one-sided. > Sorry, but that's needed to get even plausible conformance with any of > the languages MPI is likely to be used from. MPI_Recv_reduce doesn't > have the same problems. > > The point is that none of them allow more-or-less arbitrary functions > to be called asynchronously, and that has been horribly sick in every > modern system that I have looked into in any depth. It used to work > on some mainframes, but hasn't worked reliably since. That is precisely > why POSIX has deprecated calling signal handlers asynchronously. Please > don't perpetrate another feature like passive one-sided! > This is totally different than passive one-sided because it has a request and isn't guaranteed to make progress when not in the MPI stack. An implementation using comm threads also need not use interrupts. I haven't heard of any system that supports ALL combinations of built-in ops and datatypes in dedicated hardware, therefore you have some code running in a context where it could call the user-defined MPI_Op. A lot of the controversy seems to come down to not trusting the user to be able to write a pure function that is actually pure. You should realize that in many cases (including the most important ones to me), the MPI_Op is just SUM or MAX, but applied to datatypes that MPI does not cover (e.g. quad precision). The other concern people raised when this issue was last discussed for one-sided was the need for an API allowing collective specification of MPI_Ops. That isn't needed here because the reduction is specified by the receiving rank. > Even doing it for just built-in types is problematic on some systems, > just as it is for one-sided. I doubt that there are many such systems > currently in use outside the embedded arena, but there may be some there, > and they may well return to general computing. > > A much better idea would be to drop MPI_Irecv_reduce and consider just > MPI_Recv_reduce. > I hate this. It will will force the application to poll (or pay the penalty of receiving the messages in a fixed order, which may slow things down more than requiring a copy).

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 21 Sep '12

21 Sep '12

On Fri, Sep 21, 2012 at 8:34 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Sep 20, 2012 at 12:14 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> > wrote: >> >> P2P) >> >> I would like MPI_(I)RECV_REDUCE, which - as you might guess - does a >> reduction to the receive buffer instead of a simple write. This >> allows one to avoid having to manually buffer incoming messages to be >> reduced at the receiver. Torsten and I have discussed it and it seems >> there are at least a few use cases. > > > If you do this, _please_ allow a user-defined MPI_Op to be used in the > reduction (i.e., don't cripple it like one-sided). There's a reason I called it MPI_RECV_REDUCE and not MPI_TWOSIDED_ACCUMULATE :-) All of the reasons that MPI_Accumulate doesn't support active-messages do not apply here so of course user-defined reductions will be supported (if it happens at all). > Jeff Squyres, I don't know what "fix grequest" meant in your list, but I > hope that means: "provide a mechanism for users to implement nonblocking > operations with the same progress semantics as built-in nonblocking > operations". After writing the blog post below, I learned about additional > exemplar use cases in dense linear algebra. Lack of this specific feature is > causing a lot of important applications and libraries to systematically > over-synchronize and preventing them from hiding communication latency. > > https://www.ieeetcsc.org/activities/blog/user_defined_nonblocking_collectiv… > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 21 Sep '12

21 Sep '12

On Sep 21 2012, Jed Brown wrote: > On Thu, Sep 20, 2012 at 12:14 PM, Jeff Hammond > <jhammond(a)alcf.anl.gov>wrote: > >> I would like MPI_(I)RECV_REDUCE, which - as you might guess - does a >> reduction to the receive buffer instead of a simple write. This >> allows one to avoid having to manually buffer incoming messages to be >> reduced at the receiver. Torsten and I have discussed it and it seems >> there are at least a few use cases. > >If you do this, _please_ allow a user-defined MPI_Op to be used in the >reduction (i.e., don't cripple it like one-sided). On the contrary - that is essential for any kind of sane specification or implementation of MPI_Irecv_reduce, just as it is for one-sided. Sorry, but that's needed to get even plausible conformance with any of the languages MPI is likely to be used from. MPI_Recv_reduce doesn't have the same problems. The point is that none of them allow more-or-less arbitrary functions to be called asynchronously, and that has been horribly sick in every modern system that I have looked into in any depth. It used to work on some mainframes, but hasn't worked reliably since. That is precisely why POSIX has deprecated calling signal handlers asynchronously. Please don't perpetrate another feature like passive one-sided! Even doing it for just built-in types is problematic on some systems, just as it is for one-sided. I doubt that there are many such systems currently in use outside the embedded arena, but there may be some there, and they may well return to general computing. A much better idea would be to drop MPI_Irecv_reduce and consider just MPI_Recv_reduce. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 21 Sep '12

21 Sep '12

On Thu, Sep 20, 2012 at 12:14 PM, Jeff Hammond <jhammond(a)alcf.anl.gov>wrote: > P2P) > > I would like MPI_(I)RECV_REDUCE, which - as you might guess - does a > reduction to the receive buffer instead of a simple write. This > allows one to avoid having to manually buffer incoming messages to be > reduced at the receiver. Torsten and I have discussed it and it seems > there are at least a few use cases. > If you do this, _please_ allow a user-defined MPI_Op to be used in the reduction (i.e., don't cripple it like one-sided). Jeff Squyres, I don't know what "fix grequest" meant in your list, but I hope that means: "provide a mechanism for users to implement nonblocking operations with the same progress semantics as built-in nonblocking operations". After writing the blog post below, I learned about additional exemplar use cases in dense linear algebra. Lack of this specific feature is causing a lot of important applications and libraries to systematically over-synchronize and preventing them from hiding communication latency. https://www.ieeetcsc.org/activities/blog/user_defined_nonblocking_collectiv…

1 0

Pictures - Re: MPI-3.0 has passed
by Rolf Rabenseifner 21 Sep '12

21 Sep '12

And here are the pictures from our Vienna MPI-3 group after we did our final MPI-3.0 vote meeting. Rolf ----- Original Message ----- > From: "Jeff Squyres" <jsquyres(a)cisco.com> > To: "MPI Forum list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Friday, September 21, 2012 9:19:53 AM > Subject: [Mpi-forum] MPI-3.0 has passed > I am pleased to announce that the MPI Forum unanimously voted to > accept the MPI-3.0 document this morning. > > Bill is working on making the final versions of the documents (putting > in the right date, etc.). I just wanted to let those who are not here > in Vienna that the discussion and votes went well and we will soon > have a final MPI-3.0 document. > > Many thanks to all who enabled and/or participated in the effort! > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

MPI 3.0 is published
by William Gropp 21 Sep '12

21 Sep '12

The MPI 3.0 document has been placed on the MPI Forum web site - http://www.mpi-forum.org/docs/docs.html . Congratulations to all involved. I'm still working on providing an unofficial HTML version; that may take a few days. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

MPI-3.0 has passed
by Jeff Squyres 21 Sep '12

21 Sep '12

I am pleased to announce that the MPI Forum unanimously voted to accept the MPI-3.0 document this morning. Bill is working on making the final versions of the documents (putting in the right date, etc.). I just wanted to let those who are not here in Vienna that the discussion and votes went well and we will soon have a final MPI-3.0 document. Many thanks to all who enabled and/or participated in the effort! -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 20 Sep '12

20 Sep '12

RMA) I would like nonblocking variants of all the collective operations in the RMA chapter, particularly MPI_Win_{create,allocate}. I already provided justification for this to the RMA working group but there wasn't time to deal with it for 3.0. P2P) I would like MPI_(I)RECV_REDUCE, which - as you might guess - does a reduction to the receive buffer instead of a simple write. This allows one to avoid having to manually buffer incoming messages to be reduced at the receiver. Torsten and I have discussed it and it seems there are at least a few use cases. Environment) Previously posted minor changes to stipulate thread-safety of functions like MPI_Initialized, MPI_Finalized and MPI_Thread_level (?) in all MPI_THREAD_XXX modes, which was +1'd by Jeff S. I think these changes are more of the MPI 3.1 variety than MPI 4.0. Jeff On Thu, Sep 20, 2012 at 10:54 AM, Jeff Squyres <jsquyres(a)cisco.com> wrote: > Process > - We need a better system for keeping notes / minutes / etc. > > ========================================== > > Potential topics for MPI 3.1 > ---------------------------- > > - Scalable vector collectives (maybe?) > > Point made: we don't have experience with 3.0 yet to know what the > errata are going to be yet. Perhaps we should just gather errata over > the next 1-2 years and publish those as MPI-3.1. Most of our time > will be spent on MPI-4.0. > > Potential topics for MPI 4.0 > ---------------------------- > > Point made: perhaps we should just open up the topics and let groups > go do what they want for a little while. > > This turned into a brainstorming session. There's a LOT of topics > here. Some feel research-ish. We'll see what happens! > > - Fault Tolerance (Aurelien) > - Hybrid <something> > - Play nice with other systems: CUDA, OpenACC, Co-Array Fortran, > OpenMP, UPC, ...etc. (Many) > - Task-based parallelism: TBB, cilk+, etc. (Aurelien) (also listed > in thread section) > - Maybe impose some restrictions / conditions on the environment > (OS, hardware, etc.) (Brian) > - Asyncrhonous file I/O (Dave) > - Fairness (Rolf) > - Non-blocking collectives: add tags? > - If not, reply with reasons (e.g., symmetry w/ blocking, difficult > to implement in hardware, etc.) (Jeff, from MPI Comments) > - Towards exascale: > - Collate current complaints and formulate responses (similar to RMA > WG analysis/response to Bonnechea's paper) (Brian) > - Scalability issues (Christian) > - Is MPI apppropiate? (Alexander) > - What else...? > - Possible MPI_Count/MPI_Aint compatibiltiy problem with datatype > constructors. Aurelien will come up with the specific example. (Aurelien) > - Small group of minor Fortran things (Jeff) > - Persistent collectives (Christian) > - Examine possibilities of software defined networking (anything to > standardize, or is this just an implementation issue) (Bill) > - Re-examine the definition of canceling a send: weird that it's a > local operation with remote semantics (Brian) > - Is it possible to fix MPI_REQUEST_FREE to provide some kind of > guarantee as to when the request has completed? (Rolf) > - Larger question: what do people need that MPI doesn't provide? (Brian) > - E.g., point was made that much data is being stored in databases > today. Should we define an interface to databases? (Alexander) > - Things that <XYZ> should provide to support MPI well (e.g., XYZ = > batch scheduler, or ...) (Brian) > - Bindings for higher-level languages? > - C# bindings (Alexander) > - Java bindings (Alexander) > - Python bindings (Alexander) > - ...Overall suggestions for higher level languages? (Alexander) > - Subsetting (e.g., standardized "MPI for <foo> subset") (Alexander) > > - Non-technical topics: > - Hyperlink Annex A function names (and handle types) to their > prototypes (Bill) > - Should we do this: for each MPI function, list which MPI standard > version it appeared (Christian) > > Point is made: challenging the basic assumptions of MPI (e.g., > fairness, guaranteed delivery, progress) is a good way to extend > MPI. (Alexander) > > Meta issue: threading issues and consistency > > - Revisit whether "threads are dead": i.e., whether threads are too > heavyweight to be used for a non-blocking operation (e.g., we put > MPI_COMM_IDUP so that apps didn't have to spawn off threads to do > this stuff) (Bill) > - Revisit progress rules (see above -- if we force apps to use threads > to do things asynchronously, then perhaps we should also re-look at > our progress rules) (Brian) > - Fix grequest (Jeff) > - Asyncrhonous dynamic process management: > - Fix MPI_COMM_ACCEPT (Brian) > - MPI_COMM_ISPAWN (Brian) > - Task-based parallelism: TBB, cilk+, etc. (Aurelien) (also listed in > hybrid section) > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 20 Sep '12

20 Sep '12

For various reasons, the European meetings are at times I find it very hard to attend, but here is a very high-level comment from someone who teaches and advises on MPI, program design / software engineering etc. Something that I would make a plea for is less featuritis - in particular, most MPI users are NOT among the communities that want even most of MPI 2. Before I wrote my course, I did a survey of all of the scientific programs I could get my hands on, and they used a very small subset of MPI 1 with merely a couple of MPI 2 features. If anyone would like the (now rather dated, but sporadically updated) data, please ask. Quite a lot of users have expressed the desire for safely portable and simple subset - which is roughly what I teach, though it's obviously an artificial boundary. But it would help a lot when producing interfaces to other languages, ports to fairly different systems and so on, as well as with learning MPI and teaching it. So not everybody wants more and more - and it's NOT true that we can simply ignore what we don't use - though MPI does pretty well in enabling that. All I request is that people bear this in mind. Regards, Nick Maclaren.

1 0

Discussion points from the MPI-<next> discussion today
by Jeff Squyres 20 Sep '12

20 Sep '12

Process - We need a better system for keeping notes / minutes / etc. ========================================== Potential topics for MPI 3.1 ---------------------------- - Scalable vector collectives (maybe?) Point made: we don't have experience with 3.0 yet to know what the errata are going to be yet. Perhaps we should just gather errata over the next 1-2 years and publish those as MPI-3.1. Most of our time will be spent on MPI-4.0. Potential topics for MPI 4.0 ---------------------------- Point made: perhaps we should just open up the topics and let groups go do what they want for a little while. This turned into a brainstorming session. There's a LOT of topics here. Some feel research-ish. We'll see what happens! - Fault Tolerance (Aurelien) - Hybrid <something> - Play nice with other systems: CUDA, OpenACC, Co-Array Fortran, OpenMP, UPC, ...etc. (Many) - Task-based parallelism: TBB, cilk+, etc. (Aurelien) (also listed in thread section) - Maybe impose some restrictions / conditions on the environment (OS, hardware, etc.) (Brian) - Asyncrhonous file I/O (Dave) - Fairness (Rolf) - Non-blocking collectives: add tags? - If not, reply with reasons (e.g., symmetry w/ blocking, difficult to implement in hardware, etc.) (Jeff, from MPI Comments) - Towards exascale: - Collate current complaints and formulate responses (similar to RMA WG analysis/response to Bonnechea's paper) (Brian) - Scalability issues (Christian) - Is MPI apppropiate? (Alexander) - What else...? - Possible MPI_Count/MPI_Aint compatibiltiy problem with datatype constructors. Aurelien will come up with the specific example. (Aurelien) - Small group of minor Fortran things (Jeff) - Persistent collectives (Christian) - Examine possibilities of software defined networking (anything to standardize, or is this just an implementation issue) (Bill) - Re-examine the definition of canceling a send: weird that it's a local operation with remote semantics (Brian) - Is it possible to fix MPI_REQUEST_FREE to provide some kind of guarantee as to when the request has completed? (Rolf) - Larger question: what do people need that MPI doesn't provide? (Brian) - E.g., point was made that much data is being stored in databases today. Should we define an interface to databases? (Alexander) - Things that <XYZ> should provide to support MPI well (e.g., XYZ = batch scheduler, or ...) (Brian) - Bindings for higher-level languages? - C# bindings (Alexander) - Java bindings (Alexander) - Python bindings (Alexander) - ...Overall suggestions for higher level languages? (Alexander) - Subsetting (e.g., standardized "MPI for <foo> subset") (Alexander) - Non-technical topics: - Hyperlink Annex A function names (and handle types) to their prototypes (Bill) - Should we do this: for each MPI function, list which MPI standard version it appeared (Christian) Point is made: challenging the basic assumptions of MPI (e.g., fairness, guaranteed delivery, progress) is a good way to extend MPI. (Alexander) Meta issue: threading issues and consistency - Revisit whether "threads are dead": i.e., whether threads are too heavyweight to be used for a non-blocking operation (e.g., we put MPI_COMM_IDUP so that apps didn't have to spawn off threads to do this stuff) (Bill) - Revisit progress rules (see above -- if we force apps to use threads to do things asynchronously, then perhaps we should also re-look at our progress rules) (Brian) - Fix grequest (Jeff) - Asyncrhonous dynamic process management: - Fix MPI_COMM_ACCEPT (Brian) - MPI_COMM_ISPAWN (Brian) - Task-based parallelism: TBB, cilk+, etc. (Aurelien) (also listed in hybrid section) -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0