September 2012 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 21 Sep '12

21 Sep '12

On Sep 21 2012, Jeff Hammond wrote: I said that I wouldn't follow up, but I can't let this pass. >I only use MPI from C but I have never seen or heard any evidence to >justify this concern. Can you state exactly which MPI functions and >semantics are "seriously incompatible with C89 and/or C99"? Exactly >which C compilers do not work with MPI? Passive one-sided communication, for a start. I no longer have access to the sort of systems on which that would fail, and I have never seen it used. When I did have such access, I did write code that caused its underlying assumptions to break, while investigating problems on the systems I managed. >At least in C89 and C99, C doesn't have a memory model but instead >inherits this from the hardware. I think this ambiguity is a far >bigger issue than anything MPI nonblocking is doing and somehow C >manages to be used in many places. That is total nonsense. I was involved in both standardisation efforts, including as a UK expert on that very area. The model was defined in terms of sequence points, though C99 also added effective types. What you say was true for K&R C, but has not been since. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 21 Sep '12

21 Sep '12

On Fri, Sep 21, 2012 at 1:19 PM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > MPI had no option but to go outside the Fortran and C standards for > choice arguments (in the former) and non-blocking transfer for both, but > both standards have been updated to enable those aspects of MPI to > operate cleanly (though, in the case of C, it's the interpretation, not > the wording). But it is very disturbing the way that MPI has started to > specify facilities that are seriously incompatible with both language > standards and many implementations of them. I only use MPI from C but I have never seen or heard any evidence to justify this concern. Can you state exactly which MPI functions and semantics are "seriously incompatible with C89 and/or C99"? Exactly which C compilers do not work with MPI? At least in C89 and C99, C doesn't have a memory model but instead inherits this from the hardware. I think this ambiguity is a far bigger issue than anything MPI nonblocking is doing and somehow C manages to be used in many places. The notion that MPI nonblocking send/recv are seriously incompatible with the C language and its implementation in a reasonable C compiler (e.g. GCC) is completely inconsistent with enormous quantities of empirical evidence and sounds like "the sky is falling!" to me. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 21 Sep '12

21 Sep '12

On Fri, Sep 21, 2012 at 1:19 PM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > No, they most definitely do NOT apply to MPI_Irecv, because it doesn't > call user code. I accept that they apply to MPI_Ireduce, though it > could be argued not to, because there are a lot of optimisations of that > that do not involve calling a user-defined operator except at the wait. > And MPI could fix its current languages standard incompatibility by > specifying that. > The natural semantics of MPI_Irecv_reduce() say nothing about the implementation making progress while application control flow is not in the MPI stack. When it's in the stack, it's the same situation as collectives (blocking or non). As a separate matter, to enable more asynchronous progress while the application is busy in user code, MPI_Ops could have an attribute indicating that they were safe to call from a different thread (the implementation's comm thread). This need not even be a standardized thing. Note that C11 and C++11 define memory models. > But, even if they do, does making an error once justify making it again? > Oops, evidently MPI-1 made a mistake of making MPI_Op extensible. We would be so much better off if it just plain wasn't possible to MPI_Allreduce a quad precision number. Those pesky users, give them an inch and they'll go off and build something useful. ;-)

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 21 Sep '12

21 Sep '12

On Fri, Sep 21, 2012 at 12:58 PM, Supalov, Alexander < alexander.supalov(a)intel.com> wrote: > Hi, > > Thanks. There may be another interesting twist to that. > > Imagine we start challenging the initial assumptions (axioms) that lay in > the foundation of the MPI standard. Those are, e.g., reliable and pairwise > ordered message delivery, non-fairness, and maybe a few more. If we take > one out - what happens to the MPI? What part of it survives? What > applications become possible that were not possible before? This idea is > somewhat comparable to the post-Euclidean geometry, if you wish. > > E.g., take out the reliable message delivery. This may be perfect for > Exascale: you will have to have rather stable algorithms anyway, and loss > of a couple of messages should not have dramatic effect then, in this sea > of data. And what kind of speedup might be possible with that? Gee. > I recommend being _extremely_ careful with this line of thinking. By and large, these "fault-oblivious" methods converge so much slower than their "reliable" siblings that you would be better off running on today's workstation with good methods than on an exascale machine with the "oblivious" methods. Don't forget David Keyes' slide showing how algorithmic innovation has offered the same incredible number of orders of magnitude improvement in simulation capability that hardware improvements have provided over the past few decades. Most of these "oblivious" algorithms necessarily discard decades of algorithmic innovation. Furthermore, affordable assertions of correctness (and reasonable convergence rates) in this unreliable landscape require statistical characterization of the faults. If the faults are correlated in a way that you have not accounted for, the claimed result can be wrong. Even worse, the accuracy of this statistical characterization depends on the regularity of the problem being solved. Most problems of great scientific and industrial relevance (i.e., those applications that justify the existence of an exascale machine) have poorly understood regularity. I'm not saying that these "oblivious" algorithms don't have a place, but most existing work is of extraordinarily low quality and the best hope, predicated on huge advances in the quality of the underlying mathematics (e.g., from stationary iterative methods to multigrid), would be a method that was still only relevant for "toy problems" (the regularity of which is well understood). Proposals for ASCR's "Resilient Solvers" applied math call are currently in review. They contain a lot of interesting approaches, but the most practical involve the use of problem structure to identify faults and to rapidly recover lost state in the event of node failure and similar. For the methods that I proposed for our proposal, one of the most interesting things for MPI/checkpointing library to provide is the ability to connect a "spare" or restarted node to an existing communicator. Once the communicator is attached, the application will build a subcommunicator involving neighbor processes for accelerated algorithmic recovery. (We should start a new thread if you want to discuss this further.)

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 21 Sep '12

21 Sep '12

On Sep 21 2012, Anthony Skjellum wrote: > > What has been elusive, is concrete agreement beyond the 8-16 range of > basic MPI functions, are there really subsets of functions that can be > agreed on to form 'compliance points'... those could come with or without > restricted semantics (e.g., no tags in point-to-point). It's a good question, and I don't know. Only last week, someone persuaded me that I need to add a topology lecture to my course. Tags are more useful than data conversion, which is something that I have never seen used, though I would be surprised if nobody used. There are quite a lot of similar features. > Another point was made that if you use a layered library, then you don't > know what MPI functions it has used, and that subset could be quite > different from what the main application uses. That is a soluble problem, but experience is that you have to require all components to declare their subset (and not understate it) if you want that to work. A mandatory argument to MPI_Init would do, so a subsetted MPI could not be fully compatible. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 21 Sep '12

21 Sep '12

On Sep 21 2012, Jeff Hammond wrote: > >Most of the criticism here is spurious and would have sunk either >MPI_Irecv (which is just MPI_Irecv_reduce with MPI_REPLACE - yes I >know this is for MPI_Accumulate only) or MPI_Ireduce. I believe all >these comments apply to MPI_Ireduce with a user-defined reduction and >that ship has sailed. No, they most definitely do NOT apply to MPI_Irecv, because it doesn't call user code. I accept that they apply to MPI_Ireduce, though it could be argued not to, because there are a lot of optimisations of that that do not involve calling a user-defined operator except at the wait. And MPI could fix its current languages standard incompatibility by specifying that. But, even if they do, does making an error once justify making it again? >For ANY nonblocking operation, nothing is guaranteed to happen until >the MPI_Wait (or equivalent) is called. The MPI standard NEVER >species when any background activity has to occur. This is entirely >implementation defined. It is sufficient to have everything happen >during the synchronization call (e.g. MPI_Wait). That doesn't help, for the reason given above. MPI had no option but to go outside the Fortran and C standards for choice arguments (in the former) and non-blocking transfer for both, but both standards have been updated to enable those aspects of MPI to operate cleanly (though, in the case of C, it's the interpretation, not the wording). But it is very disturbing the way that MPI has started to specify facilities that are seriously incompatible with both language standards and many implementations of them. I will not follow up on this aspect after this message. On Sep 21 2012, Jed Brown wrote: > >> On the contrary - that is essential for any kind of sane specification >> or implementation of MPI_Irecv_reduce, just as it is for one-sided. >> Sorry, but that's needed to get even plausible conformance with any of >> the languages MPI is likely to be used from. MPI_Recv_reduce doesn't >> have the same problems. > >> The point is that none of them allow more-or-less arbitrary functions >> to be called asynchronously, and that has been horribly sick in every >> modern system that I have looked into in any depth. It used to work >> on some mainframes, but hasn't worked reliably since. That is precisely >> why POSIX has deprecated calling signal handlers asynchronously. Please >> don't perpetrate another feature like passive one-sided! > >This is totally different than passive one-sided because it has a request >and isn't guaranteed to make progress when not in the MPI stack. An >implementation using comm threads also need not use interrupts. Er, no. That's true in theory, but it is NOT true in practice :-( If you think that the reduction can be done in another thread, then I am afraid that you are mistaken. It can't be, any more than passive one-sided can be (though I agree that the problems are VASTLY less nasty for MPI_Irecv_reduce). It probably could be if you required the program to be run with multiple threads and to dedicate one thread for the use of MPI_Irecv_reduce, but that's a non-starter for anything even approaching standards conformance or portability. I would rather not get started on the utter ghastliess of the multiple and incompatible threading specifications, but Fortran doesn't support threading at all. Even worse, all of the others use an explicit threading model and require both sides to handshake even when one side uses an object read-only. So the user-defined operator has to know that it may be run in another thread and has to synchronise appropriately, even when it is itself entirely pure. And the program has to know that MPI is using a thread (and which). >A lot of the controversy seems to come down to not trusting the user to be >able to write a pure function that is actually pure. It has nothing whatsoever to do with trusting the user, but what the languages specify and therefore what is likely to be implemented with any degree of reliability. Fortran has the concept of a PURE function, but it does NOT specify that one can be called in the way this needs. Even worse, almost no other currently relevant languages do - and, in particular, C and C++ most assuredly do not. >You should realize >that in many cases (including the most important ones to me), the MPI_Op is >just SUM or MAX, but applied to datatypes that MPI does not cover (e.g. >quad precision). ... Well, obviously. But that doesn't help. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Supalov, Alexander 21 Sep '12

21 Sep '12

Hi, Thanks. There may be another interesting twist to that. Imagine we start challenging the initial assumptions (axioms) that lay in the foundation of the MPI standard. Those are, e.g., reliable and pairwise ordered message delivery, non-fairness, and maybe a few more. If we take one out - what happens to the MPI? What part of it survives? What applications become possible that were not possible before? This idea is somewhat comparable to the post-Euclidean geometry, if you wish. E.g., take out the reliable message delivery. This may be perfect for Exascale: you will have to have rather stable algorithms anyway, and loss of a couple of messages should not have dramatic effect then, in this sea of data. And what kind of speedup might be possible with that? Gee. Now, what to take out to get to FT MPI? What part of MPI is amenable to FT? Here we have a possible connection to the subsetting. Best regards. Alexander -----Original Message----- From: mpi-forum-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Anthony Skjellum Sent: Friday, September 21, 2012 7:46 PM To: Main MPI Forum mailing list Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today Nick, I think this is a timely e-mail... in practice users in different spaces use somewhat different subsets, and we have said as a group from the beginning that MPI (at MPI-1 level anyway) was the composition of a few concepts and a few features yielding an understandable set of functions, even though people might use a small subset. From what I recall, at the beginning of MPI-2 and MPI-3, efforts to discuss reducing and pruning met with little interest (other than the C++ bindings). What has been elusive, is concrete agreement beyond the 8-16 range of basic MPI functions, are there really subsets of functions that can be agreed on to form 'compliance points'... those could come with or without restricted semantics (e.g., no tags in point-to-point). Another point was made that if you use a layered library, then you don't know what MPI functions it has used, and that subset could be quite different from what the main application uses. In the mid-1990's, a few of us defined one or more embedded subset for constrained embedded multicomputers, such as systems where no more than O(20) functions would ever be used, and where we wanted to get more performance for applications that didn't use tags or wildcards,and where persistent communication was highly preferred, as was the use of pinned/aligned/primed memory for buffers, rather than allowing arbitrary memory. It would be very interesting for future MPI to discuss the advantages of small subsets again. In exascale, where we may have many, many small processors, the size, complexity, and scale of the library and its internal complexity might matter a lot. We might benefit from programs that only use messages from a special heap as well. I do not expect that there would be consensus on this point, but there could be considerable advantage as applications are ported to higher concurrency to remove overheads if they utilize MPI in restrictive ways, and if they are willing to use planned transfers (Send_init, Recv_init, etc). Another benefit of subsetting would be reduced complexity of implementation, with the potential for higher reliability and less dead code in implementations. Furthermore, making a subset of MPI fault tolerant, keyed to what a specific application needs, appears easier than making an entire MPI implementation fault tolerant. Regards, Tony Skjellum ----- Original Message ----- From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> Sent: Thursday, September 20, 2012 12:01:20 PM Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today For various reasons, the European meetings are at times I find it very hard to attend, but here is a very high-level comment from someone who teaches and advises on MPI, program design / software engineering etc. Something that I would make a plea for is less featuritis - in particular, most MPI users are NOT among the communities that want even most of MPI 2. Before I wrote my course, I did a survey of all of the scientific programs I could get my hands on, and they used a very small subset of MPI 1 with merely a couple of MPI 2 features. If anyone would like the (now rather dated, but sporadically updated) data, please ask. Quite a lot of users have expressed the desire for safely portable and simple subset - which is roughly what I teach, though it's obviously an artificial boundary. But it would help a lot when producing interfaces to other languages, ports to fairly different systems and so on, as well as with learning MPI and teaching it. So not everybody wants more and more - and it's NOT true that we can simply ignore what we don't use - though MPI does pretty well in enabling that. All I request is that people bear this in mind. Regards, Nick Maclaren. _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Peter Gleissner, Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Anthony Skjellum 21 Sep '12

21 Sep '12

Nick, I think this is a timely e-mail... in practice users in different spaces use somewhat different subsets, and we have said as a group from the beginning that MPI (at MPI-1 level anyway) was the composition of a few concepts and a few features yielding an understandable set of functions, even though people might use a small subset. From what I recall, at the beginning of MPI-2 and MPI-3, efforts to discuss reducing and pruning met with little interest (other than the C++ bindings). What has been elusive, is concrete agreement beyond the 8-16 range of basic MPI functions, are there really subsets of functions that can be agreed on to form 'compliance points'... those could come with or without restricted semantics (e.g., no tags in point-to-point). Another point was made that if you use a layered library, then you don't know what MPI functions it has used, and that subset could be quite different from what the main application uses. In the mid-1990's, a few of us defined one or more embedded subset for constrained embedded multicomputers, such as systems where no more than O(20) functions would ever be used, and where we wanted to get more performance for applications that didn't use tags or wildcards,and where persistent communication was highly preferred, as was the use of pinned/aligned/primed memory for buffers, rather than allowing arbitrary memory. It would be very interesting for future MPI to discuss the advantages of small subsets again. In exascale, where we may have many, many small processors, the size, complexity, and scale of the library and its internal complexity might matter a lot. We might benefit from programs that only use messages from a special heap as well. I do not expect that there would be consensus on this point, but there could be considerable advantage as applications are ported to higher concurrency to remove overheads if they utilize MPI in restrictive ways, and if they are willing to use planned transfers (Send_init, Recv_init, etc). Another benefit of subsetting would be reduced complexity of implementation, with the potential for higher reliability and less dead code in implementations. Furthermore, making a subset of MPI fault tolerant, keyed to what a specific application needs, appears easier than making an entire MPI implementation fault tolerant. Regards, Tony Skjellum ----- Original Message ----- From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> Sent: Thursday, September 20, 2012 12:01:20 PM Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today For various reasons, the European meetings are at times I find it very hard to attend, but here is a very high-level comment from someone who teaches and advises on MPI, program design / software engineering etc. Something that I would make a plea for is less featuritis - in particular, most MPI users are NOT among the communities that want even most of MPI 2. Before I wrote my course, I did a survey of all of the scientific programs I could get my hands on, and they used a very small subset of MPI 1 with merely a couple of MPI 2 features. If anyone would like the (now rather dated, but sporadically updated) data, please ask. Quite a lot of users have expressed the desire for safely portable and simple subset - which is roughly what I teach, though it's obviously an artificial boundary. But it would help a lot when producing interfaces to other languages, ports to fairly different systems and so on, as well as with learning MPI and teaching it. So not everybody wants more and more - and it's NOT true that we can simply ignore what we don't use - though MPI does pretty well in enabling that. All I request is that people bear this in mind. Regards, Nick Maclaren. _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 21 Sep '12

21 Sep '12

Note also that the possibility of performance-crippling buffering here is just as real as for MPI_SENDRECV_REPLACE. However, the opportunities for optimization of MPI_(I)RECV_REDUCE are much greater since no lock-step is needed (because it's not bidirectional) and I know plenty of networks where this function can be done asynchronously with even greater ease than MPI_Accumulate. Jeff On Fri, Sep 21, 2012 at 12:09 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > Most of the criticism here is spurious and would have sunk either > MPI_Irecv (which is just MPI_Irecv_reduce with MPI_REPLACE - yes I > know this is for MPI_Accumulate only) or MPI_Ireduce. I believe all > these comments apply to MPI_Ireduce with a user-defined reduction and > that ship has sailed. > > For ANY nonblocking operation, nothing is guaranteed to happen until > the MPI_Wait (or equivalent) is called. The MPI standard NEVER > species when any background activity has to occur. This is entirely > implementation defined. It is sufficient to have everything happen > during the synchronization call (e.g. MPI_Wait). > > Can we please keep this discussion on the topic of MPI_Irecv_reduce > and not all your issues with MPI nonblocking in general, which are a > separate issue. > > Thanks, > > Jeff > > On Fri, Sep 21, 2012 at 11:41 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >> On Fri, Sep 21, 2012 at 10:49 AM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: >>> >>> On the contrary - that is essential for any kind of sane specification >>> or implementation of MPI_Irecv_reduce, just as it is for one-sided. >>> Sorry, but that's needed to get even plausible conformance with any of >>> the languages MPI is likely to be used from. MPI_Recv_reduce doesn't >>> have the same problems. >> >> >>> >>> The point is that none of them allow more-or-less arbitrary functions >>> to be called asynchronously, and that has been horribly sick in every >>> modern system that I have looked into in any depth. It used to work >>> on some mainframes, but hasn't worked reliably since. That is precisely >>> why POSIX has deprecated calling signal handlers asynchronously. Please >>> don't perpetrate another feature like passive one-sided! >> >> >> This is totally different than passive one-sided because it has a request >> and isn't guaranteed to make progress when not in the MPI stack. An >> implementation using comm threads also need not use interrupts. >> >> I haven't heard of any system that supports ALL combinations of built-in ops >> and datatypes in dedicated hardware, therefore you have some code running in >> a context where it could call the user-defined MPI_Op. >> >> A lot of the controversy seems to come down to not trusting the user to be >> able to write a pure function that is actually pure. You should realize that >> in many cases (including the most important ones to me), the MPI_Op is just >> SUM or MAX, but applied to datatypes that MPI does not cover (e.g. quad >> precision). The other concern people raised when this issue was last >> discussed for one-sided was the need for an API allowing collective >> specification of MPI_Ops. That isn't needed here because the reduction is >> specified by the receiving rank. >> >>> >>> Even doing it for just built-in types is problematic on some systems, >>> just as it is for one-sided. I doubt that there are many such systems >>> currently in use outside the embedded arena, but there may be some there, >>> and they may well return to general computing. >>> >>> A much better idea would be to drop MPI_Irecv_reduce and consider just >>> MPI_Recv_reduce. >> >> >> I hate this. It will will force the application to poll (or pay the penalty >> of receiving the messages in a fixed order, which may slow things down more >> than requiring a copy). >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 21 Sep '12

21 Sep '12

Most of the criticism here is spurious and would have sunk either MPI_Irecv (which is just MPI_Irecv_reduce with MPI_REPLACE - yes I know this is for MPI_Accumulate only) or MPI_Ireduce. I believe all these comments apply to MPI_Ireduce with a user-defined reduction and that ship has sailed. For ANY nonblocking operation, nothing is guaranteed to happen until the MPI_Wait (or equivalent) is called. The MPI standard NEVER species when any background activity has to occur. This is entirely implementation defined. It is sufficient to have everything happen during the synchronization call (e.g. MPI_Wait). Can we please keep this discussion on the topic of MPI_Irecv_reduce and not all your issues with MPI nonblocking in general, which are a separate issue. Thanks, Jeff On Fri, Sep 21, 2012 at 11:41 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Sep 21, 2012 at 10:49 AM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: >> >> On the contrary - that is essential for any kind of sane specification >> or implementation of MPI_Irecv_reduce, just as it is for one-sided. >> Sorry, but that's needed to get even plausible conformance with any of >> the languages MPI is likely to be used from. MPI_Recv_reduce doesn't >> have the same problems. > > >> >> The point is that none of them allow more-or-less arbitrary functions >> to be called asynchronously, and that has been horribly sick in every >> modern system that I have looked into in any depth. It used to work >> on some mainframes, but hasn't worked reliably since. That is precisely >> why POSIX has deprecated calling signal handlers asynchronously. Please >> don't perpetrate another feature like passive one-sided! > > > This is totally different than passive one-sided because it has a request > and isn't guaranteed to make progress when not in the MPI stack. An > implementation using comm threads also need not use interrupts. > > I haven't heard of any system that supports ALL combinations of built-in ops > and datatypes in dedicated hardware, therefore you have some code running in > a context where it could call the user-defined MPI_Op. > > A lot of the controversy seems to come down to not trusting the user to be > able to write a pure function that is actually pure. You should realize that > in many cases (including the most important ones to me), the MPI_Op is just > SUM or MAX, but applied to datatypes that MPI does not cover (e.g. quad > precision). The other concern people raised when this issue was last > discussed for one-sided was the need for an API allowing collective > specification of MPI_Ops. That isn't needed here because the reduction is > specified by the receiving rank. > >> >> Even doing it for just built-in types is problematic on some systems, >> just as it is for one-sided. I doubt that there are many such systems >> currently in use outside the embedded arena, but there may be some there, >> and they may well return to general computing. >> >> A much better idea would be to drop MPI_Irecv_reduce and consider just >> MPI_Recv_reduce. > > > I hate this. It will will force the application to poll (or pay the penalty > of receiving the messages in a fixed order, which may slow things down more > than requiring a copy). > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0