- mpiwg-p2p - mailman3.cels.anl.gov

Re: [mpiwg-p2p] Message matching for tools
by Marc-Andre Hermanns 17 Dec '15

17 Dec '15

Hi Jeff, at the moment we don't handle MPI_THREAD_MULTIPLE at all. But we want to get there ;-) Here is a short recollection of what we do/need. Sorry for the folks who know/read this already in other context: We use, what we call "parallel replay" to analyze large event traces in parallel. Each thread has its own stream of events, such as enter and exit for tracking the calling context as well as send and receive for communication among ranks. We analyze on the same scale as the measurement, thus we have one thread per thread-local trace. Each thread processes its own thread-local trace. When encountering a communication event, it re-enacts this communication using the recorded communication parameters (rank, tag, comm). A send event leads to an issued send, a receive event leads to an issued receive. It is critical that during the analysis, the message matching is identical to the original application. However, we do not re-enact any computational time, that is the temporal distance between sends and receives is certainly different from the original application. As a consequence, while two sends may have some significant temporal distance in the original measurement, they could be issued right after each other during the analysis. Markus Geimer and I believe that creating some sort of a sequence number during measurement could help matching the right messages during the analysis, as a process could detect that it got mismatched messages and communicate with other threads to get the correct one. It is unclear, however, how to achieve this: a) Sending an additional message within the MPI wrapper at measurement time may lead to invalid matchings, as the additional message may be received by a different thread. b) Creating a derived datatype on the fly to add tool-level data to the original payload may induce a large overhead in practically _every_ send & receive operation and perturb the measurement. The best idea in the room so far is some callback mechanism into the MPI implementation that generates matching information both on sender and receiver side to generate some form of a sequence number that can then be saved during measurement. If available on both sender and receiver this information could then be used to fix incorrect matching during the analysis. Cheers, Marc-Andre On 16.12.15 16:40, Jeff Hammond wrote: > How do you handle MPI_THREAD_MULTIPLE? Understanding what your tool > does there is a good starting point for this discussion. > > Jeff > > On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns > <hermanns(a)jara.rwth-aachen.de <mailto:[email protected]>> > wrote: > > Hi all, > > CC: Tools-WG, Markus Geimer (not on either list) > > sorry for starting a new thread and being so verbose, but I subscribed > just now. I quoted Dan, Jeff, and Jim from the archive as appropriate. > > First, let me state that we do not want to prevent this assertion in > any way. For us as tools provider it is just quite a brain tickler on > how to support this in our tool and in general. > > Dan wrote: > >>> [...] The basic problem is that message matching would be > >>> non-deterministic and it would be impossible for a tool to show > >>> the user which receive operation satisfied which send operation > >>> without internally using some sort of sequence number for each > >>> send/receive operation. [...] > >>> > >>> My responses were: > >>> 1) the user asked for this behaviour so the tool could simply > >>> gracefully give up the linking function and just state the > >>> information it knows > > > Giving up can only be a temporary solution for tools. The user wants > to use this advanced feature, thus just saying: "Hey, what you're > doing is too sophisticated for us. You are on your own now." is not a > viable long-term strategy. > > >>> 2) the tool could hook all send and receive operations and > >>> piggy-back a sequence number into the message header > > We discussed piggy-backing within the tools group some time in the > past, but never came to a satisfying way of how to implement this. If, > in the process of reviving the discussion on a piggy-backing > interface, we come to a viable solution, it would certainly help with > the our issues with message matching in general. > > Scalasca's problem here is that we need to detect (and partly > recreate) the exact order of message matching to have the correct > message reach the right receivers. > > >>> 3) the tool could hook all send and receive operations and > >>> serialise them to prevent overtaking > > This is not an option for me. A "performance tool" should strive to > measure as close to the original behavior as possible. Changing > communication semantics just to make a tool "work" would have too > great of an impact on application behavior. After all, if it would > have only little impact, why should the user choose this option in the > first place. > > Jeff wrote: > >> Remember that one of the use cases of allow_overtaking is > applications that > >> have exact matching, in which case allow_overtaking is a way of > turning off > >> a feature that isn't used, in order to get a high-performing > message queue > >> implementation. In the exact matching case, tools will have no > problem > >> matching up sends and recvs. > > This is true. If the tools can identify this scenario, it could be > supported by current tools without significant change. However, as it > is not generally forbidden to have inexact matching (right?), it is > unclear on how the tools would detect this. > > What about an additional info key a user can set in this respect: > > exact_matching => true/false > > in which the user can state whether it is indeed a scenario of exact > matching or not. The tool could check this, and issue a warning. > > >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I > don't really > >> care if they can't support this assertion either. > > Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. ;-) > > As I said, we (Markus and I) see this as a trigger to come to a viable > solution for tools like ours to support either situation. > > >> And in any case, such tools can just intercept the info > operations and > >> strip this key if they can't support it. > > As I wrote above in reply to Dan, stripping options that influence > behavior is not a good option. I, personally, would rather bail out > than (silently) change messaging semantics. I can't say what Markus' > take on this is. > > Jim wrote: > > I don't really see any necessary fix to the proposal. We could > add an > > advice to users to remind them that they should ensure tools are > compatible > > with the info keys. And the reverse advice to tools writers that > they > > should check info keys for compatibility. > > I would second this idea, while emphasizing the burden to be on the > tool to check for this info key (and potentially others) and warn the > user of "undersupport". > > Cheers, > Marc-Andre > -- > Marc-Andre Hermanns > Jülich Aachen Research Alliance, > High Performance Computing (JARA-HPC) > Jülich Supercomputing Centre (JSC) > > Schinkelstrasse 2 > 52062 Aachen > Germany > > Phone: +49 2461 61 2509 | +49 241 80 24381 > Fax: +49 2461 80 6 99753 > www.jara.org/jara-hpc <http://www.jara.org/jara-hpc> > email: hermanns(a)jara.rwth-aachen.de > <mailto:[email protected]> > > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ > > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > -- Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Schinkelstrasse 2 52062 Aachen Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc email: hermanns(a)jara.rwth-aachen.de

1 0

Re: [mpiwg-p2p] Message matching for tools
by Jeff Hammond 16 Dec '15

16 Dec '15

How do you handle MPI_THREAD_MULTIPLE? Understanding what your tool does there is a good starting point for this discussion. Jeff On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns < hermanns(a)jara.rwth-aachen.de> wrote: > Hi all, > > CC: Tools-WG, Markus Geimer (not on either list) > > sorry for starting a new thread and being so verbose, but I subscribed > just now. I quoted Dan, Jeff, and Jim from the archive as appropriate. > > First, let me state that we do not want to prevent this assertion in > any way. For us as tools provider it is just quite a brain tickler on > how to support this in our tool and in general. > > Dan wrote: > >>> [...] The basic problem is that message matching would be > >>> non-deterministic and it would be impossible for a tool to show > >>> the user which receive operation satisfied which send operation > >>> without internally using some sort of sequence number for each > >>> send/receive operation. [...] > >>> > >>> My responses were: > >>> 1) the user asked for this behaviour so the tool could simply > >>> gracefully give up the linking function and just state the > >>> information it knows > > > Giving up can only be a temporary solution for tools. The user wants > to use this advanced feature, thus just saying: "Hey, what you're > doing is too sophisticated for us. You are on your own now." is not a > viable long-term strategy. > > >>> 2) the tool could hook all send and receive operations and > >>> piggy-back a sequence number into the message header > > We discussed piggy-backing within the tools group some time in the > past, but never came to a satisfying way of how to implement this. If, > in the process of reviving the discussion on a piggy-backing > interface, we come to a viable solution, it would certainly help with > the our issues with message matching in general. > > Scalasca's problem here is that we need to detect (and partly > recreate) the exact order of message matching to have the correct > message reach the right receivers. > > >>> 3) the tool could hook all send and receive operations and > >>> serialise them to prevent overtaking > > This is not an option for me. A "performance tool" should strive to > measure as close to the original behavior as possible. Changing > communication semantics just to make a tool "work" would have too > great of an impact on application behavior. After all, if it would > have only little impact, why should the user choose this option in the > first place. > > Jeff wrote: > >> Remember that one of the use cases of allow_overtaking is applications > that > >> have exact matching, in which case allow_overtaking is a way of turning > off > >> a feature that isn't used, in order to get a high-performing message > queue > >> implementation. In the exact matching case, tools will have no problem > >> matching up sends and recvs. > > This is true. If the tools can identify this scenario, it could be > supported by current tools without significant change. However, as it > is not generally forbidden to have inexact matching (right?), it is > unclear on how the tools would detect this. > > What about an additional info key a user can set in this respect: > > exact_matching => true/false > > in which the user can state whether it is indeed a scenario of exact > matching or not. The tool could check this, and issue a warning. > > >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I don't really > >> care if they can't support this assertion either. > > Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. ;-) > > As I said, we (Markus and I) see this as a trigger to come to a viable > solution for tools like ours to support either situation. > > >> And in any case, such tools can just intercept the info operations and > >> strip this key if they can't support it. > > As I wrote above in reply to Dan, stripping options that influence > behavior is not a good option. I, personally, would rather bail out > than (silently) change messaging semantics. I can't say what Markus' > take on this is. > > Jim wrote: > > I don't really see any necessary fix to the proposal. We could add an > > advice to users to remind them that they should ensure tools are > compatible > > with the info keys. And the reverse advice to tools writers that they > > should check info keys for compatibility. > > I would second this idea, while emphasizing the burden to be on the > tool to check for this info key (and potentially others) and warn the > user of "undersupport". > > Cheers, > Marc-Andre > -- > Marc-Andre Hermanns > Jülich Aachen Research Alliance, > High Performance Computing (JARA-HPC) > Jülich Supercomputing Centre (JSC) > > Schinkelstrasse 2 > 52062 Aachen > Germany > > Phone: +49 2461 61 2509 | +49 241 80 24381 > Fax: +49 2461 80 6 99753 > www.jara.org/jara-hpc > email: hermanns(a)jara.rwth-aachen.de > > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Message matching for tools
by Marc-Andre Hermanns 16 Dec '15

16 Dec '15

Hi all, CC: Tools-WG, Markus Geimer (not on either list) sorry for starting a new thread and being so verbose, but I subscribed just now. I quoted Dan, Jeff, and Jim from the archive as appropriate. First, let me state that we do not want to prevent this assertion in any way. For us as tools provider it is just quite a brain tickler on how to support this in our tool and in general. Dan wrote: >>> [...] The basic problem is that message matching would be >>> non-deterministic and it would be impossible for a tool to show >>> the user which receive operation satisfied which send operation >>> without internally using some sort of sequence number for each >>> send/receive operation. [...] >>> >>> My responses were: >>> 1) the user asked for this behaviour so the tool could simply >>> gracefully give up the linking function and just state the >>> information it knows > Giving up can only be a temporary solution for tools. The user wants to use this advanced feature, thus just saying: "Hey, what you're doing is too sophisticated for us. You are on your own now." is not a viable long-term strategy. >>> 2) the tool could hook all send and receive operations and >>> piggy-back a sequence number into the message header We discussed piggy-backing within the tools group some time in the past, but never came to a satisfying way of how to implement this. If, in the process of reviving the discussion on a piggy-backing interface, we come to a viable solution, it would certainly help with the our issues with message matching in general. Scalasca's problem here is that we need to detect (and partly recreate) the exact order of message matching to have the correct message reach the right receivers. >>> 3) the tool could hook all send and receive operations and >>> serialise them to prevent overtaking This is not an option for me. A "performance tool" should strive to measure as close to the original behavior as possible. Changing communication semantics just to make a tool "work" would have too great of an impact on application behavior. After all, if it would have only little impact, why should the user choose this option in the first place. Jeff wrote: >> Remember that one of the use cases of allow_overtaking is applications that >> have exact matching, in which case allow_overtaking is a way of turning off >> a feature that isn't used, in order to get a high-performing message queue >> implementation. In the exact matching case, tools will have no problem >> matching up sends and recvs. This is true. If the tools can identify this scenario, it could be supported by current tools without significant change. However, as it is not generally forbidden to have inexact matching (right?), it is unclear on how the tools would detect this. What about an additional info key a user can set in this respect: exact_matching => true/false in which the user can state whether it is indeed a scenario of exact matching or not. The tool could check this, and issue a warning. >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I don't really >> care if they can't support this assertion either. Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. ;-) As I said, we (Markus and I) see this as a trigger to come to a viable solution for tools like ours to support either situation. >> And in any case, such tools can just intercept the info operations and >> strip this key if they can't support it. As I wrote above in reply to Dan, stripping options that influence behavior is not a good option. I, personally, would rather bail out than (silently) change messaging semantics. I can't say what Markus' take on this is. Jim wrote: > I don't really see any necessary fix to the proposal. We could add an > advice to users to remind them that they should ensure tools are compatible > with the info keys. And the reverse advice to tools writers that they > should check info keys for compatibility. I would second this idea, while emphasizing the burden to be on the tool to check for this info key (and potentially others) and warn the user of "undersupport". Cheers, Marc-Andre -- Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Schinkelstrasse 2 52062 Aachen Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc email: hermanns(a)jara.rwth-aachen.de

1 0

Re: [mpiwg-p2p] Fwd: MPI P2P "messages may overtake" assertion
by Jim Dinan 15 Dec '15

15 Dec '15

It may be hard for tools or existing analysis techniques to provide assistance to users when this mode is selected. However, there is adequate support through the MPI interface to detect when this mode has been selected and restrict the tool's behavior -- i.e. a tool may have to ignore traffic on communicators where this info key is active. As Jeff pointed out, tools can even prevent the application from providing this info key to the library. I don't really see any necessary fix to the proposal. We could add an advice to users to remind them that they should ensure tools are compatible with the info keys. And the reverse advice to tools writers that they should check info keys for compatibility. ~Jim. On Mon, Dec 14, 2015 at 10:59 AM, Daniel Holmes <dholmes(a)epcc.ed.ac.uk> wrote: > Hi Jeff, > > ... tools can just intercept the info operations and strip this key if > they can't support it. > > Agreed: this should have been response 4 on my list (Markus and I > discussed that option too). > > Cheers, > Dan. > > > On 14/12/2015 15:54, Jeff Hammond wrote: > > Remember that one of the use cases of allow_overtaking is applications > that have exact matching, in which case allow_overtaking is a way of > turning off a feature that isn't used, in order to get a high-performing > message queue implementation. In the exact matching case, tools will have > no problem matching up sends and recvs. > > If tools cannot handle MPI_THREAD_MULTIPLE already, then I don't really > care if they can't support this assertion either. > > And in any case, such tools can just intercept the info operations and > strip this key if they can't support it. > > Jeff > > On Mon, Dec 14, 2015 at 7:44 AM, Daniel Holmes <dholmes(a)epcc.ed.ac.uk> > wrote: > >> Hi all, >> >> Markus has reminded me that we discussed the effect of the >> mpi_assert_allow_overtaking assertion on tools (see below). >> >> The basic problem is that message matching would be non-deterministic and >> it would be impossible for a tool to show the user which receive operation >> satisfied which send operation without internally using some sort of >> sequence number for each send/receive operation. As Markus mentions, this >> is a similar problem to the MPI_THREAD_MULTIPLE "logically concurrent" >> issue, with which tools already struggle. >> >> My responses were: >> 1) the user asked for this behaviour so the tool could simply gracefully >> give up the linking function and just state the information it knows >> 2) the tool could hook all send and receive operations and piggy-back a >> sequence number into the message header >> 3) the tool could hook all send and receive operations and serialise them >> to prevent overtaking >> I don't know the viability/acceptability of any of these approaches. >> >> Preferably before the formal reading at the next face-to-face meeting, we >> should discuss the possibility that this assertion will disrupt tools and >> ascertain if we can/want to do anything to mitigate that disruption. >> >> Cheers, >> Dan. >> >> -------- Forwarded Message -------- Subject: MPI P2P "messages may >> overtake" assertion Date: Mon, 14 Dec 2015 16:24:44 +0100 From: Markus >> Geimer <m.geimer(a)fz-juelich.de> <m.geimer(a)fz-juelich.de> To: >> dholmes(a)epcc.ed.ac.uk CC: Marc-André Hermanns >> <m.a.hermanns(a)fz-juelich.de> <m.a.hermanns(a)fz-juelich.de> >> >> Hi Daniel, >> >> Do you remember our conversation after the MPI BoF at SC'15 about the >> proposed "messages may overtake" assertion and that it may cause severe >> problems for tools, as correct message matching is very important for >> many of them? In fact, the issue here is similar to properly supporting >> MPI_THREAD_MULTIPLE, which most tools don't do either right now... >> >> Meanwhile I talked to Marc-Andre Hermanns from the Tools WG -- which >> kick-started a discussion on this end. But I guess this topic also >> requires some involvement of the P2P WG. For example, it would be good >> if you could summarize the ideas you had in mind how tools could handle >> this assertion. Thanks! >> >> That said, I don't think there is any urgency for discussions, as >> people are slowly leaving for the Christmas break (e.g., I'm already >> out of office). But we should definitely follow up on this early next >> year. >> >> Merry Christmas, >> Markus >> >> -- >> Dr. Markus Geimer >> Juelich Supercomputing Centre >> Institute for Advanced Simulation >> Forschungszentrum Juelich GmbH >> 52425 Juelich, Germany >> >> Phone: +49-2461-61-1773 >> Fax: +49-2461-61-6656 >> E-Mail: m.geimer(a)fz-juelich.de >> WWW: http://www.fz-juelich.de/jsc >> >> >> ------------------------------------------------------------------------------------------------ >> ------------------------------------------------------------------------------------------------ >> Forschungszentrum Juelich GmbH >> 52425 Juelich >> Sitz der Gesellschaft: Juelich >> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher >> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), >> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, >> Prof. Dr. Sebastian M. Schmidt >> ------------------------------------------------------------------------------------------------ >> ------------------------------------------------------------------------------------------------ >> >> >> >> >> -- >> Dan Holmes >> Applications Consultant in HPC Research >> EPCC, The University of Edinburgh >> James Clerk Maxwell Building >> The Kings Buildings >> Peter Guthrie Tait Road >> Edinburgh >> EH9 3FD >> T: +44(0)131 651 3465 >> E: dholmes(a)epcc.ed.ac.uk >> >> *Please consider the environment before printing this email.* >> >> >> >> >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> _______________________________________________ >> mpiwg-p2p mailing list >> mpiwg-p2p(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p >> > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > > -- > Dan Holmes > Applications Consultant in HPC Research > EPCC, The University of Edinburgh > James Clerk Maxwell Building > The Kings Buildings > Peter Guthrie Tait Road > Edinburgh > EH9 3FD > T: +44(0)131 651 3465 > E: dholmes(a)epcc.ed.ac.uk > > *Please consider the environment before printing this email.* > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p >

1 0

Re: [mpiwg-p2p] Next meeting 18th Jan 2016
by Jeff Hammond 14 Dec '15

14 Dec '15

Can someone tell me the best replacement for the following pattern if freeing active requests is made erroneous? void send_fire_and_forget_ack(int target, MPI_Comm comm) { MPI_Request request; MPI_Isend(NULL, 0, MPI_BYTE, target, special_tag, comm, &request); MPI_Request_free(&request); return; } The reason to not use MPI_Send here is it can block on resource exhaustion. Note that this is a real usage model. One of my applications send fire-and-forget acks and it was a performance issue on Blue Gene/Q. Because we send a payload - it is unclear if this is actually necessary - I replaced MPI_Send with MPI_Bsend to solve the problem (yes, Bsend really does help here), but we could instead use Isend+Request_free (which would be easy since our payload is trivial and could be a global const). The reason I bring this up even though Bsend works for us is that I would much rather deprecate Bsend that make freeing active requests erroneous. Jeff On Mon, Dec 14, 2015 at 7:55 AM, Daniel Holmes <dholmes(a)epcc.ed.ac.uk> wrote: > Hi all, > > The next point-to-point teleconference will be Monday 18th January 2016 at > 11am Central US via webex. > > I will send out another reminder nearer the time. > > Tentative agenda: > > MPI-4.0, issue #11: add communicator assertions (remember implications for > tools!) > MPI-4.0, issue #26: deprecate cancel for send requests > > MPI-4.0, issue TBD: add ARECV/FSEND > MPI-4.0, issue TBD: deal with minor correction to MPI_WAIT raised by Bill > and Rajeev (a la Bill's email 29/06/2015) > * MPI-4.0, issue TBD: deprecate request free for active receive requests > (arising from Dec 2015 meeting)* > MPI-4.0, issue TBD: define batch requests (a la Pavan's email of > 14/12/2015) > > * MPI-Next, issue TBD: disallow request free for active receive requests > (i.e. upgrade advice to erroneous)* > MPI-Next, issue #27: disallow cancel for send requests (i.e. upgrade > advice to erroneous) > > Cheers, > Dan. > > -- > Dan Holmes > Applications Consultant in HPC Research > EPCC, The University of Edinburgh > James Clerk Maxwell Building > The Kings Buildings > Peter Guthrie Tait Road > Edinburgh > EH9 3FD > T: +44(0)131 651 3465 > E: dholmes(a)epcc.ed.ac.uk > > *Please consider the environment before printing this email.* > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-p2p] Fwd: MPI P2P "messages may overtake" assertion
by Daniel Holmes 14 Dec '15

14 Dec '15

Hi Jeff, > ... tools can just intercept the info operations and strip this key if > they can't support it. Agreed: this should have been response 4 on my list (Markus and I discussed that option too). Cheers, Dan. On 14/12/2015 15:54, Jeff Hammond wrote: > Remember that one of the use cases of allow_overtaking is applications > that have exact matching, in which case allow_overtaking is a way of > turning off a feature that isn't used, in order to get a > high-performing message queue implementation. In the exact matching > case, tools will have no problem matching up sends and recvs. > > If tools cannot handle MPI_THREAD_MULTIPLE already, then I don't > really care if they can't support this assertion either. > > And in any case, such tools can just intercept the info operations and > strip this key if they can't support it. > > Jeff > > On Mon, Dec 14, 2015 at 7:44 AM, Daniel Holmes <dholmes(a)epcc.ed.ac.uk > <mailto:[email protected]>> wrote: > > Hi all, > > Markus has reminded me that we discussed the effect of the > mpi_assert_allow_overtaking assertion on tools (see below). > > The basic problem is that message matching would be > non-deterministic and it would be impossible for a tool to show > the user which receive operation satisfied which send operation > without internally using some sort of sequence number for each > send/receive operation. As Markus mentions, this is a similar > problem to the MPI_THREAD_MULTIPLE "logically concurrent" issue, > with which tools already struggle. > > My responses were: > 1) the user asked for this behaviour so the tool could simply > gracefully give up the linking function and just state the > information it knows > 2) the tool could hook all send and receive operations and > piggy-back a sequence number into the message header > 3) the tool could hook all send and receive operations and > serialise them to prevent overtaking > I don't know the viability/acceptability of any of these approaches. > > Preferably before the formal reading at the next face-to-face > meeting, we should discuss the possibility that this assertion > will disrupt tools and ascertain if we can/want to do anything to > mitigate that disruption. > > Cheers, > Dan. > > -------- Forwarded Message -------- > Subject: MPI P2P "messages may overtake" assertion > Date: Mon, 14 Dec 2015 16:24:44 +0100 > From: Markus Geimer <m.geimer(a)fz-juelich.de> > <mailto:[email protected]> > To: dholmes(a)epcc.ed.ac.uk <mailto:[email protected]> > CC: Marc-André Hermanns <m.a.hermanns(a)fz-juelich.de> > <mailto:[email protected]> > > > > Hi Daniel, > > Do you remember our conversation after the MPI BoF at SC'15 about the > proposed "messages may overtake" assertion and that it may cause severe > problems for tools, as correct message matching is very important for > many of them? In fact, the issue here is similar to properly supporting > MPI_THREAD_MULTIPLE, which most tools don't do either right now... > > Meanwhile I talked to Marc-Andre Hermanns from the Tools WG -- which > kick-started a discussion on this end. But I guess this topic also > requires some involvement of the P2P WG. For example, it would be good > if you could summarize the ideas you had in mind how tools could handle > this assertion. Thanks! > > That said, I don't think there is any urgency for discussions, as > people are slowly leaving for the Christmas break (e.g., I'm already > out of office). But we should definitely follow up on this early next > year. > > Merry Christmas, > Markus > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-Mail:[email protected] <mailto:[email protected]> > WWW:http://www.fz-juelich.de/jsc > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > > > > -- > Dan Holmes > Applications Consultant in HPC Research > EPCC, The University of Edinburgh > James Clerk Maxwell Building > The Kings Buildings > Peter Guthrie Tait Road > Edinburgh > EH9 3FD > T: +44(0)131 651 3465 > E:[email protected] <mailto:[email protected]> > > *Please consider the environment before printing this email.* > > > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Peter Guthrie Tait Road Edinburgh EH9 3FD T: +44(0)131 651 3465 E: dholmes(a)epcc.ed.ac.uk *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Next meeting 18th Jan 2016
by Daniel Holmes 14 Dec '15

14 Dec '15

Hi all, The next point-to-point teleconference will be Monday 18th January 2016 at 11am Central US via webex. I will send out another reminder nearer the time. Tentative agenda: MPI-4.0, issue #11: add communicator assertions (remember implications for tools!) MPI-4.0, issue #26: deprecate cancel for send requests MPI-4.0, issue TBD: add ARECV/FSEND MPI-4.0, issue TBD: deal with minor correction to MPI_WAIT raised by Bill and Rajeev (a la Bill's email 29/06/2015) MPI-4.0, issue TBD: deprecate request free for active receive requests (arising from Dec 2015 meeting) MPI-4.0, issue TBD: define batch requests (a la Pavan's email of 14/12/2015) MPI-Next, issue TBD: disallow request free for active receive requests (i.e. upgrade advice to erroneous) MPI-Next, issue #27: disallow cancel for send requests (i.e. upgrade advice to erroneous) Cheers, Dan. -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Peter Guthrie Tait Road Edinburgh EH9 3FD T: +44(0)131 651 3465 E: dholmes(a)epcc.ed.ac.uk *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-p2p] Fwd: MPI P2P "messages may overtake" assertion
by Jeff Hammond 14 Dec '15

14 Dec '15

Remember that one of the use cases of allow_overtaking is applications that have exact matching, in which case allow_overtaking is a way of turning off a feature that isn't used, in order to get a high-performing message queue implementation. In the exact matching case, tools will have no problem matching up sends and recvs. If tools cannot handle MPI_THREAD_MULTIPLE already, then I don't really care if they can't support this assertion either. And in any case, such tools can just intercept the info operations and strip this key if they can't support it. Jeff On Mon, Dec 14, 2015 at 7:44 AM, Daniel Holmes <dholmes(a)epcc.ed.ac.uk> wrote: > Hi all, > > Markus has reminded me that we discussed the effect of the > mpi_assert_allow_overtaking assertion on tools (see below). > > The basic problem is that message matching would be non-deterministic and > it would be impossible for a tool to show the user which receive operation > satisfied which send operation without internally using some sort of > sequence number for each send/receive operation. As Markus mentions, this > is a similar problem to the MPI_THREAD_MULTIPLE "logically concurrent" > issue, with which tools already struggle. > > My responses were: > 1) the user asked for this behaviour so the tool could simply gracefully > give up the linking function and just state the information it knows > 2) the tool could hook all send and receive operations and piggy-back a > sequence number into the message header > 3) the tool could hook all send and receive operations and serialise them > to prevent overtaking > I don't know the viability/acceptability of any of these approaches. > > Preferably before the formal reading at the next face-to-face meeting, we > should discuss the possibility that this assertion will disrupt tools and > ascertain if we can/want to do anything to mitigate that disruption. > > Cheers, > Dan. > > -------- Forwarded Message -------- Subject: MPI P2P "messages may > overtake" assertion Date: Mon, 14 Dec 2015 16:24:44 +0100 From: Markus > Geimer <m.geimer(a)fz-juelich.de> <m.geimer(a)fz-juelich.de> To: > dholmes(a)epcc.ed.ac.uk CC: Marc-André Hermanns <m.a.hermanns(a)fz-juelich.de> > <m.a.hermanns(a)fz-juelich.de> > > Hi Daniel, > > Do you remember our conversation after the MPI BoF at SC'15 about the > proposed "messages may overtake" assertion and that it may cause severe > problems for tools, as correct message matching is very important for > many of them? In fact, the issue here is similar to properly supporting > MPI_THREAD_MULTIPLE, which most tools don't do either right now... > > Meanwhile I talked to Marc-Andre Hermanns from the Tools WG -- which > kick-started a discussion on this end. But I guess this topic also > requires some involvement of the P2P WG. For example, it would be good > if you could summarize the ideas you had in mind how tools could handle > this assertion. Thanks! > > That said, I don't think there is any urgency for discussions, as > people are slowly leaving for the Christmas break (e.g., I'm already > out of office). But we should definitely follow up on this early next > year. > > Merry Christmas, > Markus > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-Mail: m.geimer(a)fz-juelich.de > WWW: http://www.fz-juelich.de/jsc > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > > > > > -- > Dan Holmes > Applications Consultant in HPC Research > EPCC, The University of Edinburgh > James Clerk Maxwell Building > The Kings Buildings > Peter Guthrie Tait Road > Edinburgh > EH9 3FD > T: +44(0)131 651 3465 > E: dholmes(a)epcc.ed.ac.uk > > *Please consider the environment before printing this email.* > > > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Fwd: MPI P2P "messages may overtake" assertion
by Daniel Holmes 14 Dec '15

14 Dec '15

Hi all, Markus has reminded me that we discussed the effect of the mpi_assert_allow_overtaking assertion on tools (see below). The basic problem is that message matching would be non-deterministic and it would be impossible for a tool to show the user which receive operation satisfied which send operation without internally using some sort of sequence number for each send/receive operation. As Markus mentions, this is a similar problem to the MPI_THREAD_MULTIPLE "logically concurrent" issue, with which tools already struggle. My responses were: 1) the user asked for this behaviour so the tool could simply gracefully give up the linking function and just state the information it knows 2) the tool could hook all send and receive operations and piggy-back a sequence number into the message header 3) the tool could hook all send and receive operations and serialise them to prevent overtaking I don't know the viability/acceptability of any of these approaches. Preferably before the formal reading at the next face-to-face meeting, we should discuss the possibility that this assertion will disrupt tools and ascertain if we can/want to do anything to mitigate that disruption. Cheers, Dan. -------- Forwarded Message -------- Subject: MPI P2P "messages may overtake" assertion Date: Mon, 14 Dec 2015 16:24:44 +0100 From: Markus Geimer <m.geimer(a)fz-juelich.de> To: dholmes(a)epcc.ed.ac.uk CC: Marc-André Hermanns <m.a.hermanns(a)fz-juelich.de> Hi Daniel, Do you remember our conversation after the MPI BoF at SC'15 about the proposed "messages may overtake" assertion and that it may cause severe problems for tools, as correct message matching is very important for many of them? In fact, the issue here is similar to properly supporting MPI_THREAD_MULTIPLE, which most tools don't do either right now... Meanwhile I talked to Marc-Andre Hermanns from the Tools WG -- which kick-started a discussion on this end. But I guess this topic also requires some involvement of the P2P WG. For example, it would be good if you could summarize the ideas you had in mind how tools could handle this assertion. Thanks! That said, I don't think there is any urgency for discussions, as people are slowly leaving for the Christmas break (e.g., I'm already out of office). But we should definitely follow up on this early next year. Merry Christmas, Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-Mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/jsc ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Peter Guthrie Tait Road Edinburgh EH9 3FD T: +44(0)131 651 3465 E: dholmes(a)epcc.ed.ac.uk *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-p2p] Next meeting: 23rd November 2015 11am Central US
by Dan Holmes 23 Nov '15

23 Nov '15

Hi all, The notes for the meeting we just had can be found on the newly created p2p git wiki: https://github.com/mpiwg-p2p/p2p-issues/wiki/notes-2015-11-23 Cheers, Dan. On 23/11/2015 15:17, Dan Holmes wrote: > Hi all, > > The next point-to-point teleconference will be today on Monday 23rd > November 2015 at 11am Central US via webex. > > Connection details are on the point-to-point wiki page: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/PtpWikiPage > > Agenda: > > 1a) Issue #11: "allowing INFO keys to change the semantic behaviour of > MPI" - preparing this for reading in December > > 1b) Issue #11: first batch of new communicator INFO keys - preparing > this for reading in December > > 2) Issue #TBD (formerly tickets #479, #480): "deprecating and then > removing MPI send cancel" - preparing this for discussion in December > > 3) Other stuff? > > Cheers, > Dan. > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0