December 2015 - mpiwg-tools - mailman3.cels.anl.gov

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Sato, Kento 18 Dec '15

18 Dec '15

Hi Marc-Andre, Thank you for the replay. RE: We analyze on the same scale as the measurement, thus we have one thread per thread-local trace. Each thread processes its own thread-local trace. When encountering a communication event, it re-enacts this communication using the recorded communication parameters (rank, tag, comm). A send event leads to an issued send, a receive event leads to an issued receive. (1) Replaying receive events Papers about “parallel replay (or record-and-replay)” uses (rank, rag, comm) for correct replay of message receive orders. Unfortunately, (rank, tag, comm) cannot replay message receive orders even in MPI_THREAD_SINGLE ** In general ** (Of course, it may work in particular case). You need to record (rank, message_id_number), and actually (tag, comm) does not work for this purpose. The details is described in Section 3.1 of this paper ( http://dl.acm.org/citation.cfm?id=2807642 ). I think we (Scalasca) does not have requirements as strict as the ones outlined in the paper. We need only to ensure that the same send/receive pairs also exchange data during the analysis (ideal case) or that we can at least detect a mismatch (in case of logically concurrent messages) and fix it locally, by exchanging the mixed up message payloads. If I understand section 3.1 correctly, the problem with out-of-order receives (Figure 3) does not pose a problem in our case, as we only care that msg1 is matched by req1 and msg2 is matched by req2, both in during measurement and replay. MPI ordering semantics should take care of that. If I correctly understand, you only need to ensure the same send/receive pairs, and you ** DO NOT ** care about the message receive orders. However, if you do not replay the message receive orders, you cannot correctly replay the same send/receive pairs because send destinations can change by message receive orders. For example, an application, e.g., N-body, changes send destinations according to some numerical result (X). In floating-point arithmetic, (a+b)+c is not necessary equal to a+(b+c). So the numerical result (X=a+b+c) can change according to message receive orders(a, b & c). It results in non-deterministic send destinations, thereby, non-deterministic send/recv pairs. I know it’s really corner case. So if detecting mismatch is find at least, this problem would be negligible for you. Kento ________________________________ Kento Sato | Center for Applied Scientific Computing (CASC) | Lawrence Livermore National Laboratory (LLNL) | http://people.llnl.gov/sato5 |

1 0

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Marc-Andre Hermanns 18 Dec '15

18 Dec '15

Hi Jeff, > at the moment we don't handle MPI_THREAD_MULTIPLE at all. But we want > to get there ;-) > > > You should vote for endpoints, as this may help you out here, > particularly if users start mapping endpoints 1:1 w/ threads. That would certainly ease things for us in these situations. Unfortunately endpoints force use to adapt other infrastructure in our measurement system. > b) Creating a derived datatype on the fly to add tool-level data to > the original payload may induce a large overhead in practically > _every_ send & receive operation and perturb the measurement. > > > You should evaluate this experimentally. I wrote a simple test > (https://github.com/jeffhammond/BigMPI/blob/master/test/perf/typepiggy.c) > and measured 1.5 us per call of overhead to create a datatype. That > is not significant except for very small messages. Thanks for the pointer. You are right. I should evaluate this further. 1.5us does indeed seem tolerable. I wonder how the influence of the derived datatype is on overall messaging performance, though. This is also something I should evaluate in the process. Cheers, Marc-Andre -- Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Schinkelstrasse 2 52062 Aachen Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc email: hermanns(a)jara.rwth-aachen.de

1 0

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Marc-Andre Hermanns 18 Dec '15

18 Dec '15

Hi Kento, >> We analyze on the same scale as the measurement, thus we have one >> thread per thread-local trace. Each thread processes its own >> thread-local trace. When encountering a communication event, it >> re-enacts this communication using the recorded communication >> parameters (rank, tag, comm). A send event leads to an issued >> send, a receive event leads to an issued receive. > > (1) Replaying receive events Papers about “parallel replay (or > record-and-replay)” uses (rank, rag, comm) for correct replay of > message receive orders. Unfortunately, (rank, tag, comm) cannot > replay message receive orders even in MPI_THREAD_SINGLE ** In > general ** (Of course, it may work in particular case). You need > to record (rank, message_id_number), and actually (tag, comm) does > not work for this purpose. The details is described in Section 3.1 > of this paper ( http://dl.acm.org/citation.cfm?id=2807642 ). I think we (Scalasca) does not have requirements as strict as the ones outlined in the paper. We need only to ensure that the same send/receive pairs also exchange data during the analysis (ideal case) or that we can at least detect a mismatch (in case of logically concurrent messages) and fix it locally, by exchanging the mixed up message payloads. If I understand section 3.1 correctly, the problem with out-of-order receives (Figure 3) does not pose a problem in our case, as we only care that msg1 is matched by req1 and msg2 is matched by req2, both in during measurement and replay. MPI ordering semantics should take care of that. > (2) Replaying send events Sorry if you’re already aware of this. > In some applications, usage of send calls, the destitution and the > message payload can change across different runs. So you also > need to record ALL non-deterministic operations which affect MPI > send behaviors. One of the examples is seed values for *void > srand(unsigned int* /seed/*)* for random numbers. time-related > function, such as gettimeofday(), can also be the example, thereby > MPI_Wtime(). I think for serial applications we are fine with what MPI provides/guarantees in message ordering semantics as outlined above. It is just in situations with logically concurrent messages that our level of replay may break down. >> b) Creating a derived datatype on the fly to add tool-level data >> to the original payload may induce a large overhead in >> practically _every_ send & receive operation and perturb the >> measurement. > > Yes, if it’s for performance analysis, it’ll somehow perturb the > measurement. This paper will help you to see if the piggybacking > overhead is acceptable or not. > http://greg.bronevetsky.com/papers/2008EuroPVM.pdf . Thanks for the pointer. I was aware of the paper, but it is quite a while back that I read it. As you pointed out, separate messages are out of the question. For large buffers, I'd think the packing would also be quite disadvantageous, as I'd need a malloc before the pack, right? So the only option left would be the datatype (struct), also suggested by Jeff in a different reply. My concerns here were not only that the construction is costly (which Jeff showed it need not be), but the fact that I turn _any_ contiguous datatype into a non-contiguous datatype and some NICs may perform significantly different with those. The paper seems to at least suggest this with the use of ad-hoc datatypes. Cheers, Marc-Andre -- Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Schinkelstrasse 2 52062 Aachen Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc email: hermanns(a)jara.rwth-aachen.de

1 0

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Jeff Hammond 17 Dec '15

17 Dec '15

On Thu, Dec 17, 2015 at 4:32 AM, Marc-Andre Hermanns < hermanns(a)jara.rwth-aachen.de> wrote: > Hi Jeff, > > at the moment we don't handle MPI_THREAD_MULTIPLE at all. But we want > to get there ;-) > > You should vote for endpoints, as this may help you out here, particularly if users start mapping endpoints 1:1 w/ threads. > Here is a short recollection of what we do/need. Sorry for the folks > who know/read this already in other context: > > We use, what we call "parallel replay" to analyze large event traces > in parallel. Each thread has its own stream of events, such as enter > and exit for tracking the calling context as well as send and receive > for communication among ranks. > > We analyze on the same scale as the measurement, thus we have one > thread per thread-local trace. Each thread processes its own > thread-local trace. When encountering a communication event, it > re-enacts this communication using the recorded communication > parameters (rank, tag, comm). A send event leads to an issued send, a > receive event leads to an issued receive. > > It is critical that during the analysis, the message matching is > identical to the original application. However, we do not re-enact any > computational time, that is the temporal distance between sends and > receives is certainly different from the original application. As a > consequence, while two sends may have some significant temporal > distance in the original measurement, they could be issued right after > each other during the analysis. > > Markus Geimer and I believe that creating some sort of a sequence > number during measurement could help matching the right messages > during the analysis, as a process could detect that it got mismatched > messages and communicate with other threads to get the correct one. > > > It is unclear, however, how to achieve this: > > a) Sending an additional message within the MPI wrapper at measurement > time may lead to invalid matchings, as the additional message may be > received by a different thread. > > b) Creating a derived datatype on the fly to add tool-level data to > the original payload may induce a large overhead in practically > _every_ send & receive operation and perturb the measurement. > > You should evaluate this experimentally. I wrote a simple test ( https://github.com/jeffhammond/BigMPI/blob/master/test/perf/typepiggy.c) and measured 1.5 us per call of overhead to create a datatype. That is not significant except for very small messages. Jeff The best idea in the room so far is some callback mechanism into the > MPI implementation that generates matching information both on sender > and receiver side to generate some form of a sequence number that can > then be saved during measurement. If available on both sender and > receiver this information could then be used to fix incorrect matching > during the analysis. > > Cheers, > Marc-Andre > > On 16.12.15 16:40, Jeff Hammond wrote: > > How do you handle MPI_THREAD_MULTIPLE? Understanding what your tool > > does there is a good starting point for this discussion. > > > > Jeff > > > > On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns > > <hermanns(a)jara.rwth-aachen.de <mailto:[email protected]>> > > wrote: > > > > Hi all, > > > > CC: Tools-WG, Markus Geimer (not on either list) > > > > sorry for starting a new thread and being so verbose, but I > subscribed > > just now. I quoted Dan, Jeff, and Jim from the archive as > appropriate. > > > > First, let me state that we do not want to prevent this assertion in > > any way. For us as tools provider it is just quite a brain tickler on > > how to support this in our tool and in general. > > > > Dan wrote: > > >>> [...] The basic problem is that message matching would be > > >>> non-deterministic and it would be impossible for a tool to show > > >>> the user which receive operation satisfied which send operation > > >>> without internally using some sort of sequence number for each > > >>> send/receive operation. [...] > > >>> > > >>> My responses were: > > >>> 1) the user asked for this behaviour so the tool could simply > > >>> gracefully give up the linking function and just state the > > >>> information it knows > > > > > Giving up can only be a temporary solution for tools. The user wants > > to use this advanced feature, thus just saying: "Hey, what you're > > doing is too sophisticated for us. You are on your own now." is not a > > viable long-term strategy. > > > > >>> 2) the tool could hook all send and receive operations and > > >>> piggy-back a sequence number into the message header > > > > We discussed piggy-backing within the tools group some time in the > > past, but never came to a satisfying way of how to implement this. > If, > > in the process of reviving the discussion on a piggy-backing > > interface, we come to a viable solution, it would certainly help with > > the our issues with message matching in general. > > > > Scalasca's problem here is that we need to detect (and partly > > recreate) the exact order of message matching to have the correct > > message reach the right receivers. > > > > >>> 3) the tool could hook all send and receive operations and > > >>> serialise them to prevent overtaking > > > > This is not an option for me. A "performance tool" should strive to > > measure as close to the original behavior as possible. Changing > > communication semantics just to make a tool "work" would have too > > great of an impact on application behavior. After all, if it would > > have only little impact, why should the user choose this option in > the > > first place. > > > > Jeff wrote: > > >> Remember that one of the use cases of allow_overtaking is > > applications that > > >> have exact matching, in which case allow_overtaking is a way of > > turning off > > >> a feature that isn't used, in order to get a high-performing > > message queue > > >> implementation. In the exact matching case, tools will have no > > problem > > >> matching up sends and recvs. > > > > This is true. If the tools can identify this scenario, it could be > > supported by current tools without significant change. However, as it > > is not generally forbidden to have inexact matching (right?), it is > > unclear on how the tools would detect this. > > > > What about an additional info key a user can set in this respect: > > > > exact_matching => true/false > > > > in which the user can state whether it is indeed a scenario of exact > > matching or not. The tool could check this, and issue a warning. > > > > >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I > > don't really > > >> care if they can't support this assertion either. > > > > Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. > ;-) > > > > As I said, we (Markus and I) see this as a trigger to come to a > viable > > solution for tools like ours to support either situation. > > > > >> And in any case, such tools can just intercept the info > > operations and > > >> strip this key if they can't support it. > > > > As I wrote above in reply to Dan, stripping options that influence > > behavior is not a good option. I, personally, would rather bail out > > than (silently) change messaging semantics. I can't say what Markus' > > take on this is. > > > > Jim wrote: > > > I don't really see any necessary fix to the proposal. We could > > add an > > > advice to users to remind them that they should ensure tools are > > compatible > > > with the info keys. And the reverse advice to tools writers that > > they > > > should check info keys for compatibility. > > > > I would second this idea, while emphasizing the burden to be on the > > tool to check for this info key (and potentially others) and warn the > > user of "undersupport". > > > > Cheers, > > Marc-Andre > > -- > > Marc-Andre Hermanns > > Jülich Aachen Research Alliance, > > High Performance Computing (JARA-HPC) > > Jülich Supercomputing Centre (JSC) > > > > Schinkelstrasse 2 > > 52062 Aachen > > Germany > > > > Phone: +49 2461 61 2509 | +49 241 80 24381 > > Fax: +49 2461 80 6 99753 > > www.jara.org/jara-hpc <http://www.jara.org/jara-hpc> > > email: hermanns(a)jara.rwth-aachen.de > > <mailto:[email protected]> > > > > > > _______________________________________________ > > mpiwg-p2p mailing list > > mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > > > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com <mailto:[email protected]> > > http://jeffhammond.github.io/ > > > > > > _______________________________________________ > > mpiwg-p2p mailing list > > mpiwg-p2p(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > > > > -- > Marc-Andre Hermanns > Jülich Aachen Research Alliance, > High Performance Computing (JARA-HPC) > Jülich Supercomputing Centre (JSC) > > Schinkelstrasse 2 > 52062 Aachen > Germany > > Phone: +49 2461 61 2509 | +49 241 80 24381 > Fax: +49 2461 80 6 99753 > www.jara.org/jara-hpc > email: hermanns(a)jara.rwth-aachen.de > > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Sato, Kento 17 Dec '15

17 Dec '15

Hi Marc-Andre, Let me leave my comments on “parallel replay”. I've just subscribed this list, so if my comment does not make sense, just drop this email. RE: We analyze on the same scale as the measurement, thus we have one thread per thread-local trace. Each thread processes its own thread-local trace. When encountering a communication event, it re-enacts this communication using the recorded communication parameters (rank, tag, comm). A send event leads to an issued send, a receive event leads to an issued receive. (1) Replaying receive events Papers about “parallel replay (or record-and-replay)” uses (rank, rag, comm) for correct replay of message receive orders. Unfortunately, (rank, tag, comm) cannot replay message receive orders even in MPI_THREAD_SINGLE ** In general ** (Of course, it may work in particular case). You need to record (rank, message_id_number), and actually (tag, comm) does not work for this purpose. The details is described in Section 3.1 of this paper ( http://dl.acm.org/citation.cfm?id=2807642 ). (2) Replaying send events Sorry if you’re already aware of this. In some applications, usage of send calls, the destitution and the message payload can change across different runs. So you also need to record ALL non-deterministic operations which affect MPI send behaviors. One of the examples is seed values for void srand(unsigned int seed) for random numbers. time-related function, such as gettimeofday(), can also be the example, thereby MPI_Wtime(). RE: a) Sending an additional message within the MPI wrapper at measurement time may lead to invalid matchings, as the additional message may be received by a different thread. Yes, that’s true. RE; b) Creating a derived datatype on the fly to add tool-level data to the original payload may induce a large overhead in practically _every_ send & receive operation and perturb the measurement. Yes, if it’s for performance analysis, it’ll somehow perturb the measurement. This paper will help you to see if the piggybacking overhead is acceptable or not. http://greg.bronevetsky.com/papers/2008EuroPVM.pdf . It evaluates three piggybacking methods: (1) Explicit Pack Operations, (2) Datatypes and (3) Separate Messages. But as you pointed out (3) would not work for MPI_THREAD_MULTIPLE . Kento ________________________________ Kento Sato | Center for Applied Scientific Computing (CASC) | Lawrence Livermore National Laboratory (LLNL) | http://people.llnl.gov/sato5 | On Dec 17, 2015, at 6:32 AM, Marc-Andre Hermanns <hermanns(a)jara.rwth-aachen.de<mailto:[email protected]>> wrote: Hi Jeff, at the moment we don't handle MPI_THREAD_MULTIPLE at all. But we want to get there ;-) Here is a short recollection of what we do/need. Sorry for the folks who know/read this already in other context: We use, what we call "parallel replay" to analyze large event traces in parallel. Each thread has its own stream of events, such as enter and exit for tracking the calling context as well as send and receive for communication among ranks. We analyze on the same scale as the measurement, thus we have one thread per thread-local trace. Each thread processes its own thread-local trace. When encountering a communication event, it re-enacts this communication using the recorded communication parameters (rank, tag, comm). A send event leads to an issued send, a receive event leads to an issued receive. It is critical that during the analysis, the message matching is identical to the original application. However, we do not re-enact any computational time, that is the temporal distance between sends and receives is certainly different from the original application. As a consequence, while two sends may have some significant temporal distance in the original measurement, they could be issued right after each other during the analysis. Markus Geimer and I believe that creating some sort of a sequence number during measurement could help matching the right messages during the analysis, as a process could detect that it got mismatched messages and communicate with other threads to get the correct one. It is unclear, however, how to achieve this: a) Sending an additional message within the MPI wrapper at measurement time may lead to invalid matchings, as the additional message may be received by a different thread. b) Creating a derived datatype on the fly to add tool-level data to the original payload may induce a large overhead in practically _every_ send & receive operation and perturb the measurement. The best idea in the room so far is some callback mechanism into the MPI implementation that generates matching information both on sender and receiver side to generate some form of a sequence number that can then be saved during measurement. If available on both sender and receiver this information could then be used to fix incorrect matching during the analysis. Cheers, Marc-Andre On 16.12.15 16:40, Jeff Hammond wrote: How do you handle MPI_THREAD_MULTIPLE? Understanding what your tool does there is a good starting point for this discussion. Jeff On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns <hermanns(a)jara.rwth-aachen.de<mailto:[email protected]> <mailto:[email protected]>> wrote: Hi all, CC: Tools-WG, Markus Geimer (not on either list) sorry for starting a new thread and being so verbose, but I subscribed just now. I quoted Dan, Jeff, and Jim from the archive as appropriate. First, let me state that we do not want to prevent this assertion in any way. For us as tools provider it is just quite a brain tickler on how to support this in our tool and in general. Dan wrote: [...] The basic problem is that message matching would be non-deterministic and it would be impossible for a tool to show the user which receive operation satisfied which send operation without internally using some sort of sequence number for each send/receive operation. [...] My responses were: 1) the user asked for this behaviour so the tool could simply gracefully give up the linking function and just state the information it knows Giving up can only be a temporary solution for tools. The user wants to use this advanced feature, thus just saying: "Hey, what you're doing is too sophisticated for us. You are on your own now." is not a viable long-term strategy. 2) the tool could hook all send and receive operations and piggy-back a sequence number into the message header We discussed piggy-backing within the tools group some time in the past, but never came to a satisfying way of how to implement this. If, in the process of reviving the discussion on a piggy-backing interface, we come to a viable solution, it would certainly help with the our issues with message matching in general. Scalasca's problem here is that we need to detect (and partly recreate) the exact order of message matching to have the correct message reach the right receivers. 3) the tool could hook all send and receive operations and serialise them to prevent overtaking This is not an option for me. A "performance tool" should strive to measure as close to the original behavior as possible. Changing communication semantics just to make a tool "work" would have too great of an impact on application behavior. After all, if it would have only little impact, why should the user choose this option in the first place. Jeff wrote: Remember that one of the use cases of allow_overtaking is applications that have exact matching, in which case allow_overtaking is a way of turning off a feature that isn't used, in order to get a high-performing message queue implementation. In the exact matching case, tools will have no problem matching up sends and recvs. This is true. If the tools can identify this scenario, it could be supported by current tools without significant change. However, as it is not generally forbidden to have inexact matching (right?), it is unclear on how the tools would detect this. What about an additional info key a user can set in this respect: exact_matching => true/false in which the user can state whether it is indeed a scenario of exact matching or not. The tool could check this, and issue a warning. If tools cannot handle MPI_THREAD_MULTIPLE already, then I don't really care if they can't support this assertion either. Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. ;-) As I said, we (Markus and I) see this as a trigger to come to a viable solution for tools like ours to support either situation. And in any case, such tools can just intercept the info operations and strip this key if they can't support it. As I wrote above in reply to Dan, stripping options that influence behavior is not a good option. I, personally, would rather bail out than (silently) change messaging semantics. I can't say what Markus' take on this is. Jim wrote: I don't really see any necessary fix to the proposal. We could add an advice to users to remind them that they should ensure tools are compatible with the info keys. And the reverse advice to tools writers that they should check info keys for compatibility. I would second this idea, while emphasizing the burden to be on the tool to check for this info key (and potentially others) and warn the user of "undersupport". Cheers, Marc-Andre -- Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Schinkelstrasse 2 52062 Aachen Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc<http://www.jara.org/jara-hpc> <http://www.jara.org/jara-hpc> email: hermanns(a)jara.rwth-aachen.de<mailto:[email protected]> <mailto:[email protected]> _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> <mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p -- Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Schinkelstrasse 2 52062 Aachen Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc<http://www.jara.org/jara-hpc> email: hermanns(a)jara.rwth-aachen.de<mailto:[email protected]> _______________________________________________ mpiwg-tools mailing list mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-tools

1 0

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Marc-Andre Hermanns 17 Dec '15

17 Dec '15

Hi Daniel, > In the existing MPI Standard, any program that relies on the tool > being able to recover the particular matching order that actually > happened from a set of possible matching orders that all satisfy > the MPI definitions (for example by relying on the tool using some > externally imposed mechanism, such as piggy-backed sequence > numbers) is relying on a particular sequentialisation of a > race-condition between logically concurrent MPI messages, which is > specifically called out as being an erroneous program. I am a little confused on what the 'program' is and what the 'tool' is. My tool is a program, and it relies on the reconstruction of the message matching that actually occurred during measurement. I agree that our tool would therefore be an erroneous program in the MPI_THREAD_MULTIPLE case. This is what we want to fix. ;-) > From MPI Forum's point-of-view, therefore, any behaviour in this > situation is allowed, including setting the data-centre on fire, > and our work here is done. From the P2P group's perspective, the work may be considered done. From the tools group's perspective, there would be further work, as it is our job to create interfaces that enable tools to support the full MPI feature set. Some of the proposed solutions may touch "P2P territory" again, which is why I want to keep the group in the discussion. > From a tool's perspective, however, the existence and consequences > of this situation should be discovered and presented to the user > [...] Alternatively, the existence of this situation could be > discovered by examination of the trace during post-processing or > analysis. As a first step, a warning would be good, but ultimately we and also a user would want a tool to be able to deal with such scenarios. > The consequences of this situation could be presented to the user > by showing all possible matching orders, as indicated by the > information in the trace concerning which messages were logically > concurrent during the actual run of the application. The tool would > not be able to tell the user which matching order actually occurred > during the measurement run but it would be able to identify that > there was a race-condition, display all the possible outcomes, and > simulate the effects of each possible route through the program. The problem at hand is that the race-condition is in the analyzer, not necessarily the application. The application may deal with ordering in a different way, but a performance tool like ours needs to reconstruct "what actually happened", as analysis results depend on it. Cheers, Marc-Andre -- Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Schinkelstrasse 2 52062 Aachen Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc email: hermanns(a)jara.rwth-aachen.de

1 0

Re: [mpiwg-tools] Meeting reminder: Thursday Dec 17
by Mohror, Kathryn 17 Dec '15

17 Dec '15

Hi all, The notes from today's meeting are here: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2015-12-17 I'm not sure if I got all the finer points, so if anyone sees an error, etc., please let me know. Our next meeting will be Jan 14. Happy holidays everybody! Kathryn _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov, http://scalability.llnl.gov/ Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA > -----Original Message----- > From: mpiwg-tools [mailto:[email protected]] On > Behalf Of Kathryn Mohror > Sent: Tuesday, December 15, 2015 7:25 AM > To: MPI Tools > Subject: [mpiwg-tools] Meeting reminder: Thursday Dec 17 > > Hi all, > > This is a reminder of our regularly scheduled Tools WG meeting Thursday Dec > 17 8:00 AM PST, 11:00 AM EST, 5:00 PM CET. > > The link for the webex is: > https://cisco.webex.com/ciscosales/j.php?MTID=m2c3f15fb156cb498d61af2 > e40e4da > 26b > > And meeting information can be found here: > https://github.com/mpiwg-tools/tools-issues/wiki/Meetings > > Agenda: > This week we will be joined by folks who are interested in using the > information provided by the MPI Handles Interface we are developing for > debuggers. They will provide their "wish list" for the MPI Objects and > information about them that they believe will be most useful in debugging > their codes. We will discuss how we can potentially support their wish list. > > Since it's close to the holidays and right after the Forum meeting, I don't have > other items on the agenda for this week. Depending on how the Handles > discussion goes, this might be a shorter meeting. > > Hope to talk with you soon! > Kathryn > > __________________________________________________________ > _______ > Kathryn Mohror, kathryn(a)llnl.gov, http://scalability.llnl.gov/ Scalability Team > @ Lawrence Livermore National Laboratory, Livermore, CA, USA > > > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-tools

1 0

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Daniel Holmes 17 Dec '15

17 Dec '15

Hi Marc-Andre, The mpi_assert_allow_overtaking INFO key essentially makes *all* send operations that have identical matching criteria logically concurrent and *all* receive operations that have overlapping matching criteria logically concurrent, irrespective of which thread issued them. So, using the INFO key can be seen as a special case of handling the full MPI_THREAD_MULTIPLE case. In the existing MPI Standard, any program that relies on the tool being able to recover the particular matching order that actually happened from a set of possible matching orders that all satisfy the MPI definitions (for example by relying on the tool using some externally imposed mechanism, such as piggy-backed sequence numbers) is relying on a particular sequentialisation of a race-condition between logically concurrent MPI messages, which is specifically called out as being an erroneous program. From MPI Forum's point-of-view, therefore, any behaviour in this situation is allowed, including setting the data-centre on fire, and our work here is done. From a tool's perspective, however, the existence and consequences of this situation should be discovered and presented to the user so that they can fix their erroneous program. This could be done by recognising when multiple send operations are logically concurrent and when multiple receive operations are logically concurrent. If two send operations are issued with the same matching criteria but by different threads then these will be logically concurrent (edit: unless an MPI synchronisation point exists such that the two cannot conflict). The same is true of receive operations. The existence of this situation could be recorded during measurement (at the cost of searching the entire trace so far for the OS process (edit: since the last MPI synchronisation point) each time a new send or receive is issued). Alternatively, the existence of this situation could be discovered by examination of the trace during post-processing or analysis. The consequences of this situation could be presented to the user by showing all possible matching orders, as indicated by the information in the trace concerning which messages were logically concurrent during the actual run of the application. The tool would not be able to tell the user which matching order actually occurred during the measurement run but it would be able to identify that there was a race-condition, display all the possible outcomes, and simulate the effects of each possible route through the program. Does this make sense and is this sufficient? Cheers, Dan. On 17/12/2015 14:32, Marc-Andre Hermanns wrote: > Hi Jeff, > > at the moment we don't handle MPI_THREAD_MULTIPLE at all. But we want > to get there ;-) > > Here is a short recollection of what we do/need. Sorry for the folks > who know/read this already in other context: > > We use, what we call "parallel replay" to analyze large event traces > in parallel. Each thread has its own stream of events, such as enter > and exit for tracking the calling context as well as send and receive > for communication among ranks. > > We analyze on the same scale as the measurement, thus we have one > thread per thread-local trace. Each thread processes its own > thread-local trace. When encountering a communication event, it > re-enacts this communication using the recorded communication > parameters (rank, tag, comm). A send event leads to an issued send, a > receive event leads to an issued receive. > > It is critical that during the analysis, the message matching is > identical to the original application. However, we do not re-enact any > computational time, that is the temporal distance between sends and > receives is certainly different from the original application. As a > consequence, while two sends may have some significant temporal > distance in the original measurement, they could be issued right after > each other during the analysis. > > Markus Geimer and I believe that creating some sort of a sequence > number during measurement could help matching the right messages > during the analysis, as a process could detect that it got mismatched > messages and communicate with other threads to get the correct one. > > > It is unclear, however, how to achieve this: > > a) Sending an additional message within the MPI wrapper at measurement > time may lead to invalid matchings, as the additional message may be > received by a different thread. > > b) Creating a derived datatype on the fly to add tool-level data to > the original payload may induce a large overhead in practically > _every_ send & receive operation and perturb the measurement. > > The best idea in the room so far is some callback mechanism into the > MPI implementation that generates matching information both on sender > and receiver side to generate some form of a sequence number that can > then be saved during measurement. If available on both sender and > receiver this information could then be used to fix incorrect matching > during the analysis. > > Cheers, > Marc-Andre > > On 16.12.15 16:40, Jeff Hammond wrote: >> How do you handle MPI_THREAD_MULTIPLE? Understanding what your tool >> does there is a good starting point for this discussion. >> >> Jeff >> >> On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns >> <hermanns(a)jara.rwth-aachen.de <mailto:[email protected]>> >> wrote: >> >> Hi all, >> >> CC: Tools-WG, Markus Geimer (not on either list) >> >> sorry for starting a new thread and being so verbose, but I subscribed >> just now. I quoted Dan, Jeff, and Jim from the archive as appropriate. >> >> First, let me state that we do not want to prevent this assertion in >> any way. For us as tools provider it is just quite a brain tickler on >> how to support this in our tool and in general. >> >> Dan wrote: >> >>> [...] The basic problem is that message matching would be >> >>> non-deterministic and it would be impossible for a tool to show >> >>> the user which receive operation satisfied which send operation >> >>> without internally using some sort of sequence number for each >> >>> send/receive operation. [...] >> >>> >> >>> My responses were: >> >>> 1) the user asked for this behaviour so the tool could simply >> >>> gracefully give up the linking function and just state the >> >>> information it knows >> > >> Giving up can only be a temporary solution for tools. The user wants >> to use this advanced feature, thus just saying: "Hey, what you're >> doing is too sophisticated for us. You are on your own now." is not a >> viable long-term strategy. >> >> >>> 2) the tool could hook all send and receive operations and >> >>> piggy-back a sequence number into the message header >> >> We discussed piggy-backing within the tools group some time in the >> past, but never came to a satisfying way of how to implement this. If, >> in the process of reviving the discussion on a piggy-backing >> interface, we come to a viable solution, it would certainly help with >> the our issues with message matching in general. >> >> Scalasca's problem here is that we need to detect (and partly >> recreate) the exact order of message matching to have the correct >> message reach the right receivers. >> >> >>> 3) the tool could hook all send and receive operations and >> >>> serialise them to prevent overtaking >> >> This is not an option for me. A "performance tool" should strive to >> measure as close to the original behavior as possible. Changing >> communication semantics just to make a tool "work" would have too >> great of an impact on application behavior. After all, if it would >> have only little impact, why should the user choose this option in the >> first place. >> >> Jeff wrote: >> >> Remember that one of the use cases of allow_overtaking is >> applications that >> >> have exact matching, in which case allow_overtaking is a way of >> turning off >> >> a feature that isn't used, in order to get a high-performing >> message queue >> >> implementation. In the exact matching case, tools will have no >> problem >> >> matching up sends and recvs. >> >> This is true. If the tools can identify this scenario, it could be >> supported by current tools without significant change. However, as it >> is not generally forbidden to have inexact matching (right?), it is >> unclear on how the tools would detect this. >> >> What about an additional info key a user can set in this respect: >> >> exact_matching => true/false >> >> in which the user can state whether it is indeed a scenario of exact >> matching or not. The tool could check this, and issue a warning. >> >> >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I >> don't really >> >> care if they can't support this assertion either. >> >> Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. ;-) >> >> As I said, we (Markus and I) see this as a trigger to come to a viable >> solution for tools like ours to support either situation. >> >> >> And in any case, such tools can just intercept the info >> operations and >> >> strip this key if they can't support it. >> >> As I wrote above in reply to Dan, stripping options that influence >> behavior is not a good option. I, personally, would rather bail out >> than (silently) change messaging semantics. I can't say what Markus' >> take on this is. >> >> Jim wrote: >> > I don't really see any necessary fix to the proposal. We could >> add an >> > advice to users to remind them that they should ensure tools are >> compatible >> > with the info keys. And the reverse advice to tools writers that >> they >> > should check info keys for compatibility. >> >> I would second this idea, while emphasizing the burden to be on the >> tool to check for this info key (and potentially others) and warn the >> user of "undersupport". >> >> Cheers, >> Marc-Andre >> -- >> Marc-Andre Hermanns >> Jï¿½lich Aachen Research Alliance, >> High Performance Computing (JARA-HPC) >> Jï¿½lich Supercomputing Centre (JSC) >> >> Schinkelstrasse 2 >> 52062 Aachen >> Germany >> >> Phone: +49 2461 61 2509 | +49 241 80 24381 >> Fax: +49 2461 80 6 99753 >> www.jara.org/jara-hpc <http://www.jara.org/jara-hpc> >> email: hermanns(a)jara.rwth-aachen.de >> <mailto:[email protected]> >> >> >> _______________________________________________ >> mpiwg-p2p mailing list >> mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com <mailto:[email protected]> >> http://jeffhammond.github.io/ >> >> >> _______________________________________________ >> mpiwg-p2p mailing list >> mpiwg-p2p(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p >> > > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Peter Guthrie Tait Road Edinburgh EH9 3FD T: +44(0)131 651 3465 E: dholmes(a)epcc.ed.ac.uk *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Marc-Andre Hermanns 17 Dec '15

17 Dec '15

Hi Jeff, at the moment we don't handle MPI_THREAD_MULTIPLE at all. But we want to get there ;-) Here is a short recollection of what we do/need. Sorry for the folks who know/read this already in other context: We use, what we call "parallel replay" to analyze large event traces in parallel. Each thread has its own stream of events, such as enter and exit for tracking the calling context as well as send and receive for communication among ranks. We analyze on the same scale as the measurement, thus we have one thread per thread-local trace. Each thread processes its own thread-local trace. When encountering a communication event, it re-enacts this communication using the recorded communication parameters (rank, tag, comm). A send event leads to an issued send, a receive event leads to an issued receive. It is critical that during the analysis, the message matching is identical to the original application. However, we do not re-enact any computational time, that is the temporal distance between sends and receives is certainly different from the original application. As a consequence, while two sends may have some significant temporal distance in the original measurement, they could be issued right after each other during the analysis. Markus Geimer and I believe that creating some sort of a sequence number during measurement could help matching the right messages during the analysis, as a process could detect that it got mismatched messages and communicate with other threads to get the correct one. It is unclear, however, how to achieve this: a) Sending an additional message within the MPI wrapper at measurement time may lead to invalid matchings, as the additional message may be received by a different thread. b) Creating a derived datatype on the fly to add tool-level data to the original payload may induce a large overhead in practically _every_ send & receive operation and perturb the measurement. The best idea in the room so far is some callback mechanism into the MPI implementation that generates matching information both on sender and receiver side to generate some form of a sequence number that can then be saved during measurement. If available on both sender and receiver this information could then be used to fix incorrect matching during the analysis. Cheers, Marc-Andre On 16.12.15 16:40, Jeff Hammond wrote: > How do you handle MPI_THREAD_MULTIPLE? Understanding what your tool > does there is a good starting point for this discussion. > > Jeff > > On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns > <hermanns(a)jara.rwth-aachen.de <mailto:[email protected]>> > wrote: > > Hi all, > > CC: Tools-WG, Markus Geimer (not on either list) > > sorry for starting a new thread and being so verbose, but I subscribed > just now. I quoted Dan, Jeff, and Jim from the archive as appropriate. > > First, let me state that we do not want to prevent this assertion in > any way. For us as tools provider it is just quite a brain tickler on > how to support this in our tool and in general. > > Dan wrote: > >>> [...] The basic problem is that message matching would be > >>> non-deterministic and it would be impossible for a tool to show > >>> the user which receive operation satisfied which send operation > >>> without internally using some sort of sequence number for each > >>> send/receive operation. [...] > >>> > >>> My responses were: > >>> 1) the user asked for this behaviour so the tool could simply > >>> gracefully give up the linking function and just state the > >>> information it knows > > > Giving up can only be a temporary solution for tools. The user wants > to use this advanced feature, thus just saying: "Hey, what you're > doing is too sophisticated for us. You are on your own now." is not a > viable long-term strategy. > > >>> 2) the tool could hook all send and receive operations and > >>> piggy-back a sequence number into the message header > > We discussed piggy-backing within the tools group some time in the > past, but never came to a satisfying way of how to implement this. If, > in the process of reviving the discussion on a piggy-backing > interface, we come to a viable solution, it would certainly help with > the our issues with message matching in general. > > Scalasca's problem here is that we need to detect (and partly > recreate) the exact order of message matching to have the correct > message reach the right receivers. > > >>> 3) the tool could hook all send and receive operations and > >>> serialise them to prevent overtaking > > This is not an option for me. A "performance tool" should strive to > measure as close to the original behavior as possible. Changing > communication semantics just to make a tool "work" would have too > great of an impact on application behavior. After all, if it would > have only little impact, why should the user choose this option in the > first place. > > Jeff wrote: > >> Remember that one of the use cases of allow_overtaking is > applications that > >> have exact matching, in which case allow_overtaking is a way of > turning off > >> a feature that isn't used, in order to get a high-performing > message queue > >> implementation. In the exact matching case, tools will have no > problem > >> matching up sends and recvs. > > This is true. If the tools can identify this scenario, it could be > supported by current tools without significant change. However, as it > is not generally forbidden to have inexact matching (right?), it is > unclear on how the tools would detect this. > > What about an additional info key a user can set in this respect: > > exact_matching => true/false > > in which the user can state whether it is indeed a scenario of exact > matching or not. The tool could check this, and issue a warning. > > >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I > don't really > >> care if they can't support this assertion either. > > Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. ;-) > > As I said, we (Markus and I) see this as a trigger to come to a viable > solution for tools like ours to support either situation. > > >> And in any case, such tools can just intercept the info > operations and > >> strip this key if they can't support it. > > As I wrote above in reply to Dan, stripping options that influence > behavior is not a good option. I, personally, would rather bail out > than (silently) change messaging semantics. I can't say what Markus' > take on this is. > > Jim wrote: > > I don't really see any necessary fix to the proposal. We could > add an > > advice to users to remind them that they should ensure tools are > compatible > > with the info keys. And the reverse advice to tools writers that > they > > should check info keys for compatibility. > > I would second this idea, while emphasizing the burden to be on the > tool to check for this info key (and potentially others) and warn the > user of "undersupport". > > Cheers, > Marc-Andre > -- > Marc-Andre Hermanns > Jülich Aachen Research Alliance, > High Performance Computing (JARA-HPC) > Jülich Supercomputing Centre (JSC) > > Schinkelstrasse 2 > 52062 Aachen > Germany > > Phone: +49 2461 61 2509 | +49 241 80 24381 > Fax: +49 2461 80 6 99753 > www.jara.org/jara-hpc <http://www.jara.org/jara-hpc> > email: hermanns(a)jara.rwth-aachen.de > <mailto:[email protected]> > > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ > > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > -- Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Schinkelstrasse 2 52062 Aachen Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc email: hermanns(a)jara.rwth-aachen.de

1 0

Re: [mpiwg-tools] [mpiwg-p2p] Message matching for tools
by Jeff Hammond 16 Dec '15

16 Dec '15

How do you handle MPI_THREAD_MULTIPLE? Understanding what your tool does there is a good starting point for this discussion. Jeff On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns < hermanns(a)jara.rwth-aachen.de> wrote: > Hi all, > > CC: Tools-WG, Markus Geimer (not on either list) > > sorry for starting a new thread and being so verbose, but I subscribed > just now. I quoted Dan, Jeff, and Jim from the archive as appropriate. > > First, let me state that we do not want to prevent this assertion in > any way. For us as tools provider it is just quite a brain tickler on > how to support this in our tool and in general. > > Dan wrote: > >>> [...] The basic problem is that message matching would be > >>> non-deterministic and it would be impossible for a tool to show > >>> the user which receive operation satisfied which send operation > >>> without internally using some sort of sequence number for each > >>> send/receive operation. [...] > >>> > >>> My responses were: > >>> 1) the user asked for this behaviour so the tool could simply > >>> gracefully give up the linking function and just state the > >>> information it knows > > > Giving up can only be a temporary solution for tools. The user wants > to use this advanced feature, thus just saying: "Hey, what you're > doing is too sophisticated for us. You are on your own now." is not a > viable long-term strategy. > > >>> 2) the tool could hook all send and receive operations and > >>> piggy-back a sequence number into the message header > > We discussed piggy-backing within the tools group some time in the > past, but never came to a satisfying way of how to implement this. If, > in the process of reviving the discussion on a piggy-backing > interface, we come to a viable solution, it would certainly help with > the our issues with message matching in general. > > Scalasca's problem here is that we need to detect (and partly > recreate) the exact order of message matching to have the correct > message reach the right receivers. > > >>> 3) the tool could hook all send and receive operations and > >>> serialise them to prevent overtaking > > This is not an option for me. A "performance tool" should strive to > measure as close to the original behavior as possible. Changing > communication semantics just to make a tool "work" would have too > great of an impact on application behavior. After all, if it would > have only little impact, why should the user choose this option in the > first place. > > Jeff wrote: > >> Remember that one of the use cases of allow_overtaking is applications > that > >> have exact matching, in which case allow_overtaking is a way of turning > off > >> a feature that isn't used, in order to get a high-performing message > queue > >> implementation. In the exact matching case, tools will have no problem > >> matching up sends and recvs. > > This is true. If the tools can identify this scenario, it could be > supported by current tools without significant change. However, as it > is not generally forbidden to have inexact matching (right?), it is > unclear on how the tools would detect this. > > What about an additional info key a user can set in this respect: > > exact_matching => true/false > > in which the user can state whether it is indeed a scenario of exact > matching or not. The tool could check this, and issue a warning. > > >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I don't really > >> care if they can't support this assertion either. > > Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. ;-) > > As I said, we (Markus and I) see this as a trigger to come to a viable > solution for tools like ours to support either situation. > > >> And in any case, such tools can just intercept the info operations and > >> strip this key if they can't support it. > > As I wrote above in reply to Dan, stripping options that influence > behavior is not a good option. I, personally, would rather bail out > than (silently) change messaging semantics. I can't say what Markus' > take on this is. > > Jim wrote: > > I don't really see any necessary fix to the proposal. We could add an > > advice to users to remind them that they should ensure tools are > compatible > > with the info keys. And the reverse advice to tools writers that they > > should check info keys for compatibility. > > I would second this idea, while emphasizing the burden to be on the > tool to check for this info key (and potentially others) and warn the > user of "undersupport". > > Cheers, > Marc-Andre > -- > Marc-Andre Hermanns > Jülich Aachen Research Alliance, > High Performance Computing (JARA-HPC) > Jülich Supercomputing Centre (JSC) > > Schinkelstrasse 2 > 52062 Aachen > Germany > > Phone: +49 2461 61 2509 | +49 241 80 24381 > Fax: +49 2461 80 6 99753 > www.jara.org/jara-hpc > email: hermanns(a)jara.rwth-aachen.de > > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0