- mpiwg-p2p - mailman3.cels.anl.gov

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by Balaji, Pavan 22 Nov '18

22 Nov '18

Jeff, all, Sorry, SC took up all of my time so I couldn't respond earlier. I'm not sure what the confusion is. I don't think the text is ambiguous. In the user program, the operations are *not* logically concurrent. They are protected by a critical section. Thus, I don't think the MPI implementation can ignore their ordering. -- Pavan > On Nov 15, 2018, at 11:09 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > Dan has convinced me that the MPI standard is terrible and, while my original interpretation is what we want and which is consistent with the principle of least surprise, it is not guaranteed by the following text. > > Per our discussion, there are a few options: > 1) make all MPI_Send logically concurrent, even on a single thread. this will break stuff and make people sad. > 2) force MPI to order injection <somehow>, which might for some implementations to add more memory ordering on the send path than they want, particularly if they do not have a TSO memory model. > 3) add something like MPI_Win_sync that logically orders sends from multiple threads explicitly. > 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the equivalent of 2 or 3 and thus doesn't cause a performance regression in MPI_THREAD_SERIALIZED. > > Jeff > If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multithreaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. The operations are logically concurrent, even if one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. > > > On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org> wrote: > Dan, > > The matching *is* ordered in this case. So the program will print 0 followed by 1. > > MPI does not order delivery of the actual data, but the first message is guaranteed to go into the first buffer. If the second message ends up going first, the MPI implementation will need to buffer it. > > — Pavan > > Sent from my iPhone > > On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org> wrote: > >> Hi Joachim, >> >> There is no guarantee of ordering between the two sends because they are logically concurrent. If they were issued on the same thread then MPI guarantees delivery order will be identical to the sequential issuing order. >> >> Many MPI libraries are very likely to deliver these messages "in order”, that is, the first one to be called chronologically at the sender process is likely to leave first and therefore likely to arrive first. Interleaving execution of the sending threads may change the issuing order on the network and out-of-order networks may change the order of arrival. >> >> On the other hand, if an MPI implementation is internally using sequence numbers (or a similar mechanism) to enforce ordering for the same-thread case, then it may also (incidentally) reconstruct the issuing order for this case. However, you cannot rely on this behaviour being portable from system to system or from MPI library to MPI library. >> >> If you wish to enforce a particular ordering of these messages, then you can use tags to differentiate each from the other. There is an argument for always using tags in this type of situation to increase program readability. >> >> Cheers, >> Dan. >> — >> Dr Daniel Holmes PhD >> Applications Consultant in HPC Research >> d.holmes(a)epcc.ed.ac.uk >> Phone: +44 (0) 131 651 3465 >> Mobile: +44 (0) 7940 524 088 >> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >> — >> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >> — >> >>> On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org> wrote: >>> >>> Hi all, >>> >>> I have a question on the "Semantics of Point-to-Point Communication" in a multithreaded context. >>> >>> For me the situation for the code below is not clear, especially with respect to the paragraph in MPI-3.1 p.41, l.10-17 : >>> >>> >>> void test(int rank) { >>> int msg = 0; >>> if (rank == 0) { >>> #pragma omp parallel num_threads(2) >>> #pragma omp critical >>> { >>> MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); >>> msg++; >>> } >>> } else if (rank == 1) { >>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); >>> printf("Received %i\n", msg); >>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); >>> printf("Received %i\n", msg); >>> } >>> } >>> >>> Two threads on the first process send a message, the first thread sends 0, the second thread send 1. From OpenMP semantics, the first send happens before the second send. >>> >>> Is there a guarantee, that the other process receives the 0 first? >>> >>> Thanks, >>> Joachim >>> >>> >>> -- >>> Dipl.-Inf. Joachim Protze >>> >>> IT Center >>> Group: High Performance Computing >>> Division: Computational Science and Engineering >>> RWTH Aachen University >>> Seffenter Weg 23 >>> D 52074 Aachen (Germany) >>> Tel: +49 241 80- 24765 >>> Fax: +49 241 80-624765 >>> protze(a)itc.rwth-aachen.de >>> www.itc.rwth-aachen.de >>> >>> _______________________________________________ >>> mpiwg-p2p mailing list >>> mpiwg-p2p(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p >> >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> _______________________________________________ >> mpiwg-p2p mailing list >> mpiwg-p2p(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by HOLMES Daniel 15 Nov '18

15 Nov '18

Hi Joachim, This is the likely best approach for now. Sorry. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 12:01, Joachim Protze <protze(a)itc.rwth-aachen.de<mailto:[email protected]>> wrote: Hi Dan, I look at this from the perspective of an correctness/debugging tool. Therefore I tried to understand which application behavior should be flagged yellow/red. As I understand now, expecting anything around this paragraph can lead to portability issues and should be reported to the programmer :) Thanks Joachi, On 11/15/18 6:57 PM, HOLMES Daniel wrote: Hi Joachim, That is technically correct (although, as Jeff points out, not at all ideal). In practice, you will probably get the behaviour you expect (i.e. ordering) because of the debate over the meaning of this text - as demonstrated by the comment from Pavan. Disambiguating this paragraph is on the to-do list for the point-to-point working group. Thanks for bringing this up again and thereby raising its priority for us. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> <mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 11:19, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]>> wrote: I think, I was once again confused by "may not" in the cited paragraph. As a non-native speaker this hits me from time to time. So if I understand it right now, the paragraph says that even if the threading semantics provide an ordering, the operations are still logically concurrent and have no ordering. Thanks Joachim On 11/15/18 6:09 PM, Jeff Hammond wrote: Dan has convinced me that the MPI standard is terrible and, while my original interpretation is what we want and which is consistent with the principle of least surprise, it is not guaranteed by the following text. Per our discussion, there are a few options: 1) make all MPI_Send logically concurrent, even on a single thread. this will break stuff and make people sad. 2) force MPI to order injection <somehow>, which might for some implementations to add more memory ordering on the send path than they want, particularly if they do not have a TSO memory model. 3) add something like MPI_Win_sync that logically orders sends from multiple threads explicitly. 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the equivalent of 2 or 3 and thus doesn't cause a performance regression in MPI_THREAD_SERIALIZED. Jeff /If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multithreaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. The operations are logically concurrent, even if one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. / On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]><mailto:[email protected]>> wrote: Dan, The matching *is* ordered in this case. So the program will print 0 followed by 1. MPI does not order delivery of the actual data, but the first message is guaranteed to go into the first buffer. If the second message ends up going first, the MPI implementation will need to buffer it. — Pavan Sent from my iPhone On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> <mailto:[email protected]>> wrote: Hi Joachim, There is no guarantee of ordering between the two sends because they are logically concurrent. If they were issued on the same thread then MPI guarantees delivery order will be identical to the sequential issuing order. Many MPI libraries are very likely to deliver these messages "in order”, that is, the first one to be called chronologically at the sender process is likely to leave first and therefore likely to arrive first. Interleaving execution of the sending threads may change the issuing order on the network and out-of-order networks may change the order of arrival. On the other hand, if an MPI implementation is internally using sequence numbers (or a similar mechanism) to enforce ordering for the same-thread case, then it may also (incidentally) reconstruct the issuing order for this case. However, you cannot rely on this behaviour being portable from system to system or from MPI library to MPI library. If you wish to enforce a particular ordering of these messages, then you can use tags to differentiate each from the other. There is an argument for always using tags in this type of situation to increase program readability. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> <mailto:[email protected]><mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> <mailto:[email protected]>> wrote: Hi all, I have a question on the "Semantics of Point-to-Point Communication" in a multithreaded context. For me the situation for the code below is not clear, especially with respect to the paragraph in MPI-3.1 p.41, l.10-17 : void test(int rank) { int msg = 0; if (rank == 0) { #pragma omp parallel num_threads(2) #pragma omp critical { MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); msg++; } } else if (rank == 1) { MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); } } Two threads on the first process send a message, the first thread sends 0, the second thread send 1. From OpenMP semantics, the first send happens before the second send. Is there a guarantee, that the other process receives the 0 first? Thanks, Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> <mailto:[email protected]><mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de> <http://www.itc.rwth-aachen.de/><http://www.itc.rwth-aachen.de <http://www.itc.rwth-aachen.de/>> _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]><mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]><mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]><mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p -- Jeff Hammond jeff.science(a)gmail.com <mailto:[email protected]><mailto:[email protected]> http://jeffhammond.github.io/ -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> <mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de> <http://www.itc.rwth-aachen.de/> _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by HOLMES Daniel 15 Nov '18

15 Nov '18

Hi Joachim, In more recent changes to MPI, the MPI Forum is trying to be consistent in its usage of terms like “shall” “can” “may” and so on. Specifically, we have adopted the meanings set out here: https://www.iec.ch/standardsdev/resources/draftingpublications/directives/p… For existing text, usage of such words is not as clear-cut. Your re-wording is one way to interpret the text as written. It is also possible to read the “may not” sentence as one example (sufficient but not necessary) of when this might happen or become relevant. Unfortunately, neither of these interpretations state clearly what happens in the converse situation, i.e. when the thread execution *does* define a relative order (as in your example OpenMP code). Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 11:57, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: I guess the paragraph is even more confusing than I thought. As I understand after reading some English grammar pages (oxford/cambridge), "may not" can be prohibitive like "shall not"/"must not"/"cannot" or optional like "might not". Searching the MPI standard for use of "may not", it is most often used in the optional sense. From this perspective, I interpreted the paragraph as: ... if the process is multithreaded and the semantics of thread execution do not define a relative order between two send operations executed by two distinct threads, then the operations are logically concurrent, even if one physically precedes the other. So, probably replacing "may not" by "cannot" would better meet the intention? Best Joachim On 11/15/18 6:19 PM, Joachim Protze wrote: I think, I was once again confused by "may not" in the cited paragraph. As a non-native speaker this hits me from time to time. So if I understand it right now, the paragraph says that even if the threading semantics provide an ordering, the operations are still logically concurrent and have no ordering. Thanks Joachim On 11/15/18 6:09 PM, Jeff Hammond wrote: Dan has convinced me that the MPI standard is terrible and, while my original interpretation is what we want and which is consistent with the principle of least surprise, it is not guaranteed by the following text. Per our discussion, there are a few options: 1) make all MPI_Send logically concurrent, even on a single thread. this will break stuff and make people sad. 2) force MPI to order injection <somehow>, which might for some implementations to add more memory ordering on the send path than they want, particularly if they do not have a TSO memory model. 3) add something like MPI_Win_sync that logically orders sends from multiple threads explicitly. 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the equivalent of 2 or 3 and thus doesn't cause a performance regression in MPI_THREAD_SERIALIZED. Jeff /If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multithreaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. The operations are logically concurrent, even if one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. / On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]>> wrote: Dan, The matching *is* ordered in this case. So the program will print 0 followed by 1. MPI does not order delivery of the actual data, but the first message is guaranteed to go into the first buffer. If the second message ends up going first, the MPI implementation will need to buffer it. — Pavan Sent from my iPhone On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]>> wrote: Hi Joachim, There is no guarantee of ordering between the two sends because they are logically concurrent. If they were issued on the same thread then MPI guarantees delivery order will be identical to the sequential issuing order. Many MPI libraries are very likely to deliver these messages "in order”, that is, the first one to be called chronologically at the sender process is likely to leave first and therefore likely to arrive first. Interleaving execution of the sending threads may change the issuing order on the network and out-of-order networks may change the order of arrival. On the other hand, if an MPI implementation is internally using sequence numbers (or a similar mechanism) to enforce ordering for the same-thread case, then it may also (incidentally) reconstruct the issuing order for this case. However, you cannot rely on this behaviour being portable from system to system or from MPI library to MPI library. If you wish to enforce a particular ordering of these messages, then you can use tags to differentiate each from the other. There is an argument for always using tags in this type of situation to increase program readability. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> <mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]>> wrote: Hi all, I have a question on the "Semantics of Point-to-Point Communication" in a multithreaded context. For me the situation for the code below is not clear, especially with respect to the paragraph in MPI-3.1 p.41, l.10-17 : void test(int rank) { int msg = 0; if (rank == 0) { #pragma omp parallel num_threads(2) #pragma omp critical { MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); msg++; } } else if (rank == 1) { MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); } } Two threads on the first process send a message, the first thread sends 0, the second thread send 1. From OpenMP semantics, the first send happens before the second send. Is there a guarantee, that the other process receives the 0 first? Thanks, Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> <mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de> <http://www.itc.rwth-aachen.de> _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> <mailto:[email protected]> http://jeffhammond.github.io/ -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by Joachim Protze 15 Nov '18

15 Nov '18

Hi Dan, I look at this from the perspective of an correctness/debugging tool. Therefore I tried to understand which application behavior should be flagged yellow/red. As I understand now, expecting anything around this paragraph can lead to portability issues and should be reported to the programmer :) Thanks Joachi, On 11/15/18 6:57 PM, HOLMES Daniel wrote: > Hi Joachim, > > That is technically correct (although, as Jeff points out, not at all > ideal). In practice, you will probably get the behaviour you expect > (i.e. ordering) because of the debate over the meaning of this text - as > demonstrated by the comment from Pavan. > > Disambiguating this paragraph is on the to-do list for the > point-to-point working group. Thanks for bringing this up again and > thereby raising its priority for us. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Applications Consultant in HPC Research > d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> > Phone: +44 (0) 131 651 3465 > Mobile: +44 (0) 7940 524 088 > Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, > EH8 9BT > — > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > — > >> On 15 Nov 2018, at 11:19, Joachim Protze via mpiwg-p2p >> <mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]>> >> wrote: >> >> I think, I was once again confused by "may not" in the cited >> paragraph. As a non-native speaker this hits me from time to time. >> >> So if I understand it right now, the paragraph says that even if the >> threading semantics provide an ordering, the operations are still >> logically concurrent and have no ordering. >> >> Thanks >> Joachim >> >> On 11/15/18 6:09 PM, Jeff Hammond wrote: >>> Dan has convinced me that the MPI standard is terrible and, while my >>> original interpretation is what we want and which is consistent with >>> the principle of least surprise, it is not guaranteed by the >>> following text. >>> Per our discussion, there are a few options: >>> 1) make all MPI_Send logically concurrent, even on a single thread. >>> this will break stuff and make people sad. >>> 2) force MPI to order injection <somehow>, which might for some >>> implementations to add more memory ordering on the send path than >>> they want, particularly if they do not have a TSO memory model. >>> 3) add something like MPI_Win_sync that logically orders sends from >>> multiple threads explicitly. >>> 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the >>> equivalent of 2 or 3 and thus doesn't cause a performance regression >>> in MPI_THREAD_SERIALIZED. >>> Jeff >>> /If a process has a single thread of execution, then any two >>> communications executed by this process are ordered. On the other >>> hand, if the process is multithreaded, then the semantics of thread >>> execution may not define a relative order between two send operations >>> executed by two distinct threads. The operations are logically >>> concurrent, even if one physically precedes the other. In such a >>> case, the two messages sent can be received in any order. Similarly, >>> if two receive operations that are logically concurrent receive two >>> successively sent messages, then the two messages can match the two >>> receives in either order. / >>> On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p >>> <mpiwg-p2p(a)lists.mpi-forum.org >>> <mailto:[email protected]><mailto:[email protected]>> >>> wrote: >>> Dan, >>> The matching *is* ordered in this case. So the program will print 0 >>> followed by 1. >>> MPI does not order delivery of the actual data, but the first >>> message is guaranteed to go into the first buffer. If the second >>> message ends up going first, the MPI implementation will need to >>> buffer it. >>> — Pavan >>> Sent from my iPhone >>> On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p >>> <mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> >>> <mailto:[email protected]>> wrote: >>>> Hi Joachim, >>>> >>>> There is no guarantee of ordering between the two sends because >>>> they are logically concurrent. If they were issued on the same >>>> thread then MPI guarantees delivery order will be identical to the >>>> sequential issuing order. >>>> >>>> Many MPI libraries are very likely to deliver these messages "in >>>> order”, that is, the first one to be called chronologically at the >>>> sender process is likely to leave first and therefore likely to >>>> arrive first. Interleaving execution of the sending threads may >>>> change the issuing order on the network and out-of-order networks >>>> may change the order of arrival. >>>> >>>> On the other hand, if an MPI implementation is internally using >>>> sequence numbers (or a similar mechanism) to enforce ordering for >>>> the same-thread case, then it may also (incidentally) reconstruct >>>> the issuing order for this case. However, you cannot rely on this >>>> behaviour being portable from system to system or from MPI library >>>> to MPI library. >>>> >>>> If you wish to enforce a particular ordering of these messages, >>>> then you can use tags to differentiate each from the other. There >>>> is an argument for always using tags in this type of situation to >>>> increase program readability. >>>> >>>> Cheers, >>>> Dan. >>>> — >>>> Dr Daniel Holmes PhD >>>> Applications Consultant in HPC Research >>>> d.holmes(a)epcc.ed.ac.uk >>>> <mailto:[email protected]><mailto:[email protected]> >>>> Phone: +44 (0) 131 651 3465 >>>> Mobile: +44 (0) 7940 524 088 >>>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, >>>> Edinburgh, EH8 9BT >>>> — >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> — >>>> >>>>> On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p >>>>> <mpiwg-p2p(a)lists.mpi-forum.org >>>>> <mailto:[email protected]> >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I have a question on the "Semantics of Point-to-Point >>>>> Communication" in a multithreaded context. >>>>> >>>>> For me the situation for the code below is not clear, especially >>>>> with respect to the paragraph in MPI-3.1 p.41, l.10-17 : >>>>> >>>>> >>>>> void test(int rank) { >>>>> int msg = 0; >>>>> if (rank == 0) { >>>>> #pragma omp parallel num_threads(2) >>>>> #pragma omp critical >>>>> { >>>>> MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); >>>>> msg++; >>>>> } >>>>> } else if (rank == 1) { >>>>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, >>>>> MPI_STATUS_IGNORE); >>>>> printf("Received %i\n", msg); >>>>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, >>>>> MPI_STATUS_IGNORE); >>>>> printf("Received %i\n", msg); >>>>> } >>>>> } >>>>> >>>>> Two threads on the first process send a message, the first thread >>>>> sends 0, the second thread send 1. From OpenMP semantics, the >>>>> first send happens before the second send. >>>>> >>>>> Is there a guarantee, that the other process receives the 0 first? >>>>> >>>>> Thanks, >>>>> Joachim >>>>> >>>>> >>>>> -- Dipl.-Inf. Joachim Protze >>>>> >>>>> IT Center >>>>> Group: High Performance Computing >>>>> Division: Computational Science and Engineering >>>>> RWTH Aachen University >>>>> Seffenter Weg 23 >>>>> D 52074 Aachen (Germany) >>>>> Tel: +49 241 80- 24765 >>>>> Fax: +49 241 80-624765 >>>>> protze(a)itc.rwth-aachen.de >>>>> <mailto:[email protected]><mailto:[email protected]> >>>>> www.itc.rwth-aachen.de >>>>> <http://www.itc.rwth-aachen.de/><http://www.itc.rwth-aachen.de >>>>> <http://www.itc.rwth-aachen.de/>> >>>>> >>>>> _______________________________________________ >>>>> mpiwg-p2p mailing list >>>>> mpiwg-p2p(a)lists.mpi-forum.org >>>>> <mailto:[email protected]><mailto:[email protected]> >>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p >>>> >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> _______________________________________________ >>>> mpiwg-p2p mailing list >>>> mpiwg-p2p(a)lists.mpi-forum.org >>>> <mailto:[email protected]><mailto:[email protected]> >>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p >>> _______________________________________________ >>> mpiwg-p2p mailing list >>> mpiwg-p2p(a)lists.mpi-forum.org >>> <mailto:[email protected]><mailto:[email protected]> >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> <mailto:[email protected]><mailto:[email protected]> >>> http://jeffhammond.github.io/ >> >> >> -- >> Dipl.-Inf. Joachim Protze >> >> IT Center >> Group: High Performance Computing >> Division: Computational Science and Engineering >> RWTH Aachen University >> Seffenter Weg 23 >> D 52074 Aachen (Germany) >> Tel: +49 241 80- 24765 >> Fax: +49 241 80-624765 >> protze(a)itc.rwth-aachen.de <mailto:[email protected]> >> www.itc.rwth-aachen.de <http://www.itc.rwth-aachen.de/> >> >> _______________________________________________ >> mpiwg-p2p mailing list >> mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by Joachim Protze 15 Nov '18

15 Nov '18

I guess the paragraph is even more confusing than I thought. As I understand after reading some English grammar pages (oxford/cambridge), "may not" can be prohibitive like "shall not"/"must not"/"cannot" or optional like "might not". Searching the MPI standard for use of "may not", it is most often used in the optional sense. From this perspective, I interpreted the paragraph as: ... if the process is multithreaded and the semantics of thread execution do not define a relative order between two send operations executed by two distinct threads, then the operations are logically concurrent, even if one physically precedes the other. So, probably replacing "may not" by "cannot" would better meet the intention? Best Joachim On 11/15/18 6:19 PM, Joachim Protze wrote: > I think, I was once again confused by "may not" in the cited paragraph. > As a non-native speaker this hits me from time to time. > > So if I understand it right now, the paragraph says that even if the > threading semantics provide an ordering, the operations are still > logically concurrent and have no ordering. > > Thanks > Joachim > > On 11/15/18 6:09 PM, Jeff Hammond wrote: >> Dan has convinced me that the MPI standard is terrible and, while my >> original interpretation is what we want and which is consistent with >> the principle of least surprise, it is not guaranteed by the following >> text. >> >> Per our discussion, there are a few options: >> 1) make all MPI_Send logically concurrent, even on a single thread. >> this will break stuff and make people sad. >> 2) force MPI to order injection <somehow>, which might for some >> implementations to add more memory ordering on the send path than they >> want, particularly if they do not have a TSO memory model. >> 3) add something like MPI_Win_sync that logically orders sends from >> multiple threads explicitly. >> 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the equivalent >> of 2 or 3 and thus doesn't cause a performance regression in >> MPI_THREAD_SERIALIZED. >> >> Jeff >> >> /If a process has a single thread of execution, then any two >> communications executed by this process are ordered. On the other >> hand, if the process is multithreaded, then the semantics of thread >> execution may not define a relative order between two send operations >> executed by two distinct threads. The operations are logically >> concurrent, even if one physically precedes the other. In such a case, >> the two messages sent can be received in any order. Similarly, if two >> receive operations that are logically concurrent receive two >> successively sent messages, then the two messages can match the two >> receives in either order. / >> >> >> On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p >> <mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]>> >> wrote: >> >> Dan, >> >> The matching *is* ordered in this case. So the program will print 0 >> followed by 1. >> >> MPI does not order delivery of the actual data, but the first >> message is guaranteed to go into the first buffer. If the second >> message ends up going first, the MPI implementation will need to >> buffer it. >> >> — Pavan >> >> Sent from my iPhone >> >> On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p >> <mpiwg-p2p(a)lists.mpi-forum.org >> <mailto:[email protected]>> wrote: >> >>> Hi Joachim, >>> >>> There is no guarantee of ordering between the two sends because >>> they are logically concurrent. If they were issued on the same >>> thread then MPI guarantees delivery order will be identical to the >>> sequential issuing order. >>> >>> Many MPI libraries are very likely to deliver these messages "in >>> order”, that is, the first one to be called chronologically at the >>> sender process is likely to leave first and therefore likely to >>> arrive first. Interleaving execution of the sending threads may >>> change the issuing order on the network and out-of-order networks >>> may change the order of arrival. >>> >>> On the other hand, if an MPI implementation is internally using >>> sequence numbers (or a similar mechanism) to enforce ordering for >>> the same-thread case, then it may also (incidentally) reconstruct >>> the issuing order for this case. However, you cannot rely on this >>> behaviour being portable from system to system or from MPI library >>> to MPI library. >>> >>> If you wish to enforce a particular ordering of these messages, >>> then you can use tags to differentiate each from the other. There >>> is an argument for always using tags in this type of situation to >>> increase program readability. >>> >>> Cheers, >>> Dan. >>> — >>> Dr Daniel Holmes PhD >>> Applications Consultant in HPC Research >>> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >>> Phone: +44 (0) 131 651 3465 >>> Mobile: +44 (0) 7940 524 088 >>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, >>> Edinburgh, EH8 9BT >>> — >>> The University of Edinburgh is a charitable body, registered in >>> Scotland, with registration number SC005336. >>> — >>> >>>> On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p >>>> <mpiwg-p2p(a)lists.mpi-forum.org >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hi all, >>>> >>>> I have a question on the "Semantics of Point-to-Point >>>> Communication" in a multithreaded context. >>>> >>>> For me the situation for the code below is not clear, especially >>>> with respect to the paragraph in MPI-3.1 p.41, l.10-17 : >>>> >>>> >>>> void test(int rank) { >>>> int msg = 0; >>>> if (rank == 0) { >>>> #pragma omp parallel num_threads(2) >>>> #pragma omp critical >>>> { >>>> MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); >>>> msg++; >>>> } >>>> } else if (rank == 1) { >>>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, >>>> MPI_STATUS_IGNORE); >>>> printf("Received %i\n", msg); >>>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, >>>> MPI_STATUS_IGNORE); >>>> printf("Received %i\n", msg); >>>> } >>>> } >>>> >>>> Two threads on the first process send a message, the first thread >>>> sends 0, the second thread send 1. From OpenMP semantics, the >>>> first send happens before the second send. >>>> >>>> Is there a guarantee, that the other process receives the 0 first? >>>> >>>> Thanks, >>>> Joachim >>>> >>>> >>>> -- Dipl.-Inf. Joachim Protze >>>> >>>> IT Center >>>> Group: High Performance Computing >>>> Division: Computational Science and Engineering >>>> RWTH Aachen University >>>> Seffenter Weg 23 >>>> D 52074 Aachen (Germany) >>>> Tel: +49 241 80- 24765 >>>> Fax: +49 241 80-624765 >>>> protze(a)itc.rwth-aachen.de <mailto:[email protected]> >>>> www.itc.rwth-aachen.de <http://www.itc.rwth-aachen.de> >>>> >>>> _______________________________________________ >>>> mpiwg-p2p mailing list >>>> mpiwg-p2p(a)lists.mpi-forum.org >>>> <mailto:[email protected]> >>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p >>> >>> The University of Edinburgh is a charitable body, registered in >>> Scotland, with registration number SC005336. >>> _______________________________________________ >>> mpiwg-p2p mailing list >>> mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p >> _______________________________________________ >> mpiwg-p2p mailing list >> mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com <mailto:[email protected]> >> http://jeffhammond.github.io/ > > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by HOLMES Daniel 15 Nov '18

15 Nov '18

Hi Joachim, That is technically correct (although, as Jeff points out, not at all ideal). In practice, you will probably get the behaviour you expect (i.e. ordering) because of the debate over the meaning of this text - as demonstrated by the comment from Pavan. Disambiguating this paragraph is on the to-do list for the point-to-point working group. Thanks for bringing this up again and thereby raising its priority for us. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 11:19, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: I think, I was once again confused by "may not" in the cited paragraph. As a non-native speaker this hits me from time to time. So if I understand it right now, the paragraph says that even if the threading semantics provide an ordering, the operations are still logically concurrent and have no ordering. Thanks Joachim On 11/15/18 6:09 PM, Jeff Hammond wrote: Dan has convinced me that the MPI standard is terrible and, while my original interpretation is what we want and which is consistent with the principle of least surprise, it is not guaranteed by the following text. Per our discussion, there are a few options: 1) make all MPI_Send logically concurrent, even on a single thread. this will break stuff and make people sad. 2) force MPI to order injection <somehow>, which might for some implementations to add more memory ordering on the send path than they want, particularly if they do not have a TSO memory model. 3) add something like MPI_Win_sync that logically orders sends from multiple threads explicitly. 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the equivalent of 2 or 3 and thus doesn't cause a performance regression in MPI_THREAD_SERIALIZED. Jeff /If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multithreaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. The operations are logically concurrent, even if one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. / On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]>> wrote: Dan, The matching *is* ordered in this case. So the program will print 0 followed by 1. MPI does not order delivery of the actual data, but the first message is guaranteed to go into the first buffer. If the second message ends up going first, the MPI implementation will need to buffer it. — Pavan Sent from my iPhone On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]>> wrote: Hi Joachim, There is no guarantee of ordering between the two sends because they are logically concurrent. If they were issued on the same thread then MPI guarantees delivery order will be identical to the sequential issuing order. Many MPI libraries are very likely to deliver these messages "in order”, that is, the first one to be called chronologically at the sender process is likely to leave first and therefore likely to arrive first. Interleaving execution of the sending threads may change the issuing order on the network and out-of-order networks may change the order of arrival. On the other hand, if an MPI implementation is internally using sequence numbers (or a similar mechanism) to enforce ordering for the same-thread case, then it may also (incidentally) reconstruct the issuing order for this case. However, you cannot rely on this behaviour being portable from system to system or from MPI library to MPI library. If you wish to enforce a particular ordering of these messages, then you can use tags to differentiate each from the other. There is an argument for always using tags in this type of situation to increase program readability. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> <mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]>> wrote: Hi all, I have a question on the "Semantics of Point-to-Point Communication" in a multithreaded context. For me the situation for the code below is not clear, especially with respect to the paragraph in MPI-3.1 p.41, l.10-17 : void test(int rank) { int msg = 0; if (rank == 0) { #pragma omp parallel num_threads(2) #pragma omp critical { MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); msg++; } } else if (rank == 1) { MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); } } Two threads on the first process send a message, the first thread sends 0, the second thread send 1. From OpenMP semantics, the first send happens before the second send. Is there a guarantee, that the other process receives the 0 first? Thanks, Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> <mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de/> <http://www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de/>> _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> <mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> <mailto:[email protected]> http://jeffhammond.github.io/ -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de/> _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by Joachim Protze 15 Nov '18

15 Nov '18

I think, I was once again confused by "may not" in the cited paragraph. As a non-native speaker this hits me from time to time. So if I understand it right now, the paragraph says that even if the threading semantics provide an ordering, the operations are still logically concurrent and have no ordering. Thanks Joachim On 11/15/18 6:09 PM, Jeff Hammond wrote: > Dan has convinced me that the MPI standard is terrible and, while my > original interpretation is what we want and which is consistent with the > principle of least surprise, it is not guaranteed by the following text. > > Per our discussion, there are a few options: > 1) make all MPI_Send logically concurrent, even on a single thread. > this will break stuff and make people sad. > 2) force MPI to order injection <somehow>, which might for some > implementations to add more memory ordering on the send path than they > want, particularly if they do not have a TSO memory model. > 3) add something like MPI_Win_sync that logically orders sends from > multiple threads explicitly. > 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the equivalent > of 2 or 3 and thus doesn't cause a performance regression in > MPI_THREAD_SERIALIZED. > > Jeff > > /If a process has a single thread of execution, then any two > communications executed by this process are ordered. On the other hand, > if the process is multithreaded, then the semantics of thread execution > may not define a relative order between two send operations executed by > two distinct threads. The operations are logically concurrent, even if > one physically precedes the other. In such a case, the two messages sent > can be received in any order. Similarly, if two receive operations that > are logically concurrent receive two successively sent messages, then > the two messages can match the two receives in either order. / > > > On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p > <mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]>> > wrote: > > Dan, > > The matching *is* ordered in this case. So the program will print 0 > followed by 1. > > MPI does not order delivery of the actual data, but the first > message is guaranteed to go into the first buffer. If the second > message ends up going first, the MPI implementation will need to > buffer it. > > — Pavan > > Sent from my iPhone > > On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p > <mpiwg-p2p(a)lists.mpi-forum.org > <mailto:[email protected]>> wrote: > >> Hi Joachim, >> >> There is no guarantee of ordering between the two sends because >> they are logically concurrent. If they were issued on the same >> thread then MPI guarantees delivery order will be identical to the >> sequential issuing order. >> >> Many MPI libraries are very likely to deliver these messages "in >> order”, that is, the first one to be called chronologically at the >> sender process is likely to leave first and therefore likely to >> arrive first. Interleaving execution of the sending threads may >> change the issuing order on the network and out-of-order networks >> may change the order of arrival. >> >> On the other hand, if an MPI implementation is internally using >> sequence numbers (or a similar mechanism) to enforce ordering for >> the same-thread case, then it may also (incidentally) reconstruct >> the issuing order for this case. However, you cannot rely on this >> behaviour being portable from system to system or from MPI library >> to MPI library. >> >> If you wish to enforce a particular ordering of these messages, >> then you can use tags to differentiate each from the other. There >> is an argument for always using tags in this type of situation to >> increase program readability. >> >> Cheers, >> Dan. >> — >> Dr Daniel Holmes PhD >> Applications Consultant in HPC Research >> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]> >> Phone: +44 (0) 131 651 3465 >> Mobile: +44 (0) 7940 524 088 >> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, >> Edinburgh, EH8 9BT >> — >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> — >> >>> On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p >>> <mpiwg-p2p(a)lists.mpi-forum.org >>> <mailto:[email protected]>> wrote: >>> >>> Hi all, >>> >>> I have a question on the "Semantics of Point-to-Point >>> Communication" in a multithreaded context. >>> >>> For me the situation for the code below is not clear, especially >>> with respect to the paragraph in MPI-3.1 p.41, l.10-17 : >>> >>> >>> void test(int rank) { >>> int msg = 0; >>> if (rank == 0) { >>> #pragma omp parallel num_threads(2) >>> #pragma omp critical >>> { >>> MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); >>> msg++; >>> } >>> } else if (rank == 1) { >>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, >>> MPI_STATUS_IGNORE); >>> printf("Received %i\n", msg); >>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, >>> MPI_STATUS_IGNORE); >>> printf("Received %i\n", msg); >>> } >>> } >>> >>> Two threads on the first process send a message, the first thread >>> sends 0, the second thread send 1. From OpenMP semantics, the >>> first send happens before the second send. >>> >>> Is there a guarantee, that the other process receives the 0 first? >>> >>> Thanks, >>> Joachim >>> >>> >>> -- >>> Dipl.-Inf. Joachim Protze >>> >>> IT Center >>> Group: High Performance Computing >>> Division: Computational Science and Engineering >>> RWTH Aachen University >>> Seffenter Weg 23 >>> D 52074 Aachen (Germany) >>> Tel: +49 241 80- 24765 >>> Fax: +49 241 80-624765 >>> protze(a)itc.rwth-aachen.de <mailto:[email protected]> >>> www.itc.rwth-aachen.de <http://www.itc.rwth-aachen.de> >>> >>> _______________________________________________ >>> mpiwg-p2p mailing list >>> mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p >> >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> _______________________________________________ >> mpiwg-p2p mailing list >> mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org <mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by Jeff Hammond 15 Nov '18

15 Nov '18

Dan has convinced me that the MPI standard is terrible and, while my original interpretation is what we want and which is consistent with the principle of least surprise, it is not guaranteed by the following text. Per our discussion, there are a few options: 1) make all MPI_Send logically concurrent, even on a single thread. this will break stuff and make people sad. 2) force MPI to order injection <somehow>, which might for some implementations to add more memory ordering on the send path than they want, particularly if they do not have a TSO memory model. 3) add something like MPI_Win_sync that logically orders sends from multiple threads explicitly. 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the equivalent of 2 or 3 and thus doesn't cause a performance regression in MPI_THREAD_SERIALIZED. Jeff *If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multithreaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. The operations are logically concurrent, even if one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. * On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p < mpiwg-p2p(a)lists.mpi-forum.org> wrote: > Dan, > > The matching *is* ordered in this case. So the program will print 0 > followed by 1. > > MPI does not order delivery of the actual data, but the first message is > guaranteed to go into the first buffer. If the second message ends up > going first, the MPI implementation will need to buffer it. > > — Pavan > > Sent from my iPhone > > On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p < > mpiwg-p2p(a)lists.mpi-forum.org> wrote: > > Hi Joachim, > > There is no guarantee of ordering between the two sends because they are > logically concurrent. If they were issued on the same thread then MPI > guarantees delivery order will be identical to the sequential issuing order. > > Many MPI libraries are very likely to deliver these messages "in order”, > that is, the first one to be called chronologically at the sender process > is likely to leave first and therefore likely to arrive first. Interleaving > execution of the sending threads may change the issuing order on the > network and out-of-order networks may change the order of arrival. > > On the other hand, if an MPI implementation is internally using sequence > numbers (or a similar mechanism) to enforce ordering for the same-thread > case, then it may also (incidentally) reconstruct the issuing order for > this case. However, you cannot rely on this behaviour being portable from > system to system or from MPI library to MPI library. > > If you wish to enforce a particular ordering of these messages, then you > can use tags to differentiate each from the other. There is an argument for > always using tags in this type of situation to increase program readability. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Applications Consultant in HPC Research > d.holmes(a)epcc.ed.ac.uk > Phone: +44 (0) 131 651 3465 > Mobile: +44 (0) 7940 524 088 > Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, > EH8 9BT > — > The University of Edinburgh is a charitable body, registered in Scotland, > with registration number SC005336. > — > > On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p < > mpiwg-p2p(a)lists.mpi-forum.org> wrote: > > Hi all, > > I have a question on the "Semantics of Point-to-Point Communication" in a > multithreaded context. > > For me the situation for the code below is not clear, especially with > respect to the paragraph in MPI-3.1 p.41, l.10-17 : > > > void test(int rank) { > int msg = 0; > if (rank == 0) { > #pragma omp parallel num_threads(2) > #pragma omp critical > { > MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); > msg++; > } > } else if (rank == 1) { > MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); > printf("Received %i\n", msg); > MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); > printf("Received %i\n", msg); > } > } > > Two threads on the first process send a message, the first thread sends 0, > the second thread send 1. From OpenMP semantics, the first send happens > before the second send. > > Is there a guarantee, that the other process receives the 0 first? > > Thanks, > Joachim > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p > > _______________________________________________ > mpiwg-p2p mailing list > mpiwg-p2p(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by Balaji, Pavan 15 Nov '18

15 Nov '18

Dan, The matching *is* ordered in this case. So the program will print 0 followed by 1. MPI does not order delivery of the actual data, but the first message is guaranteed to go into the first buffer. If the second message ends up going first, the MPI implementation will need to buffer it. — Pavan Sent from my iPhone On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Joachim, There is no guarantee of ordering between the two sends because they are logically concurrent. If they were issued on the same thread then MPI guarantees delivery order will be identical to the sequential issuing order. Many MPI libraries are very likely to deliver these messages "in order”, that is, the first one to be called chronologically at the sender process is likely to leave first and therefore likely to arrive first. Interleaving execution of the sending threads may change the issuing order on the network and out-of-order networks may change the order of arrival. On the other hand, if an MPI implementation is internally using sequence numbers (or a similar mechanism) to enforce ordering for the same-thread case, then it may also (incidentally) reconstruct the issuing order for this case. However, you cannot rely on this behaviour being portable from system to system or from MPI library to MPI library. If you wish to enforce a particular ordering of these messages, then you can use tags to differentiate each from the other. There is an argument for always using tags in this type of situation to increase program readability. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, I have a question on the "Semantics of Point-to-Point Communication" in a multithreaded context. For me the situation for the code below is not clear, especially with respect to the paragraph in MPI-3.1 p.41, l.10-17 : void test(int rank) { int msg = 0; if (rank == 0) { #pragma omp parallel num_threads(2) #pragma omp critical { MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); msg++; } } else if (rank == 1) { MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); } } Two threads on the first process send a message, the first thread sends 0, the second thread send 1. From OpenMP semantics, the first send happens before the second send. Is there a guarantee, that the other process receives the 0 first? Thanks, Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de> _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p

1 0

Re: [mpiwg-p2p] Ordering of P2P messages in multithreaded applications
by HOLMES Daniel 15 Nov '18

15 Nov '18

Hi Joachim, There is no guarantee of ordering between the two sends because they are logically concurrent. If they were issued on the same thread then MPI guarantees delivery order will be identical to the sequential issuing order. Many MPI libraries are very likely to deliver these messages "in order”, that is, the first one to be called chronologically at the sender process is likely to leave first and therefore likely to arrive first. Interleaving execution of the sending threads may change the issuing order on the network and out-of-order networks may change the order of arrival. On the other hand, if an MPI implementation is internally using sequence numbers (or a similar mechanism) to enforce ordering for the same-thread case, then it may also (incidentally) reconstruct the issuing order for this case. However, you cannot rely on this behaviour being portable from system to system or from MPI library to MPI library. If you wish to enforce a particular ordering of these messages, then you can use tags to differentiate each from the other. There is an argument for always using tags in this type of situation to increase program readability. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p <mpiwg-p2p(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, I have a question on the "Semantics of Point-to-Point Communication" in a multithreaded context. For me the situation for the code below is not clear, especially with respect to the paragraph in MPI-3.1 p.41, l.10-17 : void test(int rank) { int msg = 0; if (rank == 0) { #pragma omp parallel num_threads(2) #pragma omp critical { MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD); msg++; } } else if (rank == 1) { MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Received %i\n", msg); } } Two threads on the first process send a message, the first thread sends 0, the second thread send 1. From OpenMP semantics, the first send happens before the second send. Is there a guarantee, that the other process receives the 0 first? Thanks, Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de _______________________________________________ mpiwg-p2p mailing list mpiwg-p2p(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0