October 2008 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Graham, Richard L. 17 Oct '08

17 Oct '08

Alexander, Do you see this as different in principal from Proc 0. Proc 1 irecv(1,req_r) Irecv(0,req_r) Isend(1,req_s). Isend(0,req_s) Wait(req_s). Send(0) Send(1). Wait(req_s) I see these two cases as basically the same from a progress perspective. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 07:22:02 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Rich, Thank you. Note that the first operation is an Ibarrier. According to Torsten's explanation, process 0 will block in the Wait, which it may well do if it is Wait in process 1 that actually completes its Ibarrier. Now, how can process 0 make progress in its Wait when there's no matching Wait on the other side yet? Do we assume that any messages that may have been sent by process 0's IBarrier will be handled as unexpected messages by process 1 that is busy doing its (also unmatched) part of the Bcast? Likewise, do we assume that all messages that process 1 has to send in its Ibarrier can leave process 1 before it enters its Wait? Aren't we assuming a bit too much? In other words, aren't we assuming that all message sends and receives will be started in Ibarrier, and will be able to proceed even though the Wait is some blocking collective ops away? Given the way most MPIs implement nonblocking pt2pt ops now, something looks fishy here. In any case, I'd like to see a detailed explanation how things will work, and moreover, a proof that this does not preclude some algorithms - both in the sense of the pt2pt operations deployed, and in the sense of collective exchange pattern, for the applicable message sizes - from being used. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 11:57 AM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft To me this does not look like a deadlock, proc o will make progress in the wait, and just like with point-to-point commuications process 1 will make general mpi progress while in the bcast. This includes progress on the ibcast so that process 0 can comete it's ibcast and move on to the bcast. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 05:14:04 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Christian Siebert 17 Oct '08

17 Oct '08

Hi Torsten, > oh yeah - we should not send too bi attachments to the list (your mail > was 2.3 MiB). I fixed all remarks in the scans. See attachment at: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl I'm sorry for this, I'll give comments in pure text form in this mail. Putting the proposal into the Wiki is an excellent idea! >> 2) Clarification of MPI_Request_free() for requests from non-blocking >> collective operations. > what do you mean? The current description is not clear enough and might lead to confusion. Is the following code correct, or not? MPI_Ibcast(buf, c, dt, root, comm, req); if (myrank == root) { MPI_Request_free(req); } else { MPI_Wait(req, MPI_STATUS_IGNORE); } More background: A user might assume that the root process does act solely as a sender (even this might be dangerous, e.g. for other collectives). "Freeing a request is only useful at the sender side" suggests that the above code is valid and can potentially even be beneficial for performance (no blocking at the root)... Personally, I'd like to forbid the use of MPI_Request_free() for requests from nonblocking collectives. >> 3) Better definition/description for "matching" (there is nothing like >> "at the same time" -> logical order?). > yes, do you have a suggestion? unfortunately no, sorry (i.e. nothing proper yet) > I replaced it with simultaneously for now > - which is suboptimal too. I totally agree. Maybe someone else has some suggestions? >> 4) Define "levels of progression"? To be queried (e.g., for "Synchronous >> Progress" MPI_Tests are needed for performance, but for "Asynchronous >> Progress" they would only add unnecessary overhead)? UP >= AP >= SP? > yes, I actually erased those definitions because they don't belong in a > standard (imho). good - they might be mentioned in some "Advice to..." section >> 6) NBC gives several possible ways for optimizations. With this "General >> advice to implementers" we stick to only one, and might prevent others. >> Can we already fix a decision for optimization strategies at this stage? >> Should we fix it at all? > this is only an advice. Maybe you're right, but I don't think that it'll > be much better without the advice at all. Advices seem to be rather weak > anyway so I have no strong opinion on that. ok - maybe just weaken the statement, e.g. "Nonblocking operations can be used ..." instead of "Most nonblocking operations will be used...". (Besides some "good" interconnects, MPI history showed that this "overlapping" issue, i.e. concurrent communication and computation, was often already problematic for pt2pt ops. I totally agree that this is something really good to aim for. However, I see the main advantage of nonblocking operations slightly more in the context of reduced waiting times, i.e. a blocking receiver needs to wait for the sender but a nonblocking receiver can potentially do something worthwhile during this time. An implementation which tries to optimize for this direction might not necessarily achieve "as low CPU overhead as possible". Currently, I don't know which direction - there are surely many more - is the way to go...) To the point with matching BC and NBC: It is clearly stated that they don't match. I don't like the additional sentence "Matching them should fail ..." because this forces implementations to check for matchings (e.g. an implementation might use a separate communicator for NBC; explicit checking for an incorrect matching between BC and NBC would incur additional overhead). There are other (better) ways to detect incorrect MPI programs. However, this should not be part of the standard itself. Having an additional explanation why we decided against a matching might be beneficial. Something like "Both classes (i.e. BC and NBC) have potentially distinct optimization criteria, enforcing different implementations." might help the reader to understand this decision. To the barrier section: "initializes" is already a predefined word in the MPI standard and is reserved for persistent operations. The correct term in this context would be "initiate". Persistent collectives is another proposal... ;-) >> 7) Should there be a concrete code example in the proposal (e.g. an >> implementation of this double buffering example)? > maybe, doesn't sound too bad. Let's talk about this at the Forum Unfortunately, I'll not be there.... :-( > I just started to creat an agenda for Chicago and put hose items on > there (more will follow). great! I'm also glad that so many of my remarks made it into the proposal. So thanks again! ;-) Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Rich, Thank you. Note that the first operation is an Ibarrier. According to Torsten's explanation, process 0 will block in the Wait, which it may well do if it is Wait in process 1 that actually completes its Ibarrier. Now, how can process 0 make progress in its Wait when there's no matching Wait on the other side yet? Do we assume that any messages that may have been sent by process 0's IBarrier will be handled as unexpected messages by process 1 that is busy doing its (also unmatched) part of the Bcast? Likewise, do we assume that all messages that process 1 has to send in its Ibarrier can leave process 1 before it enters its Wait? Aren't we assuming a bit too much? In other words, aren't we assuming that all message sends and receives will be started in Ibarrier, and will be able to proceed even though the Wait is some blocking collective ops away? Given the way most MPIs implement nonblocking pt2pt ops now, something looks fishy here. In any case, I'd like to see a detailed explanation how things will work, and moreover, a proof that this does not preclude some algorithms - both in the sense of the pt2pt operations deployed, and in the sense of collective exchange pattern, for the applicable message sizes - from being used. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 11:57 AM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft To me this does not look like a deadlock, proc o will make progress in the wait, and just like with point-to-point commuications process 1 will make general mpi progress while in the bcast. This includes progress on the ibcast so that process 0 can comete it's ibcast and move on to the bcast. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 05:14:04 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Graham, Richard L. 17 Oct '08

17 Oct '08

To me this does not look like a deadlock, proc o will make progress in the wait, and just like with point-to-point commuications process 1 will make general mpi progress while in the bcast. This includes progress on the ibcast so that process 0 can comete it's ibcast and move on to the bcast. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 05:14:04 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Torsten Hoefler 16 Oct '08

16 Oct '08

Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 16 Oct '08

16 Oct '08

Dear Torsten, Thank you. The pt2pt progress comment below was taken into account when you removed the nonblocking collective progress description. As for the example, if it is legal, we should probably count the blocking operations as well when we decide how many simultaneous ops are to be supported. Why "if" above: I'm not sure what a delayed nonblocking barrier would mean for process 1. Won't one process possibly block somewhere? Can you explain this please? The format matter is complicated. I think it is worth discussing at the Forum as we're likely to have more and more proposals approaching the final stage, when editing them in PDF format will become an issue. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 10:46 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Generally, some sentences start with lower case letters that should be > capitalized. yes, I fixed some but will re-read it later this week. > Page 3, top. We probably should not explain here how to ensure progress > for nonblocking ops. This matter should be covered by the general > progress description elsewhere. I agree, I deleted it (it's already in the nonblocking point-to-point chapter). > Page 3, lower half. Look how progress is defined in this list by > referring to the pt2pt progress rules. what do you mean with that? I would just reference the current MPI definition - NBCs seem to fit this model nicely. > Ibid. There are already 3 classes of requests: pt2pt, generalized, and > file I/O ones. yes > Page 3, bottom. How many nonblocking ops can overlap with a blocking > one? I.e., can one have the following situation: > > Process 0 Process 1 > > MPI_Ibarrier(req) MPI_Ibarrier(req) > MPI_Wait(req) > MPI Bcast MPI_Bcast > MPI_Wait(req) yes, this is a legal code and should work. Any objections? > Page 5, bottom. 32767 simultaneous ops to support is a tall order. I'd > say, 1 (one) would be a good lower limit, thus allowing no overlap. Or > even 0 (zero), meaning no nonblocking collective support at all. The > rest of this passage would probably be better reformulated as an advice > to implementors. yes it is - I'm very indicisive. We have to discuss this at the Forum in Chicago. > In the margin: editing a PDF requires creativity (exemplified by the > earlier comments) or a pretty expensive Acrobat. Maybe we can find some > other way of distributing and commenting on our drafts? oh yes, I fully agree. But the final document should be in MPI style, so I thought I should start with LaTeX. Do you have any suggestions? Wiki seems suboptimal because we would have to re-format everything (which might be not too bad). LaTeX source in svn would be the best, but does everybody have LaTeX? I'm open for any suggestion! For now, I uploaded the changed version to the wikipage (to avoid attachments on the mailinglist): https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl Thanks for the comments! Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Torsten Hoefler 16 Oct '08

16 Oct '08

On Wed, Oct 15, 2008 at 04:02:41PM +0200, Christian Siebert wrote: > @Torsten: > > Thanks a lot for this first draft! I tried to make some useful comments > and attached the results to this mail (sorry, I can't comment PDFs so I > did it the "old inconvenient way"). oh yeah - we should not send too bi attachments to the list (your mail was 2.3 MiB). I fixed all remarks in the scans. See attachment at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl > @Working Group (something to discuss): > > 1) The MPI-2.1 document has 127 occurrences of "nonblocking" and only 8 > occurrences of "non-blocking". What would be the correct term? Should we > try to be more consistent? no comment ;) - see ticket #44 > 2) Clarification of MPI_Request_free() for requests from non-blocking > collective operations. what do you mean? > 3) Better definition/description for "matching" (there is nothing like > "at the same time" -> logical order?). yes, do you have a suggestion? I replaced it with simultaneously for now - which is suboptimal too. > 4) Define "levels of progression"? To be queried (e.g., for "Synchronous > Progress" MPI_Tests are needed for performance, but for "Asynchronous > Progress" they would only add unnecessary overhead)? UP >= AP >= SP? yes, I actually erased those definitions because they don't belong in a standard (imho). > 5) "Unexpected Progress" => "buffered" collectives? this is also gone - this is an implementation detail, not an interface issue. > 6) NBC gives several possible ways for optimizations. With this "General > advice to implementers" we stick to only one, and might prevent others. > Can we already fix a decision for optimization strategies at this stage? > Should we fix it at all? this is only an advice. Maybe you're right, but I don't think that it'll be much better without the advice at all. Advices seem to be rather weak anyway so I have no strong opinion on that. > 7) Should there be a concrete code example in the proposal (e.g. an > implementation of this double buffering example)? maybe, doesn't sound too bad. Let's talk about this at the Forum I just started to creat an agenda for Chicago and put hose items on there (more will follow). https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Hope this isn't too much for now. never Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Torsten Hoefler 16 Oct '08

16 Oct '08

Hello Alexander, > Generally, some sentences start with lower case letters that should be > capitalized. yes, I fixed some but will re-read it later this week. > Page 3, top. We probably should not explain here how to ensure progress > for nonblocking ops. This matter should be covered by the general > progress description elsewhere. I agree, I deleted it (it's already in the nonblocking point-to-point chapter). > Page 3, lower half. Look how progress is defined in this list by > referring to the pt2pt progress rules. what do you mean with that? I would just reference the current MPI definition - NBCs seem to fit this model nicely. > Ibid. There are already 3 classes of requests: pt2pt, generalized, and > file I/O ones. yes > Page 3, bottom. How many nonblocking ops can overlap with a blocking > one? I.e., can one have the following situation: > > Process 0 Process 1 > > MPI_Ibarrier(req) MPI_Ibarrier(req) > MPI_Wait(req) > MPI Bcast MPI_Bcast > MPI_Wait(req) yes, this is a legal code and should work. Any objections? > Page 5, bottom. 32767 simultaneous ops to support is a tall order. I'd > say, 1 (one) would be a good lower limit, thus allowing no overlap. Or > even 0 (zero), meaning no nonblocking collective support at all. The > rest of this passage would probably be better reformulated as an advice > to implementors. yes it is - I'm very indicisive. We have to discuss this at the Forum in Chicago. > In the margin: editing a PDF requires creativity (exemplified by the > earlier comments) or a pretty expensive Acrobat. Maybe we can find some > other way of distributing and commenting on our drafts? oh yes, I fully agree. But the final document should be in MPI style, so I thought I should start with LaTeX. Do you have any suggestions? Wiki seems suboptimal because we would have to re-format everything (which might be not too bad). LaTeX source in svn would be the best, but does everybody have LaTeX? I'm open for any suggestion! For now, I uploaded the changed version to the wikipage (to avoid attachments on the mailinglist): https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl Thanks for the comments! Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Jesper Larsson Traeff 16 Oct '08

16 Oct '08

(to my eyes "non-blocking" looks right... but I don't have strong arguments. but still...) jesper On Wed, Oct 15, 2008 at 09:27:12PM -0400, Torsten Hoefler wrote: > On Wed, Oct 15, 2008 at 04:42:45PM -0700, Bronis R. de Supinski wrote: > > > > Rajeev and Torsten: > > > > I believe that Rajeev is correct. The Chicago Manual of > > Style passge that Torsten cited is not germane. It deals > > with whether you are supposed to add hyphens to adjective/ > > noun combinations that are being used as adjectives, e.g., > > "graduate student" (instead of "graduate-student") in > > graduate student housing. It has nothing to do with > > hyphenating compound words or words that are synthesized > > by adding well defined prefixes to existing words. > ok, I capitulate with the hyphen :). But after reading CMS 7.90 and > 8.170, http://amaediting.blogspot.com/2008/07/hyphenated-prefixes.html > and http://www.nyu.edu/classes/copyXediting/Hyphens.html , and talking > to more native speakers, I would say that everything seems to be evolving > to closed prefixes. > > So as weird as it looks, I'd propose "nonblocking" as the way to go (I > guess I should like it as a German). Any opinions on this? > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > "Our two greatest problems are gravity and paper work. We can lick > gravity, but sometimes the paperwork is overwhelming." > -- Wernher von Braun > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0