October 2008 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Richard Graham 19 Oct '08

19 Oct '08

Here are my comments: These are from the original draft sent out, so other may have already commented on some of these could not download from the web on the plane :-) Section 1.2 paragraph 2: The matching of those operations a is ruled by the order.... This is a bit confusing, as the paragraph mentions both point-to-point operations and blocking collective operations. I would suggest changing this to ³The matching of those blocking collective operations is rules by the order ....² Section 2: ³High-quality ...² - In general I am against these sorts of comments, even though they are strewn throughout the standard. There are many tradeoffs in implementing a communications library, and what may be good in one instance, may not be appropriate in another. It may be more appropriate to state ³The enables the application to take advantage of asynchronous progress, if the implementation implements such a capability...² While I agree with the sentiment, I disagree with the categorization. Section 2.1: Instead of using the term nested collectives, multiple outstanding non-blocking collectives seems clearer to me. The comment that calling MPI_Request_free() is not useful on the send side is not quite clear to me. Why is the send side different from the receive side ? In either case, I do not think that we should allow freeing a request in the middle of a collective (like one can for ptp communications) The paragraph on the bottom of page 3 is confusing. After mentioning that we can have multiple outstanding collectives, the last sentence seems to imply that in such a case, all would have to be of the same type (such as ibcast), which I do not think is the intent. Section 2.3: As I mentioned before, I do not believe we should specify that an implementation should support more than a minimum of 1 (i.e must provide support for this). Especially as the system sizes increase markedly, we need to be careful on what sort of resource requirements we place on an implementation. Rich On 10/16/08 4:46 PM, "Torsten Hoefler" <htor(a)cs.indiana.edu> wrote: > Hello Alexander, >> > Generally, some sentences start with lower case letters that should be >> > capitalized. > yes, I fixed some but will re-read it later this week. > >> > Page 3, top. We probably should not explain here how to ensure progress >> > for nonblocking ops. This matter should be covered by the general >> > progress description elsewhere. > I agree, I deleted it (it's already in the nonblocking point-to-point > chapter). > >> > Page 3, lower half. Look how progress is defined in this list by >> > referring to the pt2pt progress rules. > what do you mean with that? I would just reference the current MPI > definition - NBCs seem to fit this model nicely. > >> > Ibid. There are already 3 classes of requests: pt2pt, generalized, and >> > file I/O ones. > yes > >> > Page 3, bottom. How many nonblocking ops can overlap with a blocking >> > one? I.e., can one have the following situation: >> > >> > Process 0 Process 1 >> > >> > MPI_Ibarrier(req) MPI_Ibarrier(req) >> > MPI_Wait(req) >> > MPI Bcast MPI_Bcast >> > MPI_Wait(req) > yes, this is a legal code and should work. Any objections? > >> > Page 5, bottom. 32767 simultaneous ops to support is a tall order. I'd >> > say, 1 (one) would be a good lower limit, thus allowing no overlap. Or >> > even 0 (zero), meaning no nonblocking collective support at all. The >> > rest of this passage would probably be better reformulated as an advice >> > to implementors. > yes it is - I'm very indicisive. We have to discuss this at the Forum in > Chicago. > >> > In the margin: editing a PDF requires creativity (exemplified by the >> > earlier comments) or a pretty expensive Acrobat. Maybe we can find some >> > other way of distributing and commenting on our drafts? > oh yes, I fully agree. But the final document should be in MPI style, so > I thought I should start with LaTeX. Do you have any suggestions? Wiki > seems suboptimal because we would have to re-format everything (which > might be not too bad). LaTeX source in svn would be the best, but does > everybody have LaTeX? I'm open for any suggestion! > > For now, I uploaded the changed version to the wikipage (to avoid > attachments on the mailinglist): > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl > > Thanks for the comments! > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Researcher > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > Lindley Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Rich, Thank you. I'm basically after making sure that we don't design something that does not work. From this discussion, it appears that the existing progress rules do cover the proposal well enough to proceed. Best regards. Alexander ________________________________ From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Richard Graham Sent: Friday, October 17, 2008 3:42 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Alexander, I am not sure what the really trying to get after. The example I gave is exactly what I intended for it to be, not a complete working MPI example, but the core as it relates to progress. The example that Torsten gave should progress and complete, whether one uses asynchronous progress mechanisms or the synchronous approach most of us take today. If the question is about the semantics of MPI_Ibarrier(), I think the only specification is that it starts when the ibarrier is posted (what ever that means, which I am sure is implementation specific) and ends with the wait - I don't think that there is any other specification here. Have I completely missed what your question is ? Rich On 10/17/08 8:51 AM, "Supalov, Alexander" <alexander.supalov(a)intel.com> wrote: Dear Rich, Thanks. You probably meant Recv(0) in Proc 1. I also miss a Wait for req_r. So, a complete example would probably look like: Proc 0 Proc 1 Irecv(1,req_r) Irecv(0,req_r) Isend(1,req_s) Isend(0,req_s) Waitall(req_s, req_r) Recv(0) Send(1) Waitall(req_s,req_r) This would work. However, let's do one change, replacing Isend by Issend. Will this still work? Irecv(1,req_r) Irecv(0,req_r) Issend(1,req_s) Issend(0,req_s) Waitall(req_s, req_r) Recv(0) Send(1) Waitall(req_s,req_r) Another possible complication: let's swap the Send and Recv, and make Send synchronous (or rendezvous): Irecv(1,req_r) Irecv(0,req_r) Issend(1,req_s) Issend(0,req_s) Waitall(req_s, req_r) Ssend(0) Recv(1) Waitall(req_s,req_r) Here I have rather strong doubts that this will work. Do you? Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 2:35 PM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Alexander, Do you see this as different in principal from Proc 0. Proc 1 irecv(1,req_r) Irecv(0,req_r) Isend(1,req_s). Isend(0,req_s) Wait(req_s). Send(0) Send(1). Wait(req_s) I see these two cases as basically the same from a progress perspective. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 07:22:02 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Rich, Thank you. Note that the first operation is an Ibarrier. According to Torsten's explanation, process 0 will block in the Wait, which it may well do if it is Wait in process 1 that actually completes its Ibarrier. Now, how can process 0 make progress in its Wait when there's no matching Wait on the other side yet? Do we assume that any messages that may have been sent by process 0's IBarrier will be handled as unexpected messages by process 1 that is busy doing its (also unmatched) part of the Bcast? Likewise, do we assume that all messages that process 1 has to send in its Ibarrier can leave process 1 before it enters its Wait? Aren't we assuming a bit too much? In other words, aren't we assuming that all message sends and receives will be started in Ibarrier, and will be able to proceed even though the Wait is some blocking collective ops away? Given the way most MPIs implement nonblocking pt2pt ops now, something looks fishy here. In any case, I'd like to see a detailed explanation how things will work, and moreover, a proof that this does not preclude some algorithms - both in the sense of the pt2pt operations deployed, and in the sense of collective exchange pattern, for the applicable message sizes - from being used. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 11:57 AM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft To me this does not look like a deadlock, proc o will make progress in the wait, and just like with point-to-point commuications process 1 will make general mpi progress while in the bcast. This includes progress on the ibcast so that process 0 can comete it's ibcast and move on to the bcast. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 05:14:04 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Got it, thanks. -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Friday, October 17, 2008 3:48 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello, > Thanks. You probably meant Recv(0) in Proc 1. I also miss a Wait for > req_r. So, a complete example would probably look like: > > Proc 0 Proc 1 > > Irecv(1,req_r) Irecv(0,req_r) > Isend(1,req_s) Isend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) > > This would work. However, let's do one change, replacing Isend by > Issend. Will this still work? > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) sorry for joining in so late - it was night here. So I would still say that it doesn't deadlock - in the send/recv as well as in the collective case. Actually, from a correctness perspective, I see no difference between Rich's Isend() and your Issend() example because an MPI implementation is free to implement Isend() with Issend() (rendezvous protocol). Thus, implementers have to assume the worst case. To your example: - the process 1 is required to progress *all* outstanding operations while it is in Recv(0) - this enables the Waitall(req_s, req_r) on proc 0 to complete and proc 0 postes the send This is required by the MPI-2.1 standard afaik. > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? It's the same thing, Ssend will also have to progress the other requests. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Christian, Thanks, this sounds convincing. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Christian Siebert Sent: Friday, October 17, 2008 3:40 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Supalov, Alexander wrote: > <snip> > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? Why do you think that it won't work? It should work! The Waitall at the left side progresses both {Irecv and Issend}. The Ssend at the right side progresses {Irecv, Issend and SSend}. Therefore the first two operations on both sides can be matched, processed and completed within the third function. The rest (i.e. recv left & ssend; waitall right) is easy: the recv matches the ssend and the right waitall will just return the status of both (already completed) nonblocking operations. Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014) _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Torsten Hoefler 17 Oct '08

17 Oct '08

Hello, > Thanks. You probably meant Recv(0) in Proc 1. I also miss a Wait for > req_r. So, a complete example would probably look like: > > Proc 0 Proc 1 > > Irecv(1,req_r) Irecv(0,req_r) > Isend(1,req_s) Isend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) > > This would work. However, let's do one change, replacing Isend by > Issend. Will this still work? > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) sorry for joining in so late - it was night here. So I would still say that it doesn't deadlock - in the send/recv as well as in the collective case. Actually, from a correctness perspective, I see no difference between Rich's Isend() and your Issend() example because an MPI implementation is free to implement Isend() with Issend() (rendezvous protocol). Thus, implementers have to assume the worst case. To your example: - the process 1 is required to progress *all* outstanding operations while it is in Recv(0) - this enables the Waitall(req_s, req_r) on proc 0 to complete and proc 0 postes the send This is required by the MPI-2.1 standard afaik. > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? It's the same thing, Ssend will also have to progress the other requests. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Richard Graham 17 Oct '08

17 Oct '08

Alexander, I am not sure what the really trying to get after. The example I gave is exactly what I intended for it to be, not a complete working MPI example, but the core as it relates to progress. The example that Torsten gave should progress and complete, whether one uses asynchronous progress mechanisms or the synchronous approach most of us take today. If the question is about the semantics of MPI_Ibarrier(), I think the only specification is that it starts when the ibarrier is posted (what ever that means, which I am sure is implementation specific) and ends with the wait I don¹t think that there is any other specification here. Have I completely missed what your question is ? Rich On 10/17/08 8:51 AM, "Supalov, Alexander" <alexander.supalov(a)intel.com> wrote: > Dear Rich, > > Thanks. You probably meant Recv(0) in Proc 1. I also miss a Wait for > req_r. So, a complete example would probably look like: > > Proc 0 Proc 1 > > Irecv(1,req_r) Irecv(0,req_r) > Isend(1,req_s) Isend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) > > This would work. However, let's do one change, replacing Isend by > Issend. Will this still work? > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) > > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Graham, > Richard L. > Sent: Friday, October 17, 2008 2:35 PM > To: mpi3-coll(a)lists.mpi-forum.org > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > Alexander, > Do you see this as different in principal from > > Proc 0. Proc 1 > irecv(1,req_r) Irecv(0,req_r) > Isend(1,req_s). Isend(0,req_s) > Wait(req_s). Send(0) > Send(1). Wait(req_s) > > I see these two cases as basically the same from a progress perspective. > > Rich > > ----- Original Message ----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > <mpi3-coll-bounces(a)lists.mpi-forum.org> > To: MPI-3 Collective Subgroup Discussions > <mpi3-coll(a)lists.mpi-forum.org> > Sent: Fri Oct 17 07:22:02 2008 > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > Dear Rich, > > Thank you. Note that the first operation is an Ibarrier. According to > Torsten's explanation, process 0 will block in the Wait, which it may > well do if it is Wait in process 1 that actually completes its Ibarrier. > > Now, how can process 0 make progress in its Wait when there's no > matching Wait on the other side yet? Do we assume that any messages that > may have been sent by process 0's IBarrier will be handled as unexpected > messages by process 1 that is busy doing its (also unmatched) part of > the Bcast? > > Likewise, do we assume that all messages that process 1 has to send in > its Ibarrier can leave process 1 before it enters its Wait? Aren't we > assuming a bit too much? In other words, aren't we assuming that all > message sends and receives will be started in Ibarrier, and will be able > to proceed even though the Wait is some blocking collective ops away? > > Given the way most MPIs implement nonblocking pt2pt ops now, something > looks fishy here. In any case, I'd like to see a detailed explanation > how things will work, and moreover, a proof that this does not preclude > some algorithms - both in the sense of the pt2pt operations deployed, > and in the sense of collective exchange pattern, for the applicable > message sizes - from being used. > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Graham, > Richard L. > Sent: Friday, October 17, 2008 11:57 AM > To: mpi3-coll(a)lists.mpi-forum.org > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > To me this does not look like a deadlock, proc o will make progress in > the wait, and just like with point-to-point commuications process 1 will > make general mpi progress while in the bcast. This includes progress on > the ibcast so that process 0 can comete it's ibcast and move on to the > bcast. > > Rich > > ----- Original Message ----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > <mpi3-coll-bounces(a)lists.mpi-forum.org> > To: MPI-3 Collective Subgroup Discussions > <mpi3-coll(a)lists.mpi-forum.org> > Sent: Fri Oct 17 05:14:04 2008 > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > Dear Torsten, > > Thanks. I'm still unsure I understand how the example would work if > processes blocked in the Wait calls as you said below. In this case > process 0 would have no chance to reach the Bcast, while process 1 would > be inside it and never reached the Wait. > > Here again is the original diagram: > > Process 0 Process 1 > > MPI_Ibarrier(req) MPI_Ibarrier(req) > MPI_Wait(req)* > MPI Bcast MPI_Bcast* > MPI_Wait(req) > > The stars mark the point where the respective processes would block, as > far as I understand. Looks like deadlock? > > Apart from this, stating that the nonblocking collectives can block in > Wait presumes a certain implementation. What if Ibarrier was able to do > all right away, and Wait were just a no-op? > > I think mixing blocking and nonblocking collectives in the proposed way > should either be very carefully considered, or simply disallowed. > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Torsten > Hoefler > Sent: Thursday, October 16, 2008 11:15 PM > To: MPI-3 Collective Subgroup Discussions > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > Hello Alexander, > >> > Thank you. The pt2pt progress comment below was taken into account > when >> > you removed the nonblocking collective progress description. > ah, ok - so I think a reference to the chapter should be sufficient. > >> > As for the example, if it is legal, we should probably count the >> > blocking operations as well when we decide how many simultaneous ops > are >> > to be supported. > yes, maybe. I am not sure but this could be an issue. It's on the agenda > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > >> > Why "if" above: I'm not sure what a delayed nonblocking barrier would >> > mean for process 1. Won't one process possibly block somewhere? Can > you >> > explain this please? > the barriers will only block in the respective MPI_Wait() (or a tight > MPI_Test() loop). But the operation can be initiated as soon as the > critical region is left (and there mihgt be some other independent > computation to do). > >> > The format matter is complicated. I think it is worth discussing at > the >> > Forum as we're likely to have more and more proposals approaching the >> > final stage, when editing them in PDF format will become an issue. > yes, it's on the WG agenda - maybe we should discuss this in the whole > forum? > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Researcher > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > Lindley Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Christian Siebert 17 Oct '08

17 Oct '08

Supalov, Alexander wrote: > <snip> > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? Why do you think that it won't work? It should work! The Waitall at the left side progresses both {Irecv and Issend}. The Ssend at the right side progresses {Irecv, Issend and SSend}. Therefore the first two operations on both sides can be matched, processed and completed within the third function. The rest (i.e. recv left & ssend; waitall right) is easy: the recv matches the ssend and the right waitall will just return the status of both (already completed) nonblocking operations. Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Christian, Thanks for the idea. I play with Ssends a little in the other reply. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Christian Siebert Sent: Friday, October 17, 2008 2:42 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Alexander, as Rich pointed out, your example will not deadlock (try to understand his good explanation). I think that your progression question is not really related to nonblocking collectives. The same question could be asked with an example using MPI-1: process 0 process 1 --------- --------- MPI_Irecv(.., req) MPI_Barrier(comm) MPI_Barrier(comm) MPI_Ssend(..) MPI_Bcast(..) MPI_Bcast(..) MPI_Wait(req, ..) The barrier only ensures that the send gets started after the incompleted Irecv returned to the user (to simulate the outstanding Ibarrier). Although the broadcast has nothing to do with the Irecv(), it will still ensure the required progress. Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014) _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Rich, Thanks. You probably meant Recv(0) in Proc 1. I also miss a Wait for req_r. So, a complete example would probably look like: Proc 0 Proc 1 Irecv(1,req_r) Irecv(0,req_r) Isend(1,req_s) Isend(0,req_s) Waitall(req_s, req_r) Recv(0) Send(1) Waitall(req_s,req_r) This would work. However, let's do one change, replacing Isend by Issend. Will this still work? Irecv(1,req_r) Irecv(0,req_r) Issend(1,req_s) Issend(0,req_s) Waitall(req_s, req_r) Recv(0) Send(1) Waitall(req_s,req_r) Another possible complication: let's swap the Send and Recv, and make Send synchronous (or rendezvous): Irecv(1,req_r) Irecv(0,req_r) Issend(1,req_s) Issend(0,req_s) Waitall(req_s, req_r) Ssend(0) Recv(1) Waitall(req_s,req_r) Here I have rather strong doubts that this will work. Do you? Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 2:35 PM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Alexander, Do you see this as different in principal from Proc 0. Proc 1 irecv(1,req_r) Irecv(0,req_r) Isend(1,req_s). Isend(0,req_s) Wait(req_s). Send(0) Send(1). Wait(req_s) I see these two cases as basically the same from a progress perspective. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 07:22:02 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Rich, Thank you. Note that the first operation is an Ibarrier. According to Torsten's explanation, process 0 will block in the Wait, which it may well do if it is Wait in process 1 that actually completes its Ibarrier. Now, how can process 0 make progress in its Wait when there's no matching Wait on the other side yet? Do we assume that any messages that may have been sent by process 0's IBarrier will be handled as unexpected messages by process 1 that is busy doing its (also unmatched) part of the Bcast? Likewise, do we assume that all messages that process 1 has to send in its Ibarrier can leave process 1 before it enters its Wait? Aren't we assuming a bit too much? In other words, aren't we assuming that all message sends and receives will be started in Ibarrier, and will be able to proceed even though the Wait is some blocking collective ops away? Given the way most MPIs implement nonblocking pt2pt ops now, something looks fishy here. In any case, I'd like to see a detailed explanation how things will work, and moreover, a proof that this does not preclude some algorithms - both in the sense of the pt2pt operations deployed, and in the sense of collective exchange pattern, for the applicable message sizes - from being used. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 11:57 AM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft To me this does not look like a deadlock, proc o will make progress in the wait, and just like with point-to-point commuications process 1 will make general mpi progress while in the bcast. This includes progress on the ibcast so that process 0 can comete it's ibcast and move on to the bcast. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 05:14:04 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Christian Siebert 17 Oct '08

17 Oct '08

Dear Alexander, as Rich pointed out, your example will not deadlock (try to understand his good explanation). I think that your progression question is not really related to nonblocking collectives. The same question could be asked with an example using MPI-1: process 0 process 1 --------- --------- MPI_Irecv(.., req) MPI_Barrier(comm) MPI_Barrier(comm) MPI_Ssend(..) MPI_Bcast(..) MPI_Bcast(..) MPI_Wait(req, ..) The barrier only ensures that the send gets started after the incompleted Irecv returned to the user (to simulate the outstanding Ibarrier). Although the broadcast has nothing to do with the Irecv(), it will still ensure the required progress. Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0