- mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Christian, Thanks, this sounds convincing. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Christian Siebert Sent: Friday, October 17, 2008 3:40 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Supalov, Alexander wrote: > <snip> > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? Why do you think that it won't work? It should work! The Waitall at the left side progresses both {Irecv and Issend}. The Ssend at the right side progresses {Irecv, Issend and SSend}. Therefore the first two operations on both sides can be matched, processed and completed within the third function. The rest (i.e. recv left & ssend; waitall right) is easy: the recv matches the ssend and the right waitall will just return the status of both (already completed) nonblocking operations. Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014) _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Torsten Hoefler 17 Oct '08

17 Oct '08

Hello, > Thanks. You probably meant Recv(0) in Proc 1. I also miss a Wait for > req_r. So, a complete example would probably look like: > > Proc 0 Proc 1 > > Irecv(1,req_r) Irecv(0,req_r) > Isend(1,req_s) Isend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) > > This would work. However, let's do one change, replacing Isend by > Issend. Will this still work? > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) sorry for joining in so late - it was night here. So I would still say that it doesn't deadlock - in the send/recv as well as in the collective case. Actually, from a correctness perspective, I see no difference between Rich's Isend() and your Issend() example because an MPI implementation is free to implement Isend() with Issend() (rendezvous protocol). Thus, implementers have to assume the worst case. To your example: - the process 1 is required to progress *all* outstanding operations while it is in Recv(0) - this enables the Waitall(req_s, req_r) on proc 0 to complete and proc 0 postes the send This is required by the MPI-2.1 standard afaik. > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? It's the same thing, Ssend will also have to progress the other requests. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Richard Graham 17 Oct '08

17 Oct '08

Alexander, I am not sure what the really trying to get after. The example I gave is exactly what I intended for it to be, not a complete working MPI example, but the core as it relates to progress. The example that Torsten gave should progress and complete, whether one uses asynchronous progress mechanisms or the synchronous approach most of us take today. If the question is about the semantics of MPI_Ibarrier(), I think the only specification is that it starts when the ibarrier is posted (what ever that means, which I am sure is implementation specific) and ends with the wait I don¹t think that there is any other specification here. Have I completely missed what your question is ? Rich On 10/17/08 8:51 AM, "Supalov, Alexander" <alexander.supalov(a)intel.com> wrote: > Dear Rich, > > Thanks. You probably meant Recv(0) in Proc 1. I also miss a Wait for > req_r. So, a complete example would probably look like: > > Proc 0 Proc 1 > > Irecv(1,req_r) Irecv(0,req_r) > Isend(1,req_s) Isend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) > > This would work. However, let's do one change, replacing Isend by > Issend. Will this still work? > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Recv(0) > Send(1) Waitall(req_s,req_r) > > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Graham, > Richard L. > Sent: Friday, October 17, 2008 2:35 PM > To: mpi3-coll(a)lists.mpi-forum.org > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > Alexander, > Do you see this as different in principal from > > Proc 0. Proc 1 > irecv(1,req_r) Irecv(0,req_r) > Isend(1,req_s). Isend(0,req_s) > Wait(req_s). Send(0) > Send(1). Wait(req_s) > > I see these two cases as basically the same from a progress perspective. > > Rich > > ----- Original Message ----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > <mpi3-coll-bounces(a)lists.mpi-forum.org> > To: MPI-3 Collective Subgroup Discussions > <mpi3-coll(a)lists.mpi-forum.org> > Sent: Fri Oct 17 07:22:02 2008 > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > Dear Rich, > > Thank you. Note that the first operation is an Ibarrier. According to > Torsten's explanation, process 0 will block in the Wait, which it may > well do if it is Wait in process 1 that actually completes its Ibarrier. > > Now, how can process 0 make progress in its Wait when there's no > matching Wait on the other side yet? Do we assume that any messages that > may have been sent by process 0's IBarrier will be handled as unexpected > messages by process 1 that is busy doing its (also unmatched) part of > the Bcast? > > Likewise, do we assume that all messages that process 1 has to send in > its Ibarrier can leave process 1 before it enters its Wait? Aren't we > assuming a bit too much? In other words, aren't we assuming that all > message sends and receives will be started in Ibarrier, and will be able > to proceed even though the Wait is some blocking collective ops away? > > Given the way most MPIs implement nonblocking pt2pt ops now, something > looks fishy here. In any case, I'd like to see a detailed explanation > how things will work, and moreover, a proof that this does not preclude > some algorithms - both in the sense of the pt2pt operations deployed, > and in the sense of collective exchange pattern, for the applicable > message sizes - from being used. > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Graham, > Richard L. > Sent: Friday, October 17, 2008 11:57 AM > To: mpi3-coll(a)lists.mpi-forum.org > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > To me this does not look like a deadlock, proc o will make progress in > the wait, and just like with point-to-point commuications process 1 will > make general mpi progress while in the bcast. This includes progress on > the ibcast so that process 0 can comete it's ibcast and move on to the > bcast. > > Rich > > ----- Original Message ----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > <mpi3-coll-bounces(a)lists.mpi-forum.org> > To: MPI-3 Collective Subgroup Discussions > <mpi3-coll(a)lists.mpi-forum.org> > Sent: Fri Oct 17 05:14:04 2008 > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > Dear Torsten, > > Thanks. I'm still unsure I understand how the example would work if > processes blocked in the Wait calls as you said below. In this case > process 0 would have no chance to reach the Bcast, while process 1 would > be inside it and never reached the Wait. > > Here again is the original diagram: > > Process 0 Process 1 > > MPI_Ibarrier(req) MPI_Ibarrier(req) > MPI_Wait(req)* > MPI Bcast MPI_Bcast* > MPI_Wait(req) > > The stars mark the point where the respective processes would block, as > far as I understand. Looks like deadlock? > > Apart from this, stating that the nonblocking collectives can block in > Wait presumes a certain implementation. What if Ibarrier was able to do > all right away, and Wait were just a no-op? > > I think mixing blocking and nonblocking collectives in the proposed way > should either be very carefully considered, or simply disallowed. > > Best regards. > > Alexander > > -----Original Message----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of Torsten > Hoefler > Sent: Thursday, October 16, 2008 11:15 PM > To: MPI-3 Collective Subgroup Discussions > Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft > > Hello Alexander, > >> > Thank you. The pt2pt progress comment below was taken into account > when >> > you removed the nonblocking collective progress description. > ah, ok - so I think a reference to the chapter should be sufficient. > >> > As for the example, if it is legal, we should probably count the >> > blocking operations as well when we decide how many simultaneous ops > are >> > to be supported. > yes, maybe. I am not sure but this could be an issue. It's on the agenda > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > >> > Why "if" above: I'm not sure what a delayed nonblocking barrier would >> > mean for process 1. Won't one process possibly block somewhere? Can > you >> > explain this please? > the barriers will only block in the respective MPI_Wait() (or a tight > MPI_Test() loop). But the operation can be initiated as soon as the > critical region is left (and there mihgt be some other independent > computation to do). > >> > The format matter is complicated. I think it is worth discussing at > the >> > Forum as we're likely to have more and more proposals approaching the >> > final stage, when editing them in PDF format will become an issue. > yes, it's on the WG agenda - maybe we should discuss this in the whole > forum? > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Researcher > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > Lindley Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Christian Siebert 17 Oct '08

17 Oct '08

Supalov, Alexander wrote: > <snip> > Another possible complication: let's swap the Send and Recv, and make > Send synchronous (or rendezvous): > > Irecv(1,req_r) Irecv(0,req_r) > Issend(1,req_s) Issend(0,req_s) > Waitall(req_s, req_r) Ssend(0) > Recv(1) Waitall(req_s,req_r) > > Here I have rather strong doubts that this will work. Do you? Why do you think that it won't work? It should work! The Waitall at the left side progresses both {Irecv and Issend}. The Ssend at the right side progresses {Irecv, Issend and SSend}. Therefore the first two operations on both sides can be matched, processed and completed within the third function. The rest (i.e. recv left & ssend; waitall right) is easy: the recv matches the ssend and the right waitall will just return the status of both (already completed) nonblocking operations. Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Christian, Thanks for the idea. I play with Ssends a little in the other reply. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Christian Siebert Sent: Friday, October 17, 2008 2:42 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Alexander, as Rich pointed out, your example will not deadlock (try to understand his good explanation). I think that your progression question is not really related to nonblocking collectives. The same question could be asked with an example using MPI-1: process 0 process 1 --------- --------- MPI_Irecv(.., req) MPI_Barrier(comm) MPI_Barrier(comm) MPI_Ssend(..) MPI_Bcast(..) MPI_Bcast(..) MPI_Wait(req, ..) The barrier only ensures that the send gets started after the incompleted Irecv returned to the user (to simulate the outstanding Ibarrier). Although the broadcast has nothing to do with the Irecv(), it will still ensure the required progress. Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014) _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Rich, Thanks. You probably meant Recv(0) in Proc 1. I also miss a Wait for req_r. So, a complete example would probably look like: Proc 0 Proc 1 Irecv(1,req_r) Irecv(0,req_r) Isend(1,req_s) Isend(0,req_s) Waitall(req_s, req_r) Recv(0) Send(1) Waitall(req_s,req_r) This would work. However, let's do one change, replacing Isend by Issend. Will this still work? Irecv(1,req_r) Irecv(0,req_r) Issend(1,req_s) Issend(0,req_s) Waitall(req_s, req_r) Recv(0) Send(1) Waitall(req_s,req_r) Another possible complication: let's swap the Send and Recv, and make Send synchronous (or rendezvous): Irecv(1,req_r) Irecv(0,req_r) Issend(1,req_s) Issend(0,req_s) Waitall(req_s, req_r) Ssend(0) Recv(1) Waitall(req_s,req_r) Here I have rather strong doubts that this will work. Do you? Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 2:35 PM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Alexander, Do you see this as different in principal from Proc 0. Proc 1 irecv(1,req_r) Irecv(0,req_r) Isend(1,req_s). Isend(0,req_s) Wait(req_s). Send(0) Send(1). Wait(req_s) I see these two cases as basically the same from a progress perspective. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 07:22:02 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Rich, Thank you. Note that the first operation is an Ibarrier. According to Torsten's explanation, process 0 will block in the Wait, which it may well do if it is Wait in process 1 that actually completes its Ibarrier. Now, how can process 0 make progress in its Wait when there's no matching Wait on the other side yet? Do we assume that any messages that may have been sent by process 0's IBarrier will be handled as unexpected messages by process 1 that is busy doing its (also unmatched) part of the Bcast? Likewise, do we assume that all messages that process 1 has to send in its Ibarrier can leave process 1 before it enters its Wait? Aren't we assuming a bit too much? In other words, aren't we assuming that all message sends and receives will be started in Ibarrier, and will be able to proceed even though the Wait is some blocking collective ops away? Given the way most MPIs implement nonblocking pt2pt ops now, something looks fishy here. In any case, I'd like to see a detailed explanation how things will work, and moreover, a proof that this does not preclude some algorithms - both in the sense of the pt2pt operations deployed, and in the sense of collective exchange pattern, for the applicable message sizes - from being used. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 11:57 AM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft To me this does not look like a deadlock, proc o will make progress in the wait, and just like with point-to-point commuications process 1 will make general mpi progress while in the bcast. This includes progress on the ibcast so that process 0 can comete it's ibcast and move on to the bcast. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 05:14:04 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Christian Siebert 17 Oct '08

17 Oct '08

Dear Alexander, as Rich pointed out, your example will not deadlock (try to understand his good explanation). I think that your progression question is not really related to nonblocking collectives. The same question could be asked with an example using MPI-1: process 0 process 1 --------- --------- MPI_Irecv(.., req) MPI_Barrier(comm) MPI_Barrier(comm) MPI_Ssend(..) MPI_Bcast(..) MPI_Bcast(..) MPI_Wait(req, ..) The barrier only ensures that the send gets started after the incompleted Irecv returned to the user (to simulate the outstanding Ibarrier). Although the broadcast has nothing to do with the Irecv(), it will still ensure the required progress. Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Graham, Richard L. 17 Oct '08

17 Oct '08

Alexander, Do you see this as different in principal from Proc 0. Proc 1 irecv(1,req_r) Irecv(0,req_r) Isend(1,req_s). Isend(0,req_s) Wait(req_s). Send(0) Send(1). Wait(req_s) I see these two cases as basically the same from a progress perspective. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 07:22:02 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Rich, Thank you. Note that the first operation is an Ibarrier. According to Torsten's explanation, process 0 will block in the Wait, which it may well do if it is Wait in process 1 that actually completes its Ibarrier. Now, how can process 0 make progress in its Wait when there's no matching Wait on the other side yet? Do we assume that any messages that may have been sent by process 0's IBarrier will be handled as unexpected messages by process 1 that is busy doing its (also unmatched) part of the Bcast? Likewise, do we assume that all messages that process 1 has to send in its Ibarrier can leave process 1 before it enters its Wait? Aren't we assuming a bit too much? In other words, aren't we assuming that all message sends and receives will be started in Ibarrier, and will be able to proceed even though the Wait is some blocking collective ops away? Given the way most MPIs implement nonblocking pt2pt ops now, something looks fishy here. In any case, I'd like to see a detailed explanation how things will work, and moreover, a proof that this does not preclude some algorithms - both in the sense of the pt2pt operations deployed, and in the sense of collective exchange pattern, for the applicable message sizes - from being used. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 11:57 AM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft To me this does not look like a deadlock, proc o will make progress in the wait, and just like with point-to-point commuications process 1 will make general mpi progress while in the bcast. This includes progress on the ibcast so that process 0 can comete it's ibcast and move on to the bcast. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 05:14:04 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Christian Siebert 17 Oct '08

17 Oct '08

Hi Torsten, > oh yeah - we should not send too bi attachments to the list (your mail > was 2.3 MiB). I fixed all remarks in the scans. See attachment at: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl I'm sorry for this, I'll give comments in pure text form in this mail. Putting the proposal into the Wiki is an excellent idea! >> 2) Clarification of MPI_Request_free() for requests from non-blocking >> collective operations. > what do you mean? The current description is not clear enough and might lead to confusion. Is the following code correct, or not? MPI_Ibcast(buf, c, dt, root, comm, req); if (myrank == root) { MPI_Request_free(req); } else { MPI_Wait(req, MPI_STATUS_IGNORE); } More background: A user might assume that the root process does act solely as a sender (even this might be dangerous, e.g. for other collectives). "Freeing a request is only useful at the sender side" suggests that the above code is valid and can potentially even be beneficial for performance (no blocking at the root)... Personally, I'd like to forbid the use of MPI_Request_free() for requests from nonblocking collectives. >> 3) Better definition/description for "matching" (there is nothing like >> "at the same time" -> logical order?). > yes, do you have a suggestion? unfortunately no, sorry (i.e. nothing proper yet) > I replaced it with simultaneously for now > - which is suboptimal too. I totally agree. Maybe someone else has some suggestions? >> 4) Define "levels of progression"? To be queried (e.g., for "Synchronous >> Progress" MPI_Tests are needed for performance, but for "Asynchronous >> Progress" they would only add unnecessary overhead)? UP >= AP >= SP? > yes, I actually erased those definitions because they don't belong in a > standard (imho). good - they might be mentioned in some "Advice to..." section >> 6) NBC gives several possible ways for optimizations. With this "General >> advice to implementers" we stick to only one, and might prevent others. >> Can we already fix a decision for optimization strategies at this stage? >> Should we fix it at all? > this is only an advice. Maybe you're right, but I don't think that it'll > be much better without the advice at all. Advices seem to be rather weak > anyway so I have no strong opinion on that. ok - maybe just weaken the statement, e.g. "Nonblocking operations can be used ..." instead of "Most nonblocking operations will be used...". (Besides some "good" interconnects, MPI history showed that this "overlapping" issue, i.e. concurrent communication and computation, was often already problematic for pt2pt ops. I totally agree that this is something really good to aim for. However, I see the main advantage of nonblocking operations slightly more in the context of reduced waiting times, i.e. a blocking receiver needs to wait for the sender but a nonblocking receiver can potentially do something worthwhile during this time. An implementation which tries to optimize for this direction might not necessarily achieve "as low CPU overhead as possible". Currently, I don't know which direction - there are surely many more - is the way to go...) To the point with matching BC and NBC: It is clearly stated that they don't match. I don't like the additional sentence "Matching them should fail ..." because this forces implementations to check for matchings (e.g. an implementation might use a separate communicator for NBC; explicit checking for an incorrect matching between BC and NBC would incur additional overhead). There are other (better) ways to detect incorrect MPI programs. However, this should not be part of the standard itself. Having an additional explanation why we decided against a matching might be beneficial. Something like "Both classes (i.e. BC and NBC) have potentially distinct optimization criteria, enforcing different implementations." might help the reader to understand this decision. To the barrier section: "initializes" is already a predefined word in the MPI standard and is reserved for persistent operations. The correct term in this context would be "initiate". Persistent collectives is another proposal... ;-) >> 7) Should there be a concrete code example in the proposal (e.g. an >> implementation of this double buffering example)? > maybe, doesn't sound too bad. Let's talk about this at the Forum Unfortunately, I'll not be there.... :-( > I just started to creat an agenda for Chicago and put hose items on > there (more will follow). great! I'm also glad that so many of my remarks made it into the proposal. So thanks again! ;-) Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Supalov, Alexander 17 Oct '08

17 Oct '08

Dear Rich, Thank you. Note that the first operation is an Ibarrier. According to Torsten's explanation, process 0 will block in the Wait, which it may well do if it is Wait in process 1 that actually completes its Ibarrier. Now, how can process 0 make progress in its Wait when there's no matching Wait on the other side yet? Do we assume that any messages that may have been sent by process 0's IBarrier will be handled as unexpected messages by process 1 that is busy doing its (also unmatched) part of the Bcast? Likewise, do we assume that all messages that process 1 has to send in its Ibarrier can leave process 1 before it enters its Wait? Aren't we assuming a bit too much? In other words, aren't we assuming that all message sends and receives will be started in Ibarrier, and will be able to proceed even though the Wait is some blocking collective ops away? Given the way most MPIs implement nonblocking pt2pt ops now, something looks fishy here. In any case, I'd like to see a detailed explanation how things will work, and moreover, a proof that this does not preclude some algorithms - both in the sense of the pt2pt operations deployed, and in the sense of collective exchange pattern, for the applicable message sizes - from being used. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Friday, October 17, 2008 11:57 AM To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft To me this does not look like a deadlock, proc o will make progress in the wait, and just like with point-to-point commuications process 1 will make general mpi progress while in the bcast. This includes progress on the ibcast so that process 0 can comete it's ibcast and move on to the bcast. Rich ----- Original Message ----- From: mpi3-coll-bounces(a)lists.mpi-forum.org <mpi3-coll-bounces(a)lists.mpi-forum.org> To: MPI-3 Collective Subgroup Discussions <mpi3-coll(a)lists.mpi-forum.org> Sent: Fri Oct 17 05:14:04 2008 Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Dear Torsten, Thanks. I'm still unsure I understand how the example would work if processes blocked in the Wait calls as you said below. In this case process 0 would have no chance to reach the Bcast, while process 1 would be inside it and never reached the Wait. Here again is the original diagram: Process 0 Process 1 MPI_Ibarrier(req) MPI_Ibarrier(req) MPI_Wait(req)* MPI Bcast MPI_Bcast* MPI_Wait(req) The stars mark the point where the respective processes would block, as far as I understand. Looks like deadlock? Apart from this, stating that the nonblocking collectives can block in Wait presumes a certain implementation. What if Ibarrier was able to do all right away, and Wait were just a no-op? I think mixing blocking and nonblocking collectives in the proposed way should either be very carefully considered, or simply disallowed. Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Thursday, October 16, 2008 11:15 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Non-blocking Collectives Proposal Draft Hello Alexander, > Thank you. The pt2pt progress comment below was taken into account when > you removed the nonblocking collective progress description. ah, ok - so I think a reference to the chapter should be sufficient. > As for the example, if it is legal, we should probably count the > blocking operations as well when we decide how many simultaneous ops are > to be supported. yes, maybe. I am not sure but this could be an issue. It's on the agenda https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/forum201008 > Why "if" above: I'm not sure what a delayed nonblocking barrier would > mean for process 1. Won't one process possibly block somewhere? Can you > explain this please? the barriers will only block in the respective MPI_Wait() (or a tight MPI_Test() loop). But the operation can be initiated as soon as the critical region is left (and there mihgt be some other independent computation to do). > The format matter is complicated. I think it is worth discussing at the > Forum as we're likely to have more and more proposals approaching the > final stage, when editing them in PDF format will become an issue. yes, it's on the WG agenda - maybe we should discuss this in the whole forum? Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0