June 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Barrett, Brian W 14 Jun '13

14 Jun '13

As an FYI, my problem with this proposal is two-fold: First, we need to deal with the fact that many of the calls are local today, meaning we need to fix some definitions, which is a big change (iflush/flush shouldn't have the same semantics). Second, the semantics of iflush (and friends) need lots of care. For example, allowing a communication event between iflush and wait would greatly hinder my implementation's performance when not using iflush. I understand the desire and like many of the non-blocking, but there are many places that we should balance non-blocking variants vs. performance loss... Brian Sent with Good (www.good.com) -----Original Message----- From: Pavan Balaji [balaji(a)mcs.anl.gov<mailto:[email protected]>] Sent: Friday, June 14, 2013 02:51 PM Mountain Standard Time To: mpi3-rma(a)lists.mpi-forum.org Subject: [EXTERNAL] Re: [Mpi3-rma] request-based ops On 06/14/2013 03:43 PM, Jim Dinan wrote: > Ahh, right, I had to miss that session at the Forum meeting. If anyone > else is looking for them, these are the slides: > > http://meetings.mpi-forum.org/secretary/2013/06/slides/2013-06-06-Nonblocki… > > I was indeed forgetting about remote versus local completion and polling > on the request. It sounds like this would definitely fill a gap in the > RMA interface. FYI, this was not meant to be an RMA proposal, since it touches blocking function calls in many chapters (e.g., COMM_ICONNECT and COMM_IACCEPT). I think I/O operations have a similar ticket as well. All these tickets need to be combined into a single nonblocking operations ticket. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] request-based ops
by Pavan Balaji 14 Jun '13

14 Jun '13

On 06/14/2013 03:43 PM, Jim Dinan wrote: > Ahh, right, I had to miss that session at the Forum meeting. If anyone > else is looking for them, these are the slides: > > http://meetings.mpi-forum.org/secretary/2013/06/slides/2013-06-06-Nonblocki… > > I was indeed forgetting about remote versus local completion and polling > on the request. It sounds like this would definitely fill a gap in the > RMA interface. FYI, this was not meant to be an RMA proposal, since it touches blocking function calls in many chapters (e.g., COMM_ICONNECT and COMM_IACCEPT). I think I/O operations have a similar ticket as well. All these tickets need to be combined into a single nonblocking operations ticket. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] request-based ops
by Jim Dinan 14 Jun '13

14 Jun '13

Ahh, right, I had to miss that session at the Forum meeting. If anyone else is looking for them, these are the slides: http://meetings.mpi-forum.org/secretary/2013/06/slides/2013-06-06-Nonblocki… I was indeed forgetting about remote versus local completion and polling on the request. It sounds like this would definitely fill a gap in the RMA interface. ~Jim. On Fri, Jun 14, 2013 at 12:56 PM, Jeff Hammond <jhammond(a)alcf.anl.gov>wrote: > "The completion of an MPI_RACCUMULATE operation indicates that the > origin buffer is free to be updated. It does not indicate that the > operation has completed at the target window." > > How does waiting on this request give me the same behavior as IWIN_FLUSH? > > Jeff > > On Fri, Jun 14, 2013 at 12:23 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Jeff, > > > > How would iwin_flush be different in practice from doing nothing? I can > > give you a trivial implementation that does nothing until you wait on the > > request, at which point I call blocking win_flush. With my implementor > hat > > on, I would view this function as a performance hint that indicates to > the > > implementation that you will want to complete all operations thus far and > > none after. Given a high quality implementation that actually makes > > background progress on one-sided communications, this should be > unnecessary > > and I would probably implement it as above. If you want to only complete > > specific operations, the request generating functions are available. > > > > ~Jim. > > > > > > > > On Fri, Jun 14, 2013 at 11:09 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> > > wrote: > >> > >> Yes, I was making a disingenuous argument to push back against the > >> fallacy that iwin_flush is not required. > >> > >> Jeff > >> > >> On Fri, Jun 14, 2013 at 10:25 AM, Jim Dinan <james.dinan(a)gmail.com> > wrote: > >> > Hey now, blocking doesn't mean that MPI /must/ block, it only means > that > >> > it > >> > /can/ block. Blocking and synchronizing are certainly not the same; > >> > Ssend > >> > synchronizes, which is a stronger semantic than blocking. > >> > > >> > ~Jim. > >> > > >> > > >> > On Wed, Jun 12, 2013 at 5:07 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> > >> > wrote: > >> >> > >> >> Sure, let's separate these issues. I think Pirate RMA (RRPUT, > >> >> RRACCUMULATE and RRRGET_ACCUMULATE) is a more _local_ solution in the > >> >> MPI standard sense :-) > >> >> > >> >> SEND is also a nonblocking operation because the implementation can > >> >> always buffer the data and return before the remote RECV is posted. > >> >> Only SSEND is blocking if one asserts that blocking is equivalent to > >> >> "requires remote agency to complete". > >> >> > >> >> So basically, I completely agree with the Forum back-pushers that > >> >> IWIN_FLUSH etc. are as useful as ISEND. > >> >> > >> >> Jeff > >> >> > >> >> On Wed, Jun 12, 2013 at 4:58 PM, Pavan Balaji <balaji(a)mcs.anl.gov> > >> >> wrote: > >> >> > > >> >> > The push-back at the Forum was that WIN_FLUSH and WIN_UNLOCK are > >> >> > local > >> >> > operations (in MPI's definition of local), so they are already > >> >> > nonblocking. > >> >> > I was not at the Forum. I pointed out to Wesley after he returned > >> >> > that > >> >> > they > >> >> > are only local in the simple case where no one else is holding the > >> >> > lock. > >> >> > Otherwise they are not. > >> >> > > >> >> > Further, the proposal is not only about WIN_FLUSH and WIN_UNLOCK. > It > >> >> > was > >> >> > for a long list of blocking functions: COMM_ACCEPT, COMM_CONNECT, > >> >> > WIN_FENCE, > >> >> > WIN_COMPLETE, WIN_WAIT, WIN_UNLOCK, WIN_FLUSH, WIN_FLUSH_LOCAL, > >> >> > WIN_FLUSH_ALL, WIN_FLUSH_LOCAL_ALL, ... The point was that MPI > >> >> > should > >> >> > deal > >> >> > with concepts, not specific function exceptions. Nonblocking > >> >> > communication > >> >> > is one such concept, and it should be valid for all operations > (that > >> >> > are > >> >> > not > >> >> > local). > >> >> > > >> >> > FWIW, my proposal was only to add nonblocking variants to blocking > >> >> > calls. > >> >> > Adding extra per-op remote completion requests is a separate issue > >> >> > and > >> >> > should not be mixed with this. I personally think it's useful (in > >> >> > fact, > >> >> > my > >> >> > original proposal for RPUT/RGET equivalents had it), but that's > still > >> >> > a > >> >> > separate issue. > >> >> > > >> >> > -- Pavan > >> >> > > >> >> > > >> >> > On 06/12/2013 09:41 AM, Jeff Hammond wrote: > >> >> >> > >> >> >> So Wes was telling me about IWIN_FLUSH yesterday (this function > is a > >> >> >> great idea, btw) and the alternative of adding double-request RPUT > >> >> >> and > >> >> >> RACCUMULATE that provide a request for both local and remote > >> >> >> completion. By induction, this would imply a triple-request > >> >> >> RGET_ACCUMULATE operation, which I also think is a good idea. > It's > >> >> >> really silly to have to wait for the result buffer to be ready to > >> >> >> reuse the origin buffer in RGET_ACCUMULATE. It seems that many > >> >> >> implementations will be able to send the entire origin buffer a > >> >> >> nontrivial amount of time before the result comes back, > particularly > >> >> >> if the remote side doesn't have hardware-based progress and the > data > >> >> >> has to wait in a network buffer. > >> >> >> > >> >> >> I'm sorry that I missed the last day of the Forum in San Jose. I > >> >> >> didn't know these operations were on the table. They sound like a > >> >> >> great idea to me. > >> >> >> > >> >> >> Jeff > >> >> >> > >> >> > > >> >> > -- > >> >> > Pavan Balaji > >> >> > http://www.mcs.anl.gov/~balaji > >> >> > >> >> > >> >> > >> >> -- > >> >> Jeff Hammond > >> >> Argonne Leadership Computing Facility > >> >> University of Chicago Computation Institute > >> >> jhammond(a)alcf.anl.gov / (630) 252-5381 > >> >> http://www.linkedin.com/in/jeffhammond > >> >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > >> >> ALCF docs: http://www.alcf.anl.gov/user-guides > >> >> _______________________________________________ > >> >> mpi3-rma mailing list > >> >> mpi3-rma(a)lists.mpi-forum.org > >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >> > > >> > > >> > > >> > _______________________________________________ > >> > mpi3-rma mailing list > >> > mpi3-rma(a)lists.mpi-forum.org > >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >> > >> > >> > >> -- > >> Jeff Hammond > >> Argonne Leadership Computing Facility > >> University of Chicago Computation Institute > >> jhammond(a)alcf.anl.gov / (630) 252-5381 > >> http://www.linkedin.com/in/jeffhammond > >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > >> ALCF docs: http://www.alcf.anl.gov/user-guides > >> _______________________________________________ > >> mpi3-rma mailing list > >> mpi3-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

Re: [Mpi3-rma] request-based ops
by Jeff Hammond 14 Jun '13

14 Jun '13

"The completion of an MPI_RACCUMULATE operation indicates that the origin buffer is free to be updated. It does not indicate that the operation has completed at the target window." How does waiting on this request give me the same behavior as IWIN_FLUSH? Jeff On Fri, Jun 14, 2013 at 12:23 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Jeff, > > How would iwin_flush be different in practice from doing nothing? I can > give you a trivial implementation that does nothing until you wait on the > request, at which point I call blocking win_flush. With my implementor hat > on, I would view this function as a performance hint that indicates to the > implementation that you will want to complete all operations thus far and > none after. Given a high quality implementation that actually makes > background progress on one-sided communications, this should be unnecessary > and I would probably implement it as above. If you want to only complete > specific operations, the request generating functions are available. > > ~Jim. > > > > On Fri, Jun 14, 2013 at 11:09 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> > wrote: >> >> Yes, I was making a disingenuous argument to push back against the >> fallacy that iwin_flush is not required. >> >> Jeff >> >> On Fri, Jun 14, 2013 at 10:25 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> > Hey now, blocking doesn't mean that MPI /must/ block, it only means that >> > it >> > /can/ block. Blocking and synchronizing are certainly not the same; >> > Ssend >> > synchronizes, which is a stronger semantic than blocking. >> > >> > ~Jim. >> > >> > >> > On Wed, Jun 12, 2013 at 5:07 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> >> > wrote: >> >> >> >> Sure, let's separate these issues. I think Pirate RMA (RRPUT, >> >> RRACCUMULATE and RRRGET_ACCUMULATE) is a more _local_ solution in the >> >> MPI standard sense :-) >> >> >> >> SEND is also a nonblocking operation because the implementation can >> >> always buffer the data and return before the remote RECV is posted. >> >> Only SSEND is blocking if one asserts that blocking is equivalent to >> >> "requires remote agency to complete". >> >> >> >> So basically, I completely agree with the Forum back-pushers that >> >> IWIN_FLUSH etc. are as useful as ISEND. >> >> >> >> Jeff >> >> >> >> On Wed, Jun 12, 2013 at 4:58 PM, Pavan Balaji <balaji(a)mcs.anl.gov> >> >> wrote: >> >> > >> >> > The push-back at the Forum was that WIN_FLUSH and WIN_UNLOCK are >> >> > local >> >> > operations (in MPI's definition of local), so they are already >> >> > nonblocking. >> >> > I was not at the Forum. I pointed out to Wesley after he returned >> >> > that >> >> > they >> >> > are only local in the simple case where no one else is holding the >> >> > lock. >> >> > Otherwise they are not. >> >> > >> >> > Further, the proposal is not only about WIN_FLUSH and WIN_UNLOCK. It >> >> > was >> >> > for a long list of blocking functions: COMM_ACCEPT, COMM_CONNECT, >> >> > WIN_FENCE, >> >> > WIN_COMPLETE, WIN_WAIT, WIN_UNLOCK, WIN_FLUSH, WIN_FLUSH_LOCAL, >> >> > WIN_FLUSH_ALL, WIN_FLUSH_LOCAL_ALL, ... The point was that MPI >> >> > should >> >> > deal >> >> > with concepts, not specific function exceptions. Nonblocking >> >> > communication >> >> > is one such concept, and it should be valid for all operations (that >> >> > are >> >> > not >> >> > local). >> >> > >> >> > FWIW, my proposal was only to add nonblocking variants to blocking >> >> > calls. >> >> > Adding extra per-op remote completion requests is a separate issue >> >> > and >> >> > should not be mixed with this. I personally think it's useful (in >> >> > fact, >> >> > my >> >> > original proposal for RPUT/RGET equivalents had it), but that's still >> >> > a >> >> > separate issue. >> >> > >> >> > -- Pavan >> >> > >> >> > >> >> > On 06/12/2013 09:41 AM, Jeff Hammond wrote: >> >> >> >> >> >> So Wes was telling me about IWIN_FLUSH yesterday (this function is a >> >> >> great idea, btw) and the alternative of adding double-request RPUT >> >> >> and >> >> >> RACCUMULATE that provide a request for both local and remote >> >> >> completion. By induction, this would imply a triple-request >> >> >> RGET_ACCUMULATE operation, which I also think is a good idea. It's >> >> >> really silly to have to wait for the result buffer to be ready to >> >> >> reuse the origin buffer in RGET_ACCUMULATE. It seems that many >> >> >> implementations will be able to send the entire origin buffer a >> >> >> nontrivial amount of time before the result comes back, particularly >> >> >> if the remote side doesn't have hardware-based progress and the data >> >> >> has to wait in a network buffer. >> >> >> >> >> >> I'm sorry that I missed the last day of the Forum in San Jose. I >> >> >> didn't know these operations were on the table. They sound like a >> >> >> great idea to me. >> >> >> >> >> >> Jeff >> >> >> >> >> > >> >> > -- >> >> > Pavan Balaji >> >> > http://www.mcs.anl.gov/~balaji >> >> >> >> >> >> >> >> -- >> >> Jeff Hammond >> >> Argonne Leadership Computing Facility >> >> University of Chicago Computation Institute >> >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> >> http://www.linkedin.com/in/jeffhammond >> >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> >> ALCF docs: http://www.alcf.anl.gov/user-guides >> >> _______________________________________________ >> >> mpi3-rma mailing list >> >> mpi3-rma(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> > >> > >> > >> > _______________________________________________ >> > mpi3-rma mailing list >> > mpi3-rma(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> ALCF docs: http://www.alcf.anl.gov/user-guides >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [Mpi3-rma] request-based ops
by Pavan Balaji 14 Jun '13

14 Jun '13

Jim, You cannot do this in WAIT, as I might do a loop on TEST. In fact, that's the primary use case for nonblocking requests, IMO. You can stick a request into an array of other requests to check on any of them. Please see the slides that Wesley presented. We should not be reiterating what was already presented at the Forum on the email list. -- Pavan On 06/14/2013 12:23 PM, Jim Dinan wrote: > Jeff, > > How would iwin_flush be different in practice from doing nothing? I can > give you a trivial implementation that does nothing until you wait on > the request, at which point I call blocking win_flush. With my > implementor hat on, I would view this function as a performance hint > that indicates to the implementation that you will want to complete all > operations thus far and none after. Given a high quality implementation > that actually makes background progress on one-sided communications, > this should be unnecessary and I would probably implement it as above. > If you want to only complete specific operations, the request > generating functions are available. > > ~Jim. > > > > On Fri, Jun 14, 2013 at 11:09 AM, Jeff Hammond <jhammond(a)alcf.anl.gov > <mailto:[email protected]>> wrote: > > Yes, I was making a disingenuous argument to push back against the > fallacy that iwin_flush is not required. > > Jeff > > On Fri, Jun 14, 2013 at 10:25 AM, Jim Dinan <james.dinan(a)gmail.com > <mailto:[email protected]>> wrote: > > Hey now, blocking doesn't mean that MPI /must/ block, it only > means that it > > /can/ block. Blocking and synchronizing are certainly not the > same; Ssend > > synchronizes, which is a stronger semantic than blocking. > > > > ~Jim. > > > > > > On Wed, Jun 12, 2013 at 5:07 PM, Jeff Hammond > <jhammond(a)alcf.anl.gov <mailto:[email protected]>> wrote: > >> > >> Sure, let's separate these issues. I think Pirate RMA (RRPUT, > >> RRACCUMULATE and RRRGET_ACCUMULATE) is a more _local_ solution > in the > >> MPI standard sense :-) > >> > >> SEND is also a nonblocking operation because the implementation can > >> always buffer the data and return before the remote RECV is posted. > >> Only SSEND is blocking if one asserts that blocking is equivalent to > >> "requires remote agency to complete". > >> > >> So basically, I completely agree with the Forum back-pushers that > >> IWIN_FLUSH etc. are as useful as ISEND. > >> > >> Jeff > >> > >> On Wed, Jun 12, 2013 at 4:58 PM, Pavan Balaji > <balaji(a)mcs.anl.gov <mailto:[email protected]>> wrote: > >> > > >> > The push-back at the Forum was that WIN_FLUSH and WIN_UNLOCK > are local > >> > operations (in MPI's definition of local), so they are already > >> > nonblocking. > >> > I was not at the Forum. I pointed out to Wesley after he > returned that > >> > they > >> > are only local in the simple case where no one else is holding > the lock. > >> > Otherwise they are not. > >> > > >> > Further, the proposal is not only about WIN_FLUSH and > WIN_UNLOCK. It > >> > was > >> > for a long list of blocking functions: COMM_ACCEPT, COMM_CONNECT, > >> > WIN_FENCE, > >> > WIN_COMPLETE, WIN_WAIT, WIN_UNLOCK, WIN_FLUSH, WIN_FLUSH_LOCAL, > >> > WIN_FLUSH_ALL, WIN_FLUSH_LOCAL_ALL, ... The point was that > MPI should > >> > deal > >> > with concepts, not specific function exceptions. Nonblocking > >> > communication > >> > is one such concept, and it should be valid for all operations > (that are > >> > not > >> > local). > >> > > >> > FWIW, my proposal was only to add nonblocking variants to blocking > >> > calls. > >> > Adding extra per-op remote completion requests is a separate > issue and > >> > should not be mixed with this. I personally think it's useful > (in fact, > >> > my > >> > original proposal for RPUT/RGET equivalents had it), but > that's still a > >> > separate issue. > >> > > >> > -- Pavan > >> > > >> > > >> > On 06/12/2013 09:41 AM, Jeff Hammond wrote: > >> >> > >> >> So Wes was telling me about IWIN_FLUSH yesterday (this > function is a > >> >> great idea, btw) and the alternative of adding double-request > RPUT and > >> >> RACCUMULATE that provide a request for both local and remote > >> >> completion. By induction, this would imply a triple-request > >> >> RGET_ACCUMULATE operation, which I also think is a good idea. > It's > >> >> really silly to have to wait for the result buffer to be ready to > >> >> reuse the origin buffer in RGET_ACCUMULATE. It seems that many > >> >> implementations will be able to send the entire origin buffer a > >> >> nontrivial amount of time before the result comes back, > particularly > >> >> if the remote side doesn't have hardware-based progress and > the data > >> >> has to wait in a network buffer. > >> >> > >> >> I'm sorry that I missed the last day of the Forum in San Jose. I > >> >> didn't know these operations were on the table. They sound > like a > >> >> great idea to me. > >> >> > >> >> Jeff > >> >> > >> > > >> > -- > >> > Pavan Balaji > >> > http://www.mcs.anl.gov/~balaji > >> > >> > >> > >> -- > >> Jeff Hammond > >> Argonne Leadership Computing Facility > >> University of Chicago Computation Institute > >> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) > 252-5381 > >> http://www.linkedin.com/in/jeffhammond > >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > >> ALCF docs: http://www.alcf.anl.gov/user-guides > >> _______________________________________________ > >> mpi3-rma mailing list > >> mpi3-rma(a)lists.mpi-forum.org <mailto:[email protected]> > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org <mailto:[email protected]> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org <mailto:[email protected]> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] request-based ops
by Jim Dinan 14 Jun '13

14 Jun '13

Jeff, How would iwin_flush be different in practice from doing nothing? I can give you a trivial implementation that does nothing until you wait on the request, at which point I call blocking win_flush. With my implementor hat on, I would view this function as a performance hint that indicates to the implementation that you will want to complete all operations thus far and none after. Given a high quality implementation that actually makes background progress on one-sided communications, this should be unnecessary and I would probably implement it as above. If you want to only complete specific operations, the request generating functions are available. ~Jim. On Fri, Jun 14, 2013 at 11:09 AM, Jeff Hammond <jhammond(a)alcf.anl.gov>wrote: > Yes, I was making a disingenuous argument to push back against the > fallacy that iwin_flush is not required. > > Jeff > > On Fri, Jun 14, 2013 at 10:25 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Hey now, blocking doesn't mean that MPI /must/ block, it only means that > it > > /can/ block. Blocking and synchronizing are certainly not the same; > Ssend > > synchronizes, which is a stronger semantic than blocking. > > > > ~Jim. > > > > > > On Wed, Jun 12, 2013 at 5:07 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> > wrote: > >> > >> Sure, let's separate these issues. I think Pirate RMA (RRPUT, > >> RRACCUMULATE and RRRGET_ACCUMULATE) is a more _local_ solution in the > >> MPI standard sense :-) > >> > >> SEND is also a nonblocking operation because the implementation can > >> always buffer the data and return before the remote RECV is posted. > >> Only SSEND is blocking if one asserts that blocking is equivalent to > >> "requires remote agency to complete". > >> > >> So basically, I completely agree with the Forum back-pushers that > >> IWIN_FLUSH etc. are as useful as ISEND. > >> > >> Jeff > >> > >> On Wed, Jun 12, 2013 at 4:58 PM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > >> > > >> > The push-back at the Forum was that WIN_FLUSH and WIN_UNLOCK are local > >> > operations (in MPI's definition of local), so they are already > >> > nonblocking. > >> > I was not at the Forum. I pointed out to Wesley after he returned > that > >> > they > >> > are only local in the simple case where no one else is holding the > lock. > >> > Otherwise they are not. > >> > > >> > Further, the proposal is not only about WIN_FLUSH and WIN_UNLOCK. It > >> > was > >> > for a long list of blocking functions: COMM_ACCEPT, COMM_CONNECT, > >> > WIN_FENCE, > >> > WIN_COMPLETE, WIN_WAIT, WIN_UNLOCK, WIN_FLUSH, WIN_FLUSH_LOCAL, > >> > WIN_FLUSH_ALL, WIN_FLUSH_LOCAL_ALL, ... The point was that MPI should > >> > deal > >> > with concepts, not specific function exceptions. Nonblocking > >> > communication > >> > is one such concept, and it should be valid for all operations (that > are > >> > not > >> > local). > >> > > >> > FWIW, my proposal was only to add nonblocking variants to blocking > >> > calls. > >> > Adding extra per-op remote completion requests is a separate issue and > >> > should not be mixed with this. I personally think it's useful (in > fact, > >> > my > >> > original proposal for RPUT/RGET equivalents had it), but that's still > a > >> > separate issue. > >> > > >> > -- Pavan > >> > > >> > > >> > On 06/12/2013 09:41 AM, Jeff Hammond wrote: > >> >> > >> >> So Wes was telling me about IWIN_FLUSH yesterday (this function is a > >> >> great idea, btw) and the alternative of adding double-request RPUT > and > >> >> RACCUMULATE that provide a request for both local and remote > >> >> completion. By induction, this would imply a triple-request > >> >> RGET_ACCUMULATE operation, which I also think is a good idea. It's > >> >> really silly to have to wait for the result buffer to be ready to > >> >> reuse the origin buffer in RGET_ACCUMULATE. It seems that many > >> >> implementations will be able to send the entire origin buffer a > >> >> nontrivial amount of time before the result comes back, particularly > >> >> if the remote side doesn't have hardware-based progress and the data > >> >> has to wait in a network buffer. > >> >> > >> >> I'm sorry that I missed the last day of the Forum in San Jose. I > >> >> didn't know these operations were on the table. They sound like a > >> >> great idea to me. > >> >> > >> >> Jeff > >> >> > >> > > >> > -- > >> > Pavan Balaji > >> > http://www.mcs.anl.gov/~balaji > >> > >> > >> > >> -- > >> Jeff Hammond > >> Argonne Leadership Computing Facility > >> University of Chicago Computation Institute > >> jhammond(a)alcf.anl.gov / (630) 252-5381 > >> http://www.linkedin.com/in/jeffhammond > >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > >> ALCF docs: http://www.alcf.anl.gov/user-guides > >> _______________________________________________ > >> mpi3-rma mailing list > >> mpi3-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

Re: [Mpi3-rma] request-based ops
by Jeff Hammond 14 Jun '13

14 Jun '13

Yes, I was making a disingenuous argument to push back against the fallacy that iwin_flush is not required. Jeff On Fri, Jun 14, 2013 at 10:25 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Hey now, blocking doesn't mean that MPI /must/ block, it only means that it > /can/ block. Blocking and synchronizing are certainly not the same; Ssend > synchronizes, which is a stronger semantic than blocking. > > ~Jim. > > > On Wed, Jun 12, 2013 at 5:07 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> >> Sure, let's separate these issues. I think Pirate RMA (RRPUT, >> RRACCUMULATE and RRRGET_ACCUMULATE) is a more _local_ solution in the >> MPI standard sense :-) >> >> SEND is also a nonblocking operation because the implementation can >> always buffer the data and return before the remote RECV is posted. >> Only SSEND is blocking if one asserts that blocking is equivalent to >> "requires remote agency to complete". >> >> So basically, I completely agree with the Forum back-pushers that >> IWIN_FLUSH etc. are as useful as ISEND. >> >> Jeff >> >> On Wed, Jun 12, 2013 at 4:58 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> > >> > The push-back at the Forum was that WIN_FLUSH and WIN_UNLOCK are local >> > operations (in MPI's definition of local), so they are already >> > nonblocking. >> > I was not at the Forum. I pointed out to Wesley after he returned that >> > they >> > are only local in the simple case where no one else is holding the lock. >> > Otherwise they are not. >> > >> > Further, the proposal is not only about WIN_FLUSH and WIN_UNLOCK. It >> > was >> > for a long list of blocking functions: COMM_ACCEPT, COMM_CONNECT, >> > WIN_FENCE, >> > WIN_COMPLETE, WIN_WAIT, WIN_UNLOCK, WIN_FLUSH, WIN_FLUSH_LOCAL, >> > WIN_FLUSH_ALL, WIN_FLUSH_LOCAL_ALL, ... The point was that MPI should >> > deal >> > with concepts, not specific function exceptions. Nonblocking >> > communication >> > is one such concept, and it should be valid for all operations (that are >> > not >> > local). >> > >> > FWIW, my proposal was only to add nonblocking variants to blocking >> > calls. >> > Adding extra per-op remote completion requests is a separate issue and >> > should not be mixed with this. I personally think it's useful (in fact, >> > my >> > original proposal for RPUT/RGET equivalents had it), but that's still a >> > separate issue. >> > >> > -- Pavan >> > >> > >> > On 06/12/2013 09:41 AM, Jeff Hammond wrote: >> >> >> >> So Wes was telling me about IWIN_FLUSH yesterday (this function is a >> >> great idea, btw) and the alternative of adding double-request RPUT and >> >> RACCUMULATE that provide a request for both local and remote >> >> completion. By induction, this would imply a triple-request >> >> RGET_ACCUMULATE operation, which I also think is a good idea. It's >> >> really silly to have to wait for the result buffer to be ready to >> >> reuse the origin buffer in RGET_ACCUMULATE. It seems that many >> >> implementations will be able to send the entire origin buffer a >> >> nontrivial amount of time before the result comes back, particularly >> >> if the remote side doesn't have hardware-based progress and the data >> >> has to wait in a network buffer. >> >> >> >> I'm sorry that I missed the last day of the Forum in San Jose. I >> >> didn't know these operations were on the table. They sound like a >> >> great idea to me. >> >> >> >> Jeff >> >> >> > >> > -- >> > Pavan Balaji >> > http://www.mcs.anl.gov/~balaji >> >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> ALCF docs: http://www.alcf.anl.gov/user-guides >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [Mpi3-rma] request-based ops
by Jim Dinan 14 Jun '13

14 Jun '13

Hey now, blocking doesn't mean that MPI /must/ block, it only means that it /can/ block. Blocking and synchronizing are certainly not the same; Ssend synchronizes, which is a stronger semantic than blocking. ~Jim. On Wed, Jun 12, 2013 at 5:07 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > Sure, let's separate these issues. I think Pirate RMA (RRPUT, > RRACCUMULATE and RRRGET_ACCUMULATE) is a more _local_ solution in the > MPI standard sense :-) > > SEND is also a nonblocking operation because the implementation can > always buffer the data and return before the remote RECV is posted. > Only SSEND is blocking if one asserts that blocking is equivalent to > "requires remote agency to complete". > > So basically, I completely agree with the Forum back-pushers that > IWIN_FLUSH etc. are as useful as ISEND. > > Jeff > > On Wed, Jun 12, 2013 at 4:58 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > > The push-back at the Forum was that WIN_FLUSH and WIN_UNLOCK are local > > operations (in MPI's definition of local), so they are already > nonblocking. > > I was not at the Forum. I pointed out to Wesley after he returned that > they > > are only local in the simple case where no one else is holding the lock. > > Otherwise they are not. > > > > Further, the proposal is not only about WIN_FLUSH and WIN_UNLOCK. It was > > for a long list of blocking functions: COMM_ACCEPT, COMM_CONNECT, > WIN_FENCE, > > WIN_COMPLETE, WIN_WAIT, WIN_UNLOCK, WIN_FLUSH, WIN_FLUSH_LOCAL, > > WIN_FLUSH_ALL, WIN_FLUSH_LOCAL_ALL, ... The point was that MPI should > deal > > with concepts, not specific function exceptions. Nonblocking > communication > > is one such concept, and it should be valid for all operations (that are > not > > local). > > > > FWIW, my proposal was only to add nonblocking variants to blocking calls. > > Adding extra per-op remote completion requests is a separate issue and > > should not be mixed with this. I personally think it's useful (in fact, > my > > original proposal for RPUT/RGET equivalents had it), but that's still a > > separate issue. > > > > -- Pavan > > > > > > On 06/12/2013 09:41 AM, Jeff Hammond wrote: > >> > >> So Wes was telling me about IWIN_FLUSH yesterday (this function is a > >> great idea, btw) and the alternative of adding double-request RPUT and > >> RACCUMULATE that provide a request for both local and remote > >> completion. By induction, this would imply a triple-request > >> RGET_ACCUMULATE operation, which I also think is a good idea. It's > >> really silly to have to wait for the result buffer to be ready to > >> reuse the origin buffer in RGET_ACCUMULATE. It seems that many > >> implementations will be able to send the entire origin buffer a > >> nontrivial amount of time before the result comes back, particularly > >> if the remote side doesn't have hardware-based progress and the data > >> has to wait in a network buffer. > >> > >> I'm sorry that I missed the last day of the Forum in San Jose. I > >> didn't know these operations were on the table. They sound like a > >> great idea to me. > >> > >> Jeff > >> > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

Re: [Mpi3-rma] RMA Example 11.10
by Pavan Balaji 13 Jun '13

13 Jun '13

On 06/12/2013 08:36 AM, Marc-Andre Hermanns wrote: >> Why will this deadlock? MPI_WIN_LOCK cannot block for uncontended locks. > > Yes. If the example is just a two-process run, then the lock will not be > contended, and everything will be fine. > > But what if a reader sees this code snippet not as a two-process example > but as a two-process snapshot in a multi-process example, where the > other processes are just not shown. > > And if there are other processes involved, there could be contention on > the lock, and there is not way for process B to know this in advance. > > The whole thing feels a little bit like dining philosophers, where it is > not such a good idea to block a resource while you are waiting for another. > >>> Consider a process C that has the same calling sequence as process B. If >>> C get's the lock granted in MPI_Lock and process B's call to MPI_Lock >>> blocks to wait for the lock to be granted, C will never release the >>> lock, as it waits for process B to join the barrier. B, however, waits >>> for the lock and will never enter the barrier. >> >> They are all using SHARED lock. > > Oops. I missed that. Sorry for that. > > Then let's change the process C to have an exclusive lock. ;-) If you add additional processes doing other things, then yes, it might deadlock. But that's not the view in most examples. You should view this as a 2-process test, for illustration purposes. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] request-based ops
by Jeff Hammond 12 Jun '13

12 Jun '13

Sure, let's separate these issues. I think Pirate RMA (RRPUT, RRACCUMULATE and RRRGET_ACCUMULATE) is a more _local_ solution in the MPI standard sense :-) SEND is also a nonblocking operation because the implementation can always buffer the data and return before the remote RECV is posted. Only SSEND is blocking if one asserts that blocking is equivalent to "requires remote agency to complete". So basically, I completely agree with the Forum back-pushers that IWIN_FLUSH etc. are as useful as ISEND. Jeff On Wed, Jun 12, 2013 at 4:58 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > The push-back at the Forum was that WIN_FLUSH and WIN_UNLOCK are local > operations (in MPI's definition of local), so they are already nonblocking. > I was not at the Forum. I pointed out to Wesley after he returned that they > are only local in the simple case where no one else is holding the lock. > Otherwise they are not. > > Further, the proposal is not only about WIN_FLUSH and WIN_UNLOCK. It was > for a long list of blocking functions: COMM_ACCEPT, COMM_CONNECT, WIN_FENCE, > WIN_COMPLETE, WIN_WAIT, WIN_UNLOCK, WIN_FLUSH, WIN_FLUSH_LOCAL, > WIN_FLUSH_ALL, WIN_FLUSH_LOCAL_ALL, ... The point was that MPI should deal > with concepts, not specific function exceptions. Nonblocking communication > is one such concept, and it should be valid for all operations (that are not > local). > > FWIW, my proposal was only to add nonblocking variants to blocking calls. > Adding extra per-op remote completion requests is a separate issue and > should not be mixed with this. I personally think it's useful (in fact, my > original proposal for RPUT/RGET equivalents had it), but that's still a > separate issue. > > -- Pavan > > > On 06/12/2013 09:41 AM, Jeff Hammond wrote: >> >> So Wes was telling me about IWIN_FLUSH yesterday (this function is a >> great idea, btw) and the alternative of adding double-request RPUT and >> RACCUMULATE that provide a request for both local and remote >> completion. By induction, this would imply a triple-request >> RGET_ACCUMULATE operation, which I also think is a good idea. It's >> really silly to have to wait for the result buffer to be ready to >> reuse the origin buffer in RGET_ACCUMULATE. It seems that many >> implementations will be able to send the entire origin buffer a >> nontrivial amount of time before the result comes back, particularly >> if the remote side doesn't have hardware-based progress and the data >> has to wait in a network buffer. >> >> I'm sorry that I missed the last day of the Forum in San Jose. I >> didn't know these operations were on the table. They sound like a >> great idea to me. >> >> Jeff >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0