June 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Jim Dinan 17 Jun '13

17 Jun '13

Sorry, I should have been more specific. An implementation of iflush that waits for the completion of all messages should be valid. Such an implementation would compare counters and return true if they are the same. This implementation could have the issue I mentioned in the previous message, where the user continuously issuing operations can prevent iflush from completing. Jim. On Jun 16, 2013 10:13 AM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > > On 06/16/2013 10:02 AM, Jim Dinan wrote: > >> If the channel is unordered, a message after the iflush can increment >> the counter, while one before the iflush has not yet completed. So, the >> counter is not enough to mark a particular point in time. >> > > Ah, good point. > > An implementation of iflush as flush should still be valid, right? Just >> > > No. You cannot do this if the user only uses TEST. > > MPI_WIN_IFLUSH(&req); > while (MPI_TEST(req) is not done); > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji >

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Pavan Balaji 16 Jun '13

16 Jun '13

On 06/16/2013 10:02 AM, Jim Dinan wrote: > If the channel is unordered, a message after the iflush can increment > the counter, while one before the iflush has not yet completed. So, the > counter is not enough to mark a particular point in time. Ah, good point. > An implementation of iflush as flush should still be valid, right? Just No. You cannot do this if the user only uses TEST. MPI_WIN_IFLUSH(&req); while (MPI_TEST(req) is not done); -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Jim Dinan 16 Jun '13

16 Jun '13

If the channel is unordered, a message after the iflush can increment the counter, while one before the iflush has not yet completed. So, the counter is not enough to mark a particular point in time. An implementation of iflush as flush should still be valid, right? Just check if the counters are equal in test/wait. It won't be as fine grain as what the user wants, but it should also have no impact on programs that don't use iflush. Although, the following program may never complete, so maybe the iflush as flush implementation is not valid. Iflush(...) Do Test(req, &flag) If (!flag) Get(...) While(!flag) Jim. On Jun 15, 2013 8:42 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: > Hi Brian > > On 06/15/2013 11:24 AM, Barrett, Brian W wrote: > >> With the current semantics of flush, a valid implementation is a counter >> of number of operations started, another of operations completed, wait >> until they're equal. On an unordered channel (and acks are usually on >> an unordered channel), this mode won't work and you either have to have >> multiple counters or start tracking events in some per-event way, >> neither of which are going to be as performant. >> > > I think this model with a global counter will still work. The IFLUSH > request object will need to know what the counter needs to be to mark this > request as complete. So when there's a test/wait on this request, you > compare this with the global counter. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > ______________________________**_________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/**mailman/listinfo.cgi/mpi3-rma<http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma> >

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Pavan Balaji 15 Jun '13

15 Jun '13

On 06/15/2013 12:42 PM, Jeff Hammond wrote: > Related to this, I was wondering about the semantics of IWIN_FLUSH > when ordering was disabled for a window (i.e. accumulate_ordering=""). > > If I do the following, can I make any assumptions about the ordering > of the two accumulates? > > MPI_Win win_with_unordered_accumulates; > MPI_Request req1, req2; > MPI_Accumulate(target=1,win=win_with_unordered_accumulates); > MPI_Iwin_flush(target=1,win=win_with_unordered_accumulates,request=req1); > MPI_Accumulate(win=win_with_unordered_accumulates); > MPI_Iwin_flush(target=1,win=win_with_unordered_accumulates,request=req2); > MPI_Waitall(requests={req1,req2}); > > The Waitall is not required to act on the requests in order, right? > Thus is req2 is completed first, then it flushes both Accumulates > simultaneously, hence does not ordering them. The completion of req1 > is then trivial and have no effect on ordering Accumulate 1 before > Accumulate 2. Correct. No additional ordering constraints of completion will be added to WAITALL. Like with any requests, they can complete in whatever order. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Pavan Balaji 15 Jun '13

15 Jun '13

Hi Brian On 06/15/2013 11:24 AM, Barrett, Brian W wrote: > With the current semantics of flush, a valid implementation is a counter > of number of operations started, another of operations completed, wait > until they're equal. On an unordered channel (and acks are usually on > an unordered channel), this mode won't work and you either have to have > multiple counters or start tracking events in some per-event way, > neither of which are going to be as performant. I think this model with a global counter will still work. The IFLUSH request object will need to know what the counter needs to be to mark this request as complete. So when there's a test/wait on this request, you compare this with the global counter. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Jeff Hammond 15 Jun '13

15 Jun '13

Related to this, I was wondering about the semantics of IWIN_FLUSH when ordering was disabled for a window (i.e. accumulate_ordering=""). If I do the following, can I make any assumptions about the ordering of the two accumulates? MPI_Win win_with_unordered_accumulates; MPI_Request req1, req2; MPI_Accumulate(target=1,win=win_with_unordered_accumulates); MPI_Iwin_flush(target=1,win=win_with_unordered_accumulates,request=req1); MPI_Accumulate(win=win_with_unordered_accumulates); MPI_Iwin_flush(target=1,win=win_with_unordered_accumulates,request=req2); MPI_Waitall(requests={req1,req2}); The Waitall is not required to act on the requests in order, right? Thus is req2 is completed first, then it flushes both Accumulates simultaneously, hence does not ordering them. The completion of req1 is then trivial and have no effect on ordering Accumulate 1 before Accumulate 2. Thanks, Jeff On Sat, Jun 15, 2013 at 11:24 AM, Barrett, Brian W <bwbarre(a)sandia.gov> wrote: > On 6/15/13 10:47 AM, "Jim Dinan" <james.dinan(a)gmail.com> wrote: > > Can you elaborate on the source of performance loss from operations between > Iflush and its completion? > > > With the current semantics of flush, a valid implementation is a counter of > number of operations started, another of operations completed, wait until > they're equal. On an unordered channel (and acks are usually on an > unordered channel), this mode won't work and you either have to have > multiple counters or start tracking events in some per-event way, neither of > which are going to be as performant. > > Brian > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Barrett, Brian W 15 Jun '13

15 Jun '13

On 6/15/13 10:47 AM, "Jim Dinan" <james.dinan(a)gmail.com<mailto:[email protected]>> wrote: Can you elaborate on the source of performance loss from operations between Iflush and its completion? With the current semantics of flush, a valid implementation is a counter of number of operations started, another of operations completed, wait until they're equal. On an unordered channel (and acks are usually on an unordered channel), this mode won't work and you either have to have multiple counters or start tracking events in some per-event way, neither of which are going to be as performant. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Jim Dinan 15 Jun '13

15 Jun '13

Brian, Can you elaborate on the source of performance loss from operations between Iflush and its completion? Jeff, I think the pirate RMA (separate per operation local/remote completion) idea is good. I'm not sure I'm sold on the interface; I'd rather not add so many new functions, if there's some other way we could do it. I think we should explore the different design options so that we can verify for the Forum that we've chosen the best one. All, Shall we schedule an RMA WG meeting some time soon? Seems like it would be good to get high bandwidth to go over these changes. I also should give the WG and update on the MPI_Aint_add session from the last Forum meeting. ~Jim. On Fri, Jun 14, 2013 at 6:06 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > That's a fair point and I agree with it. We should not add overhead to > existing functionality to support this. I guess the thought here is that > for implementations that support fully asynchronous RMA communication, test > or wait can essentially do the flush, with the caveat that test cannot > block if someone else is holding the lock. > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > "Barrett, Brian W" <bwbarre(a)sandia.gov> wrote: > > As an FYI, my problem with this proposal is two-fold: > > First, we need to deal with the fact that many of the calls are local > today, meaning we need to fix some definitions, which is a big change > (iflush/flush shouldn't have the same semantics). > > Second, the semantics of iflush (and friends) need lots of care. For > example, allowing a communication event between iflush and wait would > greatly hinder my implementation's performance when not using iflush. > > I understand the desire and like many of the non-blocking, but there are > many places that we should balance non-blocking variants vs. performance > loss... > > Brian > > > > Sent with Good (www.good.com) > > > -----Original Message----- > From: Pavan Balaji [balaji(a)mcs.anl.gov<mailto:[email protected]>] > Sent: Friday, June 14, 2013 02:51 PM Mountain Standard Time > To: mpi3-rma(a)lists.mpi-forum.org > Subject: [EXTERNAL] Re: [Mpi3-rma] request-based ops > > > > On 06/14/2013 03:43 PM, Jim Dinan wrote: > > Ahh, right, I had to miss that session at the Forum meeting. If anyone > > else is looking for them, these are the slides: > > > > > http://meetings.mpi-forum.org/secretary/2013/06/slides/2013-06-06-Nonblocki… > > > > I was indeed forgetting about remote versus local completion and polling > > on the request. It sounds like this would definitely fill a gap in the > > RMA interface. > > FYI, this was not meant to be an RMA proposal, since it touches blocking > function calls in many chapters (e.g., COMM_ICONNECT and COMM_IACCEPT). > > I think I/O operations have a similar ticket as well. > > All these tickets need to be combined into a single nonblocking > operations ticket. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Pavan Balaji 14 Jun '13

14 Jun '13

That's a fair point and I agree with it. We should not add overhead to existing functionality to support this. I guess the thought here is that for implementations that support fully asynchronous RMA communication, test or wait can essentially do the flush, with the caveat that test cannot block if someone else is holding the lock. -- Pavan Balaji http://www.mcs.anl.gov/~balaji "Barrett, Brian W" <bwbarre(a)sandia.gov> wrote: As an FYI, my problem with this proposal is two-fold: First, we need to deal with the fact that many of the calls are local today, meaning we need to fix some definitions, which is a big change (iflush/flush shouldn't have the same semantics). Second, the semantics of iflush (and friends) need lots of care. For example, allowing a communication event between iflush and wait would greatly hinder my implementation's performance when not using iflush. I understand the desire and like many of the non-blocking, but there are many places that we should balance non-blocking variants vs. performance loss... Brian Sent with Good (www.good.com) -----Original Message----- From: Pavan Balaji [balaji(a)mcs.anl.gov<mailto:[email protected]>] Sent: Friday, June 14, 2013 02:51 PM Mountain Standard Time To: mpi3-rma(a)lists.mpi-forum.org Subject: [EXTERNAL] Re: [Mpi3-rma] request-based ops On 06/14/2013 03:43 PM, Jim Dinan wrote: > Ahh, right, I had to miss that session at the Forum meeting. If anyone > else is looking for them, these are the slides: > > http://meetings.mpi-forum.org/secretary/2013/06/slides/2013-06-06-Nonblocki… > > I was indeed forgetting about remote versus local completion and polling > on the request. It sounds like this would definitely fill a gap in the > RMA interface. FYI, this was not meant to be an RMA proposal, since it touches blocking function calls in many chapters (e.g., COMM_ICONNECT and COMM_IACCEPT). I think I/O operations have a similar ticket as well. All these tickets need to be combined into a single nonblocking operations ticket. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: request-based ops
by Jeff Hammond 14 Jun '13

14 Jun '13

Do you object to Pirate RMA (requests for each completion, not just local)? Jeff On Fri, Jun 14, 2013 at 4:00 PM, Barrett, Brian W <bwbarre(a)sandia.gov> wrote: > As an FYI, my problem with this proposal is two-fold: > > First, we need to deal with the fact that many of the calls are local today, > meaning we need to fix some definitions, which is a big change (iflush/flush > shouldn't have the same semantics). > > Second, the semantics of iflush (and friends) need lots of care. For > example, allowing a communication event between iflush and wait would > greatly hinder my implementation's performance when not using iflush. > > I understand the desire and like many of the non-blocking, but there are > many places that we should balance non-blocking variants vs. performance > loss... > > Brian > > > > Sent with Good (www.good.com) > > > > -----Original Message----- > From: Pavan Balaji [balaji(a)mcs.anl.gov] > Sent: Friday, June 14, 2013 02:51 PM Mountain Standard Time > To: mpi3-rma(a)lists.mpi-forum.org > Subject: [EXTERNAL] Re: [Mpi3-rma] request-based ops > > > On 06/14/2013 03:43 PM, Jim Dinan wrote: >> Ahh, right, I had to miss that session at the Forum meeting. If anyone >> else is looking for them, these are the slides: >> >> >> http://meetings.mpi-forum.org/secretary/2013/06/slides/2013-06-06-Nonblocki… >> >> I was indeed forgetting about remote versus local completion and polling >> on the request. It sounds like this would definitely fill a gap in the >> RMA interface. > > FYI, this was not meant to be an RMA proposal, since it touches blocking > function calls in many chapters (e.g., COMM_ICONNECT and COMM_IACCEPT). > > I think I/O operations have a similar ticket as well. > > All these tickets need to be combined into a single nonblocking > operations ticket. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0