April 2011 - mpiwg-rma - mailman3.cels.anl.gov

[Mpi3-rma] Conflicting PUT/GET accesses clarification
by Pavan Balaji 27 Apr '11

27 Apr '11

Page 339 lines 21-24 of MPI-2.2 states the following: "It is erroneous to have concurrent confl icting accesses to the same memory location in a window; if a location is updated by a put or accumulate operation, then this location cannot be accessed by a load or another RMA operation until the updating operation has completed at the target." What does this mean when I do PUT with a datatype with overlapping buffers? Is that valid or erroneous? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Request-Based RMA Example
by William Gropp 21 Apr '11

21 Apr '11

I've added this and posted an update to the wiki. There are a few bugs in this; please check the wiki version. Bill On Apr 19, 2011, at 4:31 PM, James Dinan wrote: > Hi All, > > Example is below, please give any feedback. I'll work on > compressing this down the match the format in the writeup. > > Caption: The following example shows how request-based operations > can be used to overlap communication with computation. Each process > fetches, processes, and writes the result for $NSTEPS$ chunks of > data. Instead of a single buffer, $M$ local buffers are used to > allow up to $M$ communication operations to overlap with computation. > > ~Jim. > > ---snip--- > > int i, j; > MPI_Window win; > MPI_Request put_req[M] = { MPI_REQUEST_NULL }; > MPI_Request get_req; > double data[M][N]; > > /* Create win: size M*N*sizeof(double), displacement unit MPI_DOUBLE > */ > > MPI_Win_lock_all(0, win); > > for (i = 0; i < NSTEPS; i++) { > MPI_Waitany(M, put_req, &j, MPI_STATUS_IGNORE); > MPI_RGet(data[j], N, MPI_DOUBLE, target, i*N, N, MPI_DOUBLE, win, > &get_req); > MPI_Wait(get_req); > compute(i, data[M], ...); > MPI_RPut(data[j], N, MPI_DOUBLE, target, i*N, N, MPI_DOUBLE, win, > &R[j]); > } > > MPI_Waitall(M, put_req, MPI_STATUSES_IGNORE); > MPI_Win_unlock_all(win); > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

[Mpi3-rma] Request-Based RMA Example
by James Dinan 19 Apr '11

19 Apr '11

Hi All, Example is below, please give any feedback. I'll work on compressing this down the match the format in the writeup. Caption: The following example shows how request-based operations can be used to overlap communication with computation. Each process fetches, processes, and writes the result for $NSTEPS$ chunks of data. Instead of a single buffer, $M$ local buffers are used to allow up to $M$ communication operations to overlap with computation. ~Jim. ---snip--- int i, j; MPI_Window win; MPI_Request put_req[M] = { MPI_REQUEST_NULL }; MPI_Request get_req; double data[M][N]; /* Create win: size M*N*sizeof(double), displacement unit MPI_DOUBLE */ MPI_Win_lock_all(0, win); for (i = 0; i < NSTEPS; i++) { MPI_Waitany(M, put_req, &j, MPI_STATUS_IGNORE); MPI_RGet(data[j], N, MPI_DOUBLE, target, i*N, N, MPI_DOUBLE, win, &get_req); MPI_Wait(get_req); compute(i, data[M], ...); MPI_RPut(data[j], N, MPI_DOUBLE, target, i*N, N, MPI_DOUBLE, win, &R[j]); } MPI_Waitall(M, put_req, MPI_STATUSES_IGNORE); MPI_Win_unlock_all(win);

1 0

Re: [Mpi3-rma] New version of RMA chapter posted
by Barrett, Brian W 18 Apr '11

18 Apr '11

On 4/16/11 12:50 PM, "William Gropp" <wgropp(a)illinois.edu> wrote: >I found a few changes from MPI 2.2 that I missed the first time. I've >also change the handling of the few ticket o items to use consistent >colors (green is delete, red is inserted). A few of the items from >Jim's notes were updated; someone needs to look at the others and at >least propose specific updates. All - I have Jeff's changes and some of my own that we need to get inserted. Since I don't have access to the tex file, do you want me to send the long list? I'd really rather just modify the tex directly, as it will take less time, even if someone else wants to continue managing commits to the svn. As for the examples, it's pretty obvious that after the store in Example 11.7, we'd need an MPI_FOO() call to act as a memory barrier. The other way to handle the issue in the current definition would be to be in a lock shared epoch and call win_sync, in which case the example is valid for both the unified and separate models. Less clear is what to do with example 11.9. Similar to the store in Example 11.7, some synchronization is necessary for the load in Process B. One could insert a MPI_FOO call before the load and give FOO both load and store synchronization, of course. One could also be in a lock shared epoch and call win sync before the load, in which case the example is valid for both the unified and separate models. I'm actually leaning towards the second option; it doesn't require us delving into language dependent semantics, it synchronizes the local and remote views of the window, which is a semantic that we know how to provide and is fairly language independent. Brian -- Brian W. Barrett Dept. 1423: Scalable System Software Sandia National Laboratories

1 0

[Mpi3-rma] New version of RMA chapter posted
by William Gropp 16 Apr '11

16 Apr '11

I found a few changes from MPI 2.2 that I missed the first time. I've also change the handling of the few ticket o items to use consistent colors (green is delete, red is inserted). A few of the items from Jim's notes were updated; someone needs to look at the others and at least propose specific updates. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

[Mpi3-rma] Updated document with changes from MPI-2.2 marked.
by William Gropp 15 Apr '11

15 Apr '11

I have edited the MPI 3 RMA chapter to more carefully mark changes from the MPI-2.2 chapter. This version (color with old and new text) is on the wiki. Please review it and let me know if you find any mistakes WRT marking the changed text. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

Re: [Mpi3-rma] GetAccumulate restriction needed?
by Rajeev Thakur 15 Apr '11

15 Apr '11

You are right. Only origin_addr and result_addr need be distinct in compare_and_swap. Rajeev On Apr 15, 2011, at 10:33 AM, Underwood, Keith D wrote: > Since the send buffers cannot be touched by the implementation and users can now pass the same send buffer to multiple things, I don't see any reason why the orign_addr and compare_addr would need to be disjoint. > > Keith > >> -----Original Message----- >> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- >> bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur >> Sent: Friday, April 15, 2011 9:30 AM >> To: MPI 3.0 Remote Memory Access working group >> Subject: Re: [Mpi3-rma] GetAccumulate restriction needed? >> >> I assume the disjoint buffers wording needs to be added to >> >> Get_accumulate >> Fetch_and_op >> Compare_and_swap >> Rget_accumulate >> >> For compare_and_swap, do we say all 3 buffers must be disjoint >> (origin_addr, compare_addr, result_addr)? >> >> Rajeev >> >> >> On Apr 3, 2011, at 9:56 PM, Underwood, Keith D wrote: >> >>> I suppose I could live with MPI_Get_accumulate_replace, if we had a >> use case... >>> >>> BTW, Pavan, for "use case", our typical goal has been a use case in >> some program using any API/language that could not be done in a >> reasonable, high performance way using exiting semantics. We've had >> that for most of our semantics so far... >>> >>> Keith >>> >>>> -----Original Message----- >>>> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- >>>> bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur >>>> Sent: Sunday, April 03, 2011 8:51 PM >>>> To: MPI 3.0 Remote Memory Access working group >>>> Subject: Re: [Mpi3-rma] GetAccumulate restriction needed? >>>> >>>> There is also MPI_Sendrecv_replace :-) >>>> >>>> >>>> On Apr 3, 2011, at 9:32 PM, Underwood, Keith D wrote: >>>> >>>>> >>>>>> On 04/03/2011 06:14 PM, Underwood, Keith D wrote: >>>>>>> Could you perhaps provide a use case? Otherwise, it looks like >> an >>>>>>> implementation nightmare with potentially unexpected performance >>>>>>> characteristics associated with any working implementation >> choice. >>>>>> >>>>>> This would be required if the application does not want to create >>>> two >>>>>> buffers, one for the input and one for the output. The >>>> implementation >>>>>> can internally use pipelining to use a smaller buffer space. >>>>> >>>>> That is not a use case :-) My understanding of fetch-and-add style >>>> operations is that you want both your increment and the old value at >>>> the target. So, I am back to: what is the use case? >>>>> >>>>>> Also, we need this for symmetry. Two-sided calls support it. >>>>> >>>>> MPI_Sendrecv requires disjoint buffers, and that is the closest >> thing >>>> in two-sided. >>>>> >>>>>> I don't follow the implementation nightmare with this? Why is this >> a >>>>>> problem? >>>>> >>>>> Would you think the user would expect that having the same source >> and >>>> target buffer would cause the implementation to: >>>>> >>>>> -allocate a buffer >>>>> -perform a get into that buffer >>>>> -wait for get completion >>>>> -perform an atomic >>>>> -wait for atomic completion >>>>> -copy the data from the internal buffer to the user buffer >>>>> >>>>> Now, the part that sucks is that I believe the IB implementation >>>> looks like this, but I believe that even a network that provided >> native >>>> fetch-and-add semantics will wind up looking like this if it >> provides >>>> an end-to-end reliability protocol. Heck, even if it doesn't, you >> may >>>> do it rather than guesstimating the amount of network buffer between >>>> the source and destination. That means that having the same source >> and >>>> destination buffer will have radically different performance than >>>> having different buffers. So, I count that as an implementation >> that >>>> sucks ;-) >>>>> >>>>> Keith >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-rma mailing list >>>>> mpi3-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >>>> >>>> >>>> _______________________________________________ >>>> mpi3-rma mailing list >>>> mpi3-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] GetAccumulate restriction needed?
by Underwood, Keith D 15 Apr '11

15 Apr '11

Since the send buffers cannot be touched by the implementation and users can now pass the same send buffer to multiple things, I don't see any reason why the orign_addr and compare_addr would need to be disjoint. Keith > -----Original Message----- > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur > Sent: Friday, April 15, 2011 9:30 AM > To: MPI 3.0 Remote Memory Access working group > Subject: Re: [Mpi3-rma] GetAccumulate restriction needed? > > I assume the disjoint buffers wording needs to be added to > > Get_accumulate > Fetch_and_op > Compare_and_swap > Rget_accumulate > > For compare_and_swap, do we say all 3 buffers must be disjoint > (origin_addr, compare_addr, result_addr)? > > Rajeev > > > On Apr 3, 2011, at 9:56 PM, Underwood, Keith D wrote: > > > I suppose I could live with MPI_Get_accumulate_replace, if we had a > use case... > > > > BTW, Pavan, for "use case", our typical goal has been a use case in > some program using any API/language that could not be done in a > reasonable, high performance way using exiting semantics. We've had > that for most of our semantics so far... > > > > Keith > > > >> -----Original Message----- > >> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > >> bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur > >> Sent: Sunday, April 03, 2011 8:51 PM > >> To: MPI 3.0 Remote Memory Access working group > >> Subject: Re: [Mpi3-rma] GetAccumulate restriction needed? > >> > >> There is also MPI_Sendrecv_replace :-) > >> > >> > >> On Apr 3, 2011, at 9:32 PM, Underwood, Keith D wrote: > >> > >>> > >>>> On 04/03/2011 06:14 PM, Underwood, Keith D wrote: > >>>>> Could you perhaps provide a use case? Otherwise, it looks like > an > >>>>> implementation nightmare with potentially unexpected performance > >>>>> characteristics associated with any working implementation > choice. > >>>> > >>>> This would be required if the application does not want to create > >> two > >>>> buffers, one for the input and one for the output. The > >> implementation > >>>> can internally use pipelining to use a smaller buffer space. > >>> > >>> That is not a use case :-) My understanding of fetch-and-add style > >> operations is that you want both your increment and the old value at > >> the target. So, I am back to: what is the use case? > >>> > >>>> Also, we need this for symmetry. Two-sided calls support it. > >>> > >>> MPI_Sendrecv requires disjoint buffers, and that is the closest > thing > >> in two-sided. > >>> > >>>> I don't follow the implementation nightmare with this? Why is this > a > >>>> problem? > >>> > >>> Would you think the user would expect that having the same source > and > >> target buffer would cause the implementation to: > >>> > >>> -allocate a buffer > >>> -perform a get into that buffer > >>> -wait for get completion > >>> -perform an atomic > >>> -wait for atomic completion > >>> -copy the data from the internal buffer to the user buffer > >>> > >>> Now, the part that sucks is that I believe the IB implementation > >> looks like this, but I believe that even a network that provided > native > >> fetch-and-add semantics will wind up looking like this if it > provides > >> an end-to-end reliability protocol. Heck, even if it doesn't, you > may > >> do it rather than guesstimating the amount of network buffer between > >> the source and destination. That means that having the same source > and > >> destination buffer will have radically different performance than > >> having different buffers. So, I count that as an implementation > that > >> sucks ;-) > >>> > >>> Keith > >>> > >>> > >>> _______________________________________________ > >>> mpi3-rma mailing list > >>> mpi3-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >> > >> > >> _______________________________________________ > >> mpi3-rma mailing list > >> mpi3-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] GetAccumulate restriction needed?
by Rajeev Thakur 15 Apr '11

15 Apr '11

I assume the disjoint buffers wording needs to be added to Get_accumulate Fetch_and_op Compare_and_swap Rget_accumulate For compare_and_swap, do we say all 3 buffers must be disjoint (origin_addr, compare_addr, result_addr)? Rajeev On Apr 3, 2011, at 9:56 PM, Underwood, Keith D wrote: > I suppose I could live with MPI_Get_accumulate_replace, if we had a use case... > > BTW, Pavan, for "use case", our typical goal has been a use case in some program using any API/language that could not be done in a reasonable, high performance way using exiting semantics. We've had that for most of our semantics so far... > > Keith > >> -----Original Message----- >> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- >> bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur >> Sent: Sunday, April 03, 2011 8:51 PM >> To: MPI 3.0 Remote Memory Access working group >> Subject: Re: [Mpi3-rma] GetAccumulate restriction needed? >> >> There is also MPI_Sendrecv_replace :-) >> >> >> On Apr 3, 2011, at 9:32 PM, Underwood, Keith D wrote: >> >>> >>>> On 04/03/2011 06:14 PM, Underwood, Keith D wrote: >>>>> Could you perhaps provide a use case? Otherwise, it looks like an >>>>> implementation nightmare with potentially unexpected performance >>>>> characteristics associated with any working implementation choice. >>>> >>>> This would be required if the application does not want to create >> two >>>> buffers, one for the input and one for the output. The >> implementation >>>> can internally use pipelining to use a smaller buffer space. >>> >>> That is not a use case :-) My understanding of fetch-and-add style >> operations is that you want both your increment and the old value at >> the target. So, I am back to: what is the use case? >>> >>>> Also, we need this for symmetry. Two-sided calls support it. >>> >>> MPI_Sendrecv requires disjoint buffers, and that is the closest thing >> in two-sided. >>> >>>> I don't follow the implementation nightmare with this? Why is this a >>>> problem? >>> >>> Would you think the user would expect that having the same source and >> target buffer would cause the implementation to: >>> >>> -allocate a buffer >>> -perform a get into that buffer >>> -wait for get completion >>> -perform an atomic >>> -wait for atomic completion >>> -copy the data from the internal buffer to the user buffer >>> >>> Now, the part that sucks is that I believe the IB implementation >> looks like this, but I believe that even a network that provided native >> fetch-and-add semantics will wind up looking like this if it provides >> an end-to-end reliability protocol. Heck, even if it doesn't, you may >> do it rather than guesstimating the amount of network buffer between >> the source and destination. That means that having the same source and >> destination buffer will have radically different performance than >> having different buffers. So, I count that as an implementation that >> sucks ;-) >>> >>> Keith >>> >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

[Mpi3-rma] RMA Unified examples
by Pavan Balaji 12 Apr '11

12 Apr '11

I was asked to put together a list of examples that we expect to work in the unified model. I've included three examples here. Example 1: ========= Process 0 --------- Win_lock(SHARED, P0) load A store B Win_unlock() Process 1 --------- Win_lock(SHARED, P0) Put(X) Get(Y) Win_unlock() Example 2: ========= Process 0 --------- store X MPI_FOO(); /* memory barrier */ MPI_Barrier(); Process 1 --------- MPI_Barrier(); Win_lock(SHARED, P0) Get(X) Win_unlock() Example 3: ========= Process 0 --------- Win_lock(SHARED, P0) store X MPI_FOO(); /* memory barrier */ MPI_Barrier(); Win_unlock() Process 1 --------- Win_lock(SHARED, P0) MPI_Barrier(); Get(X); Win_unlock() -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0