March 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses
by Rolf Rabenseifner 22 Mar '14

22 Mar '14

Jim, thank you for your analysis. Answers see below. ----- Original Message ----- > From: "Jim Dinan" <james.dinan(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Friday, March 21, 2014 8:14:22 PM > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses > > > > Rolf, > > > This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + > MPI_MODE_NOPRECEDE, win_ rcv_buf _left ); > > > You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | > MPI_MODE_NOPRECEDE). Yes, you are right, but this cannot be the reasons for the problems according to MPI-3.0 p451:28-32. The Fortran versions show the same problems. > In halo_1sided_store_win_alloc_shared.c, you are doing stores within > the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion on > the closing fence. Yes, but I do remote stores and no local stores. But you may be right, because the text p452:9-10 does not require that the stores are local stores. When I remember correctly, then removing this hint does not produce correct results. I'll re-check. > > Following the Fence epoch, you are reading from the left/right recv > buffers. That also needs to be done within an RMA epoch, if you are > reading non-local data. This reading is only on the local recv_buf_left/right. But it is included in the RMA epoch starting with the second fence and ending with the first fence in the next loop Iteration. > > ~Jim. Best regards Rolf > > > > On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner < > rabenseifner(a)hlrs.de > wrote: > > > Dear member of the RMA group and especially the mpich developers, > > I have real problems with the new shared memory in MPI-3.0, > i.e., the load/stores together with the RMA synchronization > causes wrong execution results. > > The attached > 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip > contains > > - 1sided/halo_1sided_put_win_alloc.c > > The basis that works. It uses MPI_Put and MPI_Win_fence for > duplex left/right halo communication. > > - 1sided/halo_1sided_store_win_alloc_shared.c > > This is the same, but a shared memory window is used and > the MPU_Put is substituted by storing the data in the > neighbors window. Same MPI_Win_fence with same assertions. > > This does not work, although I'm sure that my assertions are > correct. > > Known possibilities: > - I'm wrong and was not able to understand the assertions > on MPI-3.0 p452:8-19. > - I'm wrong because it is invalid to use the MPI_Win_fence > together with the shared memory windows. > - mpich has a bug. > (The first two possibilities are the reason, why I use this > Forum email list) > > - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c > > This is a work-around-for Cray that works on our Cray > and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. > It also runs on another mpich installation. > > - 1sided/halo_1sided_store_win_alloc_shared_pscw.c > > Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait > and it does not work for any assertions. > > Same possibilities as above. > > - 1sided/halo_1sided_store_win_alloc_shared_query.c > - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c > > Same as halo_1sided_store_win_alloc_shared.c > but non-contigues windows are used. > Same problems as above. > > - 1sided/halo_1sided_store_win_alloc_shared_othersync.c > > This version uses the synchronization according to > #413 and it is tested and works on two platforms. > > Best regards > Rolf > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses
by Jim Dinan 21 Mar '14

21 Mar '14

Rolf, This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + MPI_MODE_NOPRECEDE, win_rcv_buf_left ); You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | MPI_MODE_NOPRECEDE). In halo_1sided_store_win_alloc_shared.c, you are doing stores within the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion on the closing fence. Following the Fence epoch, you are reading from the left/right recv buffers. That also needs to be done within an RMA epoch, if you are reading non-local data. ~Jim. On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de>wrote: > Dear member of the RMA group and especially the mpich developers, > > I have real problems with the new shared memory in MPI-3.0, > i.e., the load/stores together with the RMA synchronization > causes wrong execution results. > > The attached > 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip > contains > > - 1sided/halo_1sided_put_win_alloc.c > > The basis that works. It uses MPI_Put and MPI_Win_fence for > duplex left/right halo communication. > > - 1sided/halo_1sided_store_win_alloc_shared.c > > This is the same, but a shared memory window is used and > the MPU_Put is substituted by storing the data in the > neighbors window. Same MPI_Win_fence with same assertions. > > This does not work, although I'm sure that my assertions are correct. > > Known possibilities: > - I'm wrong and was not able to understand the assertions > on MPI-3.0 p452:8-19. > - I'm wrong because it is invalid to use the MPI_Win_fence > together with the shared memory windows. > - mpich has a bug. > (The first two possibilities are the reason, why I use this > Forum email list) > > - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c > > This is a work-around-for Cray that works on our Cray > and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. > It also runs on another mpich installation. > > - 1sided/halo_1sided_store_win_alloc_shared_pscw.c > > Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait > and it does not work for any assertions. > > Same possibilities as above. > > - 1sided/halo_1sided_store_win_alloc_shared_query.c > - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c > > Same as halo_1sided_store_win_alloc_shared.c > but non-contigues windows are used. > Same problems as above. > > - 1sided/halo_1sided_store_win_alloc_shared_othersync.c > > This version uses the synchronization according to > #413 and it is tested and works on two platforms. > > Best regards > Rolf > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] same_op_no_op
by Jeff Hammond 14 Mar '14

14 Mar '14

I believe we use the term "accumulate functions" to describe MPI_ACCUMULATE, MPI_GET_ACCUMULATE, MPI_FETCH_AND_OP and MPI_COMPARE_AND_SWAP and they are considered to be "the same" in some sense. If MPI_COMPARE_AND_SWAP is not interoperable with MPI_GET_ACCUMULATE (which is a superset of MPI_ACCUMULATE and MPI_FETCH_AND_OP), then SHMEM is impossible. I don't believe we intended to do this. I thought the whole point of Keith and Brian torturing the Forum with argument count, ordering-by-default, etc. was to make SHMEM over MPI-3 possible... :-) Jeff On Fri, Mar 14, 2014 at 2:37 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Does same_op_no_op_replace allow MPI_NO_OP, MPI_REPLACE, MPI_SUM, *and* > MPI_Compare_and_swap (i.e. to support shmem_*_cswap)? I don't recall if CAS > is semantically a different op, or if it's considered to be a "replace". I > had thought it was considered to be a different op. > > > On Fri, Mar 14, 2014 at 10:11 AM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: >> >> In that case impls can add new info key for same_op_no_op_replace_hardware >> or something. >> >> SHMEM only needs same_op_no_op_replace as default but UPC appears to need >> SUM and XOR to be permitted at the same time to be efficient at the UPC >> runtime level. >> >> Jeff >> >> Sent from my iPhone >> >> > On Mar 14, 2014, at 8:37 AM, "Underwood, Keith D" >> > <keith.d.underwood(a)intel.com> wrote: >> > >> > The problem here is that some existing hardware supports some atomic >> > operations. Multiply is frequently not on that list. Doing an atomic add >> > on a non-coherent NIC and a multiply somewhere else can be challenging to >> > make correct, much less atomic. Now, if "all bets are off" in the >> > definition of not atomic (i.e. any interleaving of the two implied >> > load-op-store sequencings is legal), then I would argue that the description >> > you attribute to 2.2 is the better one. >> > >> >> -----Original Message----- >> >> From: mpiwg-rma [mailto:[email protected]] On >> >> Behalf Of Balaji, Pavan >> >> Sent: Thursday, March 13, 2014 1:02 PM >> >> To: MPI WG Remote Memory Access working group >> >> Subject: Re: [mpiwg-rma] same_op_no_op >> >> >> >> >> >> MPI-2.2 says that accumulate with different ops are not atomic. >> >> >> >> MPI-3 says that accumulate with different ops are not allowed (since >> >> same_op_no_op is default). >> >> >> >> I think we screwed that up? >> >> >> >> - Pavan >> >> >> >> On Mar 13, 2014, at 11:48 AM, Jeff Hammond <jeff.science(a)gmail.com> >> >> wrote: >> >> >> >>> It is extremely difficult to see that this is what the MPI-3 standard >> >>> says. >> >>> >> >>> First we have this: >> >>> >> >>> "The outcome of concurrent accumulate operations to the same location >> >>> with the same predefined datatype is as if the accumulates were done >> >>> at that location in some serial order. Additional restrictions on the >> >>> operation apply; see the info key accumulate_ops in Section 11.2.1. >> >>> Concurrent accumulate operations with different origin and target >> >>> pairs are not ordered. Thus, there is no guarantee that the entire >> >>> call to an accumulate operation is executed atomically. The effect of >> >>> this lack of atomicity is limited: The previous correctness conditions >> >>> imply that a location updated by a call to an accumulate operation >> >>> cannot be accessed by a load or an RMA call other than accumulate >> >>> until the accumulate operation has completed (at the target). >> >>> Different interleavings can lead to different results only to the >> >>> extent that computer arithmetics are not truly associative or >> >>> commutative. The outcome of accumulate operations with overlapping >> >>> types of different sizes or target displacements is undefined." >> >>> [11.7.1 Atomicity] >> >>> >> >>> Then we have this: >> >>> >> >>> "accumulate_ops - if set to same_op, the implementation will assume >> >>> that all concurrent accumulate calls to the same target address will >> >>> use the same operation. If set to same_op_no_op, then the >> >>> implementation will assume that all concurrent accumulate calls to the >> >>> same target address will use the same operation or MPI_NO_OP. This can >> >>> eliminate the need to protect access for certain operation types where >> >>> the hardware can guarantee atomicity. The default is same_op_no_op." >> >>> [11.2.1 Window Creation] >> >>> >> >>> I was not aware that the definition of info keys was normative, given >> >>> that implementations are free to ignore them. Even if info key text >> >>> is normative, one has to infer from the fact that same_op_no_op is the >> >>> default info behavior - and thus RMA semantic - that accumulate >> >>> atomicity is restricted to the case where one uses the same op or noop >> >>> but not replace. >> >>> >> >>> The MPI-2.2 spec is unambiguous because it explicitly requires the >> >>> same operation in 11.7.1 Atomicity. This text was removed in MPI-3.0 >> >>> in favor of the info key text. >> >>> >> >>> Best, >> >>> >> >>> Jeff >> >>> >> >>>> On Tue, Mar 11, 2014 at 12:04 AM, Balaji, Pavan <balaji(a)anl.gov> >> >>>> wrote: >> >>>> >> >>>> MPI-2 defines atomicity only for the same operation, not any >> >>>> operation >> >> for MPI_ACCUMULATE. >> >>>> >> >>>> - Pavan >> >>>> >> >>>> On Mar 10, 2014, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> >> >> wrote: >> >>>> >> >>>>> So MPI-2 denied compatibility between replace and not-replace? >> >>>>> >> >>>>> Jeff >> >>>>> >> >>>>> Sent from my iPhone >> >>>>> >> >>>>>> On Mar 11, 2014, at 12:06 AM, "Balaji, Pavan" <balaji(a)anl.gov> >> >>>>>> wrote: >> >>>>>> >> >>>>>> >> >>>>>> It doesn't break backward compatibility. The info argument is >> >>>>>> still >> >> useful when you don't want to use replace. I don't see anything wrong >> >> with >> >> it. >> >>>>>> >> >>>>>>> On Mar 10, 2014, at 11:01 PM, Jeff Hammond >> >> <jeff.science(a)gmail.com> wrote: >> >>>>>>> >> >>>>>>> Does this or does this not break BW compatibility w.r.t. MPI-2.2 >> >>>>>>> and did we do it intentionally? Unless we did so intentionally >> >>>>>>> and explicitly, I will argue that the WG screwed up and the info >> >>>>>>> key+val is invalid. >> >>>>>>> >> >>>>>>> Jeff >> >>>>>>> >> >>>>>>>> On Mon, Mar 10, 2014 at 11:03 PM, Balaji, Pavan <balaji(a)anl.gov> >> >> wrote: >> >>>>>>>> >> >>>>>>>> If a hardware can implement MPI_SUM, it should be able to >> >> implement MPI_SUM with 0 as well. >> >>>>>>>> >> >>>>>>>> But that's not a generic solution. >> >>>>>>>> >> >>>>>>>> Jeff: at some point you were planning to bring in a ticket which >> >>>>>>>> does >> >> more combinations of operations than just same_op and no_op. Maybe >> >> it's >> >> worthwhile bringing that up again? >> >>>>>>>> >> >>>>>>>> - Pavan >> >>>>>>>> >> >>>>>>>>> On Mar 10, 2014, at 9:26 PM, Jim Dinan <james.dinan(a)gmail.com> >> >> wrote: >> >>>>>>>>> >> >>>>>>>>> Maybe there's a loophole that I'm forgetting? >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Mon, Mar 10, 2014 at 9:43 PM, Jeff Hammond >> >> <jeff.science(a)gmail.com> wrote: >> >>>>>>>>> How the hell can I do GA or SHMEM then? Roll my own mutexes >> >> and commit perf-suicide? >> >>>>>>>>> >> >>>>>>>>> Jeff >> >>>>>>>>> >> >>>>>>>>> Sent from my iPhone >> >>>>>>>>> >> >>>>>>>>>> On Mar 10, 2014, at 8:32 PM, Jim Dinan <james.dinan(a)gmail.com> >> >> wrote: >> >>>>>>>>>> >> >>>>>>>>>> You can't use replace and sum concurrently at a given target >> >> address. >> >>>>>>>>>> >> >>>>>>>>>> ~Jim. >> >>>>>>>>>> >> >>>>>>>>>> On Mon, Mar 10, 2014 at 4:30 PM, Jeff Hammond >> >> <jeff.science(a)gmail.com> wrote: >> >>>>>>>>>> Given the following, how do I use MPI_NO_OP, MPI_REPLACE >> >> and >> >>>>>>>>>> MPI_SUM in accumulate/atomic operations in a standard- >> >> compliant way? >> >>>>>>>>>> >> >>>>>>>>>> accumulate_ops - if set to same_op, the implementation will >> >>>>>>>>>> assume that all concurrent accumulate calls to the same target >> >>>>>>>>>> address will use the same operation. If set to same_op_no_op, >> >>>>>>>>>> then the implementation will assume that all concurrent >> >>>>>>>>>> accumulate calls to the same target address will use the same >> >>>>>>>>>> operation or MPI_NO_OP. This can eliminate the need to protect >> >>>>>>>>>> access for certain operation types where the hardware can >> >> guarantee atomicity. The default is same_op_no_op. >> >>>>>>>>>> >> >>>>>>>>>> We discuss this before and the resolution was not satisfying to >> >> me. >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> >> >>>>>>>>>> Jeff >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Jeff Hammond >> >>>>>>>>>> jeff.science(a)gmail.com >> >>>>>>>>>> _______________________________________________ >> >>>>>>>>>> mpiwg-rma mailing list >> >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>>>>> >> >>>>>>>>>> _______________________________________________ >> >>>>>>>>>> mpiwg-rma mailing list >> >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>>>> >> >>>>>>>>> _______________________________________________ >> >>>>>>>>> mpiwg-rma mailing list >> >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>>>> >> >>>>>>>>> _______________________________________________ >> >>>>>>>>> mpiwg-rma mailing list >> >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>>> >> >>>>>>>> _______________________________________________ >> >>>>>>>> mpiwg-rma mailing list >> >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Jeff Hammond >> >>>>>>> jeff.science(a)gmail.com >> >>>>>>> _______________________________________________ >> >>>>>>> mpiwg-rma mailing list >> >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>> >> >>>>>> _______________________________________________ >> >>>>>> mpiwg-rma mailing list >> >>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>> _______________________________________________ >> >>>>> mpiwg-rma mailing list >> >>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>> >> >>>> _______________________________________________ >> >>>> mpiwg-rma mailing list >> >>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >> >>> >> >>> >> >>> -- >> >>> Jeff Hammond >> >>> jeff.science(a)gmail.com >> >>> _______________________________________________ >> >>> mpiwg-rma mailing list >> >>> mpiwg-rma(a)lists.mpi-forum.org >> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> _______________________________________________ >> >> mpiwg-rma mailing list >> >> mpiwg-rma(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > _______________________________________________ >> > mpiwg-rma mailing list >> > mpiwg-rma(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] same_op_no_op
by Jim Dinan 14 Mar '14

14 Mar '14

Does same_op_no_op_replace allow MPI_NO_OP, MPI_REPLACE, MPI_SUM, *and* MPI_Compare_and_swap (i.e. to support shmem_*_cswap)? I don't recall if CAS is semantically a different op, or if it's considered to be a "replace". I had thought it was considered to be a different op. On Fri, Mar 14, 2014 at 10:11 AM, Jeff Hammond <jeff.science(a)gmail.com>wrote: > In that case impls can add new info key for same_op_no_op_replace_hardware > or something. > > SHMEM only needs same_op_no_op_replace as default but UPC appears to need > SUM and XOR to be permitted at the same time to be efficient at the UPC > runtime level. > > Jeff > > Sent from my iPhone > > > On Mar 14, 2014, at 8:37 AM, "Underwood, Keith D" < > keith.d.underwood(a)intel.com> wrote: > > > > The problem here is that some existing hardware supports some atomic > operations. Multiply is frequently not on that list. Doing an atomic add > on a non-coherent NIC and a multiply somewhere else can be challenging to > make correct, much less atomic. Now, if "all bets are off" in the > definition of not atomic (i.e. any interleaving of the two implied > load-op-store sequencings is legal), then I would argue that the > description you attribute to 2.2 is the better one. > > > >> -----Original Message----- > >> From: mpiwg-rma [mailto:[email protected]] On > >> Behalf Of Balaji, Pavan > >> Sent: Thursday, March 13, 2014 1:02 PM > >> To: MPI WG Remote Memory Access working group > >> Subject: Re: [mpiwg-rma] same_op_no_op > >> > >> > >> MPI-2.2 says that accumulate with different ops are not atomic. > >> > >> MPI-3 says that accumulate with different ops are not allowed (since > >> same_op_no_op is default). > >> > >> I think we screwed that up? > >> > >> - Pavan > >> > >> On Mar 13, 2014, at 11:48 AM, Jeff Hammond <jeff.science(a)gmail.com> > >> wrote: > >> > >>> It is extremely difficult to see that this is what the MPI-3 standard > says. > >>> > >>> First we have this: > >>> > >>> "The outcome of concurrent accumulate operations to the same location > >>> with the same predefined datatype is as if the accumulates were done > >>> at that location in some serial order. Additional restrictions on the > >>> operation apply; see the info key accumulate_ops in Section 11.2.1. > >>> Concurrent accumulate operations with different origin and target > >>> pairs are not ordered. Thus, there is no guarantee that the entire > >>> call to an accumulate operation is executed atomically. The effect of > >>> this lack of atomicity is limited: The previous correctness conditions > >>> imply that a location updated by a call to an accumulate operation > >>> cannot be accessed by a load or an RMA call other than accumulate > >>> until the accumulate operation has completed (at the target). > >>> Different interleavings can lead to different results only to the > >>> extent that computer arithmetics are not truly associative or > >>> commutative. The outcome of accumulate operations with overlapping > >>> types of different sizes or target displacements is undefined." > >>> [11.7.1 Atomicity] > >>> > >>> Then we have this: > >>> > >>> "accumulate_ops - if set to same_op, the implementation will assume > >>> that all concurrent accumulate calls to the same target address will > >>> use the same operation. If set to same_op_no_op, then the > >>> implementation will assume that all concurrent accumulate calls to the > >>> same target address will use the same operation or MPI_NO_OP. This can > >>> eliminate the need to protect access for certain operation types where > >>> the hardware can guarantee atomicity. The default is same_op_no_op." > >>> [11.2.1 Window Creation] > >>> > >>> I was not aware that the definition of info keys was normative, given > >>> that implementations are free to ignore them. Even if info key text > >>> is normative, one has to infer from the fact that same_op_no_op is the > >>> default info behavior - and thus RMA semantic - that accumulate > >>> atomicity is restricted to the case where one uses the same op or noop > >>> but not replace. > >>> > >>> The MPI-2.2 spec is unambiguous because it explicitly requires the > >>> same operation in 11.7.1 Atomicity. This text was removed in MPI-3.0 > >>> in favor of the info key text. > >>> > >>> Best, > >>> > >>> Jeff > >>> > >>>> On Tue, Mar 11, 2014 at 12:04 AM, Balaji, Pavan <balaji(a)anl.gov> > wrote: > >>>> > >>>> MPI-2 defines atomicity only for the same operation, not any operation > >> for MPI_ACCUMULATE. > >>>> > >>>> - Pavan > >>>> > >>>> On Mar 10, 2014, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> > >> wrote: > >>>> > >>>>> So MPI-2 denied compatibility between replace and not-replace? > >>>>> > >>>>> Jeff > >>>>> > >>>>> Sent from my iPhone > >>>>> > >>>>>> On Mar 11, 2014, at 12:06 AM, "Balaji, Pavan" <balaji(a)anl.gov> > wrote: > >>>>>> > >>>>>> > >>>>>> It doesn't break backward compatibility. The info argument is still > >> useful when you don't want to use replace. I don't see anything wrong > with > >> it. > >>>>>> > >>>>>>> On Mar 10, 2014, at 11:01 PM, Jeff Hammond > >> <jeff.science(a)gmail.com> wrote: > >>>>>>> > >>>>>>> Does this or does this not break BW compatibility w.r.t. MPI-2.2 > >>>>>>> and did we do it intentionally? Unless we did so intentionally > >>>>>>> and explicitly, I will argue that the WG screwed up and the info > >>>>>>> key+val is invalid. > >>>>>>> > >>>>>>> Jeff > >>>>>>> > >>>>>>>> On Mon, Mar 10, 2014 at 11:03 PM, Balaji, Pavan <balaji(a)anl.gov> > >> wrote: > >>>>>>>> > >>>>>>>> If a hardware can implement MPI_SUM, it should be able to > >> implement MPI_SUM with 0 as well. > >>>>>>>> > >>>>>>>> But that's not a generic solution. > >>>>>>>> > >>>>>>>> Jeff: at some point you were planning to bring in a ticket which > does > >> more combinations of operations than just same_op and no_op. Maybe it's > >> worthwhile bringing that up again? > >>>>>>>> > >>>>>>>> - Pavan > >>>>>>>> > >>>>>>>>> On Mar 10, 2014, at 9:26 PM, Jim Dinan <james.dinan(a)gmail.com> > >> wrote: > >>>>>>>>> > >>>>>>>>> Maybe there's a loophole that I'm forgetting? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Mon, Mar 10, 2014 at 9:43 PM, Jeff Hammond > >> <jeff.science(a)gmail.com> wrote: > >>>>>>>>> How the hell can I do GA or SHMEM then? Roll my own mutexes > >> and commit perf-suicide? > >>>>>>>>> > >>>>>>>>> Jeff > >>>>>>>>> > >>>>>>>>> Sent from my iPhone > >>>>>>>>> > >>>>>>>>>> On Mar 10, 2014, at 8:32 PM, Jim Dinan <james.dinan(a)gmail.com> > >> wrote: > >>>>>>>>>> > >>>>>>>>>> You can't use replace and sum concurrently at a given target > >> address. > >>>>>>>>>> > >>>>>>>>>> ~Jim. > >>>>>>>>>> > >>>>>>>>>> On Mon, Mar 10, 2014 at 4:30 PM, Jeff Hammond > >> <jeff.science(a)gmail.com> wrote: > >>>>>>>>>> Given the following, how do I use MPI_NO_OP, MPI_REPLACE > >> and > >>>>>>>>>> MPI_SUM in accumulate/atomic operations in a standard- > >> compliant way? > >>>>>>>>>> > >>>>>>>>>> accumulate_ops - if set to same_op, the implementation will > >>>>>>>>>> assume that all concurrent accumulate calls to the same target > >>>>>>>>>> address will use the same operation. If set to same_op_no_op, > >>>>>>>>>> then the implementation will assume that all concurrent > >>>>>>>>>> accumulate calls to the same target address will use the same > >>>>>>>>>> operation or MPI_NO_OP. This can eliminate the need to protect > >>>>>>>>>> access for certain operation types where the hardware can > >> guarantee atomicity. The default is same_op_no_op. > >>>>>>>>>> > >>>>>>>>>> We discuss this before and the resolution was not satisfying to > >> me. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> > >>>>>>>>>> Jeff > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Jeff Hammond > >>>>>>>>>> jeff.science(a)gmail.com > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> mpiwg-rma mailing list > >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>>>>> > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> mpiwg-rma mailing list > >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> mpiwg-rma mailing list > >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> mpiwg-rma mailing list > >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> mpiwg-rma mailing list > >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Jeff Hammond > >>>>>>> jeff.science(a)gmail.com > >>>>>>> _______________________________________________ > >>>>>>> mpiwg-rma mailing list > >>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>> > >>>>>> _______________________________________________ > >>>>>> mpiwg-rma mailing list > >>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>> _______________________________________________ > >>>>> mpiwg-rma mailing list > >>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>> > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> > >>> > >>> > >>> -- > >>> Jeff Hammond > >>> jeff.science(a)gmail.com > >>> _______________________________________________ > >>> mpiwg-rma mailing list > >>> mpiwg-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] same_op_no_op
by Jeff Hammond 14 Mar '14

14 Mar '14

If my compiler has a flag for -fdisable-long-double-complex, we're good, right? Jeff Sent from my iPhone On Mar 14, 2014, at 11:54 AM, "Underwood, Keith D" <keith.d.underwood(a)intel.com> wrote: >> 2. Now we expect every system to have the equivalent of CMPXCHG16B and >> that basic types will never be longer than 16B (unless a longer atomic is made >> available)? > > You mean... like long double complex?

1 0

Re: [mpiwg-rma] same_op_no_op
by Jeff Hammond 14 Mar '14

14 Mar '14

M**LOC doesn't act on standard types so it's really not relevant. I said this already. Jeff Sent from my iPhone > On Mar 14, 2014, at 11:49 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > Jeff Hammond <jeff.science(a)gmail.com> writes: > >> There's an easy solution for "bad" ops: fall back to C&S >> implementation the way one does on any shared memory arch when a >> particular atomic isn't supported but C&S is. >> >> This means that no mutexes are required, NICs that have HW support for >> NO_OP, REPLACE, SUM, XOR, etc. can use them, and when a user asks for >> something silly like PROD, the performance is degraded for those ops >> alone and not for anything else. > > 1. How does the above not justify the same for M**LOC? > > 2. Now we expect every system to have the equivalent of CMPXCHG16B and > that basic types will never be longer than 16B (unless a longer atomic > is made available)?

1 0

Re: [mpiwg-rma] same_op_no_op
by Underwood, Keith D 14 Mar '14

14 Mar '14

> 2. Now we expect every system to have the equivalent of CMPXCHG16B and > that basic types will never be longer than 16B (unless a longer atomic is made > available)? You mean... like long double complex?

1 0

Re: [mpiwg-rma] same_op_no_op
by Jed Brown 14 Mar '14

14 Mar '14

Jeff Hammond <jeff.science(a)gmail.com> writes: > There's an easy solution for "bad" ops: fall back to C&S > implementation the way one does on any shared memory arch when a > particular atomic isn't supported but C&S is. > > This means that no mutexes are required, NICs that have HW support for > NO_OP, REPLACE, SUM, XOR, etc. can use them, and when a user asks for > something silly like PROD, the performance is degraded for those ops > alone and not for anything else. 1. How does the above not justify the same for M**LOC? 2. Now we expect every system to have the equivalent of CMPXCHG16B and that basic types will never be longer than 16B (unless a longer atomic is made available)?

1 0

Re: [mpiwg-rma] RMA info keys/values not listed in appendix
by Rajeev Thakur 14 Mar '14

14 Mar '14

Ah, I didn't see that. Looks like all are there. Rajeev On Mar 14, 2014, at 11:33 AM, William Gropp <wgropp(a)illinois.edu> wrote: > There were a couple included in the latest errata. Are there others that are missing? > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > University of Illinois Urbana-Champaign > > > > > > On Mar 14, 2014, at 11:31 AM, Rajeev Thakur wrote: > >> Appendix A.1.5 and A.1.6 on pg 683 list all reserved info keys and values. The ones from the RMA chapter are not listed there though. >> >> Rajeev >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] same_op_no_op
by Underwood, Keith D 14 Mar '14

14 Mar '14

I'm sure that wasn't Keith making that objection... ;-) > -----Original Message----- > From: mpiwg-rma [mailto:[email protected]] On > Behalf Of Jeff Hammond > Sent: Friday, March 14, 2014 12:30 PM > To: MPI WG Remote Memory Access working group > Subject: Re: [mpiwg-rma] same_op_no_op > > I'm sure that the WG was too distracted by how many arguments each > function took to think about semantic issues :-) > > I think not making same_op_no_op_replace the default was just an > oversight, as you've previously suggested. I'm sure that Keith has been > consistent to objecting to atomicity of more than one non-trivial (i.e. not > NO_OP or REPLACE) op at the same time. I believe that my previous email > noting how to use C&S emulation addresses that issue though. > > Jeff > > On Fri, Mar 14, 2014 at 11:25 AM, Brian Barrett <brbarret(a)open-mpi.org> > wrote: > > I agree with Rajeev; concurrent access with different ops wasn't undefined > in MPI-2.2, it was erroneous. > > > > During the Forum meetings on RMA, there was strong opposition to > relaxing that semantic, which is how we ended up with same_op_no_op. It > sucked, but it sucked less than the semantics of MPI-2.2. > > > > I'm not opposed to loosening the restrictions (and, in fact, wanted them > looser during MPI-3), but it would be good to understand why we as a group > felt that wasn't ok 2 years ago. > > > > Brian > > > > On Mar 14, 2014, at 6:12 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > > > >> Where does it say in MPI 2.2 that concurrent accumulates with different > operations are allowed? > >> > >> On pg 365, ln 29-33, it says two concurrent accumulates that use the same > operation are allowed. > >> > >> • A location in a window must not be accessed as a target of an RMA > operation once an update to that location has started, until the update > becomes visible in the public window copy. There is one exception to this > rule, in the case where the same variable is updated by two concurrent > accumulates that use the same operation, with the same predefined > datatype, on the same window. > >> > >> > >> On Mar 14, 2014, at 12:25 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > >> > >>> All, > >>> > >>> I’ve written up these thoughts in the below ticket: > >>> > >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/416 > >>> > >>> Comments are welcome on the ticket. > >>> > >>> — Pavan > >>> > >>> On Mar 13, 2014, at 4:13 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >>> > >>>> > >>>> On Mar 13, 2014, at 4:08 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >>>>>> It was not disallowed in MPI-2. I meant that we made a mistake in > MPI-3 to disallow it since that is not backward compatible. > >>>>> > >>>>> You told me two days ago it was at best undefined, which is no > >>>>> more useful than disallowed: > >>>> > >>>> I’m not sure what you are referring to, but that was no my intention. > Perhaps I misunderstood what you were asking. > >>>> > >>>> It think it’s pretty clear that it’s disallowed in MPI-3, not undefined. > >>>> > >>>>>> IMO, we should have kept the same semantics as MPI-2, but > allowed the user to relax it with info arguments. > >>>>> > >>>>> Well we broke backwards compatibility but made it almost > >>>>> impossible for anyone to notice and certainly didn't add an advice > >>>>> to users so we are clearly all jerks. > >>>> > >>>> I like MPI-2’s approach. The data content is undefined, but not an > error. > >>>> > >>>> — Pavan > >>>> > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> > >>> _______________________________________________ > >>> mpiwg-rma mailing list > >>> mpiwg-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > > > > -- > > Brian Barrett > > > > There is an art . . . to flying. The knack lies in learning how to > > throw yourself at the ground and miss. > > Douglas Adams, 'The Hitchhikers Guide to the Galaxy' > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0