- mpiwg-rma - mailman3.cels.anl.gov

#429 - Re: Problems with RMA synchronization in combination with load/store shared memory accesses
by Rolf Rabenseifner 19 May '14

19 May '14

Jim, I mnade #429. It is reported by me, but not owned, i.e., the RMA WG should take it. If the proposed text is okay for you (or with minor updates) then you can vote on it. I'll put it on the errata reading+voting list. It is the minimal text you can do that users and implementers see, that shared Memory Windows are Special for assertions. Therefore also the change-log, because implementors had and may also in future oversee this Problem. Best regards Rolf ----- Original Message ----- > From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Monday, May 19, 2014 7:16:44 PM > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses > > Jim and RMA WG, > > There are now two questions: > > Jim asked: > > Question to WG: Do we need to update the fence assertions to better > > define interaction with local load/store accesses and remote > > stores? > > > > Rolf asked: > > Additionally, I would recommend that we add after MPI-3.0 p451:33 > > > > Note that in shared memory windows (allocated with > > MPI_WIN_ALLOCATE_SHARED), there is no difference > > between remote store accesses and local store accesses > > to the window. > > > > This would help to understand that "the local window > > was not updated by stores" does not mean "by local stores", > > see p452:1 and p452:9. > > For me, it is important to understand the meaning of the > current assertions if they are used in a shared memory window. > Therefore my proposal above as erratum to MPI-3.0. > > In MPI-3.1 and 4.0, you may want to add additional assertions. > > Your analysis below, will also show that mpich implements > Post-Start-Complete-Wait synchronization in a wrong way, > if there are no calls to RMA routines. > > Best regards > Rolf > > ----- Original Message ----- > > From: "Jim Dinan" <james.dinan(a)gmail.com> > > To: "MPI WG Remote Memory Access working group" > > <mpiwg-rma(a)lists.mpi-forum.org> > > Sent: Thursday, May 15, 2014 4:06:08 PM > > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in > > combination with load/store shared memory accesses > > > > > > > > Rolf, > > > > > > Here is an attempt to simplify your example for discussion. Given > > a > > shared memory window, shr_mem_win, with buffer, shr_mem_buf: > > > > MPI_Win_fence(MPI_MODE_NOSTORE | MPI_MODE_NOPUT | > > MPI_MODE_NOPRECEDE > > | MPI_MODE_NOSUCCEED, shr_mem_win ); > > > > shr_mem_buf[...] = ...; > > > > MPI_Win_fence(MPI_MODE_NOPUT | MPI_MODE_NOPRECEDE | > > MPI_MODE_NOSUCCEED, shr_mem_win); > > > > > > Right now, Fence assertions don't say anything special about shared > > memory windows: > > > > > > Inline image 1 > > > > > > NOPRECEDE/SUCCEED are defined in terms of MPI RMA function calls, > > and > > do not cover load/store. Thus, Rolf's usage appears to be correct > > per the current text. In the MPICH fence implementation, > > src/mpid/ch3/src/ch3u_rma_sync.c:935 we have: > > > > if (!(assert & MPI_MODE_NOSUCCEED)) win_ptr->fence_issued = 1; > > > > Because of this check, we don't actually start an active target > > epoch > > on the first fence in the example above. On the second fence, we > > therefore don't perform the necessary synchronization, leading to > > incorrect output in Rolf's example. > > > > > > Question to WG: Do we need to update the fence assertions to better > > define interaction with local load/store accesses and remote > > stores? > > > > > > If not, then Rolf's code is correct and we need to modify the check > > above in MPICH to something like: > > > > > > if (!(assert & MPI_MODE_NOSUCCEED) || win_ptr->create_ flavor == > > MPI_WIN_FLAVOR_SHARED ) > > win_ptr->fence_issued = 1; > > > > > > ~Jim. > > > > > > > > > > On Tue, Apr 8, 2014 at 12:02 PM, Rolf Rabenseifner < > > rabenseifner(a)hlrs.de > wrote: > > > > > > Jim, > > > > I'm now sure, that mpich has a bug with assertions on shared memory > > windows. > > > > in the example, rcv_buf_left and rcv_buf_right are the windows. > > the only accesses to these rcv_buf_... are stores from remote > > and fully local loads. > > Both accesses are done in different epochs surrounded by > > MPI_Win_fence. > > > > According to your interpretation (which is really okay) > > all fences can use all possible assertions (!!!), > > except after the remote stores, MPI_MODE_NOSTORE cannot be used. > > > > I updated the example and mpich is executing it wrong. > > > > Please check it yourself on your installation: > > halo_1sided_store_win_alloc_shared_w-a-2-cray.c > > > > Without the assertions, all works: > > halo_1sided_store_win_alloc_shared_w-a-2NO-cray.c > > > > Could you verify that mpich has a bug? > > > > Additionally, I would recommend that we add after MPI-3.0 p451:33 > > > > Note that in shared memory windows (allocated with > > MPI_WIN_ALLOCATE_SHARED), there is no difference > > between remote store accesses and local store accesses > > to the window. > > > > This would help to understand that "the local window > > was not updated by stores" does not mean "by local stores", > > see p452:1 and p452:9. > > > > Is it a good idea? > > > > Best regards > > Rolf > > > > > > > > ----- Original Message ----- > > > From: "Jim Dinan" < james.dinan(a)gmail.com > > > > To: "MPI WG Remote Memory Access working group" < > > > mpiwg-rma(a)lists.mpi-forum.org > > > > Sent: Friday, March 21, 2014 8:14:22 PM > > > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in > > > combination with load/store shared memory accesses > > > > > > > > > > > > > > Rolf, > > > > > > > > > This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + > > > MPI_MODE_NOPRECEDE, win_ rcv_buf _left ); > > > > > > > > > > > > > You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | > > > MPI_MODE_NOPRECEDE). > > > > > > In halo_1sided_store_win_alloc_shared.c, you are doing stores > > > within > > > the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion > > > on > > > the closing fence. > > > > > > Following the Fence epoch, you are reading from the left/right > > > recv > > > buffers. That also needs to be done within an RMA epoch, if you > > > are > > > reading non-local data. > > > > > > ~Jim. > > > > > > > > > > > > On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner < > > > rabenseifner(a)hlrs.de > wrote: > > > > > > > > > Dear member of the RMA group and especially the mpich developers, > > > > > > I have real problems with the new shared memory in MPI-3.0, > > > i.e., the load/stores together with the RMA synchronization > > > causes wrong execution results. > > > > > > The attached > > > 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip > > > contains > > > > > > - 1sided/halo_1sided_put_win_alloc.c > > > > > > The basis that works. It uses MPI_Put and MPI_Win_fence for > > > duplex left/right halo communication. > > > > > > - 1sided/halo_1sided_store_win_alloc_shared.c > > > > > > This is the same, but a shared memory window is used and > > > the MPU_Put is substituted by storing the data in the > > > neighbors window. Same MPI_Win_fence with same assertions. > > > > > > This does not work, although I'm sure that my assertions are > > > correct. > > > > > > Known possibilities: > > > - I'm wrong and was not able to understand the assertions > > > on MPI-3.0 p452:8-19. > > > - I'm wrong because it is invalid to use the MPI_Win_fence > > > together with the shared memory windows. > > > - mpich has a bug. > > > (The first two possibilities are the reason, why I use this > > > Forum email list) > > > > > > - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c > > > > > > This is a work-around-for Cray that works on our Cray > > > and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. > > > It also runs on another mpich installation. > > > > > > - 1sided/halo_1sided_store_win_alloc_shared_pscw.c > > > > > > Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait > > > and it does not work for any assertions. > > > > > > Same possibilities as above. > > > > > > - 1sided/halo_1sided_store_win_alloc_shared_query.c > > > - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c > > > > > > Same as halo_1sided_store_win_alloc_shared.c > > > but non-contigues windows are used. > > > Same problems as above. > > > > > > - 1sided/halo_1sided_store_win_alloc_shared_othersync.c > > > > > > This version uses the synchronization according to > > > #413 and it is tested and works on two platforms. > > > > > > Best regards > > > Rolf > > > > > > -- > > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > > rabenseifner(a)hlrs.de > > > High Performance Computing Center (HLRS) . phone > > > ++49(0)711/685-65530 > > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > > 685-65832 > > > Head of Dpmt Parallel Computing . . . > > > www.hlrs.de/people/rabenseifner > > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > > 1.307) > > > > > > _______________________________________________ > > > mpiwg-rma mailing list > > > mpiwg-rma(a)lists.mpi-forum.org > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > > > > _______________________________________________ > > > mpiwg-rma mailing list > > > mpiwg-rma(a)lists.mpi-forum.org > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses
by Rolf Rabenseifner 19 May '14

19 May '14

Jim and RMA WG, There are now two questions: Jim asked: > Question to WG: Do we need to update the fence assertions to better > define interaction with local load/store accesses and remote stores? > Rolf asked: > Additionally, I would recommend that we add after MPI-3.0 p451:33 > > Note that in shared memory windows (allocated with > MPI_WIN_ALLOCATE_SHARED), there is no difference > between remote store accesses and local store accesses > to the window. > > This would help to understand that "the local window > was not updated by stores" does not mean "by local stores", > see p452:1 and p452:9. For me, it is important to understand the meaning of the current assertions if they are used in a shared memory window. Therefore my proposal above as erratum to MPI-3.0. In MPI-3.1 and 4.0, you may want to add additional assertions. Your analysis below, will also show that mpich implements Post-Start-Complete-Wait synchronization in a wrong way, if there are no calls to RMA routines. Best regards Rolf ----- Original Message ----- > From: "Jim Dinan" <james.dinan(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, May 15, 2014 4:06:08 PM > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses > > > > Rolf, > > > Here is an attempt to simplify your example for discussion. Given a > shared memory window, shr_mem_win, with buffer, shr_mem_buf: > > MPI_Win_fence(MPI_MODE_NOSTORE | MPI_MODE_NOPUT | MPI_MODE_NOPRECEDE > | MPI_MODE_NOSUCCEED, shr_mem_win ); > > shr_mem_buf[...] = ...; > > MPI_Win_fence(MPI_MODE_NOPUT | MPI_MODE_NOPRECEDE | > MPI_MODE_NOSUCCEED, shr_mem_win); > > > Right now, Fence assertions don't say anything special about shared > memory windows: > > > Inline image 1 > > > NOPRECEDE/SUCCEED are defined in terms of MPI RMA function calls, and > do not cover load/store. Thus, Rolf's usage appears to be correct > per the current text. In the MPICH fence implementation, > src/mpid/ch3/src/ch3u_rma_sync.c:935 we have: > > if (!(assert & MPI_MODE_NOSUCCEED)) win_ptr->fence_issued = 1; > > Because of this check, we don't actually start an active target epoch > on the first fence in the example above. On the second fence, we > therefore don't perform the necessary synchronization, leading to > incorrect output in Rolf's example. > > > Question to WG: Do we need to update the fence assertions to better > define interaction with local load/store accesses and remote stores? > > > If not, then Rolf's code is correct and we need to modify the check > above in MPICH to something like: > > > if (!(assert & MPI_MODE_NOSUCCEED) || win_ptr->create_ flavor == > MPI_WIN_FLAVOR_SHARED ) > win_ptr->fence_issued = 1; > > > ~Jim. > > > > > On Tue, Apr 8, 2014 at 12:02 PM, Rolf Rabenseifner < > rabenseifner(a)hlrs.de > wrote: > > > Jim, > > I'm now sure, that mpich has a bug with assertions on shared memory > windows. > > in the example, rcv_buf_left and rcv_buf_right are the windows. > the only accesses to these rcv_buf_... are stores from remote > and fully local loads. > Both accesses are done in different epochs surrounded by > MPI_Win_fence. > > According to your interpretation (which is really okay) > all fences can use all possible assertions (!!!), > except after the remote stores, MPI_MODE_NOSTORE cannot be used. > > I updated the example and mpich is executing it wrong. > > Please check it yourself on your installation: > halo_1sided_store_win_alloc_shared_w-a-2-cray.c > > Without the assertions, all works: > halo_1sided_store_win_alloc_shared_w-a-2NO-cray.c > > Could you verify that mpich has a bug? > > Additionally, I would recommend that we add after MPI-3.0 p451:33 > > Note that in shared memory windows (allocated with > MPI_WIN_ALLOCATE_SHARED), there is no difference > between remote store accesses and local store accesses > to the window. > > This would help to understand that "the local window > was not updated by stores" does not mean "by local stores", > see p452:1 and p452:9. > > Is it a good idea? > > Best regards > Rolf > > > > ----- Original Message ----- > > From: "Jim Dinan" < james.dinan(a)gmail.com > > > To: "MPI WG Remote Memory Access working group" < > > mpiwg-rma(a)lists.mpi-forum.org > > > Sent: Friday, March 21, 2014 8:14:22 PM > > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in > > combination with load/store shared memory accesses > > > > > > > > > Rolf, > > > > > > This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + > > MPI_MODE_NOPRECEDE, win_ rcv_buf _left ); > > > > > > > > You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | > > MPI_MODE_NOPRECEDE). > > > > In halo_1sided_store_win_alloc_shared.c, you are doing stores > > within > > the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion on > > the closing fence. > > > > Following the Fence epoch, you are reading from the left/right recv > > buffers. That also needs to be done within an RMA epoch, if you > > are > > reading non-local data. > > > > ~Jim. > > > > > > > > On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner < > > rabenseifner(a)hlrs.de > wrote: > > > > > > Dear member of the RMA group and especially the mpich developers, > > > > I have real problems with the new shared memory in MPI-3.0, > > i.e., the load/stores together with the RMA synchronization > > causes wrong execution results. > > > > The attached > > 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip > > contains > > > > - 1sided/halo_1sided_put_win_alloc.c > > > > The basis that works. It uses MPI_Put and MPI_Win_fence for > > duplex left/right halo communication. > > > > - 1sided/halo_1sided_store_win_alloc_shared.c > > > > This is the same, but a shared memory window is used and > > the MPU_Put is substituted by storing the data in the > > neighbors window. Same MPI_Win_fence with same assertions. > > > > This does not work, although I'm sure that my assertions are > > correct. > > > > Known possibilities: > > - I'm wrong and was not able to understand the assertions > > on MPI-3.0 p452:8-19. > > - I'm wrong because it is invalid to use the MPI_Win_fence > > together with the shared memory windows. > > - mpich has a bug. > > (The first two possibilities are the reason, why I use this > > Forum email list) > > > > - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c > > > > This is a work-around-for Cray that works on our Cray > > and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. > > It also runs on another mpich installation. > > > > - 1sided/halo_1sided_store_win_alloc_shared_pscw.c > > > > Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait > > and it does not work for any assertions. > > > > Same possibilities as above. > > > > - 1sided/halo_1sided_store_win_alloc_shared_query.c > > - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c > > > > Same as halo_1sided_store_win_alloc_shared.c > > but non-contigues windows are used. > > Same problems as above. > > > > - 1sided/halo_1sided_store_win_alloc_shared_othersync.c > > > > This version uses the synchronization according to > > #413 and it is tested and works on two platforms. > > > > Best regards > > Rolf > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses
by Jim Dinan 15 May '14

15 May '14

Rolf, Here is an attempt to simplify your example for discussion. Given a shared memory window, shr_mem_win, with buffer, shr_mem_buf: MPI_Win_fence(MPI_MODE_NOSTORE | MPI_MODE_NOPUT | MPI_MODE_NOPRECEDE | MPI_MODE_NOSUCCEED, shr_mem_win ); shr_mem_buf[...] = ...; MPI_Win_fence(MPI_MODE_NOPUT | MPI_MODE_NOPRECEDE | MPI_MODE_NOSUCCEED, shr_mem_win); Right now, Fence assertions don't say anything special about shared memory windows: [image: Inline image 1] NOPRECEDE/SUCCEED are defined in terms of MPI RMA function calls, and do not cover load/store. Thus, Rolf's usage appears to be correct per the current text. In the MPICH fence implementation, src/mpid/ch3/src/ch3u_rma_sync.c:935 we have: if (!(assert & MPI_MODE_NOSUCCEED)) win_ptr->fence_issued = 1; Because of this check, we don't actually start an active target epoch on the first fence in the example above. On the second fence, we therefore don't perform the necessary synchronization, leading to incorrect output in Rolf's example. Question to WG: Do we need to update the fence assertions to better define interaction with local load/store accesses and remote stores? If not, then Rolf's code is correct and we need to modify the check above in MPICH to something like: if (!(assert & MPI_MODE_NOSUCCEED) || win_ptr->create_flavor == MPI_WIN_FLAVOR_SHARED) win_ptr->fence_issued = 1; ~Jim. On Tue, Apr 8, 2014 at 12:02 PM, Rolf Rabenseifner <rabenseifner(a)hlrs.de>wrote: > Jim, > > I'm now sure, that mpich has a bug with assertions on shared memory > windows. > > in the example, rcv_buf_left and rcv_buf_right are the windows. > the only accesses to these rcv_buf_... are stores from remote > and fully local loads. > Both accesses are done in different epochs surrounded by MPI_Win_fence. > > According to your interpretation (which is really okay) > all fences can use all possible assertions (!!!), > except after the remote stores, MPI_MODE_NOSTORE cannot be used. > > I updated the example and mpich is executing it wrong. > > Please check it yourself on your installation: > halo_1sided_store_win_alloc_shared_w-a-2-cray.c > > Without the assertions, all works: > halo_1sided_store_win_alloc_shared_w-a-2NO-cray.c > > Could you verify that mpich has a bug? > > Additionally, I would recommend that we add after MPI-3.0 p451:33 > > Note that in shared memory windows (allocated with > MPI_WIN_ALLOCATE_SHARED), there is no difference > between remote store accesses and local store accesses > to the window. > > This would help to understand that "the local window > was not updated by stores" does not mean "by local stores", > see p452:1 and p452:9. > > Is it a good idea? > > Best regards > Rolf > > > ----- Original Message ----- > > From: "Jim Dinan" <james.dinan(a)gmail.com> > > To: "MPI WG Remote Memory Access working group" < > mpiwg-rma(a)lists.mpi-forum.org> > > Sent: Friday, March 21, 2014 8:14:22 PM > > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in > combination with load/store shared memory accesses > > > > > > > > Rolf, > > > > > > This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + > > MPI_MODE_NOPRECEDE, win_ rcv_buf _left ); > > > > > > You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | > > MPI_MODE_NOPRECEDE). > > > > In halo_1sided_store_win_alloc_shared.c, you are doing stores within > > the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion on > > the closing fence. > > > > Following the Fence epoch, you are reading from the left/right recv > > buffers. That also needs to be done within an RMA epoch, if you are > > reading non-local data. > > > > ~Jim. > > > > > > > > On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner < > > rabenseifner(a)hlrs.de > wrote: > > > > > > Dear member of the RMA group and especially the mpich developers, > > > > I have real problems with the new shared memory in MPI-3.0, > > i.e., the load/stores together with the RMA synchronization > > causes wrong execution results. > > > > The attached > > 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip > > contains > > > > - 1sided/halo_1sided_put_win_alloc.c > > > > The basis that works. It uses MPI_Put and MPI_Win_fence for > > duplex left/right halo communication. > > > > - 1sided/halo_1sided_store_win_alloc_shared.c > > > > This is the same, but a shared memory window is used and > > the MPU_Put is substituted by storing the data in the > > neighbors window. Same MPI_Win_fence with same assertions. > > > > This does not work, although I'm sure that my assertions are > > correct. > > > > Known possibilities: > > - I'm wrong and was not able to understand the assertions > > on MPI-3.0 p452:8-19. > > - I'm wrong because it is invalid to use the MPI_Win_fence > > together with the shared memory windows. > > - mpich has a bug. > > (The first two possibilities are the reason, why I use this > > Forum email list) > > > > - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c > > > > This is a work-around-for Cray that works on our Cray > > and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. > > It also runs on another mpich installation. > > > > - 1sided/halo_1sided_store_win_alloc_shared_pscw.c > > > > Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait > > and it does not work for any assertions. > > > > Same possibilities as above. > > > > - 1sided/halo_1sided_store_win_alloc_shared_query.c > > - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c > > > > Same as halo_1sided_store_win_alloc_shared.c > > but non-contigues windows are used. > > Same problems as above. > > > > - 1sided/halo_1sided_store_win_alloc_shared_othersync.c > > > > This version uses the synchronization according to > > #413 and it is tested and works on two platforms. > > > > Best regards > > Rolf > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Fwd: MPI Forum / RMA WG
by Rajeev Thakur 06 May '14

06 May '14

How much working group or plenary time do we need at the June meeting? Rajeev Begin forwarded message: > From: "Schulz, Martin" <schulz6(a)llnl.gov> > Subject: MPI Forum / RMA WG > Date: May 6, 2014 2:29:47 AM CDT > To: William Gropp <wgropp(a)illinois.edu>, Rajeev Thakur <thakur(a)mcs.anl.gov> > > Hi Bill and Rajeev, > > Do you have any time requests (WG or plenary) for the June forum meeting? > > Thanks! > > Martin >

1 0

Re: [mpiwg-rma] {MAX,MIN}LOC in RMA
by Jim Dinan 09 Apr '14

09 Apr '14

FYI, we added a test case for MPI_Accumulate with MIN/MAX_LOC to the MPICH test suite: http://git.mpich.org/mpich.git/commit/a80bf84aecfc6c6152474623d418b2a3cb71f… Also, the mutex idea I had didn't pan out because of the LOC operators always chose the least loc when the values are equal. Can't think of a way to make this work.. ~Jim. On Thu, Mar 13, 2014 at 3:25 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > MAX/MIN_LOC operations have an (albeit odd) atomic compare and replace > semantic. They operate on a pair type that has <value, location> > components. The user provides the data for both value and location. The > min/max comparison (i.e. less-than/greater-than) is done on the value. If > the comparison is true, then the remote <value, location> is atomically > replaced with the supplied data. > > These operations can be useful for one-sided reductions. I think PetSC > uses them for this. In MPI-2 (there are, of course better ways in MPI-3) > you can also use these operations to implement a spinlock. The value is a > boolean indicating if the mutex is taken and the location is the rank that > hold the mutex. The mutex initially contains the unlocked value <0, > MPI_PROC_NULL>. You do a MAX_LOC accumulate of <1, me> to try to claim the > mutex then do a get to see if you succeeded (exclusive epochs because of > mixing accumulates and gets and the same op restriction). To unlock, you > write the unlocked value back to the mutex. > > I have a test case that implements this, that's been queued up for a while > waiting to go out to MPICH. Should be able to push it out soon. > > ~Jim. > > > On Thu, Mar 13, 2014 at 3:05 PM, Jeff Hammond <jeff.science(a)gmail.com>wrote: > >> What is the expected behavior of {MAX,MIN}LOC in RMA? I have a hard >> time seeing what the use would be. Does anyone want to use MAXLOC to >> find the MAX of two numbers and write in the window the rank (chosen >> from the set of the initiator and the target) of the pair member that >> is larger? It's not like one can use this to implement a one-sided >> reduction since any time the value in the window is larger, the >> information about the original LOC will be lost. >> >> Can we remove these operations from the set of accumulate operations? >> >> Thanks, >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > >

1 0

Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses
by Rolf Rabenseifner 08 Apr '14

08 Apr '14

Jim, I'm now sure, that mpich has a bug with assertions on shared memory windows. in the example, rcv_buf_left and rcv_buf_right are the windows. the only accesses to these rcv_buf_... are stores from remote and fully local loads. Both accesses are done in different epochs surrounded by MPI_Win_fence. According to your interpretation (which is really okay) all fences can use all possible assertions (!!!), except after the remote stores, MPI_MODE_NOSTORE cannot be used. I updated the example and mpich is executing it wrong. Please check it yourself on your installation: halo_1sided_store_win_alloc_shared_w-a-2-cray.c Without the assertions, all works: halo_1sided_store_win_alloc_shared_w-a-2NO-cray.c Could you verify that mpich has a bug? Additionally, I would recommend that we add after MPI-3.0 p451:33 Note that in shared memory windows (allocated with MPI_WIN_ALLOCATE_SHARED), there is no difference between remote store accesses and local store accesses to the window. This would help to understand that "the local window was not updated by stores" does not mean "by local stores", see p452:1 and p452:9. Is it a good idea? Best regards Rolf ----- Original Message ----- > From: "Jim Dinan" <james.dinan(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Friday, March 21, 2014 8:14:22 PM > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses > > > > Rolf, > > > This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + > MPI_MODE_NOPRECEDE, win_ rcv_buf _left ); > > > You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | > MPI_MODE_NOPRECEDE). > > In halo_1sided_store_win_alloc_shared.c, you are doing stores within > the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion on > the closing fence. > > Following the Fence epoch, you are reading from the left/right recv > buffers. That also needs to be done within an RMA epoch, if you are > reading non-local data. > > ~Jim. > > > > On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner < > rabenseifner(a)hlrs.de > wrote: > > > Dear member of the RMA group and especially the mpich developers, > > I have real problems with the new shared memory in MPI-3.0, > i.e., the load/stores together with the RMA synchronization > causes wrong execution results. > > The attached > 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip > contains > > - 1sided/halo_1sided_put_win_alloc.c > > The basis that works. It uses MPI_Put and MPI_Win_fence for > duplex left/right halo communication. > > - 1sided/halo_1sided_store_win_alloc_shared.c > > This is the same, but a shared memory window is used and > the MPU_Put is substituted by storing the data in the > neighbors window. Same MPI_Win_fence with same assertions. > > This does not work, although I'm sure that my assertions are > correct. > > Known possibilities: > - I'm wrong and was not able to understand the assertions > on MPI-3.0 p452:8-19. > - I'm wrong because it is invalid to use the MPI_Win_fence > together with the shared memory windows. > - mpich has a bug. > (The first two possibilities are the reason, why I use this > Forum email list) > > - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c > > This is a work-around-for Cray that works on our Cray > and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. > It also runs on another mpich installation. > > - 1sided/halo_1sided_store_win_alloc_shared_pscw.c > > Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait > and it does not work for any assertions. > > Same possibilities as above. > > - 1sided/halo_1sided_store_win_alloc_shared_query.c > - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c > > Same as halo_1sided_store_win_alloc_shared.c > but non-contigues windows are used. > Same problems as above. > > - 1sided/halo_1sided_store_win_alloc_shared_othersync.c > > This version uses the synchronization according to > #413 and it is tested and works on two platforms. > > Best regards > Rolf > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses
by Rolf Rabenseifner 22 Mar '14

22 Mar '14

Jim, thank you for your analysis. Answers see below. ----- Original Message ----- > From: "Jim Dinan" <james.dinan(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Friday, March 21, 2014 8:14:22 PM > Subject: Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses > > > > Rolf, > > > This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + > MPI_MODE_NOPRECEDE, win_ rcv_buf _left ); > > > You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | > MPI_MODE_NOPRECEDE). Yes, you are right, but this cannot be the reasons for the problems according to MPI-3.0 p451:28-32. The Fortran versions show the same problems. > In halo_1sided_store_win_alloc_shared.c, you are doing stores within > the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion on > the closing fence. Yes, but I do remote stores and no local stores. But you may be right, because the text p452:9-10 does not require that the stores are local stores. When I remember correctly, then removing this hint does not produce correct results. I'll re-check. > > Following the Fence epoch, you are reading from the left/right recv > buffers. That also needs to be done within an RMA epoch, if you are > reading non-local data. This reading is only on the local recv_buf_left/right. But it is included in the RMA epoch starting with the second fence and ending with the first fence in the next loop Iteration. > > ~Jim. Best regards Rolf > > > > On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner < > rabenseifner(a)hlrs.de > wrote: > > > Dear member of the RMA group and especially the mpich developers, > > I have real problems with the new shared memory in MPI-3.0, > i.e., the load/stores together with the RMA synchronization > causes wrong execution results. > > The attached > 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip > contains > > - 1sided/halo_1sided_put_win_alloc.c > > The basis that works. It uses MPI_Put and MPI_Win_fence for > duplex left/right halo communication. > > - 1sided/halo_1sided_store_win_alloc_shared.c > > This is the same, but a shared memory window is used and > the MPU_Put is substituted by storing the data in the > neighbors window. Same MPI_Win_fence with same assertions. > > This does not work, although I'm sure that my assertions are > correct. > > Known possibilities: > - I'm wrong and was not able to understand the assertions > on MPI-3.0 p452:8-19. > - I'm wrong because it is invalid to use the MPI_Win_fence > together with the shared memory windows. > - mpich has a bug. > (The first two possibilities are the reason, why I use this > Forum email list) > > - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c > > This is a work-around-for Cray that works on our Cray > and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. > It also runs on another mpich installation. > > - 1sided/halo_1sided_store_win_alloc_shared_pscw.c > > Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait > and it does not work for any assertions. > > Same possibilities as above. > > - 1sided/halo_1sided_store_win_alloc_shared_query.c > - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c > > Same as halo_1sided_store_win_alloc_shared.c > but non-contigues windows are used. > Same problems as above. > > - 1sided/halo_1sided_store_win_alloc_shared_othersync.c > > This version uses the synchronization according to > #413 and it is tested and works on two platforms. > > Best regards > Rolf > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Problems with RMA synchronization in combination with load/store shared memory accesses
by Jim Dinan 21 Mar '14

21 Mar '14

Rolf, This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + MPI_MODE_NOPRECEDE, win_rcv_buf_left ); You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | MPI_MODE_NOPRECEDE). In halo_1sided_store_win_alloc_shared.c, you are doing stores within the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion on the closing fence. Following the Fence epoch, you are reading from the left/right recv buffers. That also needs to be done within an RMA epoch, if you are reading non-local data. ~Jim. On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de>wrote: > Dear member of the RMA group and especially the mpich developers, > > I have real problems with the new shared memory in MPI-3.0, > i.e., the load/stores together with the RMA synchronization > causes wrong execution results. > > The attached > 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip > contains > > - 1sided/halo_1sided_put_win_alloc.c > > The basis that works. It uses MPI_Put and MPI_Win_fence for > duplex left/right halo communication. > > - 1sided/halo_1sided_store_win_alloc_shared.c > > This is the same, but a shared memory window is used and > the MPU_Put is substituted by storing the data in the > neighbors window. Same MPI_Win_fence with same assertions. > > This does not work, although I'm sure that my assertions are correct. > > Known possibilities: > - I'm wrong and was not able to understand the assertions > on MPI-3.0 p452:8-19. > - I'm wrong because it is invalid to use the MPI_Win_fence > together with the shared memory windows. > - mpich has a bug. > (The first two possibilities are the reason, why I use this > Forum email list) > > - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c > > This is a work-around-for Cray that works on our Cray > and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. > It also runs on another mpich installation. > > - 1sided/halo_1sided_store_win_alloc_shared_pscw.c > > Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait > and it does not work for any assertions. > > Same possibilities as above. > > - 1sided/halo_1sided_store_win_alloc_shared_query.c > - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c > > Same as halo_1sided_store_win_alloc_shared.c > but non-contigues windows are used. > Same problems as above. > > - 1sided/halo_1sided_store_win_alloc_shared_othersync.c > > This version uses the synchronization according to > #413 and it is tested and works on two platforms. > > Best regards > Rolf > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] same_op_no_op
by Jeff Hammond 14 Mar '14

14 Mar '14

I believe we use the term "accumulate functions" to describe MPI_ACCUMULATE, MPI_GET_ACCUMULATE, MPI_FETCH_AND_OP and MPI_COMPARE_AND_SWAP and they are considered to be "the same" in some sense. If MPI_COMPARE_AND_SWAP is not interoperable with MPI_GET_ACCUMULATE (which is a superset of MPI_ACCUMULATE and MPI_FETCH_AND_OP), then SHMEM is impossible. I don't believe we intended to do this. I thought the whole point of Keith and Brian torturing the Forum with argument count, ordering-by-default, etc. was to make SHMEM over MPI-3 possible... :-) Jeff On Fri, Mar 14, 2014 at 2:37 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Does same_op_no_op_replace allow MPI_NO_OP, MPI_REPLACE, MPI_SUM, *and* > MPI_Compare_and_swap (i.e. to support shmem_*_cswap)? I don't recall if CAS > is semantically a different op, or if it's considered to be a "replace". I > had thought it was considered to be a different op. > > > On Fri, Mar 14, 2014 at 10:11 AM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: >> >> In that case impls can add new info key for same_op_no_op_replace_hardware >> or something. >> >> SHMEM only needs same_op_no_op_replace as default but UPC appears to need >> SUM and XOR to be permitted at the same time to be efficient at the UPC >> runtime level. >> >> Jeff >> >> Sent from my iPhone >> >> > On Mar 14, 2014, at 8:37 AM, "Underwood, Keith D" >> > <keith.d.underwood(a)intel.com> wrote: >> > >> > The problem here is that some existing hardware supports some atomic >> > operations. Multiply is frequently not on that list. Doing an atomic add >> > on a non-coherent NIC and a multiply somewhere else can be challenging to >> > make correct, much less atomic. Now, if "all bets are off" in the >> > definition of not atomic (i.e. any interleaving of the two implied >> > load-op-store sequencings is legal), then I would argue that the description >> > you attribute to 2.2 is the better one. >> > >> >> -----Original Message----- >> >> From: mpiwg-rma [mailto:[email protected]] On >> >> Behalf Of Balaji, Pavan >> >> Sent: Thursday, March 13, 2014 1:02 PM >> >> To: MPI WG Remote Memory Access working group >> >> Subject: Re: [mpiwg-rma] same_op_no_op >> >> >> >> >> >> MPI-2.2 says that accumulate with different ops are not atomic. >> >> >> >> MPI-3 says that accumulate with different ops are not allowed (since >> >> same_op_no_op is default). >> >> >> >> I think we screwed that up? >> >> >> >> - Pavan >> >> >> >> On Mar 13, 2014, at 11:48 AM, Jeff Hammond <jeff.science(a)gmail.com> >> >> wrote: >> >> >> >>> It is extremely difficult to see that this is what the MPI-3 standard >> >>> says. >> >>> >> >>> First we have this: >> >>> >> >>> "The outcome of concurrent accumulate operations to the same location >> >>> with the same predefined datatype is as if the accumulates were done >> >>> at that location in some serial order. Additional restrictions on the >> >>> operation apply; see the info key accumulate_ops in Section 11.2.1. >> >>> Concurrent accumulate operations with different origin and target >> >>> pairs are not ordered. Thus, there is no guarantee that the entire >> >>> call to an accumulate operation is executed atomically. The effect of >> >>> this lack of atomicity is limited: The previous correctness conditions >> >>> imply that a location updated by a call to an accumulate operation >> >>> cannot be accessed by a load or an RMA call other than accumulate >> >>> until the accumulate operation has completed (at the target). >> >>> Different interleavings can lead to different results only to the >> >>> extent that computer arithmetics are not truly associative or >> >>> commutative. The outcome of accumulate operations with overlapping >> >>> types of different sizes or target displacements is undefined." >> >>> [11.7.1 Atomicity] >> >>> >> >>> Then we have this: >> >>> >> >>> "accumulate_ops - if set to same_op, the implementation will assume >> >>> that all concurrent accumulate calls to the same target address will >> >>> use the same operation. If set to same_op_no_op, then the >> >>> implementation will assume that all concurrent accumulate calls to the >> >>> same target address will use the same operation or MPI_NO_OP. This can >> >>> eliminate the need to protect access for certain operation types where >> >>> the hardware can guarantee atomicity. The default is same_op_no_op." >> >>> [11.2.1 Window Creation] >> >>> >> >>> I was not aware that the definition of info keys was normative, given >> >>> that implementations are free to ignore them. Even if info key text >> >>> is normative, one has to infer from the fact that same_op_no_op is the >> >>> default info behavior - and thus RMA semantic - that accumulate >> >>> atomicity is restricted to the case where one uses the same op or noop >> >>> but not replace. >> >>> >> >>> The MPI-2.2 spec is unambiguous because it explicitly requires the >> >>> same operation in 11.7.1 Atomicity. This text was removed in MPI-3.0 >> >>> in favor of the info key text. >> >>> >> >>> Best, >> >>> >> >>> Jeff >> >>> >> >>>> On Tue, Mar 11, 2014 at 12:04 AM, Balaji, Pavan <balaji(a)anl.gov> >> >>>> wrote: >> >>>> >> >>>> MPI-2 defines atomicity only for the same operation, not any >> >>>> operation >> >> for MPI_ACCUMULATE. >> >>>> >> >>>> - Pavan >> >>>> >> >>>> On Mar 10, 2014, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> >> >> wrote: >> >>>> >> >>>>> So MPI-2 denied compatibility between replace and not-replace? >> >>>>> >> >>>>> Jeff >> >>>>> >> >>>>> Sent from my iPhone >> >>>>> >> >>>>>> On Mar 11, 2014, at 12:06 AM, "Balaji, Pavan" <balaji(a)anl.gov> >> >>>>>> wrote: >> >>>>>> >> >>>>>> >> >>>>>> It doesn't break backward compatibility. The info argument is >> >>>>>> still >> >> useful when you don't want to use replace. I don't see anything wrong >> >> with >> >> it. >> >>>>>> >> >>>>>>> On Mar 10, 2014, at 11:01 PM, Jeff Hammond >> >> <jeff.science(a)gmail.com> wrote: >> >>>>>>> >> >>>>>>> Does this or does this not break BW compatibility w.r.t. MPI-2.2 >> >>>>>>> and did we do it intentionally? Unless we did so intentionally >> >>>>>>> and explicitly, I will argue that the WG screwed up and the info >> >>>>>>> key+val is invalid. >> >>>>>>> >> >>>>>>> Jeff >> >>>>>>> >> >>>>>>>> On Mon, Mar 10, 2014 at 11:03 PM, Balaji, Pavan <balaji(a)anl.gov> >> >> wrote: >> >>>>>>>> >> >>>>>>>> If a hardware can implement MPI_SUM, it should be able to >> >> implement MPI_SUM with 0 as well. >> >>>>>>>> >> >>>>>>>> But that's not a generic solution. >> >>>>>>>> >> >>>>>>>> Jeff: at some point you were planning to bring in a ticket which >> >>>>>>>> does >> >> more combinations of operations than just same_op and no_op. Maybe >> >> it's >> >> worthwhile bringing that up again? >> >>>>>>>> >> >>>>>>>> - Pavan >> >>>>>>>> >> >>>>>>>>> On Mar 10, 2014, at 9:26 PM, Jim Dinan <james.dinan(a)gmail.com> >> >> wrote: >> >>>>>>>>> >> >>>>>>>>> Maybe there's a loophole that I'm forgetting? >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Mon, Mar 10, 2014 at 9:43 PM, Jeff Hammond >> >> <jeff.science(a)gmail.com> wrote: >> >>>>>>>>> How the hell can I do GA or SHMEM then? Roll my own mutexes >> >> and commit perf-suicide? >> >>>>>>>>> >> >>>>>>>>> Jeff >> >>>>>>>>> >> >>>>>>>>> Sent from my iPhone >> >>>>>>>>> >> >>>>>>>>>> On Mar 10, 2014, at 8:32 PM, Jim Dinan <james.dinan(a)gmail.com> >> >> wrote: >> >>>>>>>>>> >> >>>>>>>>>> You can't use replace and sum concurrently at a given target >> >> address. >> >>>>>>>>>> >> >>>>>>>>>> ~Jim. >> >>>>>>>>>> >> >>>>>>>>>> On Mon, Mar 10, 2014 at 4:30 PM, Jeff Hammond >> >> <jeff.science(a)gmail.com> wrote: >> >>>>>>>>>> Given the following, how do I use MPI_NO_OP, MPI_REPLACE >> >> and >> >>>>>>>>>> MPI_SUM in accumulate/atomic operations in a standard- >> >> compliant way? >> >>>>>>>>>> >> >>>>>>>>>> accumulate_ops - if set to same_op, the implementation will >> >>>>>>>>>> assume that all concurrent accumulate calls to the same target >> >>>>>>>>>> address will use the same operation. If set to same_op_no_op, >> >>>>>>>>>> then the implementation will assume that all concurrent >> >>>>>>>>>> accumulate calls to the same target address will use the same >> >>>>>>>>>> operation or MPI_NO_OP. This can eliminate the need to protect >> >>>>>>>>>> access for certain operation types where the hardware can >> >> guarantee atomicity. The default is same_op_no_op. >> >>>>>>>>>> >> >>>>>>>>>> We discuss this before and the resolution was not satisfying to >> >> me. >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> >> >>>>>>>>>> Jeff >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Jeff Hammond >> >>>>>>>>>> jeff.science(a)gmail.com >> >>>>>>>>>> _______________________________________________ >> >>>>>>>>>> mpiwg-rma mailing list >> >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>>>>> >> >>>>>>>>>> _______________________________________________ >> >>>>>>>>>> mpiwg-rma mailing list >> >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>>>> >> >>>>>>>>> _______________________________________________ >> >>>>>>>>> mpiwg-rma mailing list >> >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>>>> >> >>>>>>>>> _______________________________________________ >> >>>>>>>>> mpiwg-rma mailing list >> >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>>> >> >>>>>>>> _______________________________________________ >> >>>>>>>> mpiwg-rma mailing list >> >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Jeff Hammond >> >>>>>>> jeff.science(a)gmail.com >> >>>>>>> _______________________________________________ >> >>>>>>> mpiwg-rma mailing list >> >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>>> >> >>>>>> _______________________________________________ >> >>>>>> mpiwg-rma mailing list >> >>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>>> _______________________________________________ >> >>>>> mpiwg-rma mailing list >> >>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>> >> >>>> _______________________________________________ >> >>>> mpiwg-rma mailing list >> >>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >> >>> >> >>> >> >>> -- >> >>> Jeff Hammond >> >>> jeff.science(a)gmail.com >> >>> _______________________________________________ >> >>> mpiwg-rma mailing list >> >>> mpiwg-rma(a)lists.mpi-forum.org >> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> _______________________________________________ >> >> mpiwg-rma mailing list >> >> mpiwg-rma(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > _______________________________________________ >> > mpiwg-rma mailing list >> > mpiwg-rma(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] same_op_no_op
by Jim Dinan 14 Mar '14

14 Mar '14

Does same_op_no_op_replace allow MPI_NO_OP, MPI_REPLACE, MPI_SUM, *and* MPI_Compare_and_swap (i.e. to support shmem_*_cswap)? I don't recall if CAS is semantically a different op, or if it's considered to be a "replace". I had thought it was considered to be a different op. On Fri, Mar 14, 2014 at 10:11 AM, Jeff Hammond <jeff.science(a)gmail.com>wrote: > In that case impls can add new info key for same_op_no_op_replace_hardware > or something. > > SHMEM only needs same_op_no_op_replace as default but UPC appears to need > SUM and XOR to be permitted at the same time to be efficient at the UPC > runtime level. > > Jeff > > Sent from my iPhone > > > On Mar 14, 2014, at 8:37 AM, "Underwood, Keith D" < > keith.d.underwood(a)intel.com> wrote: > > > > The problem here is that some existing hardware supports some atomic > operations. Multiply is frequently not on that list. Doing an atomic add > on a non-coherent NIC and a multiply somewhere else can be challenging to > make correct, much less atomic. Now, if "all bets are off" in the > definition of not atomic (i.e. any interleaving of the two implied > load-op-store sequencings is legal), then I would argue that the > description you attribute to 2.2 is the better one. > > > >> -----Original Message----- > >> From: mpiwg-rma [mailto:[email protected]] On > >> Behalf Of Balaji, Pavan > >> Sent: Thursday, March 13, 2014 1:02 PM > >> To: MPI WG Remote Memory Access working group > >> Subject: Re: [mpiwg-rma] same_op_no_op > >> > >> > >> MPI-2.2 says that accumulate with different ops are not atomic. > >> > >> MPI-3 says that accumulate with different ops are not allowed (since > >> same_op_no_op is default). > >> > >> I think we screwed that up? > >> > >> - Pavan > >> > >> On Mar 13, 2014, at 11:48 AM, Jeff Hammond <jeff.science(a)gmail.com> > >> wrote: > >> > >>> It is extremely difficult to see that this is what the MPI-3 standard > says. > >>> > >>> First we have this: > >>> > >>> "The outcome of concurrent accumulate operations to the same location > >>> with the same predefined datatype is as if the accumulates were done > >>> at that location in some serial order. Additional restrictions on the > >>> operation apply; see the info key accumulate_ops in Section 11.2.1. > >>> Concurrent accumulate operations with different origin and target > >>> pairs are not ordered. Thus, there is no guarantee that the entire > >>> call to an accumulate operation is executed atomically. The effect of > >>> this lack of atomicity is limited: The previous correctness conditions > >>> imply that a location updated by a call to an accumulate operation > >>> cannot be accessed by a load or an RMA call other than accumulate > >>> until the accumulate operation has completed (at the target). > >>> Different interleavings can lead to different results only to the > >>> extent that computer arithmetics are not truly associative or > >>> commutative. The outcome of accumulate operations with overlapping > >>> types of different sizes or target displacements is undefined." > >>> [11.7.1 Atomicity] > >>> > >>> Then we have this: > >>> > >>> "accumulate_ops - if set to same_op, the implementation will assume > >>> that all concurrent accumulate calls to the same target address will > >>> use the same operation. If set to same_op_no_op, then the > >>> implementation will assume that all concurrent accumulate calls to the > >>> same target address will use the same operation or MPI_NO_OP. This can > >>> eliminate the need to protect access for certain operation types where > >>> the hardware can guarantee atomicity. The default is same_op_no_op." > >>> [11.2.1 Window Creation] > >>> > >>> I was not aware that the definition of info keys was normative, given > >>> that implementations are free to ignore them. Even if info key text > >>> is normative, one has to infer from the fact that same_op_no_op is the > >>> default info behavior - and thus RMA semantic - that accumulate > >>> atomicity is restricted to the case where one uses the same op or noop > >>> but not replace. > >>> > >>> The MPI-2.2 spec is unambiguous because it explicitly requires the > >>> same operation in 11.7.1 Atomicity. This text was removed in MPI-3.0 > >>> in favor of the info key text. > >>> > >>> Best, > >>> > >>> Jeff > >>> > >>>> On Tue, Mar 11, 2014 at 12:04 AM, Balaji, Pavan <balaji(a)anl.gov> > wrote: > >>>> > >>>> MPI-2 defines atomicity only for the same operation, not any operation > >> for MPI_ACCUMULATE. > >>>> > >>>> - Pavan > >>>> > >>>> On Mar 10, 2014, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> > >> wrote: > >>>> > >>>>> So MPI-2 denied compatibility between replace and not-replace? > >>>>> > >>>>> Jeff > >>>>> > >>>>> Sent from my iPhone > >>>>> > >>>>>> On Mar 11, 2014, at 12:06 AM, "Balaji, Pavan" <balaji(a)anl.gov> > wrote: > >>>>>> > >>>>>> > >>>>>> It doesn't break backward compatibility. The info argument is still > >> useful when you don't want to use replace. I don't see anything wrong > with > >> it. > >>>>>> > >>>>>>> On Mar 10, 2014, at 11:01 PM, Jeff Hammond > >> <jeff.science(a)gmail.com> wrote: > >>>>>>> > >>>>>>> Does this or does this not break BW compatibility w.r.t. MPI-2.2 > >>>>>>> and did we do it intentionally? Unless we did so intentionally > >>>>>>> and explicitly, I will argue that the WG screwed up and the info > >>>>>>> key+val is invalid. > >>>>>>> > >>>>>>> Jeff > >>>>>>> > >>>>>>>> On Mon, Mar 10, 2014 at 11:03 PM, Balaji, Pavan <balaji(a)anl.gov> > >> wrote: > >>>>>>>> > >>>>>>>> If a hardware can implement MPI_SUM, it should be able to > >> implement MPI_SUM with 0 as well. > >>>>>>>> > >>>>>>>> But that's not a generic solution. > >>>>>>>> > >>>>>>>> Jeff: at some point you were planning to bring in a ticket which > does > >> more combinations of operations than just same_op and no_op. Maybe it's > >> worthwhile bringing that up again? > >>>>>>>> > >>>>>>>> - Pavan > >>>>>>>> > >>>>>>>>> On Mar 10, 2014, at 9:26 PM, Jim Dinan <james.dinan(a)gmail.com> > >> wrote: > >>>>>>>>> > >>>>>>>>> Maybe there's a loophole that I'm forgetting? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Mon, Mar 10, 2014 at 9:43 PM, Jeff Hammond > >> <jeff.science(a)gmail.com> wrote: > >>>>>>>>> How the hell can I do GA or SHMEM then? Roll my own mutexes > >> and commit perf-suicide? > >>>>>>>>> > >>>>>>>>> Jeff > >>>>>>>>> > >>>>>>>>> Sent from my iPhone > >>>>>>>>> > >>>>>>>>>> On Mar 10, 2014, at 8:32 PM, Jim Dinan <james.dinan(a)gmail.com> > >> wrote: > >>>>>>>>>> > >>>>>>>>>> You can't use replace and sum concurrently at a given target > >> address. > >>>>>>>>>> > >>>>>>>>>> ~Jim. > >>>>>>>>>> > >>>>>>>>>> On Mon, Mar 10, 2014 at 4:30 PM, Jeff Hammond > >> <jeff.science(a)gmail.com> wrote: > >>>>>>>>>> Given the following, how do I use MPI_NO_OP, MPI_REPLACE > >> and > >>>>>>>>>> MPI_SUM in accumulate/atomic operations in a standard- > >> compliant way? > >>>>>>>>>> > >>>>>>>>>> accumulate_ops - if set to same_op, the implementation will > >>>>>>>>>> assume that all concurrent accumulate calls to the same target > >>>>>>>>>> address will use the same operation. If set to same_op_no_op, > >>>>>>>>>> then the implementation will assume that all concurrent > >>>>>>>>>> accumulate calls to the same target address will use the same > >>>>>>>>>> operation or MPI_NO_OP. This can eliminate the need to protect > >>>>>>>>>> access for certain operation types where the hardware can > >> guarantee atomicity. The default is same_op_no_op. > >>>>>>>>>> > >>>>>>>>>> We discuss this before and the resolution was not satisfying to > >> me. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> > >>>>>>>>>> Jeff > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Jeff Hammond > >>>>>>>>>> jeff.science(a)gmail.com > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> mpiwg-rma mailing list > >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>>>>> > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> mpiwg-rma mailing list > >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> mpiwg-rma mailing list > >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> mpiwg-rma mailing list > >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> mpiwg-rma mailing list > >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Jeff Hammond > >>>>>>> jeff.science(a)gmail.com > >>>>>>> _______________________________________________ > >>>>>>> mpiwg-rma mailing list > >>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>> > >>>>>> _______________________________________________ > >>>>>> mpiwg-rma mailing list > >>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>> _______________________________________________ > >>>>> mpiwg-rma mailing list > >>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>> > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> > >>> > >>> > >>> -- > >>> Jeff Hammond > >>> jeff.science(a)gmail.com > >>> _______________________________________________ > >>> mpiwg-rma mailing list > >>> mpiwg-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0