February 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Rolf Rabenseifner 04 Feb '14

04 Feb '14

Brian and all, No wording in the MPI-3.0 tells anything about the compiler problems. They are addressed by MPI_F_SYNC_REG (or work around with MPI_Get_address) to be sure that the store instruction is issued before I do any synchronization calls. If I must do an additional MPI_WIN_SYNC, when must it be done? My example is simple: X is part of a shared memory window and should mean the same memory location in both processes Process A Process B x=13 MPI_F_SYNC_REG(X) MPI_Barrier MPI_Barrier MPI_F_SYNC_REG(X) print X Where exactly do I need in which process an additional MPI_WIN_SYNC? Which wording in the MPI-3.0 does tell this need? Best regards Rolf ----- Original Message ----- > From: "Brian W Barrett" <bwbarre(a)sandia.gov> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> > Sent: Tuesday, February 4, 2014 6:06:02 PM > Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows > > On 2/4/14 9:59 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: > > >Jeff, > > > >thank you for the MPI_FREE_MEM hint. Yes, I'll fix it in my > >examples. > > > >About the synchronization problems: > >If I use shared memory windows and direct remote load and store > >instead of the RMA functions PUT or GET, > >is it then correct, when I never use MPI_WIN-synchronization > >routines? > > > >I would expect this, because in the unified RMA modell, > >the load and store accesses to the neighbors memory are done > >directly and MPI is not involved. > >Because of the unified RMA modell (must be for shared memory > >windows), > >there should be no need for cache flush routines. > >Correct? > > No. Maybe. Sometimes. > > Basically, unified is saying that the memory has the same eventually > completeness that users have come to expect with shared memory > programming. But what users have come to expect depends on the > architecture and the programming language. So in some processor / > language / compiler combinations, you might not need any > synchronization. > In other combinations, you might need compiler or processor memory > barriers in your code. MPI guarantees, however, that MPI_WIN_SYNC > will > implement whatever compiler / processor memory barriers you need, so > it > can be used as the portable hammer in your toolchest of shared memory > programming. > > If you're only using load/stores in shared memory windows, you should > not > need the other synchronization routines (FLUSH, etc.). > > Brian > > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Synchronization on shared memory windows
by Jeff Hammond 04 Feb '14

04 Feb '14

To get interprocess memory barrier, right? Because no RMA means no sync other than the obvious shared memory consistency business. Jeff On Tue, Feb 4, 2014 at 11:01 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Rolf, > > You still need MPI_WIN_SYNC in UNIFIED to ensure memory access ordering. > > — Pavan > > On Feb 4, 2014, at 10:59 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > >> Jeff, >> >> thank you for the MPI_FREE_MEM hint. Yes, I'll fix it in my examples. >> >> About the synchronization problems: >> If I use shared memory windows and direct remote load and store >> instead of the RMA functions PUT or GET, >> is it then correct, when I never use MPI_WIN-synchronization routines? >> >> I would expect this, because in the unified RMA modell, >> the load and store accesses to the neighbors memory are done >> directly and MPI is not involved. >> Because of the unified RMA modell (must be for shared memory windows), >> there should be no need for cache flush routines. >> Correct? >> >> Therefore, my halo_1sided_put_win_alloc_shared_othersync_20.f90 >> should be correct? >> >> Best regards >> Rolf >> >> ----- Original Message ----- >>> From: "Jeff Hammond" <jeff.science(a)gmail.com> >>> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >>> Sent: Tuesday, February 4, 2014 5:46:28 PM >>> Subject: Re: [mpiwg-rma] Synchronization on shared memory windows >>> >>> Using 2-sided with window memory is no different from load-store. >>> You >>> need to synchronize to maintain consistency if you are doing both RMA >>> and non-RMA on the window. If you merely use the WIN calls to >>> allocate memory and only access the private copy, you shouldn't need >>> any sync. >>> >>> I have not explored if your use of active-target sync is valid or not >>> because I think these modes are almost entirely useless and pay >>> almost >>> no attention to their semantics. This is a matter of opinion that no >>> one else is required to share, of course. >>> >>> Jeff >>> >>> On Tue, Feb 4, 2014 at 10:03 AM, Rolf Rabenseifner >>> <rabenseifner(a)hlrs.de> wrote: >>>> Dear all, >>>> >>>> I tested MPI-3.0 1-sided synchronization on shared memory windows. >>>> A lot didn't worked as I expected: >>>> >>>> - assertions fail with MPI_Win_fence >>>> (there is no restriction defined on MPI-3.0 p452:8-19), >>>> >>>> - Post-Start-Complete-Wait fully fails >>>> (compare MPI-3.0 p410:16-19, especially the ref to Sect.11.5), >>>> >>>> - and MPI_Free_mem also fails for the shared memory windows >>>> (compare MI-3.0 p409:23-24, especially that MPI_FREE_MEM >>>> is mentioned here). >>>> >>>> Attached are some files: >>>> - halo_1sided_put_win_alloc_20.f90 >>>> >>>> This is the basis and works. >>>> It is with normal (distributed) windows. >>>> >>>> - halo_1sided_put_win_alloc_shared_20.f90 >>>> >>>> It is the first shared memory example. >>>> It causes several errors with the test on our Cray system: >>>> - The assertions on MPI_WIN_FENCE do not work >>>> - The MPI_FREE_MEM does not work for the shared buffers >>>> >>>> Is my program wrong? it is a simple left and right >>>> 1-dim halo exchange. >>>> >>>> - halo_1sided_put_win_alloc_shared_20_w-a-cray.f90 >>>> >>>> This is a workaround that works on our Cray: >>>> - assertions MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED removed >>>> - MPI_FREE_MEM removed >>>> >>>> - halo_1sided_put_win_alloc_shared_pscw_20.f90 >>>> >>>> With Post-Start-Complete-Wait, nothing works! >>>> No work-around. >>>> >>>> - halo_1sided_put_win_alloc_shared_othersync_20.f90 >>>> >>>> In this example, I fully substituted the RMA synchronization >>>> by point-to-point synchronization with Irecv, send and waitall. >>>> Is this allowed? >>>> Was it intended? >>>> Is there any wording about in MPI-3.0? >>>> Will we have any wording in MPI-next? (3.1 or 4.0)? >>>> >>>> I hope, someone knows the answers. >>>> >>>> Best regards >>>> Rolf >>>> >>>> -- >>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>> rabenseifner(a)hlrs.de >>>> High Performance Computing Center (HLRS) . phone >>>> ++49(0)711/685-65530 >>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>> 685-65832 >>>> Head of Dpmt Parallel Computing . . . >>>> www.hlrs.de/people/rabenseifner >>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>> 1.307) >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Barrett, Brian W 04 Feb '14

04 Feb '14

On 2/4/14 9:59 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: >Jeff, > >thank you for the MPI_FREE_MEM hint. Yes, I'll fix it in my examples. > >About the synchronization problems: >If I use shared memory windows and direct remote load and store >instead of the RMA functions PUT or GET, >is it then correct, when I never use MPI_WIN-synchronization routines? > >I would expect this, because in the unified RMA modell, >the load and store accesses to the neighbors memory are done >directly and MPI is not involved. >Because of the unified RMA modell (must be for shared memory windows), >there should be no need for cache flush routines. >Correct? No. Maybe. Sometimes. Basically, unified is saying that the memory has the same eventually completeness that users have come to expect with shared memory programming. But what users have come to expect depends on the architecture and the programming language. So in some processor / language / compiler combinations, you might not need any synchronization. In other combinations, you might need compiler or processor memory barriers in your code. MPI guarantees, however, that MPI_WIN_SYNC will implement whatever compiler / processor memory barriers you need, so it can be used as the portable hammer in your toolchest of shared memory programming. If you're only using load/stores in shared memory windows, you should not need the other synchronization routines (FLUSH, etc.). Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] Synchronization on shared memory windows
by Balaji, Pavan 04 Feb '14

04 Feb '14

Rolf, You still need MPI_WIN_SYNC in UNIFIED to ensure memory access ordering. — Pavan On Feb 4, 2014, at 10:59 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Jeff, > > thank you for the MPI_FREE_MEM hint. Yes, I'll fix it in my examples. > > About the synchronization problems: > If I use shared memory windows and direct remote load and store > instead of the RMA functions PUT or GET, > is it then correct, when I never use MPI_WIN-synchronization routines? > > I would expect this, because in the unified RMA modell, > the load and store accesses to the neighbors memory are done > directly and MPI is not involved. > Because of the unified RMA modell (must be for shared memory windows), > there should be no need for cache flush routines. > Correct? > > Therefore, my halo_1sided_put_win_alloc_shared_othersync_20.f90 > should be correct? > > Best regards > Rolf > > ----- Original Message ----- >> From: "Jeff Hammond" <jeff.science(a)gmail.com> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >> Sent: Tuesday, February 4, 2014 5:46:28 PM >> Subject: Re: [mpiwg-rma] Synchronization on shared memory windows >> >> Using 2-sided with window memory is no different from load-store. >> You >> need to synchronize to maintain consistency if you are doing both RMA >> and non-RMA on the window. If you merely use the WIN calls to >> allocate memory and only access the private copy, you shouldn't need >> any sync. >> >> I have not explored if your use of active-target sync is valid or not >> because I think these modes are almost entirely useless and pay >> almost >> no attention to their semantics. This is a matter of opinion that no >> one else is required to share, of course. >> >> Jeff >> >> On Tue, Feb 4, 2014 at 10:03 AM, Rolf Rabenseifner >> <rabenseifner(a)hlrs.de> wrote: >>> Dear all, >>> >>> I tested MPI-3.0 1-sided synchronization on shared memory windows. >>> A lot didn't worked as I expected: >>> >>> - assertions fail with MPI_Win_fence >>> (there is no restriction defined on MPI-3.0 p452:8-19), >>> >>> - Post-Start-Complete-Wait fully fails >>> (compare MPI-3.0 p410:16-19, especially the ref to Sect.11.5), >>> >>> - and MPI_Free_mem also fails for the shared memory windows >>> (compare MI-3.0 p409:23-24, especially that MPI_FREE_MEM >>> is mentioned here). >>> >>> Attached are some files: >>> - halo_1sided_put_win_alloc_20.f90 >>> >>> This is the basis and works. >>> It is with normal (distributed) windows. >>> >>> - halo_1sided_put_win_alloc_shared_20.f90 >>> >>> It is the first shared memory example. >>> It causes several errors with the test on our Cray system: >>> - The assertions on MPI_WIN_FENCE do not work >>> - The MPI_FREE_MEM does not work for the shared buffers >>> >>> Is my program wrong? it is a simple left and right >>> 1-dim halo exchange. >>> >>> - halo_1sided_put_win_alloc_shared_20_w-a-cray.f90 >>> >>> This is a workaround that works on our Cray: >>> - assertions MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED removed >>> - MPI_FREE_MEM removed >>> >>> - halo_1sided_put_win_alloc_shared_pscw_20.f90 >>> >>> With Post-Start-Complete-Wait, nothing works! >>> No work-around. >>> >>> - halo_1sided_put_win_alloc_shared_othersync_20.f90 >>> >>> In this example, I fully substituted the RMA synchronization >>> by point-to-point synchronization with Irecv, send and waitall. >>> Is this allowed? >>> Was it intended? >>> Is there any wording about in MPI-3.0? >>> Will we have any wording in MPI-next? (3.1 or 4.0)? >>> >>> I hope, someone knows the answers. >>> >>> Best regards >>> Rolf >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>> rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone >>> ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>> 685-65832 >>> Head of Dpmt Parallel Computing . . . >>> www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>> 1.307) >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Synchronization on shared memory windows
by Rolf Rabenseifner 04 Feb '14

04 Feb '14

Jeff, thank you for the MPI_FREE_MEM hint. Yes, I'll fix it in my examples. About the synchronization problems: If I use shared memory windows and direct remote load and store instead of the RMA functions PUT or GET, is it then correct, when I never use MPI_WIN-synchronization routines? I would expect this, because in the unified RMA modell, the load and store accesses to the neighbors memory are done directly and MPI is not involved. Because of the unified RMA modell (must be for shared memory windows), there should be no need for cache flush routines. Correct? Therefore, my halo_1sided_put_win_alloc_shared_othersync_20.f90 should be correct? Best regards Rolf ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> > Sent: Tuesday, February 4, 2014 5:46:28 PM > Subject: Re: [mpiwg-rma] Synchronization on shared memory windows > > Using 2-sided with window memory is no different from load-store. > You > need to synchronize to maintain consistency if you are doing both RMA > and non-RMA on the window. If you merely use the WIN calls to > allocate memory and only access the private copy, you shouldn't need > any sync. > > I have not explored if your use of active-target sync is valid or not > because I think these modes are almost entirely useless and pay > almost > no attention to their semantics. This is a matter of opinion that no > one else is required to share, of course. > > Jeff > > On Tue, Feb 4, 2014 at 10:03 AM, Rolf Rabenseifner > <rabenseifner(a)hlrs.de> wrote: > > Dear all, > > > > I tested MPI-3.0 1-sided synchronization on shared memory windows. > > A lot didn't worked as I expected: > > > > - assertions fail with MPI_Win_fence > > (there is no restriction defined on MPI-3.0 p452:8-19), > > > > - Post-Start-Complete-Wait fully fails > > (compare MPI-3.0 p410:16-19, especially the ref to Sect.11.5), > > > > - and MPI_Free_mem also fails for the shared memory windows > > (compare MI-3.0 p409:23-24, especially that MPI_FREE_MEM > > is mentioned here). > > > > Attached are some files: > > - halo_1sided_put_win_alloc_20.f90 > > > > This is the basis and works. > > It is with normal (distributed) windows. > > > > - halo_1sided_put_win_alloc_shared_20.f90 > > > > It is the first shared memory example. > > It causes several errors with the test on our Cray system: > > - The assertions on MPI_WIN_FENCE do not work > > - The MPI_FREE_MEM does not work for the shared buffers > > > > Is my program wrong? it is a simple left and right > > 1-dim halo exchange. > > > > - halo_1sided_put_win_alloc_shared_20_w-a-cray.f90 > > > > This is a workaround that works on our Cray: > > - assertions MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED removed > > - MPI_FREE_MEM removed > > > > - halo_1sided_put_win_alloc_shared_pscw_20.f90 > > > > With Post-Start-Complete-Wait, nothing works! > > No work-around. > > > > - halo_1sided_put_win_alloc_shared_othersync_20.f90 > > > > In this example, I fully substituted the RMA synchronization > > by point-to-point synchronization with Irecv, send and waitall. > > Is this allowed? > > Was it intended? > > Is there any wording about in MPI-3.0? > > Will we have any wording in MPI-next? (3.1 or 4.0)? > > > > I hope, someone knows the answers. > > > > Best regards > > Rolf > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Synchronization on shared memory windows
by Jeff Hammond 04 Feb '14

04 Feb '14

Using 2-sided with window memory is no different from load-store. You need to synchronize to maintain consistency if you are doing both RMA and non-RMA on the window. If you merely use the WIN calls to allocate memory and only access the private copy, you shouldn't need any sync. I have not explored if your use of active-target sync is valid or not because I think these modes are almost entirely useless and pay almost no attention to their semantics. This is a matter of opinion that no one else is required to share, of course. Jeff On Tue, Feb 4, 2014 at 10:03 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dear all, > > I tested MPI-3.0 1-sided synchronization on shared memory windows. > A lot didn't worked as I expected: > > - assertions fail with MPI_Win_fence > (there is no restriction defined on MPI-3.0 p452:8-19), > > - Post-Start-Complete-Wait fully fails > (compare MPI-3.0 p410:16-19, especially the ref to Sect.11.5), > > - and MPI_Free_mem also fails for the shared memory windows > (compare MI-3.0 p409:23-24, especially that MPI_FREE_MEM > is mentioned here). > > Attached are some files: > - halo_1sided_put_win_alloc_20.f90 > > This is the basis and works. > It is with normal (distributed) windows. > > - halo_1sided_put_win_alloc_shared_20.f90 > > It is the first shared memory example. > It causes several errors with the test on our Cray system: > - The assertions on MPI_WIN_FENCE do not work > - The MPI_FREE_MEM does not work for the shared buffers > > Is my program wrong? it is a simple left and right > 1-dim halo exchange. > > - halo_1sided_put_win_alloc_shared_20_w-a-cray.f90 > > This is a workaround that works on our Cray: > - assertions MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED removed > - MPI_FREE_MEM removed > > - halo_1sided_put_win_alloc_shared_pscw_20.f90 > > With Post-Start-Complete-Wait, nothing works! > No work-around. > > - halo_1sided_put_win_alloc_shared_othersync_20.f90 > > In this example, I fully substituted the RMA synchronization > by point-to-point synchronization with Irecv, send and waitall. > Is this allowed? > Was it intended? > Is there any wording about in MPI-3.0? > Will we have any wording in MPI-next? (3.1 or 4.0)? > > I hope, someone knows the answers. > > Best regards > Rolf > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] Synchronization on shared memory windows
by Jeff Hammond 04 Feb '14

04 Feb '14

See page 415 ("If the window was created with MPI_WIN_ALLOCATE_SHARED, MPI_WIN_FREE will free the window memory that was allocated in MPI_WIN_ALLOCATE_SHARED.") regarding whether or not it is legal to MPI_FREE_MEM that which was allocated by MPI_WIN_ALLOCATE_SHARED. So, to be explicit, halo_1sided_put_win_alloc_20 and halo_1sided_put_win_alloc_shared_20 are incorrect programs. I didn't look at the others. Best, Jeff On Tue, Feb 4, 2014 at 10:03 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dear all, > > I tested MPI-3.0 1-sided synchronization on shared memory windows. > A lot didn't worked as I expected: > > - assertions fail with MPI_Win_fence > (there is no restriction defined on MPI-3.0 p452:8-19), > > - Post-Start-Complete-Wait fully fails > (compare MPI-3.0 p410:16-19, especially the ref to Sect.11.5), > > - and MPI_Free_mem also fails for the shared memory windows > (compare MI-3.0 p409:23-24, especially that MPI_FREE_MEM > is mentioned here). > > Attached are some files: > - halo_1sided_put_win_alloc_20.f90 > > This is the basis and works. > It is with normal (distributed) windows. > > - halo_1sided_put_win_alloc_shared_20.f90 > > It is the first shared memory example. > It causes several errors with the test on our Cray system: > - The assertions on MPI_WIN_FENCE do not work > - The MPI_FREE_MEM does not work for the shared buffers > > Is my program wrong? it is a simple left and right > 1-dim halo exchange. > > - halo_1sided_put_win_alloc_shared_20_w-a-cray.f90 > > This is a workaround that works on our Cray: > - assertions MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED removed > - MPI_FREE_MEM removed > > - halo_1sided_put_win_alloc_shared_pscw_20.f90 > > With Post-Start-Complete-Wait, nothing works! > No work-around. > > - halo_1sided_put_win_alloc_shared_othersync_20.f90 > > In this example, I fully substituted the RMA synchronization > by point-to-point synchronization with Irecv, send and waitall. > Is this allowed? > Was it intended? > Is there any wording about in MPI-3.0? > Will we have any wording in MPI-next? (3.1 or 4.0)? > > I hope, someone knows the answers. > > Best regards > Rolf > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] Synchronization on shared memory windows
by Jeff Hammond 04 Feb '14

04 Feb '14

I can only assume you tested a particular implementation of MPI-3, since the MPI standard itself does not execute on any platform of which I know. Can I assume you have identified bugs or semantic misinterpretations present in CrayMPI, which is a derivative of MPICH? If so, I recommend replacing the MPI-3 RMA working group with the Cray and MPICH developers. Best, Jeff On Tue, Feb 4, 2014 at 10:03 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dear all, > > I tested MPI-3.0 1-sided synchronization on shared memory windows. > A lot didn't worked as I expected: > > - assertions fail with MPI_Win_fence > (there is no restriction defined on MPI-3.0 p452:8-19), > > - Post-Start-Complete-Wait fully fails > (compare MPI-3.0 p410:16-19, especially the ref to Sect.11.5), > > - and MPI_Free_mem also fails for the shared memory windows > (compare MI-3.0 p409:23-24, especially that MPI_FREE_MEM > is mentioned here). > > Attached are some files: > - halo_1sided_put_win_alloc_20.f90 > > This is the basis and works. > It is with normal (distributed) windows. > > - halo_1sided_put_win_alloc_shared_20.f90 > > It is the first shared memory example. > It causes several errors with the test on our Cray system: > - The assertions on MPI_WIN_FENCE do not work > - The MPI_FREE_MEM does not work for the shared buffers > > Is my program wrong? it is a simple left and right > 1-dim halo exchange. > > - halo_1sided_put_win_alloc_shared_20_w-a-cray.f90 > > This is a workaround that works on our Cray: > - assertions MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED removed > - MPI_FREE_MEM removed > > - halo_1sided_put_win_alloc_shared_pscw_20.f90 > > With Post-Start-Complete-Wait, nothing works! > No work-around. > > - halo_1sided_put_win_alloc_shared_othersync_20.f90 > > In this example, I fully substituted the RMA synchronization > by point-to-point synchronization with Irecv, send and waitall. > Is this allowed? > Was it intended? > Is there any wording about in MPI-3.0? > Will we have any wording in MPI-next? (3.1 or 4.0)? > > I hope, someone knows the answers. > > Best regards > Rolf > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Synchronization on shared memory windows
by Rolf Rabenseifner 04 Feb '14

04 Feb '14

Dear all, I tested MPI-3.0 1-sided synchronization on shared memory windows. A lot didn't worked as I expected: - assertions fail with MPI_Win_fence (there is no restriction defined on MPI-3.0 p452:8-19), - Post-Start-Complete-Wait fully fails (compare MPI-3.0 p410:16-19, especially the ref to Sect.11.5), - and MPI_Free_mem also fails for the shared memory windows (compare MI-3.0 p409:23-24, especially that MPI_FREE_MEM is mentioned here). Attached are some files: - halo_1sided_put_win_alloc_20.f90 This is the basis and works. It is with normal (distributed) windows. - halo_1sided_put_win_alloc_shared_20.f90 It is the first shared memory example. It causes several errors with the test on our Cray system: - The assertions on MPI_WIN_FENCE do not work - The MPI_FREE_MEM does not work for the shared buffers Is my program wrong? it is a simple left and right 1-dim halo exchange. - halo_1sided_put_win_alloc_shared_20_w-a-cray.f90 This is a workaround that works on our Cray: - assertions MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED removed - MPI_FREE_MEM removed - halo_1sided_put_win_alloc_shared_pscw_20.f90 With Post-Start-Complete-Wait, nothing works! No work-around. - halo_1sided_put_win_alloc_shared_othersync_20.f90 In this example, I fully substituted the RMA synchronization by point-to-point synchronization with Irecv, send and waitall. Is this allowed? Was it intended? Is there any wording about in MPI-3.0? Will we have any wording in MPI-next? (3.1 or 4.0)? I hope, someone knows the answers. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0