February 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Balaji, Pavan 05 Feb '14

05 Feb '14

Yes, this is a correct program. — Pavan On Feb 5, 2014, at 10:30 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Jeff and all, > > it looks like that it works as MPI-3 is designed: > > I need to add once at begin a MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, win) > and once at end a MPI_WIN_UNLOCK_ALL(win) > and then all works fine with MPI_WIN_SYNC in each iteration. > > Is this usage consistent with the definition in the MPI-3 standard? > > Here the total scenario that I use: > > -------------------- > X is part of a shared memory window and should mean the same > memory location in both processes > > Process A Process B > > MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( > MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) > > DO ... DO ... > x=... > MPI_F_SYNC_REG(X) > MPI_Barrier MPI_Barrier > MPI_F_SYNC_REG(X) > print X > END DO END DO > > MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) > > -------------------- > > Best regards > Rolf > > > > ----- Original Message ----- >> From: "Jeff Hammond" <jeff.science(a)gmail.com> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >> Sent: Tuesday, February 4, 2014 7:42:58 PM >> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows >> >> "For the purposes of synchronizing the private and public window, >> MPI_WIN_SYNC has the effect of ending and reopening an access and >> exposure epoch on the window (note that it does not actually end an >> epoch or complete any pending MPI RMA operations)." >> >> I think this is interpreted to mean that this call is only valid >> inside of an existing epoch and thus if you want to call it, you need >> to use it inside of a passive-target epoch. Thus, it is not merely a >> portable abstraction for a memory barrier. >> >> I think we should fix MPICH and/or MPI-Next to allow the more general >> use such that your code is standard-compliant and executes correctly. >> >> I await violent disagreement from others :-) >> >> Jeff >> >> On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner >> <rabenseifner(a)hlrs.de> wrote: >>> Brian, Pavan, and Jeff, >>> >>> you convinced me. I did it, see attached file, and my mpich based >>> Cray lib tells >>> >>> Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in >>> MPI_Win_sync: Wrong synchronization of RMA calls , error stack: >>> MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed >>> MPIDI_Win_sync(2495): Wrong synchronization of RMA calls >>> >>> (only once in each process). >>> >>> I expect, that this is now an implementation bug that should be >>> fixed by mpich and cray? >>> >>> Best regards >>> Rolf >>> >>> ----- Original Message ----- >>>> From: "Brian W Barrett" <bwbarre(a)sandia.gov> >>>> To: "MPI WG Remote Memory Access working group" >>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>>> <longb(a)cray.com> >>>> Sent: Tuesday, February 4, 2014 7:09:02 PM >>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared >>>> memory windows >>>> >>>> On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >>>> wrote: >>>> >>>>> The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) >>>>> has no meaning in the unified memory model if all accesses >>>>> are done without RMA routines. >>>>> It has only a meaning if different public and privat copy is >>>>> there (MPI-3.0 p450:46-p451:2). >>>>> MPI-3.0 p456:3 - p457:7 define the rules for the unified memory >>>>> model >>>>> but there is no need to use MPI_WIN_SYNC. >>>> >>>> Right, there's no need from an MPI point of view, but that doesn't >>>> mean >>>> that the language/compiler/processor doesn't have a need for extra >>>> synchronization. >>>> >>>>> The combination of X=13 and MPI_F_SYNC_REG(X) >>>>> before MPI_Barrier should guarantee that all bytes of X are >>>>> stored in memory. The same should be valid in C, >>>>> because the C compiler has no chance to see whether >>>>> MPI_Barrier will access the bytes of X or not. >>>>> And if it is guaranteed to be in the unified memory, >>>>> then the other process (B) should be able to correctly >>>>> read the data after the return from its barrier. >>>>> >>>>> What is wrong with my thinking? >>>>> Which detail do I miss? >>>> >>>> According to my reading of the spec, MPI_F_SYNC_REG only prevents >>>> the >>>> language/compiler from moving the store, but does not say anything >>>> about >>>> processor ordering. So the WIN_SYNC in my last e-mail will add >>>> the >>>> processor memory barrier, which will give you all the semantics >>>> you >>>> need. >>>> >>>> Shared memory programming is a disaster in most languages today, >>>> so >>>> we >>>> decided to pass that disaster on to the user. We really can't >>>> help, >>>> without adding lots of overhead (ie, using put/get/rma >>>> synchronization). >>>> So if a user already knows how to do shared memory programming, >>>> this >>>> will >>>> feel natural. If they don't, it's going to hurt badly :/. >>>> >>>> >>>> Brian >>>> >>>> -- >>>> Brian W. Barrett >>>> Scalable System Software Group >>>> Sandia National Laboratories >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>> rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone >>> ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>> 685-65832 >>> Head of Dpmt Parallel Computing . . . >>> www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>> 1.307) >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Rolf Rabenseifner 05 Feb '14

05 Feb '14

Jeff and all, it looks like that it works as MPI-3 is designed: I need to add once at begin a MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, win) and once at end a MPI_WIN_UNLOCK_ALL(win) and then all works fine with MPI_WIN_SYNC in each iteration. Is this usage consistent with the definition in the MPI-3 standard? Here the total scenario that I use: -------------------- X is part of a shared memory window and should mean the same memory location in both processes Process A Process B MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) DO ... DO ... x=... MPI_F_SYNC_REG(X) MPI_Barrier MPI_Barrier MPI_F_SYNC_REG(X) print X END DO END DO MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) -------------------- Best regards Rolf ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> > Sent: Tuesday, February 4, 2014 7:42:58 PM > Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows > > "For the purposes of synchronizing the private and public window, > MPI_WIN_SYNC has the effect of ending and reopening an access and > exposure epoch on the window (note that it does not actually end an > epoch or complete any pending MPI RMA operations)." > > I think this is interpreted to mean that this call is only valid > inside of an existing epoch and thus if you want to call it, you need > to use it inside of a passive-target epoch. Thus, it is not merely a > portable abstraction for a memory barrier. > > I think we should fix MPICH and/or MPI-Next to allow the more general > use such that your code is standard-compliant and executes correctly. > > I await violent disagreement from others :-) > > Jeff > > On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner > <rabenseifner(a)hlrs.de> wrote: > > Brian, Pavan, and Jeff, > > > > you convinced me. I did it, see attached file, and my mpich based > > Cray lib tells > > > > Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in > > MPI_Win_sync: Wrong synchronization of RMA calls , error stack: > > MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed > > MPIDI_Win_sync(2495): Wrong synchronization of RMA calls > > > > (only once in each process). > > > > I expect, that this is now an implementation bug that should be > > fixed by mpich and cray? > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "Brian W Barrett" <bwbarre(a)sandia.gov> > >> To: "MPI WG Remote Memory Access working group" > >> <mpiwg-rma(a)lists.mpi-forum.org> > >> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" > >> <longb(a)cray.com> > >> Sent: Tuesday, February 4, 2014 7:09:02 PM > >> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared > >> memory windows > >> > >> On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > >> wrote: > >> > >> >The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) > >> >has no meaning in the unified memory model if all accesses > >> >are done without RMA routines. > >> >It has only a meaning if different public and privat copy is > >> >there (MPI-3.0 p450:46-p451:2). > >> >MPI-3.0 p456:3 - p457:7 define the rules for the unified memory > >> >model > >> >but there is no need to use MPI_WIN_SYNC. > >> > >> Right, there's no need from an MPI point of view, but that doesn't > >> mean > >> that the language/compiler/processor doesn't have a need for extra > >> synchronization. > >> > >> >The combination of X=13 and MPI_F_SYNC_REG(X) > >> >before MPI_Barrier should guarantee that all bytes of X are > >> >stored in memory. The same should be valid in C, > >> >because the C compiler has no chance to see whether > >> >MPI_Barrier will access the bytes of X or not. > >> >And if it is guaranteed to be in the unified memory, > >> >then the other process (B) should be able to correctly > >> >read the data after the return from its barrier. > >> > > >> >What is wrong with my thinking? > >> >Which detail do I miss? > >> > >> According to my reading of the spec, MPI_F_SYNC_REG only prevents > >> the > >> language/compiler from moving the store, but does not say anything > >> about > >> processor ordering. So the WIN_SYNC in my last e-mail will add > >> the > >> processor memory barrier, which will give you all the semantics > >> you > >> need. > >> > >> Shared memory programming is a disaster in most languages today, > >> so > >> we > >> decided to pass that disaster on to the user. We really can't > >> help, > >> without adding lots of overhead (ie, using put/get/rma > >> synchronization). > >> So if a user already knows how to do shared memory programming, > >> this > >> will > >> feel natural. If they don't, it's going to hurt badly :/. > >> > >> > >> Brian > >> > >> -- > >> Brian W. Barrett > >> Scalable System Software Group > >> Sandia National Laboratories > >> > >> > >> > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Balaji, Pavan 04 Feb '14

04 Feb '14

On Feb 4, 2014, at 12:34 PM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > you convinced me. I did it, see attached file, and my mpich based Cray lib tells Any chance this can be rewritten in C/C++? Also, better to drop off from the MPI WG mailing list at this point. — Pavan

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Jeff Hammond 04 Feb '14

04 Feb '14

"For the purposes of synchronizing the private and public window, MPI_WIN_SYNC has the effect of ending and reopening an access and exposure epoch on the window (note that it does not actually end an epoch or complete any pending MPI RMA operations)." I think this is interpreted to mean that this call is only valid inside of an existing epoch and thus if you want to call it, you need to use it inside of a passive-target epoch. Thus, it is not merely a portable abstraction for a memory barrier. I think we should fix MPICH and/or MPI-Next to allow the more general use such that your code is standard-compliant and executes correctly. I await violent disagreement from others :-) Jeff On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Brian, Pavan, and Jeff, > > you convinced me. I did it, see attached file, and my mpich based Cray lib tells > > Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in MPI_Win_sync: Wrong synchronization of RMA calls , error stack: > MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed > MPIDI_Win_sync(2495): Wrong synchronization of RMA calls > > (only once in each process). > > I expect, that this is now an implementation bug that should be > fixed by mpich and cray? > > Best regards > Rolf > > ----- Original Message ----- >> From: "Brian W Barrett" <bwbarre(a)sandia.gov> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >> Sent: Tuesday, February 4, 2014 7:09:02 PM >> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows >> >> On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: >> >> >The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) >> >has no meaning in the unified memory model if all accesses >> >are done without RMA routines. >> >It has only a meaning if different public and privat copy is >> >there (MPI-3.0 p450:46-p451:2). >> >MPI-3.0 p456:3 - p457:7 define the rules for the unified memory >> >model >> >but there is no need to use MPI_WIN_SYNC. >> >> Right, there's no need from an MPI point of view, but that doesn't >> mean >> that the language/compiler/processor doesn't have a need for extra >> synchronization. >> >> >The combination of X=13 and MPI_F_SYNC_REG(X) >> >before MPI_Barrier should guarantee that all bytes of X are >> >stored in memory. The same should be valid in C, >> >because the C compiler has no chance to see whether >> >MPI_Barrier will access the bytes of X or not. >> >And if it is guaranteed to be in the unified memory, >> >then the other process (B) should be able to correctly >> >read the data after the return from its barrier. >> > >> >What is wrong with my thinking? >> >Which detail do I miss? >> >> According to my reading of the spec, MPI_F_SYNC_REG only prevents the >> language/compiler from moving the store, but does not say anything >> about >> processor ordering. So the WIN_SYNC in my last e-mail will add the >> processor memory barrier, which will give you all the semantics you >> need. >> >> Shared memory programming is a disaster in most languages today, so >> we >> decided to pass that disaster on to the user. We really can't help, >> without adding lots of overhead (ie, using put/get/rma >> synchronization). >> So if a user already knows how to do shared memory programming, this >> will >> feel natural. If they don't, it's going to hurt badly :/. >> >> >> Brian >> >> -- >> Brian W. Barrett >> Scalable System Software Group >> Sandia National Laboratories >> >> >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Rolf Rabenseifner 04 Feb '14

04 Feb '14

Brian, Pavan, and Jeff, you convinced me. I did it, see attached file, and my mpich based Cray lib tells Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in MPI_Win_sync: Wrong synchronization of RMA calls , error stack: MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed MPIDI_Win_sync(2495): Wrong synchronization of RMA calls (only once in each process). I expect, that this is now an implementation bug that should be fixed by mpich and cray? Best regards Rolf ----- Original Message ----- > From: "Brian W Barrett" <bwbarre(a)sandia.gov> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> > Sent: Tuesday, February 4, 2014 7:09:02 PM > Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows > > On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: > > >The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) > >has no meaning in the unified memory model if all accesses > >are done without RMA routines. > >It has only a meaning if different public and privat copy is > >there (MPI-3.0 p450:46-p451:2). > >MPI-3.0 p456:3 - p457:7 define the rules for the unified memory > >model > >but there is no need to use MPI_WIN_SYNC. > > Right, there's no need from an MPI point of view, but that doesn't > mean > that the language/compiler/processor doesn't have a need for extra > synchronization. > > >The combination of X=13 and MPI_F_SYNC_REG(X) > >before MPI_Barrier should guarantee that all bytes of X are > >stored in memory. The same should be valid in C, > >because the C compiler has no chance to see whether > >MPI_Barrier will access the bytes of X or not. > >And if it is guaranteed to be in the unified memory, > >then the other process (B) should be able to correctly > >read the data after the return from its barrier. > > > >What is wrong with my thinking? > >Which detail do I miss? > > According to my reading of the spec, MPI_F_SYNC_REG only prevents the > language/compiler from moving the store, but does not say anything > about > processor ordering. So the WIN_SYNC in my last e-mail will add the > processor memory barrier, which will give you all the semantics you > need. > > Shared memory programming is a disaster in most languages today, so > we > decided to pass that disaster on to the user. We really can't help, > without adding lots of overhead (ie, using put/get/rma > synchronization). > So if a user already knows how to do shared memory programming, this > will > feel natural. If they don't, it's going to hurt badly :/. > > > Brian > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Barrett, Brian W 04 Feb '14

04 Feb '14

On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: >The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) >has no meaning in the unified memory model if all accesses >are done without RMA routines. >It has only a meaning if different public and privat copy is >there (MPI-3.0 p450:46-p451:2). >MPI-3.0 p456:3 - p457:7 define the rules for the unified memory model >but there is no need to use MPI_WIN_SYNC. Right, there's no need from an MPI point of view, but that doesn't mean that the language/compiler/processor doesn't have a need for extra synchronization. >The combination of X=13 and MPI_F_SYNC_REG(X) >before MPI_Barrier should guarantee that all bytes of X are >stored in memory. The same should be valid in C, >because the C compiler has no chance to see whether >MPI_Barrier will access the bytes of X or not. >And if it is guaranteed to be in the unified memory, >then the other process (B) should be able to correctly >read the data after the return from its barrier. > >What is wrong with my thinking? >Which detail do I miss? According to my reading of the spec, MPI_F_SYNC_REG only prevents the language/compiler from moving the store, but does not say anything about processor ordering. So the WIN_SYNC in my last e-mail will add the processor memory barrier, which will give you all the semantics you need. Shared memory programming is a disaster in most languages today, so we decided to pass that disaster on to the user. We really can't help, without adding lots of overhead (ie, using put/get/rma synchronization). So if a user already knows how to do shared memory programming, this will feel natural. If they don't, it's going to hurt badly :/. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Balaji, Pavan 04 Feb '14

04 Feb '14

On Feb 4, 2014, at 12:01 PM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) > has no meaning in the unified memory model if all accesses > are done without RMA routines. No. On shared memory windows, if you are using load/store, you still need WIN_SYNC. > It has only a meaning if different public and privat copy is > there (MPI-3.0 p450:46-p451:2). > MPI-3.0 p456:3 - p457:7 define the rules for the unified memory model > but there is no need to use MPI_WIN_SYNC. There probably needs to be some tweaks to the wording, but the intention was that the accesses will become visible “eventually”, not “immediately”. If you want them to become visible immediately, you need to do a WIN_SYNC. — Pavan

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Barrett, Brian W 04 Feb '14

04 Feb '14

On 2/4/14 10:39 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: >Brian and all, > >No wording in the MPI-3.0 tells anything about the compiler problems. >They are addressed by MPI_F_SYNC_REG (or work around with >MPI_Get_address) >to be sure that the store instruction is issued before I do any >synchronization calls. Jeff already pointed out the text that's important here. It's also important to note that we don't explicitly say how to program shared memory directly because it's not something we can standardize on because of how platform, compiler, and language dependent it is. >If I must do an additional MPI_WIN_SYNC, when must it be done? It's hard to say, but generally any time you want 1) ordering of reads and writes or 2) to force the compiler/language to re-issue a load. >My example is simple: >X is part of a shared memory window and should mean the same >memory location in both processes > >Process A Process B > >x=13 >MPI_F_SYNC_REG(X) >MPI_Barrier MPI_Barrier > MPI_F_SYNC_REG(X) > print X > >Where exactly do I need in which process an additional MPI_WIN_SYNC? If you wanted to be absolutely sure that print X works, you would need a MPI_WIN_SYNC after MPI_F_SYNC_REG on process A (to make sure the store isn't hoisted after the barrier) and another MPI_WIN_SYNC before MPI_F_SYNC_REG on process B (to make sure the load isn't hoisted to before the barrier). Note that the hoisting / reordering can occur at the compiler or processor level and nothing in MPI_Barrier requires a processor memory barrier, which is why it's not to assume it will do the right thing. Now, on x86, your code will likely work as written because the processor is pretty much memory ordered and the SYNC_REG / Barrier will keep the compiler from moving the store or load. >Which wording in the MPI-3.0 does tell this need? None. This is in the realm of undefined behavior for better performance (because it's so processor/language/compiler specific). Note that we do define a mechanism you can use (put/get/synchronization), at a cost in performance. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Rolf Rabenseifner 04 Feb '14

04 Feb '14

The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) has no meaning in the unified memory model if all accesses are done without RMA routines. It has only a meaning if different public and privat copy is there (MPI-3.0 p450:46-p451:2). MPI-3.0 p456:3 - p457:7 define the rules for the unified memory model but there is no need to use MPI_WIN_SYNC. The combination of X=13 and MPI_F_SYNC_REG(X) before MPI_Barrier should guarantee that all bytes of X are stored in memory. The same should be valid in C, because the C compiler has no chance to see whether MPI_Barrier will access the bytes of X or not. And if it is guaranteed to be in the unified memory, then the other process (B) should be able to correctly read the data after the return from its barrier. What is wrong with my thinking? Which detail do I miss? Best regards Rolf ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> > Sent: Tuesday, February 4, 2014 6:42:52 PM > Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows > > On Tue, Feb 4, 2014 at 11:39 AM, Rolf Rabenseifner > <rabenseifner(a)hlrs.de> wrote: > > Brian and all, > > > > No wording in the MPI-3.0 tells anything about the compiler > > problems. > > They are addressed by MPI_F_SYNC_REG (or work around with > > MPI_Get_address) > > to be sure that the store instruction is issued before I do any > > synchronization calls. > > See MPI-3 Section 11.7.4. This is not new text. It's been there for > years. > > > If I must do an additional MPI_WIN_SYNC, when must it be done? > > Whenever you might have a race condition due to two threads or > processes accessing the same memory via: > - RMA + load/store > - load/store on shared memory windows > - RMA + non-RMA > > This may not be an inclusive list. > > Jeff > > > My example is simple: > > X is part of a shared memory window and should mean the same > > memory location in both processes > > > > Process A Process B > > > > x=13 > > MPI_F_SYNC_REG(X) > > MPI_Barrier MPI_Barrier > > MPI_F_SYNC_REG(X) > > print X > > > > Where exactly do I need in which process an additional > > MPI_WIN_SYNC? > > Which wording in the MPI-3.0 does tell this need? > > > > Best regards > > Rolf > > > > > > > > ----- Original Message ----- > >> From: "Brian W Barrett" <bwbarre(a)sandia.gov> > >> To: "MPI WG Remote Memory Access working group" > >> <mpiwg-rma(a)lists.mpi-forum.org> > >> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" > >> <longb(a)cray.com> > >> Sent: Tuesday, February 4, 2014 6:06:02 PM > >> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared > >> memory windows > >> > >> On 2/4/14 9:59 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > >> wrote: > >> > >> >Jeff, > >> > > >> >thank you for the MPI_FREE_MEM hint. Yes, I'll fix it in my > >> >examples. > >> > > >> >About the synchronization problems: > >> >If I use shared memory windows and direct remote load and store > >> >instead of the RMA functions PUT or GET, > >> >is it then correct, when I never use MPI_WIN-synchronization > >> >routines? > >> > > >> >I would expect this, because in the unified RMA modell, > >> >the load and store accesses to the neighbors memory are done > >> >directly and MPI is not involved. > >> >Because of the unified RMA modell (must be for shared memory > >> >windows), > >> >there should be no need for cache flush routines. > >> >Correct? > >> > >> No. Maybe. Sometimes. > >> > >> Basically, unified is saying that the memory has the same > >> eventually > >> completeness that users have come to expect with shared memory > >> programming. But what users have come to expect depends on the > >> architecture and the programming language. So in some processor / > >> language / compiler combinations, you might not need any > >> synchronization. > >> In other combinations, you might need compiler or processor memory > >> barriers in your code. MPI guarantees, however, that MPI_WIN_SYNC > >> will > >> implement whatever compiler / processor memory barriers you need, > >> so > >> it > >> can be used as the portable hammer in your toolchest of shared > >> memory > >> programming. > >> > >> If you're only using load/stores in shared memory windows, you > >> should > >> not > >> need the other synchronization routines (FLUSH, etc.). > >> > >> Brian > >> > >> > >> -- > >> Brian W. Barrett > >> Scalable System Software Group > >> Sandia National Laboratories > >> > >> > >> > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Jeff Hammond 04 Feb '14

04 Feb '14

On Tue, Feb 4, 2014 at 11:39 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Brian and all, > > No wording in the MPI-3.0 tells anything about the compiler problems. > They are addressed by MPI_F_SYNC_REG (or work around with MPI_Get_address) > to be sure that the store instruction is issued before I do any > synchronization calls. See MPI-3 Section 11.7.4. This is not new text. It's been there for years. > If I must do an additional MPI_WIN_SYNC, when must it be done? Whenever you might have a race condition due to two threads or processes accessing the same memory via: - RMA + load/store - load/store on shared memory windows - RMA + non-RMA This may not be an inclusive list. Jeff > My example is simple: > X is part of a shared memory window and should mean the same > memory location in both processes > > Process A Process B > > x=13 > MPI_F_SYNC_REG(X) > MPI_Barrier MPI_Barrier > MPI_F_SYNC_REG(X) > print X > > Where exactly do I need in which process an additional MPI_WIN_SYNC? > Which wording in the MPI-3.0 does tell this need? > > Best regards > Rolf > > > > ----- Original Message ----- >> From: "Brian W Barrett" <bwbarre(a)sandia.gov> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >> Sent: Tuesday, February 4, 2014 6:06:02 PM >> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows >> >> On 2/4/14 9:59 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: >> >> >Jeff, >> > >> >thank you for the MPI_FREE_MEM hint. Yes, I'll fix it in my >> >examples. >> > >> >About the synchronization problems: >> >If I use shared memory windows and direct remote load and store >> >instead of the RMA functions PUT or GET, >> >is it then correct, when I never use MPI_WIN-synchronization >> >routines? >> > >> >I would expect this, because in the unified RMA modell, >> >the load and store accesses to the neighbors memory are done >> >directly and MPI is not involved. >> >Because of the unified RMA modell (must be for shared memory >> >windows), >> >there should be no need for cache flush routines. >> >Correct? >> >> No. Maybe. Sometimes. >> >> Basically, unified is saying that the memory has the same eventually >> completeness that users have come to expect with shared memory >> programming. But what users have come to expect depends on the >> architecture and the programming language. So in some processor / >> language / compiler combinations, you might not need any >> synchronization. >> In other combinations, you might need compiler or processor memory >> barriers in your code. MPI guarantees, however, that MPI_WIN_SYNC >> will >> implement whatever compiler / processor memory barriers you need, so >> it >> can be used as the portable hammer in your toolchest of shared memory >> programming. >> >> If you're only using load/stores in shared memory windows, you should >> not >> need the other synchronization routines (FLUSH, etc.). >> >> Brian >> >> >> -- >> Brian W. Barrett >> Scalable System Software Group >> Sandia National Laboratories >> >> >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0