February 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Jeff Hammond 05 Feb '14

05 Feb '14

On Wed, Feb 5, 2014 at 11:57 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > On Feb 5, 2014, at 11:54 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> I think advice to users regarding WIN_SYNC wouldn't hurt given that >> some of us are talking about it as if it is a portable abstraction for >> a memory barrier when it is now only correct to use it within an RMA >> sync epoch. Or we need to relax this restriction. > > Without this restriction, there’s no connection to what memory is being made consistent with the WIN_SYNC call. I’d be against removing it. The memory being made consistent is obviously the public and private window. In the case of UNIFIED, it's just a memory barrier. Right? I suppose someone could always WIN_LOCK+WIN_SYNC+WIN_UNLOCK as a portable abstraction for a memory barrier in the case where an epoch is not already established. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Balaji, Pavan 05 Feb '14

05 Feb '14

On Feb 5, 2014, at 11:54 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I think advice to users regarding WIN_SYNC wouldn't hurt given that > some of us are talking about it as if it is a portable abstraction for > a memory barrier when it is now only correct to use it within an RMA > sync epoch. Or we need to relax this restriction. Without this restriction, there’s no connection to what memory is being made consistent with the WIN_SYNC call. I’d be against removing it. — Pavan

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Jeff Hammond 05 Feb '14

05 Feb '14

I think advice to users regarding WIN_SYNC wouldn't hurt given that some of us are talking about it as if it is a portable abstraction for a memory barrier when it is now only correct to use it within an RMA sync epoch. Or we need to relax this restriction. Jeff On Wed, Feb 5, 2014 at 11:53 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > It’s correct now. > > In the MPI standard, we have stayed away from tutorial-style material. But I’m personally not against it. > > — Pavan > > On Feb 5, 2014, at 11:43 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > >> Pavan, Dave, Jeff, and Brian, >> >> is it now correct? >> Should it go in MPI-next into Sect.11.7? >> >>> -------------------- >>> X is part of a shared memory window and should mean the same >>> memory location in both processes produced with >>> MPI_WIN_ALLOCATE_SHARED. >>> >>> Process A Process B >>> >>> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >>> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >>> >>> DO ... DO ... >>> x=... >>> MPI_F_SYNC_REG(X) >>> MPI_WIN_SYNC(win) >>> MPI_Barrier MPI_Barrier >>> MPI_WIN_SYNC(win) >>> MPI_F_SYNC_REG(X) >>> print X >>> END DO END DO >>> >>> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >>> -------------------- >> >> Best regards >> Rolf >> >> ----- Original Message ----- >>> From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >>> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >>> Sent: Wednesday, February 5, 2014 5:59:03 PM >>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows >>> >>> That was my fault. Here the program that should be examined, >>> whether it is correct according to MPI-3.0: >>> >>> -------------------- >>> X is part of a shared memory window and should mean the same >>> memory location in both processes produced with >>> MPI_WIN_ALLOCATE_SHARED. >>> >>> Process A Process B >>> >>> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >>> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >>> >>> DO ... DO ... >>> x=... >>> MPI_F_SYNC_REG(X) >>> MPI_WIN_SYNC(win) >>> MPI_Barrier MPI_Barrier >>> MPI_WIN_SYNC(win) >>> MPI_F_SYNC_REG(X) >>> print X >>> END DO END DO >>> >>> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >>> -------------------- >>> >>> Is it now correct according to MPI-3.0? >>> And perhaps also according other rules for >>> real shared memory programming? >>> >>> Would it be helpful to add it to at the end of Sect.11.7 >>> because it would definitely clarify the rules >>> how to use shared memory windows. >>> >>> Best regards >>> Rolf >>> >>> ----- Original Message ----- >>>> From: "Pavan Balaji" <balaji(a)anl.gov> >>>> To: "MPI WG Remote Memory Access working group" >>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>>> <longb(a)cray.com> >>>> Sent: Wednesday, February 5, 2014 5:46:25 PM >>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared >>>> memory windows >>>> >>>> >>>> Whoops, I read MPI_F_SYNC_REG as MPI_WIN_SYNC. There need to be >>>> WIN_SYNCs on both processes. >>>> >>>> — Pavan >>>> >>>> On Feb 5, 2014, at 10:40 AM, Dave Goodell (dgoodell) >>>> <dgoodell(a)cisco.com> wrote: >>>> >>>>> Pavan, is it? >>>>> >>>>> Rolf, here is the supposed MPI_WIN_SYNC call? I assume you meant >>>>> to put it between the MPI_F_SYNC_REG and MPI_Barrier in both >>>>> processes? >>>>> >>>>> -Dave >>>>> >>>>> On Feb 5, 2014, at 10:31 AM, "Balaji, Pavan" <balaji(a)anl.gov> >>>>> wrote: >>>>> >>>>>> >>>>>> Yes, this is a correct program. >>>>>> >>>>>> — Pavan >>>>>> >>>>>> On Feb 5, 2014, at 10:30 AM, Rolf Rabenseifner >>>>>> <rabenseifner(a)hlrs.de> wrote: >>>>>> >>>>>>> Jeff and all, >>>>>>> >>>>>>> it looks like that it works as MPI-3 is designed: >>>>>>> >>>>>>> I need to add once at begin a >>>>>>> MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, >>>>>>> win) >>>>>>> and once at end a MPI_WIN_UNLOCK_ALL(win) >>>>>>> and then all works fine with MPI_WIN_SYNC in each iteration. >>>>>>> >>>>>>> Is this usage consistent with the definition in the MPI-3 >>>>>>> standard? >>>>>>> >>>>>>> Here the total scenario that I use: >>>>>>> >>>>>>> -------------------- >>>>>>> X is part of a shared memory window and should mean the same >>>>>>> memory location in both processes >>>>>>> >>>>>>> Process A Process B >>>>>>> >>>>>>> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >>>>>>> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >>>>>>> >>>>>>> DO ... DO ... >>>>>>> x=... >>>>>>> MPI_F_SYNC_REG(X) >>>>>>> MPI_Barrier MPI_Barrier >>>>>>> MPI_F_SYNC_REG(X) >>>>>>> print X >>>>>>> END DO END DO >>>>>>> >>>>>>> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >>>>>>> >>>>>>> -------------------- >>>>>>> >>>>>>> Best regards >>>>>>> Rolf >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Jeff Hammond" <jeff.science(a)gmail.com> >>>>>>>> To: "MPI WG Remote Memory Access working group" >>>>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>>>>>>> <longb(a)cray.com> >>>>>>>> Sent: Tuesday, February 4, 2014 7:42:58 PM >>>>>>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on >>>>>>>> shared memory windows >>>>>>>> >>>>>>>> "For the purposes of synchronizing the private and public >>>>>>>> window, >>>>>>>> MPI_WIN_SYNC has the effect of ending and reopening an access >>>>>>>> and >>>>>>>> exposure epoch on the window (note that it does not actually >>>>>>>> end >>>>>>>> an >>>>>>>> epoch or complete any pending MPI RMA operations)." >>>>>>>> >>>>>>>> I think this is interpreted to mean that this call is only >>>>>>>> valid >>>>>>>> inside of an existing epoch and thus if you want to call it, >>>>>>>> you >>>>>>>> need >>>>>>>> to use it inside of a passive-target epoch. Thus, it is not >>>>>>>> merely a >>>>>>>> portable abstraction for a memory barrier. >>>>>>>> >>>>>>>> I think we should fix MPICH and/or MPI-Next to allow the more >>>>>>>> general >>>>>>>> use such that your code is standard-compliant and executes >>>>>>>> correctly. >>>>>>>> >>>>>>>> I await violent disagreement from others :-) >>>>>>>> >>>>>>>> Jeff >>>>>>>> >>>>>>>> On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner >>>>>>>> <rabenseifner(a)hlrs.de> wrote: >>>>>>>>> Brian, Pavan, and Jeff, >>>>>>>>> >>>>>>>>> you convinced me. I did it, see attached file, and my mpich >>>>>>>>> based >>>>>>>>> Cray lib tells >>>>>>>>> >>>>>>>>> Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in >>>>>>>>> MPI_Win_sync: Wrong synchronization of RMA calls , error >>>>>>>>> stack: >>>>>>>>> MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed >>>>>>>>> MPIDI_Win_sync(2495): Wrong synchronization of RMA calls >>>>>>>>> >>>>>>>>> (only once in each process). >>>>>>>>> >>>>>>>>> I expect, that this is now an implementation bug that should >>>>>>>>> be >>>>>>>>> fixed by mpich and cray? >>>>>>>>> >>>>>>>>> Best regards >>>>>>>>> Rolf >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Brian W Barrett" <bwbarre(a)sandia.gov> >>>>>>>>>> To: "MPI WG Remote Memory Access working group" >>>>>>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>>>>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>>>>>>>>> <longb(a)cray.com> >>>>>>>>>> Sent: Tuesday, February 4, 2014 7:09:02 PM >>>>>>>>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on >>>>>>>>>> shared >>>>>>>>>> memory windows >>>>>>>>>> >>>>>>>>>> On 2/4/14 11:01 AM, "Rolf Rabenseifner" >>>>>>>>>> <rabenseifner(a)hlrs.de> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) >>>>>>>>>>> has no meaning in the unified memory model if all accesses >>>>>>>>>>> are done without RMA routines. >>>>>>>>>>> It has only a meaning if different public and privat copy >>>>>>>>>>> is >>>>>>>>>>> there (MPI-3.0 p450:46-p451:2). >>>>>>>>>>> MPI-3.0 p456:3 - p457:7 define the rules for the unified >>>>>>>>>>> memory >>>>>>>>>>> model >>>>>>>>>>> but there is no need to use MPI_WIN_SYNC. >>>>>>>>>> >>>>>>>>>> Right, there's no need from an MPI point of view, but that >>>>>>>>>> doesn't >>>>>>>>>> mean >>>>>>>>>> that the language/compiler/processor doesn't have a need for >>>>>>>>>> extra >>>>>>>>>> synchronization. >>>>>>>>>> >>>>>>>>>>> The combination of X=13 and MPI_F_SYNC_REG(X) >>>>>>>>>>> before MPI_Barrier should guarantee that all bytes of X are >>>>>>>>>>> stored in memory. The same should be valid in C, >>>>>>>>>>> because the C compiler has no chance to see whether >>>>>>>>>>> MPI_Barrier will access the bytes of X or not. >>>>>>>>>>> And if it is guaranteed to be in the unified memory, >>>>>>>>>>> then the other process (B) should be able to correctly >>>>>>>>>>> read the data after the return from its barrier. >>>>>>>>>>> >>>>>>>>>>> What is wrong with my thinking? >>>>>>>>>>> Which detail do I miss? >>>>>>>>>> >>>>>>>>>> According to my reading of the spec, MPI_F_SYNC_REG only >>>>>>>>>> prevents >>>>>>>>>> the >>>>>>>>>> language/compiler from moving the store, but does not say >>>>>>>>>> anything >>>>>>>>>> about >>>>>>>>>> processor ordering. So the WIN_SYNC in my last e-mail will >>>>>>>>>> add >>>>>>>>>> the >>>>>>>>>> processor memory barrier, which will give you all the >>>>>>>>>> semantics >>>>>>>>>> you >>>>>>>>>> need. >>>>>>>>>> >>>>>>>>>> Shared memory programming is a disaster in most languages >>>>>>>>>> today, >>>>>>>>>> so >>>>>>>>>> we >>>>>>>>>> decided to pass that disaster on to the user. We really >>>>>>>>>> can't >>>>>>>>>> help, >>>>>>>>>> without adding lots of overhead (ie, using put/get/rma >>>>>>>>>> synchronization). >>>>>>>>>> So if a user already knows how to do shared memory >>>>>>>>>> programming, >>>>>>>>>> this >>>>>>>>>> will >>>>>>>>>> feel natural. If they don't, it's going to hurt badly :/. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Brian >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Brian W. Barrett >>>>>>>>>> Scalable System Software Group >>>>>>>>>> Sandia National Laboratories >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> mpiwg-rma mailing list >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>>>>>> rabenseifner(a)hlrs.de >>>>>>>>> High Performance Computing Center (HLRS) . phone >>>>>>>>> ++49(0)711/685-65530 >>>>>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>>>>>> 685-65832 >>>>>>>>> Head of Dpmt Parallel Computing . . . >>>>>>>>> www.hlrs.de/people/rabenseifner >>>>>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: >>>>>>>>> Room >>>>>>>>> 1.307) >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> mpiwg-rma mailing list >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jeff Hammond >>>>>>>> jeff.science(a)gmail.com >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>>>> rabenseifner(a)hlrs.de >>>>>>> High Performance Computing Center (HLRS) . phone >>>>>>> ++49(0)711/685-65530 >>>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>>>> 685-65832 >>>>>>> Head of Dpmt Parallel Computing . . . >>>>>>> www.hlrs.de/people/rabenseifner >>>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>>>>> 1.307) >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Balaji, Pavan 05 Feb '14

05 Feb '14

It’s correct now. In the MPI standard, we have stayed away from tutorial-style material. But I’m personally not against it. — Pavan On Feb 5, 2014, at 11:43 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Pavan, Dave, Jeff, and Brian, > > is it now correct? > Should it go in MPI-next into Sect.11.7? > >> -------------------- >> X is part of a shared memory window and should mean the same >> memory location in both processes produced with >> MPI_WIN_ALLOCATE_SHARED. >> >> Process A Process B >> >> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >> >> DO ... DO ... >> x=... >> MPI_F_SYNC_REG(X) >> MPI_WIN_SYNC(win) >> MPI_Barrier MPI_Barrier >> MPI_WIN_SYNC(win) >> MPI_F_SYNC_REG(X) >> print X >> END DO END DO >> >> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >> -------------------- > > Best regards > Rolf > > ----- Original Message ----- >> From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >> Sent: Wednesday, February 5, 2014 5:59:03 PM >> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows >> >> That was my fault. Here the program that should be examined, >> whether it is correct according to MPI-3.0: >> >> -------------------- >> X is part of a shared memory window and should mean the same >> memory location in both processes produced with >> MPI_WIN_ALLOCATE_SHARED. >> >> Process A Process B >> >> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >> >> DO ... DO ... >> x=... >> MPI_F_SYNC_REG(X) >> MPI_WIN_SYNC(win) >> MPI_Barrier MPI_Barrier >> MPI_WIN_SYNC(win) >> MPI_F_SYNC_REG(X) >> print X >> END DO END DO >> >> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >> -------------------- >> >> Is it now correct according to MPI-3.0? >> And perhaps also according other rules for >> real shared memory programming? >> >> Would it be helpful to add it to at the end of Sect.11.7 >> because it would definitely clarify the rules >> how to use shared memory windows. >> >> Best regards >> Rolf >> >> ----- Original Message ----- >>> From: "Pavan Balaji" <balaji(a)anl.gov> >>> To: "MPI WG Remote Memory Access working group" >>> <mpiwg-rma(a)lists.mpi-forum.org> >>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>> <longb(a)cray.com> >>> Sent: Wednesday, February 5, 2014 5:46:25 PM >>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared >>> memory windows >>> >>> >>> Whoops, I read MPI_F_SYNC_REG as MPI_WIN_SYNC. There need to be >>> WIN_SYNCs on both processes. >>> >>> — Pavan >>> >>> On Feb 5, 2014, at 10:40 AM, Dave Goodell (dgoodell) >>> <dgoodell(a)cisco.com> wrote: >>> >>>> Pavan, is it? >>>> >>>> Rolf, here is the supposed MPI_WIN_SYNC call? I assume you meant >>>> to put it between the MPI_F_SYNC_REG and MPI_Barrier in both >>>> processes? >>>> >>>> -Dave >>>> >>>> On Feb 5, 2014, at 10:31 AM, "Balaji, Pavan" <balaji(a)anl.gov> >>>> wrote: >>>> >>>>> >>>>> Yes, this is a correct program. >>>>> >>>>> — Pavan >>>>> >>>>> On Feb 5, 2014, at 10:30 AM, Rolf Rabenseifner >>>>> <rabenseifner(a)hlrs.de> wrote: >>>>> >>>>>> Jeff and all, >>>>>> >>>>>> it looks like that it works as MPI-3 is designed: >>>>>> >>>>>> I need to add once at begin a >>>>>> MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, >>>>>> win) >>>>>> and once at end a MPI_WIN_UNLOCK_ALL(win) >>>>>> and then all works fine with MPI_WIN_SYNC in each iteration. >>>>>> >>>>>> Is this usage consistent with the definition in the MPI-3 >>>>>> standard? >>>>>> >>>>>> Here the total scenario that I use: >>>>>> >>>>>> -------------------- >>>>>> X is part of a shared memory window and should mean the same >>>>>> memory location in both processes >>>>>> >>>>>> Process A Process B >>>>>> >>>>>> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >>>>>> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >>>>>> >>>>>> DO ... DO ... >>>>>> x=... >>>>>> MPI_F_SYNC_REG(X) >>>>>> MPI_Barrier MPI_Barrier >>>>>> MPI_F_SYNC_REG(X) >>>>>> print X >>>>>> END DO END DO >>>>>> >>>>>> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >>>>>> >>>>>> -------------------- >>>>>> >>>>>> Best regards >>>>>> Rolf >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Jeff Hammond" <jeff.science(a)gmail.com> >>>>>>> To: "MPI WG Remote Memory Access working group" >>>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>>>>>> <longb(a)cray.com> >>>>>>> Sent: Tuesday, February 4, 2014 7:42:58 PM >>>>>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on >>>>>>> shared memory windows >>>>>>> >>>>>>> "For the purposes of synchronizing the private and public >>>>>>> window, >>>>>>> MPI_WIN_SYNC has the effect of ending and reopening an access >>>>>>> and >>>>>>> exposure epoch on the window (note that it does not actually >>>>>>> end >>>>>>> an >>>>>>> epoch or complete any pending MPI RMA operations)." >>>>>>> >>>>>>> I think this is interpreted to mean that this call is only >>>>>>> valid >>>>>>> inside of an existing epoch and thus if you want to call it, >>>>>>> you >>>>>>> need >>>>>>> to use it inside of a passive-target epoch. Thus, it is not >>>>>>> merely a >>>>>>> portable abstraction for a memory barrier. >>>>>>> >>>>>>> I think we should fix MPICH and/or MPI-Next to allow the more >>>>>>> general >>>>>>> use such that your code is standard-compliant and executes >>>>>>> correctly. >>>>>>> >>>>>>> I await violent disagreement from others :-) >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>> On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner >>>>>>> <rabenseifner(a)hlrs.de> wrote: >>>>>>>> Brian, Pavan, and Jeff, >>>>>>>> >>>>>>>> you convinced me. I did it, see attached file, and my mpich >>>>>>>> based >>>>>>>> Cray lib tells >>>>>>>> >>>>>>>> Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in >>>>>>>> MPI_Win_sync: Wrong synchronization of RMA calls , error >>>>>>>> stack: >>>>>>>> MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed >>>>>>>> MPIDI_Win_sync(2495): Wrong synchronization of RMA calls >>>>>>>> >>>>>>>> (only once in each process). >>>>>>>> >>>>>>>> I expect, that this is now an implementation bug that should >>>>>>>> be >>>>>>>> fixed by mpich and cray? >>>>>>>> >>>>>>>> Best regards >>>>>>>> Rolf >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Brian W Barrett" <bwbarre(a)sandia.gov> >>>>>>>>> To: "MPI WG Remote Memory Access working group" >>>>>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>>>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>>>>>>>> <longb(a)cray.com> >>>>>>>>> Sent: Tuesday, February 4, 2014 7:09:02 PM >>>>>>>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on >>>>>>>>> shared >>>>>>>>> memory windows >>>>>>>>> >>>>>>>>> On 2/4/14 11:01 AM, "Rolf Rabenseifner" >>>>>>>>> <rabenseifner(a)hlrs.de> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) >>>>>>>>>> has no meaning in the unified memory model if all accesses >>>>>>>>>> are done without RMA routines. >>>>>>>>>> It has only a meaning if different public and privat copy >>>>>>>>>> is >>>>>>>>>> there (MPI-3.0 p450:46-p451:2). >>>>>>>>>> MPI-3.0 p456:3 - p457:7 define the rules for the unified >>>>>>>>>> memory >>>>>>>>>> model >>>>>>>>>> but there is no need to use MPI_WIN_SYNC. >>>>>>>>> >>>>>>>>> Right, there's no need from an MPI point of view, but that >>>>>>>>> doesn't >>>>>>>>> mean >>>>>>>>> that the language/compiler/processor doesn't have a need for >>>>>>>>> extra >>>>>>>>> synchronization. >>>>>>>>> >>>>>>>>>> The combination of X=13 and MPI_F_SYNC_REG(X) >>>>>>>>>> before MPI_Barrier should guarantee that all bytes of X are >>>>>>>>>> stored in memory. The same should be valid in C, >>>>>>>>>> because the C compiler has no chance to see whether >>>>>>>>>> MPI_Barrier will access the bytes of X or not. >>>>>>>>>> And if it is guaranteed to be in the unified memory, >>>>>>>>>> then the other process (B) should be able to correctly >>>>>>>>>> read the data after the return from its barrier. >>>>>>>>>> >>>>>>>>>> What is wrong with my thinking? >>>>>>>>>> Which detail do I miss? >>>>>>>>> >>>>>>>>> According to my reading of the spec, MPI_F_SYNC_REG only >>>>>>>>> prevents >>>>>>>>> the >>>>>>>>> language/compiler from moving the store, but does not say >>>>>>>>> anything >>>>>>>>> about >>>>>>>>> processor ordering. So the WIN_SYNC in my last e-mail will >>>>>>>>> add >>>>>>>>> the >>>>>>>>> processor memory barrier, which will give you all the >>>>>>>>> semantics >>>>>>>>> you >>>>>>>>> need. >>>>>>>>> >>>>>>>>> Shared memory programming is a disaster in most languages >>>>>>>>> today, >>>>>>>>> so >>>>>>>>> we >>>>>>>>> decided to pass that disaster on to the user. We really >>>>>>>>> can't >>>>>>>>> help, >>>>>>>>> without adding lots of overhead (ie, using put/get/rma >>>>>>>>> synchronization). >>>>>>>>> So if a user already knows how to do shared memory >>>>>>>>> programming, >>>>>>>>> this >>>>>>>>> will >>>>>>>>> feel natural. If they don't, it's going to hurt badly :/. >>>>>>>>> >>>>>>>>> >>>>>>>>> Brian >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Brian W. Barrett >>>>>>>>> Scalable System Software Group >>>>>>>>> Sandia National Laboratories >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> mpiwg-rma mailing list >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>>>>> rabenseifner(a)hlrs.de >>>>>>>> High Performance Computing Center (HLRS) . phone >>>>>>>> ++49(0)711/685-65530 >>>>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>>>>> 685-65832 >>>>>>>> Head of Dpmt Parallel Computing . . . >>>>>>>> www.hlrs.de/people/rabenseifner >>>>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: >>>>>>>> Room >>>>>>>> 1.307) >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jeff Hammond >>>>>>> jeff.science(a)gmail.com >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>> >>>>>> -- >>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>>> rabenseifner(a)hlrs.de >>>>>> High Performance Computing Center (HLRS) . phone >>>>>> ++49(0)711/685-65530 >>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>>> 685-65832 >>>>>> Head of Dpmt Parallel Computing . . . >>>>>> www.hlrs.de/people/rabenseifner >>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>>>> 1.307) >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Jeff Hammond 05 Feb '14

05 Feb '14

I'd like to see the text changes that are proposed before discussing MPI-Next. Jeff On Wed, Feb 5, 2014 at 11:47 AM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: > On Feb 5, 2014, at 11:43 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > >> Pavan, Dave, Jeff, and Brian, >> >> is it now correct? > > As far as I can tell it is correct, modulo any Fortran behavior that I don't understand. > >> Should it go in MPI-next into Sect.11.7? > > That seems reasonable to me. > > -Dave > >>> -------------------- >>> X is part of a shared memory window and should mean the same >>> memory location in both processes produced with >>> MPI_WIN_ALLOCATE_SHARED. >>> >>> Process A Process B >>> >>> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >>> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >>> >>> DO ... DO ... >>> x=... >>> MPI_F_SYNC_REG(X) >>> MPI_WIN_SYNC(win) >>> MPI_Barrier MPI_Barrier >>> MPI_WIN_SYNC(win) >>> MPI_F_SYNC_REG(X) >>> print X >>> END DO END DO >>> >>> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >>> -------------------- > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Dave Goodell (dgoodell) 05 Feb '14

05 Feb '14

On Feb 5, 2014, at 11:43 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Pavan, Dave, Jeff, and Brian, > > is it now correct? As far as I can tell it is correct, modulo any Fortran behavior that I don't understand. > Should it go in MPI-next into Sect.11.7? That seems reasonable to me. -Dave >> -------------------- >> X is part of a shared memory window and should mean the same >> memory location in both processes produced with >> MPI_WIN_ALLOCATE_SHARED. >> >> Process A Process B >> >> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >> >> DO ... DO ... >> x=... >> MPI_F_SYNC_REG(X) >> MPI_WIN_SYNC(win) >> MPI_Barrier MPI_Barrier >> MPI_WIN_SYNC(win) >> MPI_F_SYNC_REG(X) >> print X >> END DO END DO >> >> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >> --------------------

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Rolf Rabenseifner 05 Feb '14

05 Feb '14

Pavan, Dave, Jeff, and Brian, is it now correct? Should it go in MPI-next into Sect.11.7? > -------------------- > X is part of a shared memory window and should mean the same > memory location in both processes produced with > MPI_WIN_ALLOCATE_SHARED. > > Process A Process B > > MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( > MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) > > DO ... DO ... > x=... > MPI_F_SYNC_REG(X) > MPI_WIN_SYNC(win) > MPI_Barrier MPI_Barrier > MPI_WIN_SYNC(win) > MPI_F_SYNC_REG(X) > print X > END DO END DO > > MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) > -------------------- Best regards Rolf ----- Original Message ----- > From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> > Sent: Wednesday, February 5, 2014 5:59:03 PM > Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows > > That was my fault. Here the program that should be examined, > whether it is correct according to MPI-3.0: > > -------------------- > X is part of a shared memory window and should mean the same > memory location in both processes produced with > MPI_WIN_ALLOCATE_SHARED. > > Process A Process B > > MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( > MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) > > DO ... DO ... > x=... > MPI_F_SYNC_REG(X) > MPI_WIN_SYNC(win) > MPI_Barrier MPI_Barrier > MPI_WIN_SYNC(win) > MPI_F_SYNC_REG(X) > print X > END DO END DO > > MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) > -------------------- > > Is it now correct according to MPI-3.0? > And perhaps also according other rules for > real shared memory programming? > > Would it be helpful to add it to at the end of Sect.11.7 > because it would definitely clarify the rules > how to use shared memory windows. > > Best regards > Rolf > > ----- Original Message ----- > > From: "Pavan Balaji" <balaji(a)anl.gov> > > To: "MPI WG Remote Memory Access working group" > > <mpiwg-rma(a)lists.mpi-forum.org> > > Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" > > <longb(a)cray.com> > > Sent: Wednesday, February 5, 2014 5:46:25 PM > > Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared > > memory windows > > > > > > Whoops, I read MPI_F_SYNC_REG as MPI_WIN_SYNC. There need to be > > WIN_SYNCs on both processes. > > > > — Pavan > > > > On Feb 5, 2014, at 10:40 AM, Dave Goodell (dgoodell) > > <dgoodell(a)cisco.com> wrote: > > > > > Pavan, is it? > > > > > > Rolf, here is the supposed MPI_WIN_SYNC call? I assume you meant > > > to put it between the MPI_F_SYNC_REG and MPI_Barrier in both > > > processes? > > > > > > -Dave > > > > > > On Feb 5, 2014, at 10:31 AM, "Balaji, Pavan" <balaji(a)anl.gov> > > > wrote: > > > > > >> > > >> Yes, this is a correct program. > > >> > > >> — Pavan > > >> > > >> On Feb 5, 2014, at 10:30 AM, Rolf Rabenseifner > > >> <rabenseifner(a)hlrs.de> wrote: > > >> > > >>> Jeff and all, > > >>> > > >>> it looks like that it works as MPI-3 is designed: > > >>> > > >>> I need to add once at begin a > > >>> MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, > > >>> win) > > >>> and once at end a MPI_WIN_UNLOCK_ALL(win) > > >>> and then all works fine with MPI_WIN_SYNC in each iteration. > > >>> > > >>> Is this usage consistent with the definition in the MPI-3 > > >>> standard? > > >>> > > >>> Here the total scenario that I use: > > >>> > > >>> -------------------- > > >>> X is part of a shared memory window and should mean the same > > >>> memory location in both processes > > >>> > > >>> Process A Process B > > >>> > > >>> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( > > >>> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) > > >>> > > >>> DO ... DO ... > > >>> x=... > > >>> MPI_F_SYNC_REG(X) > > >>> MPI_Barrier MPI_Barrier > > >>> MPI_F_SYNC_REG(X) > > >>> print X > > >>> END DO END DO > > >>> > > >>> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) > > >>> > > >>> -------------------- > > >>> > > >>> Best regards > > >>> Rolf > > >>> > > >>> > > >>> > > >>> ----- Original Message ----- > > >>>> From: "Jeff Hammond" <jeff.science(a)gmail.com> > > >>>> To: "MPI WG Remote Memory Access working group" > > >>>> <mpiwg-rma(a)lists.mpi-forum.org> > > >>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" > > >>>> <longb(a)cray.com> > > >>>> Sent: Tuesday, February 4, 2014 7:42:58 PM > > >>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on > > >>>> shared memory windows > > >>>> > > >>>> "For the purposes of synchronizing the private and public > > >>>> window, > > >>>> MPI_WIN_SYNC has the effect of ending and reopening an access > > >>>> and > > >>>> exposure epoch on the window (note that it does not actually > > >>>> end > > >>>> an > > >>>> epoch or complete any pending MPI RMA operations)." > > >>>> > > >>>> I think this is interpreted to mean that this call is only > > >>>> valid > > >>>> inside of an existing epoch and thus if you want to call it, > > >>>> you > > >>>> need > > >>>> to use it inside of a passive-target epoch. Thus, it is not > > >>>> merely a > > >>>> portable abstraction for a memory barrier. > > >>>> > > >>>> I think we should fix MPICH and/or MPI-Next to allow the more > > >>>> general > > >>>> use such that your code is standard-compliant and executes > > >>>> correctly. > > >>>> > > >>>> I await violent disagreement from others :-) > > >>>> > > >>>> Jeff > > >>>> > > >>>> On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner > > >>>> <rabenseifner(a)hlrs.de> wrote: > > >>>>> Brian, Pavan, and Jeff, > > >>>>> > > >>>>> you convinced me. I did it, see attached file, and my mpich > > >>>>> based > > >>>>> Cray lib tells > > >>>>> > > >>>>> Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in > > >>>>> MPI_Win_sync: Wrong synchronization of RMA calls , error > > >>>>> stack: > > >>>>> MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed > > >>>>> MPIDI_Win_sync(2495): Wrong synchronization of RMA calls > > >>>>> > > >>>>> (only once in each process). > > >>>>> > > >>>>> I expect, that this is now an implementation bug that should > > >>>>> be > > >>>>> fixed by mpich and cray? > > >>>>> > > >>>>> Best regards > > >>>>> Rolf > > >>>>> > > >>>>> ----- Original Message ----- > > >>>>>> From: "Brian W Barrett" <bwbarre(a)sandia.gov> > > >>>>>> To: "MPI WG Remote Memory Access working group" > > >>>>>> <mpiwg-rma(a)lists.mpi-forum.org> > > >>>>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" > > >>>>>> <longb(a)cray.com> > > >>>>>> Sent: Tuesday, February 4, 2014 7:09:02 PM > > >>>>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on > > >>>>>> shared > > >>>>>> memory windows > > >>>>>> > > >>>>>> On 2/4/14 11:01 AM, "Rolf Rabenseifner" > > >>>>>> <rabenseifner(a)hlrs.de> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) > > >>>>>>> has no meaning in the unified memory model if all accesses > > >>>>>>> are done without RMA routines. > > >>>>>>> It has only a meaning if different public and privat copy > > >>>>>>> is > > >>>>>>> there (MPI-3.0 p450:46-p451:2). > > >>>>>>> MPI-3.0 p456:3 - p457:7 define the rules for the unified > > >>>>>>> memory > > >>>>>>> model > > >>>>>>> but there is no need to use MPI_WIN_SYNC. > > >>>>>> > > >>>>>> Right, there's no need from an MPI point of view, but that > > >>>>>> doesn't > > >>>>>> mean > > >>>>>> that the language/compiler/processor doesn't have a need for > > >>>>>> extra > > >>>>>> synchronization. > > >>>>>> > > >>>>>>> The combination of X=13 and MPI_F_SYNC_REG(X) > > >>>>>>> before MPI_Barrier should guarantee that all bytes of X are > > >>>>>>> stored in memory. The same should be valid in C, > > >>>>>>> because the C compiler has no chance to see whether > > >>>>>>> MPI_Barrier will access the bytes of X or not. > > >>>>>>> And if it is guaranteed to be in the unified memory, > > >>>>>>> then the other process (B) should be able to correctly > > >>>>>>> read the data after the return from its barrier. > > >>>>>>> > > >>>>>>> What is wrong with my thinking? > > >>>>>>> Which detail do I miss? > > >>>>>> > > >>>>>> According to my reading of the spec, MPI_F_SYNC_REG only > > >>>>>> prevents > > >>>>>> the > > >>>>>> language/compiler from moving the store, but does not say > > >>>>>> anything > > >>>>>> about > > >>>>>> processor ordering. So the WIN_SYNC in my last e-mail will > > >>>>>> add > > >>>>>> the > > >>>>>> processor memory barrier, which will give you all the > > >>>>>> semantics > > >>>>>> you > > >>>>>> need. > > >>>>>> > > >>>>>> Shared memory programming is a disaster in most languages > > >>>>>> today, > > >>>>>> so > > >>>>>> we > > >>>>>> decided to pass that disaster on to the user. We really > > >>>>>> can't > > >>>>>> help, > > >>>>>> without adding lots of overhead (ie, using put/get/rma > > >>>>>> synchronization). > > >>>>>> So if a user already knows how to do shared memory > > >>>>>> programming, > > >>>>>> this > > >>>>>> will > > >>>>>> feel natural. If they don't, it's going to hurt badly :/. > > >>>>>> > > >>>>>> > > >>>>>> Brian > > >>>>>> > > >>>>>> -- > > >>>>>> Brian W. Barrett > > >>>>>> Scalable System Software Group > > >>>>>> Sandia National Laboratories > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> _______________________________________________ > > >>>>>> mpiwg-rma mailing list > > >>>>>> mpiwg-rma(a)lists.mpi-forum.org > > >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >>>>>> > > >>>>> > > >>>>> -- > > >>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > > >>>>> rabenseifner(a)hlrs.de > > >>>>> High Performance Computing Center (HLRS) . phone > > >>>>> ++49(0)711/685-65530 > > >>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > >>>>> 685-65832 > > >>>>> Head of Dpmt Parallel Computing . . . > > >>>>> www.hlrs.de/people/rabenseifner > > >>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: > > >>>>> Room > > >>>>> 1.307) > > >>>>> > > >>>>> _______________________________________________ > > >>>>> mpiwg-rma mailing list > > >>>>> mpiwg-rma(a)lists.mpi-forum.org > > >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Jeff Hammond > > >>>> jeff.science(a)gmail.com > > >>>> _______________________________________________ > > >>>> mpiwg-rma mailing list > > >>>> mpiwg-rma(a)lists.mpi-forum.org > > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >>>> > > >>> > > >>> -- > > >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > > >>> rabenseifner(a)hlrs.de > > >>> High Performance Computing Center (HLRS) . phone > > >>> ++49(0)711/685-65530 > > >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > >>> 685-65832 > > >>> Head of Dpmt Parallel Computing . . . > > >>> www.hlrs.de/people/rabenseifner > > >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > >>> 1.307) > > >>> _______________________________________________ > > >>> mpiwg-rma mailing list > > >>> mpiwg-rma(a)lists.mpi-forum.org > > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >> > > >> _______________________________________________ > > >> mpiwg-rma mailing list > > >> mpiwg-rma(a)lists.mpi-forum.org > > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > _______________________________________________ > > > mpiwg-rma mailing list > > > mpiwg-rma(a)lists.mpi-forum.org > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Rolf Rabenseifner 05 Feb '14

05 Feb '14

That was my fault. Here the program that should be examined, whether it is correct according to MPI-3.0: -------------------- X is part of a shared memory window and should mean the same memory location in both processes produced with MPI_WIN_ALLOCATE_SHARED. Process A Process B MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) DO ... DO ... x=... MPI_F_SYNC_REG(X) MPI_WIN_SYNC(win) MPI_Barrier MPI_Barrier MPI_WIN_SYNC(win) MPI_F_SYNC_REG(X) print X END DO END DO MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) -------------------- Is it now correct according to MPI-3.0? And perhaps also according other rules for real shared memory programming? Would it be helpful to add it to at the end of Sect.11.7 because it would definitely clarify the rules how to use shared memory windows. Best regards Rolf ----- Original Message ----- > From: "Pavan Balaji" <balaji(a)anl.gov> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> > Sent: Wednesday, February 5, 2014 5:46:25 PM > Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows > > > Whoops, I read MPI_F_SYNC_REG as MPI_WIN_SYNC. There need to be > WIN_SYNCs on both processes. > > — Pavan > > On Feb 5, 2014, at 10:40 AM, Dave Goodell (dgoodell) > <dgoodell(a)cisco.com> wrote: > > > Pavan, is it? > > > > Rolf, here is the supposed MPI_WIN_SYNC call? I assume you meant > > to put it between the MPI_F_SYNC_REG and MPI_Barrier in both > > processes? > > > > -Dave > > > > On Feb 5, 2014, at 10:31 AM, "Balaji, Pavan" <balaji(a)anl.gov> > > wrote: > > > >> > >> Yes, this is a correct program. > >> > >> — Pavan > >> > >> On Feb 5, 2014, at 10:30 AM, Rolf Rabenseifner > >> <rabenseifner(a)hlrs.de> wrote: > >> > >>> Jeff and all, > >>> > >>> it looks like that it works as MPI-3 is designed: > >>> > >>> I need to add once at begin a MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, > >>> win) > >>> and once at end a MPI_WIN_UNLOCK_ALL(win) > >>> and then all works fine with MPI_WIN_SYNC in each iteration. > >>> > >>> Is this usage consistent with the definition in the MPI-3 > >>> standard? > >>> > >>> Here the total scenario that I use: > >>> > >>> -------------------- > >>> X is part of a shared memory window and should mean the same > >>> memory location in both processes > >>> > >>> Process A Process B > >>> > >>> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( > >>> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) > >>> > >>> DO ... DO ... > >>> x=... > >>> MPI_F_SYNC_REG(X) > >>> MPI_Barrier MPI_Barrier > >>> MPI_F_SYNC_REG(X) > >>> print X > >>> END DO END DO > >>> > >>> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) > >>> > >>> -------------------- > >>> > >>> Best regards > >>> Rolf > >>> > >>> > >>> > >>> ----- Original Message ----- > >>>> From: "Jeff Hammond" <jeff.science(a)gmail.com> > >>>> To: "MPI WG Remote Memory Access working group" > >>>> <mpiwg-rma(a)lists.mpi-forum.org> > >>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" > >>>> <longb(a)cray.com> > >>>> Sent: Tuesday, February 4, 2014 7:42:58 PM > >>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on > >>>> shared memory windows > >>>> > >>>> "For the purposes of synchronizing the private and public > >>>> window, > >>>> MPI_WIN_SYNC has the effect of ending and reopening an access > >>>> and > >>>> exposure epoch on the window (note that it does not actually end > >>>> an > >>>> epoch or complete any pending MPI RMA operations)." > >>>> > >>>> I think this is interpreted to mean that this call is only valid > >>>> inside of an existing epoch and thus if you want to call it, you > >>>> need > >>>> to use it inside of a passive-target epoch. Thus, it is not > >>>> merely a > >>>> portable abstraction for a memory barrier. > >>>> > >>>> I think we should fix MPICH and/or MPI-Next to allow the more > >>>> general > >>>> use such that your code is standard-compliant and executes > >>>> correctly. > >>>> > >>>> I await violent disagreement from others :-) > >>>> > >>>> Jeff > >>>> > >>>> On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner > >>>> <rabenseifner(a)hlrs.de> wrote: > >>>>> Brian, Pavan, and Jeff, > >>>>> > >>>>> you convinced me. I did it, see attached file, and my mpich > >>>>> based > >>>>> Cray lib tells > >>>>> > >>>>> Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in > >>>>> MPI_Win_sync: Wrong synchronization of RMA calls , error stack: > >>>>> MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed > >>>>> MPIDI_Win_sync(2495): Wrong synchronization of RMA calls > >>>>> > >>>>> (only once in each process). > >>>>> > >>>>> I expect, that this is now an implementation bug that should be > >>>>> fixed by mpich and cray? > >>>>> > >>>>> Best regards > >>>>> Rolf > >>>>> > >>>>> ----- Original Message ----- > >>>>>> From: "Brian W Barrett" <bwbarre(a)sandia.gov> > >>>>>> To: "MPI WG Remote Memory Access working group" > >>>>>> <mpiwg-rma(a)lists.mpi-forum.org> > >>>>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" > >>>>>> <longb(a)cray.com> > >>>>>> Sent: Tuesday, February 4, 2014 7:09:02 PM > >>>>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on > >>>>>> shared > >>>>>> memory windows > >>>>>> > >>>>>> On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > >>>>>> wrote: > >>>>>> > >>>>>>> The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) > >>>>>>> has no meaning in the unified memory model if all accesses > >>>>>>> are done without RMA routines. > >>>>>>> It has only a meaning if different public and privat copy is > >>>>>>> there (MPI-3.0 p450:46-p451:2). > >>>>>>> MPI-3.0 p456:3 - p457:7 define the rules for the unified > >>>>>>> memory > >>>>>>> model > >>>>>>> but there is no need to use MPI_WIN_SYNC. > >>>>>> > >>>>>> Right, there's no need from an MPI point of view, but that > >>>>>> doesn't > >>>>>> mean > >>>>>> that the language/compiler/processor doesn't have a need for > >>>>>> extra > >>>>>> synchronization. > >>>>>> > >>>>>>> The combination of X=13 and MPI_F_SYNC_REG(X) > >>>>>>> before MPI_Barrier should guarantee that all bytes of X are > >>>>>>> stored in memory. The same should be valid in C, > >>>>>>> because the C compiler has no chance to see whether > >>>>>>> MPI_Barrier will access the bytes of X or not. > >>>>>>> And if it is guaranteed to be in the unified memory, > >>>>>>> then the other process (B) should be able to correctly > >>>>>>> read the data after the return from its barrier. > >>>>>>> > >>>>>>> What is wrong with my thinking? > >>>>>>> Which detail do I miss? > >>>>>> > >>>>>> According to my reading of the spec, MPI_F_SYNC_REG only > >>>>>> prevents > >>>>>> the > >>>>>> language/compiler from moving the store, but does not say > >>>>>> anything > >>>>>> about > >>>>>> processor ordering. So the WIN_SYNC in my last e-mail will > >>>>>> add > >>>>>> the > >>>>>> processor memory barrier, which will give you all the > >>>>>> semantics > >>>>>> you > >>>>>> need. > >>>>>> > >>>>>> Shared memory programming is a disaster in most languages > >>>>>> today, > >>>>>> so > >>>>>> we > >>>>>> decided to pass that disaster on to the user. We really can't > >>>>>> help, > >>>>>> without adding lots of overhead (ie, using put/get/rma > >>>>>> synchronization). > >>>>>> So if a user already knows how to do shared memory > >>>>>> programming, > >>>>>> this > >>>>>> will > >>>>>> feel natural. If they don't, it's going to hurt badly :/. > >>>>>> > >>>>>> > >>>>>> Brian > >>>>>> > >>>>>> -- > >>>>>> Brian W. Barrett > >>>>>> Scalable System Software Group > >>>>>> Sandia National Laboratories > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> mpiwg-rma mailing list > >>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>>>> > >>>>> > >>>>> -- > >>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > >>>>> rabenseifner(a)hlrs.de > >>>>> High Performance Computing Center (HLRS) . phone > >>>>> ++49(0)711/685-65530 > >>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >>>>> 685-65832 > >>>>> Head of Dpmt Parallel Computing . . . > >>>>> www.hlrs.de/people/rabenseifner > >>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >>>>> 1.307) > >>>>> > >>>>> _______________________________________________ > >>>>> mpiwg-rma mailing list > >>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>> > >>>> > >>>> > >>>> -- > >>>> Jeff Hammond > >>>> jeff.science(a)gmail.com > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>> > >>> > >>> -- > >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > >>> rabenseifner(a)hlrs.de > >>> High Performance Computing Center (HLRS) . phone > >>> ++49(0)711/685-65530 > >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >>> 685-65832 > >>> Head of Dpmt Parallel Computing . . . > >>> www.hlrs.de/people/rabenseifner > >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >>> 1.307) > >>> _______________________________________________ > >>> mpiwg-rma mailing list > >>> mpiwg-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Balaji, Pavan 05 Feb '14

05 Feb '14

Whoops, I read MPI_F_SYNC_REG as MPI_WIN_SYNC. There need to be WIN_SYNCs on both processes. — Pavan On Feb 5, 2014, at 10:40 AM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: > Pavan, is it? > > Rolf, here is the supposed MPI_WIN_SYNC call? I assume you meant to put it between the MPI_F_SYNC_REG and MPI_Barrier in both processes? > > -Dave > > On Feb 5, 2014, at 10:31 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > >> >> Yes, this is a correct program. >> >> — Pavan >> >> On Feb 5, 2014, at 10:30 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: >> >>> Jeff and all, >>> >>> it looks like that it works as MPI-3 is designed: >>> >>> I need to add once at begin a MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, win) >>> and once at end a MPI_WIN_UNLOCK_ALL(win) >>> and then all works fine with MPI_WIN_SYNC in each iteration. >>> >>> Is this usage consistent with the definition in the MPI-3 standard? >>> >>> Here the total scenario that I use: >>> >>> -------------------- >>> X is part of a shared memory window and should mean the same >>> memory location in both processes >>> >>> Process A Process B >>> >>> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >>> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >>> >>> DO ... DO ... >>> x=... >>> MPI_F_SYNC_REG(X) >>> MPI_Barrier MPI_Barrier >>> MPI_F_SYNC_REG(X) >>> print X >>> END DO END DO >>> >>> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >>> >>> -------------------- >>> >>> Best regards >>> Rolf >>> >>> >>> >>> ----- Original Message ----- >>>> From: "Jeff Hammond" <jeff.science(a)gmail.com> >>>> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >>>> Sent: Tuesday, February 4, 2014 7:42:58 PM >>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows >>>> >>>> "For the purposes of synchronizing the private and public window, >>>> MPI_WIN_SYNC has the effect of ending and reopening an access and >>>> exposure epoch on the window (note that it does not actually end an >>>> epoch or complete any pending MPI RMA operations)." >>>> >>>> I think this is interpreted to mean that this call is only valid >>>> inside of an existing epoch and thus if you want to call it, you need >>>> to use it inside of a passive-target epoch. Thus, it is not merely a >>>> portable abstraction for a memory barrier. >>>> >>>> I think we should fix MPICH and/or MPI-Next to allow the more general >>>> use such that your code is standard-compliant and executes correctly. >>>> >>>> I await violent disagreement from others :-) >>>> >>>> Jeff >>>> >>>> On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner >>>> <rabenseifner(a)hlrs.de> wrote: >>>>> Brian, Pavan, and Jeff, >>>>> >>>>> you convinced me. I did it, see attached file, and my mpich based >>>>> Cray lib tells >>>>> >>>>> Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in >>>>> MPI_Win_sync: Wrong synchronization of RMA calls , error stack: >>>>> MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed >>>>> MPIDI_Win_sync(2495): Wrong synchronization of RMA calls >>>>> >>>>> (only once in each process). >>>>> >>>>> I expect, that this is now an implementation bug that should be >>>>> fixed by mpich and cray? >>>>> >>>>> Best regards >>>>> Rolf >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Brian W Barrett" <bwbarre(a)sandia.gov> >>>>>> To: "MPI WG Remote Memory Access working group" >>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>>>>> <longb(a)cray.com> >>>>>> Sent: Tuesday, February 4, 2014 7:09:02 PM >>>>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared >>>>>> memory windows >>>>>> >>>>>> On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >>>>>> wrote: >>>>>> >>>>>>> The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) >>>>>>> has no meaning in the unified memory model if all accesses >>>>>>> are done without RMA routines. >>>>>>> It has only a meaning if different public and privat copy is >>>>>>> there (MPI-3.0 p450:46-p451:2). >>>>>>> MPI-3.0 p456:3 - p457:7 define the rules for the unified memory >>>>>>> model >>>>>>> but there is no need to use MPI_WIN_SYNC. >>>>>> >>>>>> Right, there's no need from an MPI point of view, but that doesn't >>>>>> mean >>>>>> that the language/compiler/processor doesn't have a need for extra >>>>>> synchronization. >>>>>> >>>>>>> The combination of X=13 and MPI_F_SYNC_REG(X) >>>>>>> before MPI_Barrier should guarantee that all bytes of X are >>>>>>> stored in memory. The same should be valid in C, >>>>>>> because the C compiler has no chance to see whether >>>>>>> MPI_Barrier will access the bytes of X or not. >>>>>>> And if it is guaranteed to be in the unified memory, >>>>>>> then the other process (B) should be able to correctly >>>>>>> read the data after the return from its barrier. >>>>>>> >>>>>>> What is wrong with my thinking? >>>>>>> Which detail do I miss? >>>>>> >>>>>> According to my reading of the spec, MPI_F_SYNC_REG only prevents >>>>>> the >>>>>> language/compiler from moving the store, but does not say anything >>>>>> about >>>>>> processor ordering. So the WIN_SYNC in my last e-mail will add >>>>>> the >>>>>> processor memory barrier, which will give you all the semantics >>>>>> you >>>>>> need. >>>>>> >>>>>> Shared memory programming is a disaster in most languages today, >>>>>> so >>>>>> we >>>>>> decided to pass that disaster on to the user. We really can't >>>>>> help, >>>>>> without adding lots of overhead (ie, using put/get/rma >>>>>> synchronization). >>>>>> So if a user already knows how to do shared memory programming, >>>>>> this >>>>>> will >>>>>> feel natural. If they don't, it's going to hurt badly :/. >>>>>> >>>>>> >>>>>> Brian >>>>>> >>>>>> -- >>>>>> Brian W. Barrett >>>>>> Scalable System Software Group >>>>>> Sandia National Laboratories >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>> >>>>> -- >>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>> rabenseifner(a)hlrs.de >>>>> High Performance Computing Center (HLRS) . phone >>>>> ++49(0)711/685-65530 >>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>> 685-65832 >>>>> Head of Dpmt Parallel Computing . . . >>>>> www.hlrs.de/people/rabenseifner >>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>>> 1.307) >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Dave Goodell (dgoodell) 05 Feb '14

05 Feb '14

Pavan, is it? Rolf, here is the supposed MPI_WIN_SYNC call? I assume you meant to put it between the MPI_F_SYNC_REG and MPI_Barrier in both processes? -Dave On Feb 5, 2014, at 10:31 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > Yes, this is a correct program. > > — Pavan > > On Feb 5, 2014, at 10:30 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > >> Jeff and all, >> >> it looks like that it works as MPI-3 is designed: >> >> I need to add once at begin a MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, win) >> and once at end a MPI_WIN_UNLOCK_ALL(win) >> and then all works fine with MPI_WIN_SYNC in each iteration. >> >> Is this usage consistent with the definition in the MPI-3 standard? >> >> Here the total scenario that I use: >> >> -------------------- >> X is part of a shared memory window and should mean the same >> memory location in both processes >> >> Process A Process B >> >> MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( >> MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) >> >> DO ... DO ... >> x=... >> MPI_F_SYNC_REG(X) >> MPI_Barrier MPI_Barrier >> MPI_F_SYNC_REG(X) >> print X >> END DO END DO >> >> MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) >> >> -------------------- >> >> Best regards >> Rolf >> >> >> >> ----- Original Message ----- >>> From: "Jeff Hammond" <jeff.science(a)gmail.com> >>> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" <longb(a)cray.com> >>> Sent: Tuesday, February 4, 2014 7:42:58 PM >>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows >>> >>> "For the purposes of synchronizing the private and public window, >>> MPI_WIN_SYNC has the effect of ending and reopening an access and >>> exposure epoch on the window (note that it does not actually end an >>> epoch or complete any pending MPI RMA operations)." >>> >>> I think this is interpreted to mean that this call is only valid >>> inside of an existing epoch and thus if you want to call it, you need >>> to use it inside of a passive-target epoch. Thus, it is not merely a >>> portable abstraction for a memory barrier. >>> >>> I think we should fix MPICH and/or MPI-Next to allow the more general >>> use such that your code is standard-compliant and executes correctly. >>> >>> I await violent disagreement from others :-) >>> >>> Jeff >>> >>> On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner >>> <rabenseifner(a)hlrs.de> wrote: >>>> Brian, Pavan, and Jeff, >>>> >>>> you convinced me. I did it, see attached file, and my mpich based >>>> Cray lib tells >>>> >>>> Rank 0 [Tue Feb 4 19:31:28 2014] [c9-1c2s7n0] Fatal error in >>>> MPI_Win_sync: Wrong synchronization of RMA calls , error stack: >>>> MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed >>>> MPIDI_Win_sync(2495): Wrong synchronization of RMA calls >>>> >>>> (only once in each process). >>>> >>>> I expect, that this is now an implementation bug that should be >>>> fixed by mpich and cray? >>>> >>>> Best regards >>>> Rolf >>>> >>>> ----- Original Message ----- >>>>> From: "Brian W Barrett" <bwbarre(a)sandia.gov> >>>>> To: "MPI WG Remote Memory Access working group" >>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>> Cc: "Stefan Andersson" <stefan(a)cray.com>, "Bill Long" >>>>> <longb(a)cray.com> >>>>> Sent: Tuesday, February 4, 2014 7:09:02 PM >>>>> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared >>>>> memory windows >>>>> >>>>> On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >>>>> wrote: >>>>> >>>>>> The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG) >>>>>> has no meaning in the unified memory model if all accesses >>>>>> are done without RMA routines. >>>>>> It has only a meaning if different public and privat copy is >>>>>> there (MPI-3.0 p450:46-p451:2). >>>>>> MPI-3.0 p456:3 - p457:7 define the rules for the unified memory >>>>>> model >>>>>> but there is no need to use MPI_WIN_SYNC. >>>>> >>>>> Right, there's no need from an MPI point of view, but that doesn't >>>>> mean >>>>> that the language/compiler/processor doesn't have a need for extra >>>>> synchronization. >>>>> >>>>>> The combination of X=13 and MPI_F_SYNC_REG(X) >>>>>> before MPI_Barrier should guarantee that all bytes of X are >>>>>> stored in memory. The same should be valid in C, >>>>>> because the C compiler has no chance to see whether >>>>>> MPI_Barrier will access the bytes of X or not. >>>>>> And if it is guaranteed to be in the unified memory, >>>>>> then the other process (B) should be able to correctly >>>>>> read the data after the return from its barrier. >>>>>> >>>>>> What is wrong with my thinking? >>>>>> Which detail do I miss? >>>>> >>>>> According to my reading of the spec, MPI_F_SYNC_REG only prevents >>>>> the >>>>> language/compiler from moving the store, but does not say anything >>>>> about >>>>> processor ordering. So the WIN_SYNC in my last e-mail will add >>>>> the >>>>> processor memory barrier, which will give you all the semantics >>>>> you >>>>> need. >>>>> >>>>> Shared memory programming is a disaster in most languages today, >>>>> so >>>>> we >>>>> decided to pass that disaster on to the user. We really can't >>>>> help, >>>>> without adding lots of overhead (ie, using put/get/rma >>>>> synchronization). >>>>> So if a user already knows how to do shared memory programming, >>>>> this >>>>> will >>>>> feel natural. If they don't, it's going to hurt badly :/. >>>>> >>>>> >>>>> Brian >>>>> >>>>> -- >>>>> Brian W. Barrett >>>>> Scalable System Software Group >>>>> Sandia National Laboratories >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>> >>>> -- >>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>> rabenseifner(a)hlrs.de >>>> High Performance Computing Center (HLRS) . phone >>>> ++49(0)711/685-65530 >>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>> 685-65832 >>>> Head of Dpmt Parallel Computing . . . >>>> www.hlrs.de/people/rabenseifner >>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>> 1.307) >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0