August 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jeff Hammond 27 Aug '13

27 Aug '13

The MPI standard will not define by what mechanism the SHMEM implementation will establish the necessary consistency. This is what I mean by undefined = implementation-defined. I do not think that Keith is proposing to enumerate all processor memory models and the appropriate fences required to achieve the affect of win_sync but at lower cost. Thus, it is the implementation - or perhaps more accurately, the platform on which the implementation resides - that will define these semantics. Jeff On Tue, Aug 27, 2013 at 10:16 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 08/27/2013 11:39 AM, Jeff Hammond wrote: > >> I agree with Keith. Losing SHMEM as a client of MPI-3 is not worth >> symmetry or other esoteric properties. >> >> It's entirely reasonable to say the standard leaves no-sync unified as >> undefined = implementation-defined. This would not be the first time. >> > > Your statements are inconsistent. You said you agree with Keith, but then > said that the no-sync unified option should be undefined. That's not what > Keith is saying; he wants that to have *defined* behavior, i.e., it is > valid for the client to do platform-specific memory consistency. > > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jim Dinan 27 Aug '13

27 Aug '13

So, we should define it as not defined? OpenSHMEM itself does not define what should happen if you poll on a location that is the target of an RMA update. The "proper" way to poll in SHMEM is to call one of the built-in polling functions (e.g. shmem_int_wait(location, value)), which can deal with the data consistency problem. ~Jim. On Tue, Aug 27, 2013 at 1:16 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 08/27/2013 11:39 AM, Jeff Hammond wrote: > >> I agree with Keith. Losing SHMEM as a client of MPI-3 is not worth >> symmetry or other esoteric properties. >> >> It's entirely reasonable to say the standard leaves no-sync unified as >> undefined = implementation-defined. This would not be the first time. >> > > Your statements are inconsistent. You said you agree with Keith, but then > said that the no-sync unified option should be undefined. That's not what > Keith is saying; he wants that to have *defined* behavior, i.e., it is > valid for the client to do platform-specific memory consistency. > > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > ______________________________**_________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/**mailman/listinfo.cgi/mpiwg-rma<http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma> >

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 27 Aug '13

27 Aug '13

On 08/27/2013 11:39 AM, Jeff Hammond wrote: > I agree with Keith. Losing SHMEM as a client of MPI-3 is not worth > symmetry or other esoteric properties. > > It's entirely reasonable to say the standard leaves no-sync unified as > undefined = implementation-defined. This would not be the first time. Your statements are inconsistent. You said you agree with Keith, but then said that the no-sync unified option should be undefined. That's not what Keith is saying; he wants that to have *defined* behavior, i.e., it is valid for the client to do platform-specific memory consistency. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jeff Hammond 27 Aug '13

27 Aug '13

I agree with Keith. Losing SHMEM as a client of MPI-3 is not worth symmetry or other esoteric properties. It's entirely reasonable to say the standard leaves no-sync unified as undefined = implementation-defined. This would not be the first time. Jeff Sent from my iPhone On Aug 27, 2013, at 7:16 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 08/27/2013 08:56 AM, Underwood, Keith D wrote: >>> On 08/27/2013 07:15 AM, Underwood, Keith D wrote: >>>>>> I prefer option #1 -- that a window synchronization (e.g. Win_sync) >>>>>> can be used to order load/store operations with respect to actions >>>>>> performed by other processes in the target's window. If no ordering >>>>>> is enforced, the MPI standard does not define what is seen by load >>>>>> operations at the target process. As a rationale, the local process' >>>>>> view of the window may not be consistent with the window because of >>>>>> performance optimizations or the consistency model of the underlying >>>>>> architecture. This would allow e.g. SHMEM implementations to still >>>>>> use MPI-3 RMA, but they would have to rely on a behavior that is >>>>>> defined by the architecture/implementation, as they currently do. >>>> >>>> My statement: this seems to be what SHMEM does - except they don't >>>> provide the equivalent of Win_sync. If we are going to hack on the >>>> text, I would prefer that we be *very* careful in our wording. >>>> Specifically, we need to make it very clear that the SHMEM approach is >>>> *legal*, but the burden is on the user to do potentially >>>> implementation specific things to make it work right. >>> >>> Hmm. OK. Just so there's no confusion, by "SHMEM approach", you are >>> saying that the application can use its own platform-specific memory >>> consistency calls and avoid using WIN_SYNC, correct? >> >> Yes. WIN_SYNC will do that for them, but they can do everything >> without calling WIN_SYNC as long as they call the right >> platform-specific things and the platform-specific approach might be >> faster/lower overhead. > > OK. I'm not disagreeing with this option, but I just want to point out that this will essentially discard MPI implementations that would implement UNIFIED in software (e.g., by trapping store operations). That's probably not a big deal, but from a pure CS perspective, it's not symmetric. > > Thoughts from others? > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 27 Aug '13

27 Aug '13

On 08/27/2013 08:56 AM, Underwood, Keith D wrote: >> On 08/27/2013 07:15 AM, Underwood, Keith D wrote: >>>>> I prefer option #1 -- that a window synchronization (e.g. Win_sync) >>>>> can be used to order load/store operations with respect to actions >>>>> performed by other processes in the target's window. If no ordering >>>>> is enforced, the MPI standard does not define what is seen by load >>>>> operations at the target process. As a rationale, the local process' >>>>> view of the window may not be consistent with the window because of >>>>> performance optimizations or the consistency model of the underlying >>>>> architecture. This would allow e.g. SHMEM implementations to still >>>>> use MPI-3 RMA, but they would have to rely on a behavior that is >>>>> defined by the architecture/implementation, as they currently do. >>> >>> My statement: this seems to be what SHMEM does - except they don't >>> provide the equivalent of Win_sync. If we are going to hack on the >>> text, I would prefer that we be *very* careful in our wording. >>> Specifically, we need to make it very clear that the SHMEM approach is >>> *legal*, but the burden is on the user to do potentially >>> implementation specific things to make it work right. >> >> Hmm. OK. Just so there's no confusion, by "SHMEM approach", you are >> saying that the application can use its own platform-specific memory >> consistency calls and avoid using WIN_SYNC, correct? > > Yes. WIN_SYNC will do that for them, but they can do everything > without calling WIN_SYNC as long as they call the right > platform-specific things and the platform-specific approach might be > faster/lower overhead. OK. I'm not disagreeing with this option, but I just want to point out that this will essentially discard MPI implementations that would implement UNIFIED in software (e.g., by trapping store operations). That's probably not a big deal, but from a pure CS perspective, it's not symmetric. Thoughts from others? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Underwood, Keith D 27 Aug '13

27 Aug '13

> On 08/27/2013 07:15 AM, Underwood, Keith D wrote: > >>> I prefer option #1 -- that a window synchronization (e.g. Win_sync) > >>> can be used to order load/store operations with respect to actions > >>> performed by other processes in the target's window. If no ordering > >>> is enforced, the MPI standard does not define what is seen by load > >>> operations at the target process. As a rationale, the local process' > >>> view of the window may not be consistent with the window because of > >>> performance optimizations or the consistency model of the underlying > >>> architecture. This would allow e.g. SHMEM implementations to still > >>> use MPI-3 RMA, but they would have to rely on a behavior that is > >>> defined by the architecture/implementation, as they currently do. > > > > My statement: this seems to be what SHMEM does - except they don't > > provide the equivalent of Win_sync. If we are going to hack on the > > text, I would prefer that we be *very* careful in our wording. > > Specifically, we need to make it very clear that the SHMEM approach is > > *legal*, but the burden is on the user to do potentially > > implementation specific things to make it work right. > > Hmm. OK. Just so there's no confusion, by "SHMEM approach", you are > saying that the application can use its own platform-specific memory > consistency calls and avoid using WIN_SYNC, correct? Yes. WIN_SYNC will do that for them, but they can do everything without calling WIN_SYNC as long as they call the right platform-specific things and the platform-specific approach might be faster/lower overhead.

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 27 Aug '13

27 Aug '13

On 08/27/2013 07:15 AM, Underwood, Keith D wrote: >>> I prefer option #1 -- that a window synchronization (e.g. Win_sync) >>> can be used to order load/store operations with respect to actions >>> performed by other processes in the target's window. If no ordering >>> is enforced, the MPI standard does not define what is seen by load >>> operations at the target process. As a rationale, the local process' >>> view of the window may not be consistent with the window because of >>> performance optimizations or the consistency model of the underlying >>> architecture. This would allow e.g. SHMEM implementations to still >>> use MPI-3 RMA, but they would have to rely on a behavior that is >>> defined by the architecture/implementation, as they currently do. > > My statement: this seems to be what SHMEM does - except they don't > provide the equivalent of Win_sync. If we are going to hack on the > text, I would prefer that we be *very* careful in our wording. > Specifically, we need to make it very clear that the SHMEM approach > is *legal*, but the burden is on the user to do potentially > implementation specific things to make it work right. Hmm. OK. Just so there's no confusion, by "SHMEM approach", you are saying that the application can use its own platform-specific memory consistency calls and avoid using WIN_SYNC, correct? -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Underwood, Keith D 27 Aug '13

27 Aug '13

Sorry, let me try to put it in slightly more clear language. The email thread I replied to said (with general agreement): >> I prefer option #1 -- that a window synchronization (e.g. Win_sync) >> can be used to order load/store operations with respect to actions >> performed by other processes in the target's window. If no ordering >> is enforced, the MPI standard does not define what is seen by load >> operations at the target process. As a rationale, the local process' >> view of the window may not be consistent with the window because of >> performance optimizations or the consistency model of the underlying >> architecture. This would allow e.g. SHMEM implementations to still >> use MPI-3 RMA, but they would have to rely on a behavior that is >> defined by the architecture/implementation, as they currently do. My statement: this seems to be what SHMEM does - except they don't provide the equivalent of Win_sync. If we are going to hack on the text, I would prefer that we be *very* careful in our wording. Specifically, we need to make it very clear that the SHMEM approach is *legal*, but the burden is on the user to do potentially implementation specific things to make it work right. > -----Original Message----- > From: Pavan Balaji [mailto:[email protected]] > Sent: Monday, August 26, 2013 11:38 PM > To: mpiwg-rma(a)lists.mpi-forum.org > Cc: Underwood, Keith D > Subject: Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED > model updates > > Keith, > > On 08/26/2013 08:00 PM, Underwood, Keith D wrote: > >> Well, after asking somebody what happens with SHMEM on such > >> architectures, I believe they basically tell the user "program this > >> like you would a multithreaded program - all such rules apply". I > >> don't see why we don't take that approach. It seems to me to be the > >> closest to the objective of unified... > > I think your comment is really talking about the previous thread of emails on > this topic. Let me try to clarify this here, but that was not the intent of this > email thread. > > There are really three memory models: SEPARATE, > UNIFIED_REQUIRING_WIN_SYNC, and > UNIFIED_NOT_REQUIRING_WIN_SYNC. > > SEPARATE is what we already have in the standard. > UNIFIED_REQUIRING_WIN_SYNC is where the target has to call WIN_SYNC > to synchronize the public/private windows for overlapping data, but > nonoverlapping data can be accessed without synchronization. > UNIFIED_NOT_REQUIRING_WIN_SYNC is where the target data is always > synchronized between the public/private windows. > > Some folks on the WG thought that "UNIFIED" in the MPI-3 standard meant > UNIFIED_REQUIRING_WIN_SYNC, while others thought it meant > UNIFIED_NOT_REQUIRING_WIN_SYNC. So the chapter is in an inconsistent > state right now. Most of the previous email thread was to come to a > consensus on these different memory models and we agreed that these > three exist, irrespective of whether we want to support them in the MPI > standard or not. > > In this email thread, we are trying to pick one of two options: > > 1. Change the wording in the MPI standard so "UNIFIED" means > UNIFIED_REQUIRING_WIN_SYNC. This seems to be the option that got most > votes so far. In fact, folks feel that this can be done as an errata item, though > whether the Forum will accept that as an errata remains to be seen. It is > important to note that this option would mean that we will not have a way to > represent the UNIFIED_NOT_REQUIRING_WIN_SYNC model. > > 2. Change the wording in the MPI standard so "UNIFIED" means > UNIFIED_NOT_REQUIRING_WIN_SYNC, but add another memory model for > UNIFIED_REQUIRING_WIN_SYNC. This option only got my vote so far :-). > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 26 Aug '13

26 Aug '13

Keith, On 08/26/2013 08:00 PM, Underwood, Keith D wrote: >> Well, after asking somebody what happens with SHMEM on such >> architectures, I believe they basically tell the user "program this like you >> would a multithreaded program - all such rules apply". I don't see why we >> don't take that approach. It seems to me to be the closest to the objective >> of unified... I think your comment is really talking about the previous thread of emails on this topic. Let me try to clarify this here, but that was not the intent of this email thread. There are really three memory models: SEPARATE, UNIFIED_REQUIRING_WIN_SYNC, and UNIFIED_NOT_REQUIRING_WIN_SYNC. SEPARATE is what we already have in the standard. UNIFIED_REQUIRING_WIN_SYNC is where the target has to call WIN_SYNC to synchronize the public/private windows for overlapping data, but nonoverlapping data can be accessed without synchronization. UNIFIED_NOT_REQUIRING_WIN_SYNC is where the target data is always synchronized between the public/private windows. Some folks on the WG thought that "UNIFIED" in the MPI-3 standard meant UNIFIED_REQUIRING_WIN_SYNC, while others thought it meant UNIFIED_NOT_REQUIRING_WIN_SYNC. So the chapter is in an inconsistent state right now. Most of the previous email thread was to come to a consensus on these different memory models and we agreed that these three exist, irrespective of whether we want to support them in the MPI standard or not. In this email thread, we are trying to pick one of two options: 1. Change the wording in the MPI standard so "UNIFIED" means UNIFIED_REQUIRING_WIN_SYNC. This seems to be the option that got most votes so far. In fact, folks feel that this can be done as an errata item, though whether the Forum will accept that as an errata remains to be seen. It is important to note that this option would mean that we will not have a way to represent the UNIFIED_NOT_REQUIRING_WIN_SYNC model. 2. Change the wording in the MPI standard so "UNIFIED" means UNIFIED_NOT_REQUIRING_WIN_SYNC, but add another memory model for UNIFIED_REQUIRING_WIN_SYNC. This option only got my vote so far :-). -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Underwood, Keith D 26 Aug '13

26 Aug '13

Bitten by the migration :-) > -----Original Message----- > From: Underwood, Keith D > Sent: Monday, August 26, 2013 8:46 PM > To: 'MPI 3.0 Remote Memory Access working group' > Subject: RE: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates > > Well, after asking somebody what happens with SHMEM on such > architectures, I believe they basically tell the user "program this like you > would a multithreaded program - all such rules apply". I don't see why we > don't take that approach. It seems to me to be the closest to the objective > of unified... > > > -----Original Message----- > > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > > bounces(a)lists.mpi-forum.org] On Behalf Of Pavan Balaji > > Sent: Thursday, August 15, 2013 12:26 PM > > To: mpi3-rma(a)lists.mpi-forum.org > > Subject: Re: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates > > > > > > Before I go out and start writing up the text for this, are there > > other comments on this? > > > > Keith/Torsten/Jeff/Bill/Rajeev: thoughts? > > > > On 08/06/2013 10:09 AM, Barrett, Brian W wrote: > > > On 8/6/13 7:30 AM, "Jim Dinan" <james.dinan(a)gmail.com> wrote: > > > > > >> My preference would be to have the fewer memory models in the > > standard. > > >> I prefer option #1 -- that a window synchronization (e.g. Win_sync) > > >> can be used to order load/store operations with respect to actions > > >> performed by other processes in the target's window. If no > > >> ordering is enforced, the MPI standard does not define what is seen > > >> by load operations at the target process. As a rationale, the local > process' > > >> view of the window may not be consistent with the window because of > > >> performance optimizations or the consistency model of the > > >> underlying architecture. This would allow e.g. SHMEM > > >> implementations to still use MPI-3 RMA, but they would have to rely > > >> on a behavior that is defined by the architecture/implementation, as > they currently do. > > >> > > >> I think there's also a good chance that this could be an erratum, > > >> whereas a new memory model would have to go into a new version of > > the > > >> spec. If we were to decide later that we want a stronger memory > > >> model that defines the ordering seen by the target in the absence > > >> of synchronizations, this option would still allow us to add it later. > > > > > > I tend to agree. I also think the change can be quite small. > > > Section > > > 11.4 (pg 436, line 37-38) already says that updates will eventually > > > be seen by a load in UNIFIED, meaning that the FLUSH/SYNC example > > > that started this thread already requires a SYNC in order to be > > > correct (or a while (!updated) loop). So the primary thing we need > > > to do is to clarify that the "identical" refers to their view from > > > memory, and not their view from a reordering processor, so Sync (or > > > other system-specific operations) are required to force ordering. > > > This really feels > > like an erratum to me. > > > > > > Brian > > > > > > -- > > > Brian W. Barrett > > > Scalable System Software Group > > > Sandia National Laboratories > > > > > > > > > > > > _______________________________________________ > > > mpi3-rma mailing list > > > mpi3-rma(a)lists.mpi-forum.org > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0