August 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jeff Hammond 27 Aug '13

27 Aug '13

I thought OpenSHMEM 1.0 was required to be SGI-SHMEM with minor corrections for legal reasons but that future specs can be improved to behave in a self-consistent way, have a finalize routine, etc. We should consider writing something for IWOSH if we think OpenSHMEM needs to be changed to work over MPI-3. Now is a good time to figure out the give and take between the two specs. Jeff Sent from my iPhone On Aug 27, 2013, at 6:20 PM, "Underwood, Keith D" <keith.d.underwood(a)intel.com> wrote: >>> I'm having trouble with the exact phrasing, but my thought is that we >>> should have an advice to users that says 1) while the view of memory >>> is consistent in UNIFIED, other artifacts (such as processor or memory >>> controller reordering) can result in unexpected behavior 2) >>> MPI_WIN_SYNC may be used to provide stronger ordering guarantees and >>> 3) an application programmer may use other, implementation defined, >>> semantics to provide required ordering guarantees. >> >> Isn't (3) true for any MPI call in general, since the user can use >> implementation-defined alternate mechanisms for any of them? >> >> Also, I think we are ignoring the point Jim raised: does OpenSHMEM really >> need these semantics? > > Having pinged Jim, OpenSHMEM doesn't really say anything one way or the other. They have shmem_*_wait(), but don't actually *say* that you *have* to use it. Then again, there are a lot of things left unsaid... > > (3) might be generally true, but we do have to be careful not to say these things are "undefined". And, since we are making this hubbub, wouldn't it be nice to include the "why" in the spec? heck, if we don't, we won't all agree about what we meant by it when next August rolls around... > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Underwood, Keith D 27 Aug '13

27 Aug '13

> > I'm having trouble with the exact phrasing, but my thought is that we > > should have an advice to users that says 1) while the view of memory > > is consistent in UNIFIED, other artifacts (such as processor or memory > > controller reordering) can result in unexpected behavior 2) > > MPI_WIN_SYNC may be used to provide stronger ordering guarantees and > > 3) an application programmer may use other, implementation defined, > > semantics to provide required ordering guarantees. > > Isn't (3) true for any MPI call in general, since the user can use > implementation-defined alternate mechanisms for any of them? > > Also, I think we are ignoring the point Jim raised: does OpenSHMEM really > need these semantics? Having pinged Jim, OpenSHMEM doesn't really say anything one way or the other. They have shmem_*_wait(), but don't actually *say* that you *have* to use it. Then again, there are a lot of things left unsaid... (3) might be generally true, but we do have to be careful not to say these things are "undefined". And, since we are making this hubbub, wouldn't it be nice to include the "why" in the spec? heck, if we don't, we won't all agree about what we meant by it when next August rolls around...

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Underwood, Keith D 27 Aug '13

27 Aug '13

I could live with "implementation defined", but I don't think we tend to use that phrasing. When MPI uses the words "undefined", they tend to mean "the implementation may return the results of /dev/random". From: mpiwg-rma [mailto:[email protected]] On Behalf Of Jeff Hammond Sent: Tuesday, August 27, 2013 3:20 PM To: MPI WG Remote Memory Access working group Subject: Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates How does what you want not meet the definition of "implementation defined" rather than "defined"? I know this is semantic hair-splitting, but you are not proposing to define the semantics of RMA in the way that one normally thinks of the term "defined". I don't see how your "definition" is any more of a definition than saying that one can either use MPI_RECV or "some combination of function calls and instructions that would allow the user to receive the data and set the message queue in the proper state as if it were RECVed" since what you are saying we're going to "define" is the user can employ MPI_WIN_SYNC or "some combination of function calls and instructions that would allow the user to synchronize the public and private windows in the same manner as WIN_SYNC would do". Jeff On Tue, Aug 27, 2013 at 11:38 AM, Underwood, Keith D <keith.d.underwood(a)intel.com<mailto:[email protected]>> wrote: Yes, I want it to be defined. I don't think that version of defined is a problem. The definition just has to happen in terms of what is in memory and say "an implementation might reorder your loads in fun and exciting ways" ;-) This is pretty close to just saying the public and private window are the same. Undefined in the way MPI phrases it allows a unified where it is impossible to use it without the MPI_WIN_SYNC. That isn't ok. From: mpiwg-rma [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Jeff Hammond Sent: Tuesday, August 27, 2013 2:31 PM To: Pavan Balaji Cc: MPI Forum Subject: Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates Keith's definition is not a definition in the sense of MPI semantics being "defined". What he is proposing is to be pedantic about the fact that the user may want to rely upon implementation-defined behavior to realize the best performance or to avoid changing existing SHMEM code. The MPI standard isn't defining something if it's not explicitly defined in the standard. Therefore, it must be called "undefined". If you or Keith want to call this "defined" behavior, then I'm going to insist that it be defined in the usual sense, which means the text of the standard will enumerate the behavior of MPI in an architecture-independent way. Jeff On Tue, Aug 27, 2013 at 11:19 AM, Pavan Balaji <balaji(a)mcs.anl.gov<mailto:[email protected]>> wrote: On 08/27/2013 12:57 PM, Jeff Hammond wrote: The MPI standard will not define by what mechanism the SHMEM implementation will establish the necessary consistency. This is what I mean by undefined = implementation-defined. I do not think that Keith is proposing to enumerate all processor memory models and the appropriate fences required to achieve the affect of win_sync but at lower cost. Thus, it is the implementation - or perhaps more accurately, the platform on which the implementation resides - that will define these semantics. Here's what I understood from Keith -- He wants a statement in the standard that says the application can do platform-specific memory consistency to synchronize the public and private windows in UNIFIED. I do agree that that's a very flimsy statement and doesn't really mean much. But I'm calling it a definition, while I think you are saying that it's pretty much undefined. I like leaving it totally undefined better than the above "definition". -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]>

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 27 Aug '13

27 Aug '13

Hi Brian, On 08/27/2013 01:49 PM, Barrett, Brian W wrote: > I'm having trouble with the exact phrasing, but my thought is that we > should have an advice to users that says 1) while the view of memory is > consistent in UNIFIED, other artifacts (such as processor or memory > controller reordering) can result in unexpected behavior 2) MPI_WIN_SYNC > may be used to provide stronger ordering guarantees and 3) an application > programmer may use other, implementation defined, semantics to provide > required ordering guarantees. Isn't (3) true for any MPI call in general, since the user can use implementation-defined alternate mechanisms for any of them? Also, I think we are ignoring the point Jim raised: does OpenSHMEM really need these semantics? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jeff Hammond 27 Aug '13

27 Aug '13

How does what you want not meet the definition of "implementation defined" rather than "defined"? I know this is semantic hair-splitting, but you are not proposing to define the semantics of RMA in the way that one normally thinks of the term "defined". I don't see how your "definition" is any more of a definition than saying that one can either use MPI_RECV or "some combination of function calls and instructions that would allow the user to receive the data and set the message queue in the proper state as if it were RECVed" since what you are saying we're going to "define" is the user can employ MPI_WIN_SYNC or "some combination of function calls and instructions that would allow the user to synchronize the public and private windows in the same manner as WIN_SYNC would do". Jeff On Tue, Aug 27, 2013 at 11:38 AM, Underwood, Keith D < keith.d.underwood(a)intel.com> wrote: > Yes, I want it to be defined. I don’t think that version of defined is > a problem. The definition just has to happen in terms of what is in memory > and say “an implementation might reorder your loads in fun and exciting > ways” ;-) This is pretty close to just saying the public and private > window are the same. **** > > ** ** > > Undefined in the way MPI phrases it allows a unified where it is > impossible to use it without the MPI_WIN_SYNC. That isn’t ok.**** > > ** ** > > *From:* mpiwg-rma [mailto:[email protected]] *On > Behalf Of *Jeff Hammond > *Sent:* Tuesday, August 27, 2013 2:31 PM > *To:* Pavan Balaji > *Cc:* MPI Forum > > *Subject:* Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED > model updates**** > > ** ** > > Keith's definition is not a definition in the sense of MPI semantics being > "defined". What he is proposing is to be pedantic about the fact that the > user may want to rely upon implementation-defined behavior to realize the > best performance or to avoid changing existing SHMEM code.**** > > ** ** > > The MPI standard isn't defining something if it's not explicitly defined > in the standard. Therefore, it must be called "undefined". If you or > Keith want to call this "defined" behavior, then I'm going to insist that > it be defined in the usual sense, which means the text of the standard will > enumerate the behavior of MPI in an architecture-independent way.**** > > ** ** > > Jeff**** > > ** ** > > On Tue, Aug 27, 2013 at 11:19 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > **** > > > On 08/27/2013 12:57 PM, Jeff Hammond wrote:**** > > The MPI standard will not define by what mechanism the SHMEM > implementation will establish the necessary consistency. This is what I > mean by undefined = implementation-defined. I do not think that Keith > is proposing to enumerate all processor memory models and the > appropriate fences required to achieve the affect of win_sync but at > lower cost. Thus, it is the implementation - or perhaps more > accurately, the platform on which the implementation resides - that will > define these semantics.**** > > ** ** > > Here's what I understood from Keith -- > > He wants a statement in the standard that says the application can do > platform-specific memory consistency to synchronize the public and private > windows in UNIFIED. > > I do agree that that's a very flimsy statement and doesn't really mean > much. But I'm calling it a definition, while I think you are saying that > it's pretty much undefined. > > I like leaving it totally undefined better than the above "definition".*** > * > > > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji**** > > > > **** > > ** ** > > -- > Jeff Hammond > jeff.science(a)gmail.com **** > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Barrett, Brian W 27 Aug '13

27 Aug '13

On 8/27/13 12:19 PM, "Pavan Balaji" <balaji(a)mcs.anl.gov> wrote: >On 08/27/2013 12:57 PM, Jeff Hammond wrote: >> The MPI standard will not define by what mechanism the SHMEM >> implementation will establish the necessary consistency. This is what I >> mean by undefined = implementation-defined. I do not think that Keith >> is proposing to enumerate all processor memory models and the >> appropriate fences required to achieve the affect of win_sync but at >> lower cost. Thus, it is the implementation - or perhaps more >> accurately, the platform on which the implementation resides - that will >> define these semantics. > >Here's what I understood from Keith -- > >He wants a statement in the standard that says the application can do >platform-specific memory consistency to synchronize the public and >private windows in UNIFIED. > >I do agree that that's a very flimsy statement and doesn't really mean >much. But I'm calling it a definition, while I think you are saying >that it's pretty much undefined. > >I like leaving it totally undefined better than the above "definition". I'm having trouble with the exact phrasing, but my thought is that we should have an advice to users that says 1) while the view of memory is consistent in UNIFIED, other artifacts (such as processor or memory controller reordering) can result in unexpected behavior 2) MPI_WIN_SYNC may be used to provide stronger ordering guarantees and 3) an application programmer may use other, implementation defined, semantics to provide required ordering guarantees. That is, we should say explicitly that the user can cause oddness, that MPI has a mechanism to fix that oddness, and that the user may also seek other mechanisms to solve the oddness, should she so desire. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 27 Aug '13

27 Aug '13

Hi Keith, Did you see Jim's email? He raised a good point that even with OpenSHMEM this is undefined. You need to use a function call to make it have defined behavior. So is it still true that OpenSHMEM cannot be implemented on top of MPI? -- Pavan On 08/27/2013 01:38 PM, Underwood, Keith D wrote: > Yes, I want it to be defined. I donï¿½t think that version of defined is > a problem. The definition just has to happen in terms of what is in > memory and say ï¿½an implementation might reorder your loads in fun and > exciting waysï¿½ ;-) This is pretty close to just saying the public and > private window are the same. > > Undefined in the way MPI phrases it allows a unified where it is > impossible to use it without the MPI_WIN_SYNC. That isnï¿½t ok. > > *From:*mpiwg-rma [mailto:[email protected]] *On > Behalf Of *Jeff Hammond > *Sent:* Tuesday, August 27, 2013 2:31 PM > *To:* Pavan Balaji > *Cc:* MPI Forum > *Subject:* Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED > model updates > > Keith's definition is not a definition in the sense of MPI semantics > being "defined". What he is proposing is to be pedantic about the fact > that the user may want to rely upon implementation-defined behavior to > realize the best performance or to avoid changing existing SHMEM code. > > The MPI standard isn't defining something if it's not explicitly defined > in the standard. Therefore, it must be called "undefined". If you or > Keith want to call this "defined" behavior, then I'm going to insist > that it be defined in the usual sense, which means the text of the > standard will enumerate the behavior of MPI in an > architecture-independent way. > > Jeff > > On Tue, Aug 27, 2013 at 11:19 AM, Pavan Balaji <balaji(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > > On 08/27/2013 12:57 PM, Jeff Hammond wrote: > > The MPI standard will not define by what mechanism the SHMEM > implementation will establish the necessary consistency. This is what I > mean by undefined = implementation-defined. I do not think that Keith > is proposing to enumerate all processor memory models and the > appropriate fences required to achieve the affect of win_sync but at > lower cost. Thus, it is the implementation - or perhaps more > accurately, the platform on which the implementation resides - that will > define these semantics. > > Here's what I understood from Keith -- > > He wants a statement in the standard that says the application can do > platform-specific memory consistency to synchronize the public and > private windows in UNIFIED. > > I do agree that that's a very flimsy statement and doesn't really mean > much. But I'm calling it a definition, while I think you are saying > that it's pretty much undefined. > > I like leaving it totally undefined better than the above "definition". > > > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Underwood, Keith D 27 Aug '13

27 Aug '13

Yes, I want it to be defined. I don't think that version of defined is a problem. The definition just has to happen in terms of what is in memory and say "an implementation might reorder your loads in fun and exciting ways" ;-) This is pretty close to just saying the public and private window are the same. Undefined in the way MPI phrases it allows a unified where it is impossible to use it without the MPI_WIN_SYNC. That isn't ok. From: mpiwg-rma [mailto:[email protected]] On Behalf Of Jeff Hammond Sent: Tuesday, August 27, 2013 2:31 PM To: Pavan Balaji Cc: MPI Forum Subject: Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates Keith's definition is not a definition in the sense of MPI semantics being "defined". What he is proposing is to be pedantic about the fact that the user may want to rely upon implementation-defined behavior to realize the best performance or to avoid changing existing SHMEM code. The MPI standard isn't defining something if it's not explicitly defined in the standard. Therefore, it must be called "undefined". If you or Keith want to call this "defined" behavior, then I'm going to insist that it be defined in the usual sense, which means the text of the standard will enumerate the behavior of MPI in an architecture-independent way. Jeff On Tue, Aug 27, 2013 at 11:19 AM, Pavan Balaji <balaji(a)mcs.anl.gov<mailto:[email protected]>> wrote: On 08/27/2013 12:57 PM, Jeff Hammond wrote: The MPI standard will not define by what mechanism the SHMEM implementation will establish the necessary consistency. This is what I mean by undefined = implementation-defined. I do not think that Keith is proposing to enumerate all processor memory models and the appropriate fences required to achieve the affect of win_sync but at lower cost. Thus, it is the implementation - or perhaps more accurately, the platform on which the implementation resides - that will define these semantics. Here's what I understood from Keith -- He wants a statement in the standard that says the application can do platform-specific memory consistency to synchronize the public and private windows in UNIFIED. I do agree that that's a very flimsy statement and doesn't really mean much. But I'm calling it a definition, while I think you are saying that it's pretty much undefined. I like leaving it totally undefined better than the above "definition". -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]>

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Jeff Hammond 27 Aug '13

27 Aug '13

Keith's definition is not a definition in the sense of MPI semantics being "defined". What he is proposing is to be pedantic about the fact that the user may want to rely upon implementation-defined behavior to realize the best performance or to avoid changing existing SHMEM code. The MPI standard isn't defining something if it's not explicitly defined in the standard. Therefore, it must be called "undefined". If you or Keith want to call this "defined" behavior, then I'm going to insist that it be defined in the usual sense, which means the text of the standard will enumerate the behavior of MPI in an architecture-independent way. Jeff On Tue, Aug 27, 2013 at 11:19 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 08/27/2013 12:57 PM, Jeff Hammond wrote: > >> The MPI standard will not define by what mechanism the SHMEM >> implementation will establish the necessary consistency. This is what I >> mean by undefined = implementation-defined. I do not think that Keith >> is proposing to enumerate all processor memory models and the >> appropriate fences required to achieve the affect of win_sync but at >> lower cost. Thus, it is the implementation - or perhaps more >> accurately, the platform on which the implementation resides - that will >> define these semantics. >> > > Here's what I understood from Keith -- > > He wants a statement in the standard that says the application can do > platform-specific memory consistency to synchronize the public and private > windows in UNIFIED. > > I do agree that that's a very flimsy statement and doesn't really mean > much. But I'm calling it a definition, while I think you are saying that > it's pretty much undefined. > > I like leaving it totally undefined better than the above "definition". > > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] FW: [Mpi3-rma] [EXTERNAL] Re: MPI-3 UNIFIED model updates
by Pavan Balaji 27 Aug '13

27 Aug '13

On 08/27/2013 12:57 PM, Jeff Hammond wrote: > The MPI standard will not define by what mechanism the SHMEM > implementation will establish the necessary consistency. This is what I > mean by undefined = implementation-defined. I do not think that Keith > is proposing to enumerate all processor memory models and the > appropriate fences required to achieve the affect of win_sync but at > lower cost. Thus, it is the implementation - or perhaps more > accurately, the platform on which the implementation resides - that will > define these semantics. Here's what I understood from Keith -- He wants a statement in the standard that says the application can do platform-specific memory consistency to synchronize the public and private windows in UNIFIED. I do agree that that's a very flimsy statement and doesn't really mean much. But I'm calling it a definition, while I think you are saying that it's pretty much undefined. I like leaving it totally undefined better than the above "definition". -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0