December 2012 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Rajeev Thakur 03 Dec '12

03 Dec '12

Looking at the exact sentence in the standard, 455:32-35, maybe it is already correct as it is. "A put or accumulate must not access a target window once a load/store update or a put or accumulate update to another (overlapping) target window has started on a location in the target window, until the update becomes visible in the public copy of the window. ....., the restriction applies to operations even if they access disjoint locations in the window." The overlapping in "A put or accumulate update to another (overlapping) target window" is probably intended there because if it is the same window, concurrent puts/accumulates to distinct parts of the same window are allowed (in the absence of local loads/stores). Rajeev On Dec 3, 2012, at 2:22 PM, Rajeev Thakur wrote: > The change suggested on slide 2 is not right. > > In the existing standard, for the separate memory model, the restriction is that a put/accumulate must not access a target window once a local update (to anywhere in the window) has started or a put/get (to *overlapping memory locations* in the window) has started. > > You are allowed to do concurrent put/gets to distinct memory locations in the same window. > > So the change needed is say "put/get to overlapping memory locations" instead of "put/get to an overlapping window". > > Rajeev > > > > On Dec 1, 2012, at 8:21 PM, Pavan Balaji wrote: > >> All, >> >> Please see the attached slides as consideration items for MPI-3.1. >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> <mpi31-rma.pptx>_______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Rajeev Thakur 03 Dec '12

03 Dec '12

The change suggested on slide 2 is not right. In the existing standard, for the separate memory model, the restriction is that a put/accumulate must not access a target window once a local update (to anywhere in the window) has started or a put/get (to *overlapping memory locations* in the window) has started. You are allowed to do concurrent put/gets to distinct memory locations in the same window. So the change needed is say "put/get to overlapping memory locations" instead of "put/get to an overlapping window". Rajeev On Dec 1, 2012, at 8:21 PM, Pavan Balaji wrote: > All, > > Please see the attached slides as consideration items for MPI-3.1. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > <mpi31-rma.pptx>_______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Jim Dinan 03 Dec '12

03 Dec '12

Hubert, Thanks for your comment -- in general, the RMA group is thinking about the vector systems. Is there a desire from users to have shared memory windows defined for this platform? Keeping in mind that the synchronization required will be harder to reason about than get/put? ~Jim. On 12/3/12 10:24 AM, Hubert Ritzdorf wrote: > NEC SX (including the Earth Simulator) memory is not cache coherent and > MPI-2 RMA is working since many years on these systems. > > Hubert > ________________________________________ > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mpi3-rma-bounces(a)lists.mpi-forum.org] on behalf of Jim Dinan [dinan(a)mcs.anl.gov] > Sent: Monday, December 03, 2012 4:39 PM > To: mpi3-rma(a)lists.mpi-forum.org > Subject: Re: [Mpi3-rma] MPI-3.1 consideration slides > > On 12/2/12 7:44 AM, Jeff Hammond wrote: >>>> (1) Define semantics of shared memory windows in separate model. >>>> >>>> In Section 11.2.3 it says "MPI does not define semantics for accessing >>>> shared memory windows in the separate memory model." >>>> >>>> I would like to try to define these semantics in MPI 3.1. Shouldn't >>>> it at least be possible to use MPI_WIN_(UN)LOCK with >>>> MPI_LOCK_EXCLUSIVE here? >>>> >>>> I believe that supporting RMA for weakly- or non- coherent memory >>>> architectures is vital for future systems. >>> I agree. We had this discussion and decided to postpone it because we >>> had other more pressing issues. Now may be the time. A discussion would >>> be good. >> >> Yes, there was no urgency. I'm glad the standard left this issue wide >> open instead of defining something restrictive. > > IMHO, defining shared memory windows in the separate model is not a > worthy use of the Forum's time. This will be hard to get right, and > AFAIK no (reasonable) system requires it. As we've currently left it, > if this functionality is needed, one can define the semantics as an MPI > extension. I think this is a better option that defining a new model in > the absence of a platform that uses it. > > ~Jim. > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > Click https://www.mailcontrol.com/sr/BMhj+WAEk+nGX2PQPOmvUpFBc4ZD8M8UQDU1HKsp9BM+… to report this email as spam. > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

Re: [Mpi3-rma] missing RMA info keys
by Jim Dinan 03 Dec '12

03 Dec '12

On 12/3/12 10:33 AM, Jeff Hammond wrote: > yes, to summarize what Hubert and Jim said and add my own comments: > > * accumulate_ordering and accumulate_ops should be valid for all > window types (unless shared memory windows allow ONLY load-store, then > we can remove them from MPI_WIN_ALLOCATE_SHARED). Shared memory windows are not restricted to only load/store accesses. This info key should be valid for all window types. > * same_size is primarily useful for MPI_WIN_ALLOCATE but it would be > valid and mildly useful for MPI_WIN_CREATE. for example, if i want to > do valid-range checking and i don't know same_size, i may need an O(N) > list of (base,max_offset), whereas same_size allows me to have only > O(N) for base and O(1)=1 for max_offset. same_size is potentially useful for MPI_Win_allocate_shared, and is not useful for MPI_Win_create_dynamic. > * alloc_shared_noncontig only makes sense for MPI_WIN_ALLOCATE_SHARED Yep. ~Jim.

1 0

Re: [Mpi3-rma] missing RMA info keys
by Jeff Hammond 03 Dec '12

03 Dec '12

yes, to summarize what Hubert and Jim said and add my own comments: * accumulate_ordering and accumulate_ops should be valid for all window types (unless shared memory windows allow ONLY load-store, then we can remove them from MPI_WIN_ALLOCATE_SHARED). * same_size is primarily useful for MPI_WIN_ALLOCATE but it would be valid and mildly useful for MPI_WIN_CREATE. for example, if i want to do valid-range checking and i don't know same_size, i may need an O(N) list of (base,max_offset), whereas same_size allows me to have only O(N) for base and O(1)=1 for max_offset. * alloc_shared_noncontig only makes sense for MPI_WIN_ALLOCATE_SHARED best, jeff On Mon, Dec 3, 2012 at 8:17 AM, Hubert Ritzdorf <Hubert.Ritzdorf(a)emea.nec.com> wrote: > Hi, > > I think that the info argument "same_size" is defined only for > MPI_Win_Allocate. It's defined in the context of this function > and it's explicitly an additional info relative to the info keys > of MPI_Win_create. > > MPI_Win_allocate is not mentioned in the list of Page 409 Line 48 > and "same_size" doesn't make sense for MPI_Win_allocate_shared. > > MPI_Win_create_dynamic specifies only the info keys of MPI_Win_create > (Page 412 Line 22-23) > and "same_size" doesn't make sense since MPI_Win_attach is not collective. > > Hubert > > > ________________________________________ > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mpi3-rma-bounces(a)lists.mpi-forum.org] on behalf of Jim Dinan [dinan(a)mcs.anl.gov] > Sent: Monday, December 03, 2012 4:07 PM > To: mpi3-rma(a)lists.mpi-forum.org > Subject: Re: [Mpi3-rma] missing RMA info keys > > Hi Jeff, > > I haven't looked at the text in a while -- alloc_shared_noncontig only > makes sense in a call to MPI_Win_allocate_shared(). The other info keys > were intended to be valid for all communicator creation routines. If > this isn't clear, then we should definitely correct it. I created a > ticket to make sure we remember to look at this: > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/346 > > ~Jim. > > On 12/1/12 8:50 PM, Jeff Hammond wrote: >> It is not clear if these info keys are predefined for MPI_WIN_ALLOCATE >> or MPI_WIN_CREATE_DYNAMIC. >> >> Does the following text mean that the info keys for MPI_WIN_ALLOCATE >> are inclusive of those for MPI_WIN_CREATE? It really does not seem >> that way to me. >> >> ======================== >> The info argument can be used to specify hints similar to the info >> argument for MPI_WIN_CREATE and MPI_ALLOC_MEM. The following info key >> is predefined: >> >> same_size ï¿½ if set to true, then the implementation may assume that >> the argument size is identical on all processes. >> ======================== >> >> If the keys "accumulate_ordering, accumulate_ops, same_size, >> alloc_shared_noncontig" are not valid for MPI_WIN_ALLOCATE and/or >> MPI_WIN_CREATE_DYNAMIC already, they should be this task should be >> added to the TODO list for MPI 3.1. >> >> Jeff >> >> On Tue, Nov 6, 2012 at 4:41 AM, Hubert Ritzdorf >> <Hubert.Ritzdorf(a)emea.nec.com> wrote: >>> Hi, >>> >>> the new RMA info keys "accumulate_ordering, accumulate_ops, same_size, >>> alloc_shared_noncontig" >>> and the corresponding values are missing in Sections A.1.5 and A.1.6 (Pages >>> 683 and 684). >>> >>> Hubert >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> >> > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > Click https://www.mailcontrol.com/sr/Bnj6dm!ZzevGX2PQPOmvUvSoGVaHHSWCqoZfLcnlBGwq… to report this email as spam. > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Hubert Ritzdorf 03 Dec '12

03 Dec '12

NEC SX (including the Earth Simulator) memory is not cache coherent and MPI-2 RMA is working since many years on these systems. Hubert ________________________________________ From: mpi3-rma-bounces(a)lists.mpi-forum.org [mpi3-rma-bounces(a)lists.mpi-forum.org] on behalf of Jim Dinan [dinan(a)mcs.anl.gov] Sent: Monday, December 03, 2012 4:39 PM To: mpi3-rma(a)lists.mpi-forum.org Subject: Re: [Mpi3-rma] MPI-3.1 consideration slides On 12/2/12 7:44 AM, Jeff Hammond wrote: >>> (1) Define semantics of shared memory windows in separate model. >>> >>> In Section 11.2.3 it says "MPI does not define semantics for accessing >>> shared memory windows in the separate memory model." >>> >>> I would like to try to define these semantics in MPI 3.1. Shouldn't >>> it at least be possible to use MPI_WIN_(UN)LOCK with >>> MPI_LOCK_EXCLUSIVE here? >>> >>> I believe that supporting RMA for weakly- or non- coherent memory >>> architectures is vital for future systems. >> I agree. We had this discussion and decided to postpone it because we >> had other more pressing issues. Now may be the time. A discussion would >> be good. > > Yes, there was no urgency. I'm glad the standard left this issue wide > open instead of defining something restrictive. IMHO, defining shared memory windows in the separate model is not a worthy use of the Forum's time. This will be hard to get right, and AFAIK no (reasonable) system requires it. As we've currently left it, if this functionality is needed, one can define the semantics as an MPI extension. I think this is a better option that defining a new model in the absence of a platform that uses it. ~Jim. _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma Click https://www.mailcontrol.com/sr/BMhj+WAEk+nGX2PQPOmvUpFBc4ZD8M8UQDU1HKsp9BM+… to report this email as spam.

1 0

Re: [Mpi3-rma] missing RMA info keys
by Hubert Ritzdorf 03 Dec '12

03 Dec '12

Hi, I think that the info argument "same_size" is defined only for MPI_Win_Allocate. It's defined in the context of this function and it's explicitly an additional info relative to the info keys of MPI_Win_create. MPI_Win_allocate is not mentioned in the list of Page 409 Line 48 and "same_size" doesn't make sense for MPI_Win_allocate_shared. MPI_Win_create_dynamic specifies only the info keys of MPI_Win_create (Page 412 Line 22-23) and "same_size" doesn't make sense since MPI_Win_attach is not collective. Hubert ________________________________________ From: mpi3-rma-bounces(a)lists.mpi-forum.org [mpi3-rma-bounces(a)lists.mpi-forum.org] on behalf of Jim Dinan [dinan(a)mcs.anl.gov] Sent: Monday, December 03, 2012 4:07 PM To: mpi3-rma(a)lists.mpi-forum.org Subject: Re: [Mpi3-rma] missing RMA info keys Hi Jeff, I haven't looked at the text in a while -- alloc_shared_noncontig only makes sense in a call to MPI_Win_allocate_shared(). The other info keys were intended to be valid for all communicator creation routines. If this isn't clear, then we should definitely correct it. I created a ticket to make sure we remember to look at this: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/346 ~Jim. On 12/1/12 8:50 PM, Jeff Hammond wrote: > It is not clear if these info keys are predefined for MPI_WIN_ALLOCATE > or MPI_WIN_CREATE_DYNAMIC. > > Does the following text mean that the info keys for MPI_WIN_ALLOCATE > are inclusive of those for MPI_WIN_CREATE? It really does not seem > that way to me. > > ======================== > The info argument can be used to specify hints similar to the info > argument for MPI_WIN_CREATE and MPI_ALLOC_MEM. The following info key > is predefined: > > same_size ï¿½ if set to true, then the implementation may assume that > the argument size is identical on all processes. > ======================== > > If the keys "accumulate_ordering, accumulate_ops, same_size, > alloc_shared_noncontig" are not valid for MPI_WIN_ALLOCATE and/or > MPI_WIN_CREATE_DYNAMIC already, they should be this task should be > added to the TODO list for MPI 3.1. > > Jeff > > On Tue, Nov 6, 2012 at 4:41 AM, Hubert Ritzdorf > <Hubert.Ritzdorf(a)emea.nec.com> wrote: >> Hi, >> >> the new RMA info keys "accumulate_ordering, accumulate_ops, same_size, >> alloc_shared_noncontig" >> and the corresponding values are missing in Sections A.1.5 and A.1.6 (Pages >> 683 and 684). >> >> Hubert >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma Click https://www.mailcontrol.com/sr/Bnj6dm!ZzevGX2PQPOmvUvSoGVaHHSWCqoZfLcnlBGwq… to report this email as spam.

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Jeff Hammond 03 Dec '12

03 Dec '12

Yes, and it is unfortunate that there is a dearth of representative platforms. However, MPI-2 RMA worked to get this right and it's generally accepted that those semantics support non-coherent platforms, so in the absence of an argument that MPI-2 RMA semantics don't work for non-coherent platforms, I would argue that we can just use those. Jeff On Mon, Dec 3, 2012 at 7:55 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > On 12/3/12 9:44 AM, Jeff Hammond wrote: >> >> On Mon, Dec 3, 2012 at 7:39 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >>> >>> On 12/2/12 7:44 AM, Jeff Hammond wrote: >>>>>> >>>>>> >>>>>> (1) Define semantics of shared memory windows in separate model. >>>>>> >>>>>> In Section 11.2.3 it says "MPI does not define semantics for accessing >>>>>> shared memory windows in the separate memory model." >>>>>> >>>>>> I would like to try to define these semantics in MPI 3.1. Shouldn't >>>>>> it at least be possible to use MPI_WIN_(UN)LOCK with >>>>>> MPI_LOCK_EXCLUSIVE here? >>>>>> >>>>>> I believe that supporting RMA for weakly- or non- coherent memory >>>>>> architectures is vital for future systems. >>>>> >>>>> >>>>> I agree. We had this discussion and decided to postpone it because we >>>>> had other more pressing issues. Now may be the time. A discussion would >>>>> be good. >>>> >>>> >>>> >>>> Yes, there was no urgency. I'm glad the standard left this issue wide >>>> open instead of defining something restrictive. >>> >>> >>> >>> IMHO, defining shared memory windows in the separate model is not a >>> worthy >>> use of the Forum's time. This will be hard to get right, and AFAIK no >>> (reasonable) system requires it. As we've currently left it, if this >>> functionality is needed, one can define the semantics as an MPI >>> extension. >>> I think this is a better option that defining a new model in the absence >>> of >>> a platform that uses it. >> >> >> Intel SCC is non-coherent but supports POSIX shared memory. It's >> really quite simple to use there, but one has to have synchronization >> primitives. I just want the Forum to define those in the same manner >> as for MPI-2 RMA, which supports non-coherent platforms. >> >> What's the harm in saying that MPI_WIN_UNLOCK approach causes global >> visibility of the window to all processes? > > > I think that what you suggested would be a useful and reasonable extension > to an MPI implementation. However, the SCC is still a research prototype; I > don't think it has reached the level of maturity needed to define semantics > in a standards document. > > > ~Jim. > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Jim Dinan 03 Dec '12

03 Dec '12

Hi All, Unfortunately, I won't be able to attend the meeting this week. I can join the working group sessions by phone if we are able to set something up. I would like to add the following items to the working group's agenda: MPI 3.0 errata: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/346 https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/347 https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/348 https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/350 Corrections for MPI 3.next: Shared memory window attribute query mechanism: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/344 Dynamic window displacements: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/349 Thanks, ~Jim. On 12/2/12 8:53 PM, Pavan Balaji wrote: > > Attached is a slightly updated version of the slides. I had forgotten > to include one additional item (in the last slide of the attached slide > deck). > > -- Pavan > > On 12/02/2012 09:30 AM US Central Time, Pavan Balaji wrote: >> >> On 12/02/2012 05:27 AM US Central Time, Torsten Hoefler wrote: >>> The only thing I disagree to is slide 6. Win_sync really does nothing >>> in the separate case. Your example would work on shared memory if >>> flush issues a memory barrier *and* MPI_Barrier() is an optimization >>> barrier (which is is in C and needs to be ensured in Fortran anyway). >>> But we can discuss next week. >> >> The problem here is that MPI_BARRIER is a process barrier, not a memory >> barrier. FWIW, with current compilers, which do not reorder >> instructions across functions, this should work correctly. But this is >> a correctness concern when compilers do more than that, e.g., IPC >> optimizations. >> >> An alternative would be to leave it as-is, and add a note that we assume >> the compiler will not reorder instructions here because it's a function >> call, though I don't prefer this alternative. >> >> -- Pavan >> > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Jim Dinan 03 Dec '12

03 Dec '12

On 12/3/12 9:44 AM, Jeff Hammond wrote: > On Mon, Dec 3, 2012 at 7:39 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >> On 12/2/12 7:44 AM, Jeff Hammond wrote: >>>>> >>>>> (1) Define semantics of shared memory windows in separate model. >>>>> >>>>> In Section 11.2.3 it says "MPI does not define semantics for accessing >>>>> shared memory windows in the separate memory model." >>>>> >>>>> I would like to try to define these semantics in MPI 3.1. Shouldn't >>>>> it at least be possible to use MPI_WIN_(UN)LOCK with >>>>> MPI_LOCK_EXCLUSIVE here? >>>>> >>>>> I believe that supporting RMA for weakly- or non- coherent memory >>>>> architectures is vital for future systems. >>>> >>>> I agree. We had this discussion and decided to postpone it because we >>>> had other more pressing issues. Now may be the time. A discussion would >>>> be good. >>> >>> >>> Yes, there was no urgency. I'm glad the standard left this issue wide >>> open instead of defining something restrictive. >> >> >> IMHO, defining shared memory windows in the separate model is not a worthy >> use of the Forum's time. This will be hard to get right, and AFAIK no >> (reasonable) system requires it. As we've currently left it, if this >> functionality is needed, one can define the semantics as an MPI extension. >> I think this is a better option that defining a new model in the absence of >> a platform that uses it. > > Intel SCC is non-coherent but supports POSIX shared memory. It's > really quite simple to use there, but one has to have synchronization > primitives. I just want the Forum to define those in the same manner > as for MPI-2 RMA, which supports non-coherent platforms. > > What's the harm in saying that MPI_WIN_UNLOCK approach causes global > visibility of the window to all processes? I think that what you suggested would be a useful and reasonable extension to an MPI implementation. However, the SCC is still a research prototype; I don't think it has reached the level of maturity needed to define semantics in a standards document. ~Jim.

1 0