December 2012 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Jeff Hammond 03 Dec '12

03 Dec '12

On Mon, Dec 3, 2012 at 7:39 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > On 12/2/12 7:44 AM, Jeff Hammond wrote: >>>> >>>> (1) Define semantics of shared memory windows in separate model. >>>> >>>> In Section 11.2.3 it says "MPI does not define semantics for accessing >>>> shared memory windows in the separate memory model." >>>> >>>> I would like to try to define these semantics in MPI 3.1. Shouldn't >>>> it at least be possible to use MPI_WIN_(UN)LOCK with >>>> MPI_LOCK_EXCLUSIVE here? >>>> >>>> I believe that supporting RMA for weakly- or non- coherent memory >>>> architectures is vital for future systems. >>> >>> I agree. We had this discussion and decided to postpone it because we >>> had other more pressing issues. Now may be the time. A discussion would >>> be good. >> >> >> Yes, there was no urgency. I'm glad the standard left this issue wide >> open instead of defining something restrictive. > > > IMHO, defining shared memory windows in the separate model is not a worthy > use of the Forum's time. This will be hard to get right, and AFAIK no > (reasonable) system requires it. As we've currently left it, if this > functionality is needed, one can define the semantics as an MPI extension. > I think this is a better option that defining a new model in the absence of > a platform that uses it. Intel SCC is non-coherent but supports POSIX shared memory. It's really quite simple to use there, but one has to have synchronization primitives. I just want the Forum to define those in the same manner as for MPI-2 RMA, which supports non-coherent platforms. What's the harm in saying that MPI_WIN_UNLOCK approach causes global visibility of the window to all processes? Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Jim Dinan 03 Dec '12

03 Dec '12

On 12/2/12 7:44 AM, Jeff Hammond wrote: >>> (1) Define semantics of shared memory windows in separate model. >>> >>> In Section 11.2.3 it says "MPI does not define semantics for accessing >>> shared memory windows in the separate memory model." >>> >>> I would like to try to define these semantics in MPI 3.1. Shouldn't >>> it at least be possible to use MPI_WIN_(UN)LOCK with >>> MPI_LOCK_EXCLUSIVE here? >>> >>> I believe that supporting RMA for weakly- or non- coherent memory >>> architectures is vital for future systems. >> I agree. We had this discussion and decided to postpone it because we >> had other more pressing issues. Now may be the time. A discussion would >> be good. > > Yes, there was no urgency. I'm glad the standard left this issue wide > open instead of defining something restrictive. IMHO, defining shared memory windows in the separate model is not a worthy use of the Forum's time. This will be hard to get right, and AFAIK no (reasonable) system requires it. As we've currently left it, if this functionality is needed, one can define the semantics as an MPI extension. I think this is a better option that defining a new model in the absence of a platform that uses it. ~Jim.

1 0

Re: [Mpi3-rma] missing RMA info keys
by Jim Dinan 03 Dec '12

03 Dec '12

Hi Jeff, I haven't looked at the text in a while -- alloc_shared_noncontig only makes sense in a call to MPI_Win_allocate_shared(). The other info keys were intended to be valid for all communicator creation routines. If this isn't clear, then we should definitely correct it. I created a ticket to make sure we remember to look at this: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/346 ~Jim. On 12/1/12 8:50 PM, Jeff Hammond wrote: > It is not clear if these info keys are predefined for MPI_WIN_ALLOCATE > or MPI_WIN_CREATE_DYNAMIC. > > Does the following text mean that the info keys for MPI_WIN_ALLOCATE > are inclusive of those for MPI_WIN_CREATE? It really does not seem > that way to me. > > ======================== > The info argument can be used to specify hints similar to the info > argument for MPI_WIN_CREATE and MPI_ALLOC_MEM. The following info key > is predefined: > > same_size ï¿½ if set to true, then the implementation may assume that > the argument size is identical on all processes. > ======================== > > If the keys "accumulate_ordering, accumulate_ops, same_size, > alloc_shared_noncontig" are not valid for MPI_WIN_ALLOCATE and/or > MPI_WIN_CREATE_DYNAMIC already, they should be this task should be > added to the TODO list for MPI 3.1. > > Jeff > > On Tue, Nov 6, 2012 at 4:41 AM, Hubert Ritzdorf > <Hubert.Ritzdorf(a)emea.nec.com> wrote: >> Hi, >> >> the new RMA info keys "accumulate_ordering, accumulate_ops, same_size, >> alloc_shared_noncontig" >> and the corresponding values are missing in Sections A.1.5 and A.1.6 (Pages >> 683 and 684). >> >> Hubert >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > >

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Pavan Balaji 02 Dec '12

02 Dec '12

Attached is a slightly updated version of the slides. I had forgotten to include one additional item (in the last slide of the attached slide deck). -- Pavan On 12/02/2012 09:30 AM US Central Time, Pavan Balaji wrote: > > On 12/02/2012 05:27 AM US Central Time, Torsten Hoefler wrote: >> The only thing I disagree to is slide 6. Win_sync really does nothing >> in the separate case. Your example would work on shared memory if >> flush issues a memory barrier *and* MPI_Barrier() is an optimization >> barrier (which is is in C and needs to be ensured in Fortran anyway). >> But we can discuss next week. > > The problem here is that MPI_BARRIER is a process barrier, not a memory > barrier. FWIW, with current compilers, which do not reorder > instructions across functions, this should work correctly. But this is > a correctness concern when compilers do more than that, e.g., IPC > optimizations. > > An alternative would be to leave it as-is, and add a note that we assume > the compiler will not reorder instructions here because it's a function > call, though I don't prefer this alternative. > > -- Pavan > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Pavan Balaji 02 Dec '12

02 Dec '12

On 12/02/2012 05:27 AM US Central Time, Torsten Hoefler wrote: > The only thing I disagree to is slide 6. Win_sync really does nothing > in the separate case. Your example would work on shared memory if > flush issues a memory barrier *and* MPI_Barrier() is an optimization > barrier (which is is in C and needs to be ensured in Fortran anyway). > But we can discuss next week. The problem here is that MPI_BARRIER is a process barrier, not a memory barrier. FWIW, with current compilers, which do not reorder instructions across functions, this should work correctly. But this is a correctness concern when compilers do more than that, e.g., IPC optimizations. An alternative would be to leave it as-is, and add a note that we assume the compiler will not reorder instructions here because it's a function call, though I don't prefer this alternative. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Jeff Hammond 02 Dec '12

02 Dec '12

>> I propose the following additions. I will be at the Forum this week >> and can discuss them in person with any interested parties. If you >> want to poop on my ideas, doing so in person will probably go over >> better. > I assure you that the outcome will not be different. Unless you include overall personal happiness in the outcome. >> (1) Define semantics of shared memory windows in separate model. >> >> In Section 11.2.3 it says "MPI does not define semantics for accessing >> shared memory windows in the separate memory model." >> >> I would like to try to define these semantics in MPI 3.1. Shouldn't >> it at least be possible to use MPI_WIN_(UN)LOCK with >> MPI_LOCK_EXCLUSIVE here? >> >> I believe that supporting RMA for weakly- or non- coherent memory >> architectures is vital for future systems. > I agree. We had this discussion and decided to postpone it because we > had other more pressing issues. Now may be the time. A discussion would > be good. Yes, there was no urgency. I'm glad the standard left this issue wide open instead of defining something restrictive. >> (2) Add info keys for restricted usage of RMA ops. >> >> Add info keys to windows to restrict usage to subsets of RMA, >> particular MPI_FETCH_AND_OP and/or MPI_COMPARE_AND_SWAP. >> >> As noted on a recent discussion involving Hubert, Jim and me on this >> list, this has potential performance benefits. > Yes, this should probably be an assertion though. Info keys are too slow > to interpret (unless you want to define it on a per-window and not > per-operation basis). We shall discuss. I would like it to be per-window. It would help if there was a way to collectively change info keys but perhaps someone will say that just creating N windows with different keys on to of the same memory is valid here if one barriers/syncs between epochs. >> (3) Unify and add info keys across window types. >> >> Unify info keys across different window creation mechanisms. As I >> noted in my previous email, it is not clear if "accumulate_ordering, >> accumulate_ops, same_size, alloc_shared_noncontig" are valid for >> MPI_WIN_ALLOCATE and MPI_WIN_CREATE_DYNAMIC. > That's what was intended. Maybe we should update the wording to reflect > this. > >> Furthermore, I would like the "same_size" info key to be defined for >> MPI_WIN_CREATE as well since it may eliminate an unnecessary >> collective (but not all of them, so I don't mind if people don't like >> this suggestion). >> >> Finally, I would like an info key for >> "only_one_rank_gives_nonzero_size" (probably needs a better name) >> since this means that one needs only to do a bcast and not an >> allreduce or allgather during window creation/allocation. This usage >> pattern is common for certain types of load-balancing counters (not >> necessarily globally shared, although GA's NXTVAL is this way) and >> perhaps also for some implementations of mutexes. > We should discuss. Yes, there are perhaps generalizations of this that I haven't thought of. Best, Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Torsten Hoefler 02 Dec '12

02 Dec '12

... I sent this a while ago but from my @ethz account: On 12/02/2012 04:01 AM, Jeff Hammond wrote: > I propose the following additions. I will be at the Forum this week > and can discuss them in person with any interested parties. If you > want to poop on my ideas, doing so in person will probably go over > better. I assure you that the outcome will not be different. > (1) Define semantics of shared memory windows in separate model. > > In Section 11.2.3 it says "MPI does not define semantics for accessing > shared memory windows in the separate memory model." > > I would like to try to define these semantics in MPI 3.1. Shouldn't > it at least be possible to use MPI_WIN_(UN)LOCK with > MPI_LOCK_EXCLUSIVE here? > > I believe that supporting RMA for weakly- or non- coherent memory > architectures is vital for future systems. I agree. We had this discussion and decided to postpone it because we had other more pressing issues. Now may be the time. A discussion would be good. > (2) Add info keys for restricted usage of RMA ops. > > Add info keys to windows to restrict usage to subsets of RMA, > particular MPI_FETCH_AND_OP and/or MPI_COMPARE_AND_SWAP. > > As noted on a recent discussion involving Hubert, Jim and me on this > list, this has potential performance benefits. Yes, this should probably be an assertion though. Info keys are too slow to interpret (unless you want to define it on a per-window and not per-operation basis). We shall discuss. > (3) Unify and add info keys across window types. > > Unify info keys across different window creation mechanisms. As I > noted in my previous email, it is not clear if "accumulate_ordering, > accumulate_ops, same_size, alloc_shared_noncontig" are valid for > MPI_WIN_ALLOCATE and MPI_WIN_CREATE_DYNAMIC. That's what was intended. Maybe we should update the wording to reflect this. > Furthermore, I would like the "same_size" info key to be defined for > MPI_WIN_CREATE as well since it may eliminate an unnecessary > collective (but not all of them, so I don't mind if people don't like > this suggestion). > > Finally, I would like an info key for > "only_one_rank_gives_nonzero_size" (probably needs a better name) > since this means that one needs only to do a bcast and not an > allreduce or allgather during window creation/allocation. This usage > pattern is common for certain types of load-balancing counters (not > necessarily globally shared, although GA's NXTVAL is this way) and > perhaps also for some implementations of mutexes. We should discuss. Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Torsten Hoefler 02 Dec '12

02 Dec '12

... I sent this a while ago but from my @ethz account: Pavan, Good points. I believe we wanted to include many of them in the errata (which may or may not happen ...). The only thing I disagree to is slide 6. Win_sync really does nothing in the separate case. Your example would work on shared memory if flush issues a memory barrier *and* MPI_Barrier() is an optimization barrier (which is is in C and needs to be ensured in Fortran anyway). But we can discuss next week. Best, Torsten On Sat, Dec 01, 2012 at 08:21:52PM -0600, Pavan Balaji wrote: > All, > > Please see the attached slides as consideration items for MPI-3.1. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

Re: [Mpi3-rma] MPI-3.1 consideration slides
by Jeff Hammond 01 Dec '12

01 Dec '12

I propose the following additions. I will be at the Forum this week and can discuss them in person with any interested parties. If you want to poop on my ideas, doing so in person will probably go over better. (1) Define semantics of shared memory windows in separate model. In Section 11.2.3 it says "MPI does not define semantics for accessing shared memory windows in the separate memory model." I would like to try to define these semantics in MPI 3.1. Shouldn't it at least be possible to use MPI_WIN_(UN)LOCK with MPI_LOCK_EXCLUSIVE here? I believe that supporting RMA for weakly- or non- coherent memory architectures is vital for future systems. (2) Add info keys for restricted usage of RMA ops. Add info keys to windows to restrict usage to subsets of RMA, particular MPI_FETCH_AND_OP and/or MPI_COMPARE_AND_SWAP. As noted on a recent discussion involving Hubert, Jim and me on this list, this has potential performance benefits. (3) Unify and add info keys across window types. Unify info keys across different window creation mechanisms. As I noted in my previous email, it is not clear if "accumulate_ordering, accumulate_ops, same_size, alloc_shared_noncontig" are valid for MPI_WIN_ALLOCATE and MPI_WIN_CREATE_DYNAMIC. Furthermore, I would like the "same_size" info key to be defined for MPI_WIN_CREATE as well since it may eliminate an unnecessary collective (but not all of them, so I don't mind if people don't like this suggestion). Finally, I would like an info key for "only_one_rank_gives_nonzero_size" (probably needs a better name) since this means that one needs only to do a bcast and not an allreduce or allgather during window creation/allocation. This usage pattern is common for certain types of load-balancing counters (not necessarily globally shared, although GA's NXTVAL is this way) and perhaps also for some implementations of mutexes. Thanks, Jeff On Sat, Dec 1, 2012 at 8:21 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > All, > > Please see the attached slides as consideration items for MPI-3.1. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

[Mpi3-rma] missing RMA info keys
by Jeff Hammond 01 Dec '12

01 Dec '12

It is not clear if these info keys are predefined for MPI_WIN_ALLOCATE or MPI_WIN_CREATE_DYNAMIC. Does the following text mean that the info keys for MPI_WIN_ALLOCATE are inclusive of those for MPI_WIN_CREATE? It really does not seem that way to me. ======================== The info argument can be used to specify hints similar to the info argument for MPI_WIN_CREATE and MPI_ALLOC_MEM. The following info key is predefined: same_size ï¿½ if set to true, then the implementation may assume that the argument size is identical on all processes. ======================== If the keys "accumulate_ordering, accumulate_ops, same_size, alloc_shared_noncontig" are not valid for MPI_WIN_ALLOCATE and/or MPI_WIN_CREATE_DYNAMIC already, they should be this task should be added to the TODO list for MPI 3.1. Jeff On Tue, Nov 6, 2012 at 4:41 AM, Hubert Ritzdorf <Hubert.Ritzdorf(a)emea.nec.com> wrote: > Hi, > > the new RMA info keys "accumulate_ordering, accumulate_ops, same_size, > alloc_shared_noncontig" > and the corresponding values are missing in Sections A.1.5 and A.1.6 (Pages > 683 and 684). > > Hubert > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0