Yes, I think what you are saying is what I was trying to say in option #2. I'd prefer an explicit function to change "sync mode" rather than try and overload fence. But I guess I can see the logic is using fence for this purpose as well. Part of the problem I had with MPI2 RMA was the ambiguity of fence, though. Perhaps I was reading more into the proposal, but I thought things like "lockall" and/or "alllockall" could conceptually replace fence. I would assume that if a platform had hardware-assist for MPI_Win_fence that it also could be used for lockall/alllockall, but maybe that is a stretch. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 [email protected] Douglas Miller/Rochester/IBM William Gropp <wgropp@illinois. edu> To Sent by: "MPI 3.0 Remote Memory Access mpi3-rma-bounces@ working group" lists.mpi-forum.o <[email protected]> rg cc Subject 06/21/2010 08:49 Re: [Mpi3-rma] RMA synchronization AM optimization [was: Updated MPI-3 RMA proposal 1] Please respond to "MPI 3.0 Remote Memory Access working group" <[email protected] pi-forum.org> I believe that the original motivation for permitting the mixed sync model was for applications that did something like this: # Initialize a global data area fence various put or accumulate updates fence # passive-target access of the area various lock/get/unlock accesses Another option would be to require an explicit and collective change to the sync mode - as there is only one passive target mode, and the "scalable sync" mode in practice involves all processes, this would be possible. An info (already (mis)used for the no_locks property could be used with win_create to specify that all changes in sync mode would be signaled with a routine (either win_fence with an assert about sync mode changing or a new win_sync_mode routine). Would something like that address the implementation issues that you see (remembering that some systems provided special hardware for a fast win_fence, and a single sync model probably isn't sufficient)? Bill On Jun 21, 2010, at 7:38 AM, Douglas Miller wrote:

At the risk of prolonging an already difficult-to-follow set of e-mail threads, I have to re-iterate my concerns for implementation efficiency.

The impediment I see to creating efficient implementations of MPI RMA is that the synchronization primitives provide too much freedom. By allowing one to switch back and forth between different synchronization methods on the same window, an implementation must keep track of more state information and handle complex corner-cases, all of which precludes an optimized implementation. It's been my (admittedly limited) experience that the synchronization adds a significant overhead, and that overhead is largely due to the handling of state and special cases involving mixed synchronization methods. I have two high-level suggestions, listed in the order I prefer them:

1. Leave MPI2 One-Sided as-is (and hope to deprecate it someday), create a new and separate RMA scheme which is intended to replace the old, which uses a single synchronization method (say, the *lock* methods being proposed). I prefer this path because it gives us the flexibility to design exactly what we want without being tied to the previous, possibly flawed, design. There is, admittedly, extra work involved with have two sets of APIs, but I think there is some room for re-use and common code, and I feel the extra work is worth the benefit.

2. Augment the MPI2 One-Sided specification with the ability for the user to specify a single synchronization method to be used exclusively on a given window. This could be by adding Win_create/allocate functions that take an "assert" which specifies the synchronization method to be used, and/or a way to specify "eras" of epochs that will use a single synchronization method - for example, a program can declare at some point that a given window will use only lock/unlock until the next declaration call (specifying another synchronization method, or "all"). At least with such capabilities, an implementation could allow programs to be more efficient if they choose to take the optimization of using a single synchronization method. I know that Win_create does not currently have asserts, but there is a way to add a new function for creating windows that does have asserts (and ensure Win_allocate also has asserts) and then define that the current Win_create is equivalent to Win_create_assert (e.g.) with "asserts" set to zero. Depending on the asserts defined, of course, that should allow the existing Win_create to maintain backward compatibility with MPI2.

thanks, _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 [email protected] Douglas Miller/Rochester/IBM

William Gropp <wgropp@illinois.

edu> To Sent by: "MPI 3.0 Remote Memory Access mpi3-rma-bounces@ working group" lists.mpi-forum.o <[email protected]>

rg cc

Subject 06/21/2010 12:05 Re: [Mpi3-rma] Updated MPI-3 RMA AM proposal 1

Please respond to "MPI 3.0 Remote Memory Access working group" <[email protected] pi-forum.org>

I agree with Rajeev. And I think we strayed somewhat from the original plan.

The goal for the MPI RMA was to make enlarge the set of applications that could be efficiently implemented with a one-sided model. The current model *is* a good one for *some* applications; the complaints about it are often because it doesn't fit some other application. The RMA extensions for MPI-3, in my mind, needed to address *some*, not all, of the important application areas that the current model does not handle. It is interesting to consider whether a reasonable functional implementation of, say, UPC, could be implemented with it, but I do not see the MPI RMA as supplying the universal implementation layer for other programming models.

Making MPI RMA suitable for implementing all other parallel programming models will require more than I think we want to do - look at the short word and aligned move routines in GASNET as an example. You don't need these functionally, but you may need them for performance.

That's why we asked for *application* use cases - those would drive the design. We have a few but we haven't focused on them as much as I think we should. What I wanted in proposal 1 was a set of operations, each of which (a) was consistent with the others and (b) was clearly driven by some *application* need (where an application in this case is *not* implementing another programming model). This very well may have required some options to deal with things like selecting different ordering and overlapping update semantics, though that could be very coarse grained.

Note that in this interpretation, proposal 1 is not a "bare minimum"; rather, it is a consensus collection of consistent extensions that enlarge the space of applications that can be efficiently coded using MPI-3 one-sided. It will leave some useful features out and some programming models should focus on interoperability with MPI rather than having MPI-3 RMA provide the specific features that they need. It is fine if there is something important that can't be done with MPI-3 RMA, as long as there are other important things that can be done with it.

Bill

On Jun 20, 2010, at 6:03 PM, Rajeev Thakur wrote:

...

Are you refering to Accumulate_get :-)? Maybe it should be in Proposal 2.

Maybe we also need a "journal of development" as in MPI-2 :-).

But, seriously, we need to present a united front at least in proposal 1. Otherwise the Forum will have no confidence in us.

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Pavan Balaji Sent: Sunday, June 20, 2010 5:57 PM To: MPI 3.0 Remote Memory Access working group Subject: Re: [Mpi3-rma] Updated MPI-3 RMA proposal 1

On 06/20/2010 05:48 PM, Rajeev Thakur wrote:

...

Proposal 1: This is what the RMA experts agree is the bare minimum needed to fix what is considered broken in MPI-2 RMA.

I don't agree that whatever is there in proposal 1 is the "bare minimum". Maybe this policy should be reworded as: *all* members of the working group should agree that this is needed.

This makes both proposal 1 and proposal 2 contain random pieces of unrelated features, though.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ mpi3-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

_______________________________________________ mpi3-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

_______________________________________________ mpi3-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

_______________________________________________ mpi3-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign _______________________________________________ mpi3-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma