I don't know about illegal as much as "doesn't make sense". Keith brough this up, but I think it got lost in other discussions... What is the semantic of a collective operation during a point-to-point epoch? We're constrained by the existing API (as we've decided not to write a new API) I can see a couple of scenarios for allflushall and passive target, none of which I like. The first is that allflushall only flushes with peers it currently holds a window open to. This means tracking such state, which I think is a bad idea due to state required. The second option is that it's erroneous to call allflushall unless there's a passive access epoch to every peer in the window. This seems to encourage a behavior I dont' like (namely, having to have such an epoch open at all times). I could possibly see the benefit of an allflushall in an active target, where group semantics are a bit more well-defined, but that's an entirely different discussion. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories ________________________________________ From: [email protected] [[email protected]] On Behalf Of Underwood, Keith D [[email protected]] Sent: Friday, May 21, 2010 1:44 PM To: MPI 3.0 Remote Memory Access working group Subject: Re: [Mpi3-rma] RMA proposal 1 update That's an interesting point. I would think that would be illegal? Keith

-----Original Message----- From: [email protected] [mailto:mpi3-rma- [email protected]] On Behalf Of Rajeev Thakur Sent: Friday, May 21, 2010 1:38 PM To: 'MPI 3.0 Remote Memory Access working group' Subject: Re: [Mpi3-rma] RMA proposal 1 update

Lock-unlock is not collective. If we add an allflushall, how would the processes that don't call lock-unlock call it? Just directly?

Rajeev

...

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Underwood, Keith D Sent: Friday, May 21, 2010 11:22 AM To: MPI 3.0 Remote Memory Access working group Subject: Re: [Mpi3-rma] RMA proposal 1 update

There is a proposal on the table to add flush(rank) and flushall() as local calls for passive target to allow incremental remote completion without calling unlock(). There is an additional proposal to also include allflushall(), which would be a collective remote completion semantic added to passive target (nominally, we might not want to call it passive target anymore if we did that, but that is a different discussion). The question I had posed was: can you really get a measurable performance advantage in realistic usage scenarios by having allflushall() instead of just doing flushall() + barrier()? I was asking if someone could provide an implementation sketch - preferably along with a real usage scenario where that implementation sketch would yield a performance advantage. A back-of-the-envelope assessment of that would be a really nice to have.

My contention was that the number of targets with outstanding requests from a given between one flush/flushall and the next one would frequently not be large enough to justify the cost of a collective. Alternatively, if messages in that window are "large" (for some definition of "large" that is likely less than 4KB and is certainly no larger than the rendezvous threshold), I would contend that generating a software ack for each one would be essentially zero overhead and would allow source side tracking of remote completion such that a flushall could be a local operation.

There is a completely separate discussion that needs to occur on whether it is better to add collective completion to passive target or one-sided completion to active target (both of which are likely to meet some resistance because of the impurity the introduce into the model/naming) or whether the two need to be mixed at all.

Keith

...

-----Original Message----- From: [email protected] [mailto:mpi3-rma- [email protected]] On Behalf Of Douglas Miller Sent: Friday, May 21, 2010 7:16 AM To: MPI 3.0 Remote Memory Access working group Subject: Re: [Mpi3-rma] RMA proposal 1 update

It is now obvious that I did not do a sufficient job of explaining things before. It's not even clear to me anymore exactly what we're talking about - is this ARMCI? MPI2? MPI3? Active-target one-sided? Passive target? My previous explanation was for BG/P MPI2 active-target one-sided using MPI_Win_fence synchronization. BG/P MPI2 implementations for MPI_Win_post/start/complete/wait and passive-target MPI_Win_lock/unlock use different methods for determining target completion, although internally they may use some/all of the same structures.

What was the original question?

_______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 [email protected] Douglas Miller/Rochester/IBM

"Underwood, Keith D" <keith.d.underwoo To [email protected]> "MPI 3.0 Remote Memory Access Sent by: working group" mpi3-rma-bounces@ <[email protected]> lists.mpi-forum.o cc rg

Subject Re: [Mpi3-rma] RMA proposal 1 05/20/2010 04:56 update PM

Please respond to "MPI 3.0 Remote Memory Access working group" <[email protected] pi-forum.org>

My point was, the way Jeff is doing synchronization in NWChem is via a fenceall(); barrier(); on the equivalent of MPI_COMM_WORLD. If I knew he was going to be primarily doing this (ie, that he wanted to know that all nodes were synched), I would do something like maintain counts of sent and received messages on each node. I could then do something like an allreduce of those 2 ints over the tree to determine if everyone is synched. There are probably some technical details that would have to be worked out to ensure this works but it seems good from 10000 feet.

Right now we do numprocs 0-byte get operations to make sure the torus is flushed on each node. A torus operation is ~3us on a 512-way. It grows slowly with number of midplanes. I'm sure a 72 rack longest Manhattan distance noncongested pingpong is <10us, but I don't have the data in front of me.

Based on Doug's email, I had assumed you would know who you have sent messages to. If you knew that in a given fence interval the node had only sent distinct messages to 1K other cores, you would only have 1K gets to issue. Suck? Yes. Worse than the tree messages? Maybe, maybe not. There is definitely a cross-over between 1 and np outstanding messages between fences where on the 1 side of things the tree messages are worse and on the np side of things the tree messages are better. There is another spectrum based on request size where getting a response for every request becomes an inconsequential overhead. I would have to know the cost of processing a message, the size of a response, and the cost of generating that response to create a proper graph of that.

A tree int/sum is roughly 5us on a 512-way and grows similarly. I would postulate that a 72 rack MPI allreduce int/sum is on the order of 10us.

So you generate np*np messages vs 1 tree message. Contention and all the overhead of that many messages will be significantly worse than even several tree messages. Oh, wait, so, you would sum all sent and sum all received and then check if they were equal? And then (presumably) iterate until the answer was yes? Hrm. That is more interesting. Can you easily separate one-sided and two sided messages in your counting while maintaining the performance of one-sided messages? Doug's earlier answer implied you were going to allreduce a vector of counts (one per rank) and that would have been ugly. I am assuming you would do at least 2 tree messages in what I believe you are describing, so there is still a crossover between n*np messages and m tree messages (where n is the number of outstanding requests between fencealls and 2 <= m <= 10), and the locality of communications impacts that crossover. BTW, can you actually generate messages fast enough to cause contention with tiny messages? Anytime I know that an operation is collective, I can almost guarantee I can do it better than even a good pt2pt algorithm if I am utilizing our collective network. I think on machines that have remote completion notification an allfenceall() is just a barrier(), and since fenceall(); barrier(); is going to be replaced by allfenceall(), it doesn't seem to me like it is any extra overhead if allfenceall() is just a barrier() for you.

My concerns are twofold: 1) we are talking about adding collective completion to passive target when active target was the one designed to have collective completion. That is semantically and API-wise a bit ugly. 2) I think the allfenceall() as a collective will optimize to the case where you have outstanding requests to everybody and I believe that will be slower than the typical case of having outstanding requests to some people. I think that users would typically call allfenceall() rather than fenceall() + barrier() and then they would see a performance paradox: the fenceall() + barrier() could be substantially faster when you have a "small" number of peers you are communicating with in this iteration. I am not at all worried about the overhead of allfenceall() for networks with remote completion. Keith

From: "Underwood, Keith D" <[email protected]>

To: "MPI 3.0 Remote Memory Access working group" <[email protected]>

Date: 05/20/2010 09:19 AM

Subject Re: [Mpi3-rma] RMA proposal 1 update :

Sent [email protected] by:

_______________________________________________ mpi3-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

_______________________________________________ mpi3-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

_______________________________________________ mpi3-rma mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma