- mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] RMA proposal 1 update
by Jeff Hammond 21 May '10

21 May '10

>> I've been talking about allflushall because ARMCI_Barrier is partially >> redundant with MPI_Barrier, which is clearly already in MPI-3. > > Intuitively named as ARMCI_Barrier - I imagine why I missed that ;-) ï¿½So, this offers some opportunity for experimentation... ï¿½ ARMCI_Barrier should be what GA_Sync calls, right? ï¿½How is the implementation of ARMCI_Barrier done on BG? ï¿½It would seem that we could try 3 implementations in some real world usage scenarios and get some data: Yes, and not to be confused with armci_msg_barrier (<http://www.emsl.pnl.gov/docs/parsoft/armci/documentation.htm#collect>) :( > 1) ARMCI_Barrier the "really bad" way: ï¿½i.e. np^2 messages get generated and then a barrier is done > 2) ARMCI_Barrier as a collective: ï¿½allreduce is used to agree about completion > 3) ARMCI_Barrier assuming a "small" number of targets between barriers/fences: ï¿½you track who has outstanding messages and only sync with them. Yeah, I'm going to work on this as soon as I have some free time. I have (1) and (3) already but I need to test them outside of NWChem to have any hope of making sense of the data. I think I can at least hack (2). I think I can implement a fourth option that beats (2) but it has the same complexity model (logarithmic). > Anyway, just to clarify, GA regularly uses both one-sided and collective completion calls? ï¿½Or, is it dominated by one or the other? ï¿½I look at GA_Fence() and see the equivalent of flushall() and GA_Sync = GA_Fence() + MPI_Barrier(). ï¿½If you call both, then you have this mixture of passive and active target, but... if GA_Sync is going to perform significantly better than GA_Fence(), couldn't you just switch to calling GA_Sync? ï¿½It would seem like users would rather have the barrier too (especially if it was cheaper than calling GA_Fence()). ï¿½Or, put another way, the online guide for GA essentially says "um, don't call GA_Sync() very often", so is allfenceall optimizing the infrequent case? The GA manual can say that calling GA_Sync will bring on the apocalypse, but that won't stop quantum chemistry software developers from calling it at the top and bottom of every subroutine. =O While there are too many GA_Sync calls in NWChem, removing the majority of them requires a complete rewrite of very complex algorithms and problem redesigning the entire code from top to bottom. I have never seen GA_Fence used. All ~remote~ completion in NWChem is collective. All three modes of local completion - trivial (blocking), individual (request-based) and bulk (fenced target) are all used explicitly via GA or implicitly within GA but invisible to the GA user. GA_Fence was probably added much later in response to NWChem users, so it isn't surprising it is never called in NWChem except in a "test all GA functionality" utility routine. > UPC has a similar scenario: ï¿½barrier implies a strict access that causes a flushall, but everybody knows that you avoid calling barrier to the extent you possibly can. ï¿½So, the model you use is actually focused on minimizing either strict accesses or barriers, but you would typically rather do a strict access than a barrier. ï¿½If GA code is optimized the same way, how often is GA_Sync() called? GA_Sync is called like crazy in NWChem. There is an effort to fix this pathological behavior in one particularly heinous abuser (TCE) but it will be a long time before NWChem learns to be kind to network hardware. Jeff -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Underwood, Keith D 21 May '10

21 May '10

> > My contention was that the number of targets with outstanding > requests from a given between one flush/flushall and the next one would > frequently not be large enough to justify the cost of a collective. > > (1) There exists a counterexample to your contention, and it is > NWChem. The number of messages NWChem dumps onto the network in some > simulations is ridiculously. I stuck a profiler on DCMF to measure > the number of events that a call to advance() processes during a > single call (it is called continuously with lock-cycling by the > communication helper thread). The biggest number I've seen so far is > more than 60,000. That number might not be equivalent to the number > of ARMCI messages due to the way non-contiguous is handled (or not > handled), but I think it indicates the order of magnitude that one has > to deal with. The number of messages in flight is not the same as the number of targets with a message in flight. Is there any chance of collecting that piece of data? Two ratios are relevant: 1) the number of targets to the number of messages (1000 messages each to 1000000 targets between flushalls would make the 1 message to each target for a non-collective flushall irrelevant), and 2) the number of targets to the time to do the allreduce (e.g. 600 messages to each of 100 targets would probably mean that you could do those 100 messages faster than you could do 2+ allreduces in the collective completion). Since I don't have enough low level detail on BGP (or, more importantly, BGQ, since BGP will be out of service by the time we have an MPI3 RMA library that is optimized), I don't know where the crossover is. > (2) In GA allflushall is going to be called along with a barrier, so > the extra cost is negligible. > > > Alternatively, if messages in that window are "large" (for some > definition of "large" that is likely less than 4KB and is certainly no > larger than the rendezvous threshold), I would contend that generating > a software ack for each one would be essentially zero overhead and > would allow source side tracking of remote completion such that a > flushall could be a local operation. > > There are going to be two contexts worth address: > (1) bigger messages, where flushall is local and allflushall is just a > barrier. > (2) smaller messages, where allflushall matters. > > Smaller messages matters because on networks with high-injection rate, > no hardware support for strided messages and/or relatively good DMA > performance relative to memcopy (low-frequency CPU ala BGP), it makes > sense to do ARMCI non-contiguous operations by just sending one > message per contiguous section. In this scenario, software acks are > not free and one would rather use a nonlocal flush, hence allflushall It would take some BGQ data to figure out where the crossover is between "small" and "big". E.g. say that a link is 1 GB/s per direction (total speculation on what those number might be - 1's are convenient for some of this math). So, let's say an ack needs 16 bytes of network traffic. An ack on a 1KB message would be 1.5% network traffic overhead - pretty tolerable. Now, let's say you can process a header in 1 microsecond and that it adds 50 ns to generate an ack. 5% processing overhead... Not too bad. Oh, and that 1 microsecond means that you want to send 1KB messages on a 1 GB/s network, so, voila, you're good. Now, the problem is, I don't know what ANY of those numbers actually are for BGQ ;-) And, BG is the only platform that has been raised as a big issue for this. > is useful. ARMCI used to do allflushall in GA_Sync by flushing all > links from every node, which deadlocked the network above 1K due to > the nproc^2 total injection. I rewrote it so it flushes active > connections one at a time (added granularity helps prevent deadlock > but is not good for performance), but this is not going to come close > to the performance I could get with a collective on BGP. Deadlock? Eww. How did you manage that? An application (or library) should never be able to deadlock the network while using the lowest level network API... eww... Keith

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Underwood, Keith D 21 May '10

21 May '10

> > The struggle here is that GA has both one-sided and collective > > completion. ï¿½Collective completion can obviously be emulated by one- > > sided completion + a barrier, but you have indicated that that is a > > performance issue. ï¿½Unfortunately, there is not an obvious place inside > > of the existing MPI RMA interface where the mixture of collective and > > one-sided completion fit. ï¿½The need for both one-sided and collective > > completion was clearly not architected into MPI one-sided and may very > > well break the architecture. ï¿½It certainly breaks the naming (there is > > nothing "passive" about a target that calls allfenceall and there is > > nothing "active" about a target when an initiator does one-side > > completion). > > Clearly, allflushall is not passive, but it is a convenience operation > to complete what would otherwise be a passive-target epoch. That it > would be inter-operable with all passive-target functionality, and > perhaps not start/post/complete (which still makes no sense to me), > makes it clearly part of the passive-target set. Well, that is part of the issue... we are going to have to figure out the language, but I don't think it "completes the epoch" even for the flushall variant (that would be the unlock). > > Does anybody know of another model (other than GA) that calls for a > > mixture of collective and one-sided completion? ï¿½CoArray Fortran uses > > collective completion, UPC expects one-sided completion, SHMEM only > > exposes one-sided completion, ARMCI only exposes one-sided > > completion... ï¿½If we could look at a second model that needed a > > mixture, it might help us formulate a better solution. > > ARMCI has collective completion. From > <http://www.emsl.pnl.gov/docs/parsoft/armci/documentation.htm#compl>: > > int ARMCI_Barrier() > PURPOSE: Synchronize processors and memory. This operation combines > functionality of > MPI_Barrier and ARMCI_AllFence. > > I've been talking about allflushall because ARMCI_Barrier is partially > redundant with MPI_Barrier, which is clearly already in MPI-3. Intuitively named as ARMCI_Barrier - I imagine why I missed that ;-) So, this offers some opportunity for experimentation... ARMCI_Barrier should be what GA_Sync calls, right? How is the implementation of ARMCI_Barrier done on BG? It would seem that we could try 3 implementations in some real world usage scenarios and get some data: 1) ARMCI_Barrier the "really bad" way: i.e. np^2 messages get generated and then a barrier is done 2) ARMCI_Barrier as a collective: allreduce is used to agree about completion 3) ARMCI_Barrier assuming a "small" number of targets between barriers/fences: you track who has outstanding messages and only sync with them. Anyway, just to clarify, GA regularly uses both one-sided and collective completion calls? Or, is it dominated by one or the other? I look at GA_Fence() and see the equivalent of flushall() and GA_Sync = GA_Fence() + MPI_Barrier(). If you call both, then you have this mixture of passive and active target, but... if GA_Sync is going to perform significantly better than GA_Fence(), couldn't you just switch to calling GA_Sync? It would seem like users would rather have the barrier too (especially if it was cheaper than calling GA_Fence()). Or, put another way, the online guide for GA essentially says "um, don't call GA_Sync() very often", so is allfenceall optimizing the infrequent case? UPC has a similar scenario: barrier implies a strict access that causes a flushall, but everybody knows that you avoid calling barrier to the extent you possibly can. So, the model you use is actually focused on minimizing either strict accesses or barriers, but you would typically rather do a strict access than a barrier. If GA code is optimized the same way, how often is GA_Sync() called? Keith

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Jeff Hammond 21 May '10

21 May '10

On Fri, May 21, 2010 at 3:27 PM, Underwood, Keith D <keith.d.underwood(a)intel.com> wrote: > I think (although I am not an ARMCI expert) that passive target with the lockall/unlockall and the flush/flushall extensions we have proposed gives you (approximately) ARMCI. ï¿½The question is then: ï¿½where does the allflushall to support GA belong? ï¿½ARMCI is actually an easier target than GA, because (like many models) ARMCI only has one-sided completion. ï¿½But, I thought you wanted better than ARMCI? I completely agree with the first part. All I'm asking is that, in addition to flush and flushall, we add allflushall so that some of us can defend against pathological behavior when GA_Sync calls flushall from every node. It is in this respect that I want better than ARMCI. ARMCI is effectively a p2p transport layer, where as GA's programming model leads to much collective-like behavior. It seems natural to want MPI-3 to do a better job supporting GA than ARMCI. Adding allflushall is one step in this direction. > The struggle here is that GA has both one-sided and collective completion. ï¿½Collective completion can obviously be emulated by one-sided completion + a barrier, but you have indicated that that is a performance issue. ï¿½Unfortunately, there is not an obvious place inside of the existing MPI RMA interface where the mixture of collective and one-sided completion fit. ï¿½The need for both one-sided and collective completion was clearly not architected into MPI one-sided and may very well break the architecture. ï¿½It certainly breaks the naming (there is nothing "passive" about a target that calls allfenceall and there is nothing "active" about a target when an initiator does one-side completion). Clearly, allflushall is not passive, but it is a convenience operation to complete what would otherwise be a passive-target epoch. That it would be inter-operable with all passive-target functionality, and perhaps not start/post/complete (which still makes no sense to me), makes it clearly part of the passive-target set. > Does anybody know of another model (other than GA) that calls for a mixture of collective and one-sided completion? ï¿½CoArray Fortran uses collective completion, UPC expects one-sided completion, SHMEM only exposes one-sided completion, ARMCI only exposes one-sided completion... ï¿½If we could look at a second model that needed a mixture, it might help us formulate a better solution. ARMCI has collective completion. From <http://www.emsl.pnl.gov/docs/parsoft/armci/documentation.htm#compl>: int ARMCI_Barrier() PURPOSE: Synchronize processors and memory. This operation combines functionality of MPI_Barrier and ARMCI_AllFence. I've been talking about allflushall because ARMCI_Barrier is partially redundant with MPI_Barrier, which is clearly already in MPI-3. Jeff -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Jeff Hammond 21 May '10

21 May '10

> My contention was that the number of targets with outstanding requests from a given between one flush/flushall and the next one would frequently not be large enough to justify the cost of a collective. (1) There exists a counterexample to your contention, and it is NWChem. The number of messages NWChem dumps onto the network in some simulations is ridiculously. I stuck a profiler on DCMF to measure the number of events that a call to advance() processes during a single call (it is called continuously with lock-cycling by the communication helper thread). The biggest number I've seen so far is more than 60,000. That number might not be equivalent to the number of ARMCI messages due to the way non-contiguous is handled (or not handled), but I think it indicates the order of magnitude that one has to deal with. (2) In GA allflushall is going to be called along with a barrier, so the extra cost is negligible. > Alternatively, if messages in that window are "large" (for some definition of "large" that is likely less than 4KB and is certainly no larger than the rendezvous threshold), I would contend that generating a software ack for each one would be essentially zero overhead and would allow source side tracking of remote completion such that a flushall could be a local operation. There are going to be two contexts worth address: (1) bigger messages, where flushall is local and allflushall is just a barrier. (2) smaller messages, where allflushall matters. Smaller messages matters because on networks with high-injection rate, no hardware support for strided messages and/or relatively good DMA performance relative to memcopy (low-frequency CPU ala BGP), it makes sense to do ARMCI non-contiguous operations by just sending one message per contiguous section. In this scenario, software acks are not free and one would rather use a nonlocal flush, hence allflushall is useful. ARMCI used to do allflushall in GA_Sync by flushing all links from every node, which deadlocked the network above 1K due to the nproc^2 total injection. I rewrote it so it flushes active connections one at a time (added granularity helps prevent deadlock but is not good for performance), but this is not going to come close to the performance I could get with a collective on BGP. Jeff -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Underwood, Keith D 21 May '10

21 May '10

I think (although I am not an ARMCI expert) that passive target with the lockall/unlockall and the flush/flushall extensions we have proposed gives you (approximately) ARMCI. The question is then: where does the allflushall to support GA belong? ARMCI is actually an easier target than GA, because (like many models) ARMCI only has one-sided completion. But, I thought you wanted better than ARMCI? The struggle here is that GA has both one-sided and collective completion. Collective completion can obviously be emulated by one-sided completion + a barrier, but you have indicated that that is a performance issue. Unfortunately, there is not an obvious place inside of the existing MPI RMA interface where the mixture of collective and one-sided completion fit. The need for both one-sided and collective completion was clearly not architected into MPI one-sided and may very well break the architecture. It certainly breaks the naming (there is nothing "passive" about a target that calls allfenceall and there is nothing "active" about a target when an initiator does one-side completion). Does anybody know of another model (other than GA) that calls for a mixture of collective and one-sided completion? CoArray Fortran uses collective completion, UPC expects one-sided completion, SHMEM only exposes one-sided completiong, ARMCI only exposes one-sided completion... If we could look at a second model that needed a mixture, it might help us formulate a better solution. Keith > -----Original Message----- > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond > Sent: Friday, May 21, 2010 2:11 PM > To: MPI 3.0 Remote Memory Access working group > Cc: MPI 3.0 Remote Memory Access working group > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > Please explain how active target gives me ARMCI. > > Jeff > > Sent from my iPhone > > On May 21, 2010, at 3:05 PM, "Barrett, Brian W" <bwbarre(a)sandia.gov> > wrote: > > > I don't know about illegal as much as "doesn't make sense". Keith > > brough this up, but I think it got lost in other discussions... > > What is the semantic of a collective operation during a point-to- > > point epoch? We're constrained by the existing API (as we've > > decided not to write a new API) > > > > I can see a couple of scenarios for allflushall and passive target, > > none of which I like. The first is that allflushall only flushes > > with peers it currently holds a window open to. This means tracking > > such state, which I think is a bad idea due to state required. The > > second option is that it's erroneous to call allflushall unless > > there's a passive access epoch to every peer in the window. This > > seems to encourage a behavior I dont' like (namely, having to have > > such an epoch open at all times). > > > > I could possibly see the benefit of an allflushall in an active > > target, where group semantics are a bit more well-defined, but > > that's an entirely different discussion. > > > > Brian > > > > -- > > Brian W. Barrett > > Scalable System Software Group > > Sandia National Laboratories > > ________________________________________ > > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mpi3-rma- > > bounces(a)lists.mpi-forum.org] On Behalf Of Underwood, Keith D > > [keith.d.underwood(a)intel.com] > > Sent: Friday, May 21, 2010 1:44 PM > > To: MPI 3.0 Remote Memory Access working group > > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > > > That's an interesting point. I would think that would be illegal? > > > > Keith > > > >> -----Original Message----- > >> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > >> bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur > >> Sent: Friday, May 21, 2010 1:38 PM > >> To: 'MPI 3.0 Remote Memory Access working group' > >> Subject: Re: [Mpi3-rma] RMA proposal 1 update > >> > >> Lock-unlock is not collective. If we add an allflushall, how would > >> the > >> processes that don't call lock-unlock call it? Just directly? > >> > >> Rajeev > >> > >> > >>> -----Original Message----- > >>> From: mpi3-rma-bounces(a)lists.mpi-forum.org > >>> [mailto:[email protected]] On Behalf Of > >>> Underwood, Keith D > >>> Sent: Friday, May 21, 2010 11:22 AM > >>> To: MPI 3.0 Remote Memory Access working group > >>> Subject: Re: [Mpi3-rma] RMA proposal 1 update > >>> > >>> There is a proposal on the table to add flush(rank) and > >>> flushall() as local calls for passive target to allow > >>> incremental remote completion without calling unlock(). > >>> There is an additional proposal to also include > >>> allflushall(), which would be a collective remote completion > >>> semantic added to passive target (nominally, we might not > >>> want to call it passive target anymore if we did that, but > >>> that is a different discussion). The question I had posed > >>> was: can you really get a measurable performance advantage > >>> in realistic usage scenarios by having allflushall() instead > >>> of just doing flushall() + barrier()? I was asking if > >>> someone could provide an implementation sketch - preferably > >>> along with a real usage scenario where that implementation > >>> sketch would yield a performance advantage. A > >>> back-of-the-envelope assessment of that would be a really > >>> nice to have. > >>> > >>> My contention was that the number of targets with outstanding > >>> requests from a given between one flush/flushall and the next > >>> one would frequently not be large enough to justify the cost > >>> of a collective. Alternatively, if messages in that window > >>> are "large" (for some definition of "large" that is likely > >>> less than 4KB and is certainly no larger than the rendezvous > >>> threshold), I would contend that generating a software ack > >>> for each one would be essentially zero overhead and would > >>> allow source side tracking of remote completion such that a > >>> flushall could be a local operation. > >>> > >>> There is a completely separate discussion that needs to occur > >>> on whether it is better to add collective completion to > >>> passive target or one-sided completion to active target (both > >>> of which are likely to meet some resistance because of the > >>> impurity the introduce into the model/naming) or whether the > >>> two need to be mixed at all. > >>> > >>> Keith > >>> > >>>> -----Original Message----- > >>>> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > >>>> bounces(a)lists.mpi-forum.org] On Behalf Of Douglas Miller > >>>> Sent: Friday, May 21, 2010 7:16 AM > >>>> To: MPI 3.0 Remote Memory Access working group > >>>> Subject: Re: [Mpi3-rma] RMA proposal 1 update > >>>> > >>>> It is now obvious that I did not do a sufficient job of explaining > >>>> things > >>>> before. It's not even clear to me anymore exactly what we're > >> talking > >>>> about > >>>> - is this ARMCI? MPI2? MPI3? Active-target one-sided? > >>> Passive target? > >>>> My > >>>> previous explanation was for BG/P MPI2 active-target one-sided > >> using > >>>> MPI_Win_fence synchronization. BG/P MPI2 implementations for > >>>> MPI_Win_post/start/complete/wait and passive-target > >>> MPI_Win_lock/unlock > >>>> use > >>>> different methods for determining target completion, although > >>>> internally > >>>> they may use some/all of the same structures. > >>>> > >>>> What was the original question? > >>>> > >>>> _______________________________________________ > >>>> Douglas Miller BlueGene Messaging Development > >>>> IBM Corp., Rochester, MN USA Bldg 030-2 A410 > >>>> dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > >>>> > >>>> > >>>> > >>>> "Underwood, Keith > >>>> D" > >>>> <keith.d.underwoo > >>>> To > >>>> d(a)intel.com> "MPI 3.0 Remote Memory > >> Access > >>>> Sent by: working group" > >>>> mpi3-rma-bounces@ > >>> <mpi3-rma(a)lists.mpi-forum.org> > >>>> lists.mpi-forum.o > >>>> cc > >>>> rg > >>>> > >>>> Subject > >>>> Re: [Mpi3-rma] RMA proposal > >> 1 > >>>> 05/20/2010 04:56 update > >>>> PM > >>>> > >>>> > >>>> Please respond to > >>>> "MPI 3.0 Remote > >>>> Memory Access > >>>> working group" > >>>> <mpi3-rma(a)lists.m > >>>> pi-forum.org> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> My point was, the way Jeff is doing synchronization in > >>> NWChem is via a > >>>> fenceall(); barrier(); on the equivalent of MPI_COMM_WORLD. > >>> If I knew > >>>> he > >>>> was going to be primarily doing this (ie, that he wanted to > >>> know that > >>>> all > >>>> nodes were synched), I would do something like maintain > >>> counts of sent > >>>> and > >>>> received messages on each node. I could then do something like an > >>>> allreduce > >>>> of those 2 ints over the tree to determine if everyone is synched. > >>>> There > >>>> are probably some technical details that would have to be > >>> worked out to > >>>> ensure this works but it seems good from 10000 feet. > >>>> > >>>> Right now we do numprocs 0-byte get operations to make sure > >>> the torus > >>>> is > >>>> flushed on each node. A torus operation is ~3us on a > >>> 512-way. It grows > >>>> slowly with number of midplanes. I'm sure a 72 rack longest > >>> Manhattan > >>>> distance noncongested pingpong is <10us, but I don't have > >>> the data in > >>>> front > >>>> of me. > >>>> > >>>> Based on Doug's email, I had assumed you would know who you > >>> have sent > >>>> messages to. If you knew that in a given fence interval > >>> the node had > >>>> only > >>>> sent distinct messages to 1K other cores, you would only > >>> have 1K gets > >>>> to > >>>> issue. Suck? Yes. Worse than the tree messages? Maybe, > >>> maybe not. > >>>> There is definitely a cross-over between 1 and np > >>> outstanding messages > >>>> between fences where on the 1 side of things the tree messages are > >>>> worse > >>>> and on the np side of things the tree messages are better. There > >> is > >>>> another spectrum based on request size where getting a response > for > >>>> every > >>>> request becomes an inconsequential overhead. I would have > >>> to know the > >>>> cost > >>>> of processing a message, the size of a response, and the cost of > >>>> generating > >>>> that response to create a proper graph of that. > >>>> > >>>> A tree int/sum is roughly 5us on a 512-way and grows > >>> similarly. I would > >>>> postulate that a 72 rack MPI allreduce int/sum is on the > >>> order of 10us. > >>>> > >>>> So you generate np*np messages vs 1 tree message. Contention and > >> all > >>>> the > >>>> overhead of that many messages will be significantly worse than > >> even > >>>> several tree messages. > >>>> Oh, wait, so, you would sum all sent and sum all received and then > >>>> check if > >>>> they were equal? And then (presumably) iterate until the answer > >> was > >>>> yes? > >>>> Hrm. That is more interesting. Can you easily separate > >>> one-sided and > >>>> two > >>>> sided messages in your counting while maintaining the performance > >> of > >>>> one-sided messages? > >>>> Doug's earlier answer implied you were going to allreduce a > >>> vector of > >>>> counts (one per rank) and that would have been ugly. I am > >> assuming > >>>> you > >>>> would do at least 2 tree messages in what I believe you are > >>> describing, > >>>> so > >>>> there is still a crossover between n*np messages and m tree > >> messages > >>>> (where > >>>> n is the number of outstanding requests between fencealls > >>> and 2 <= m <= > >>>> 10), and the locality of communications impacts that crossover. > >>>> BTW, can you actually generate messages fast enough to > >>> cause contention > >>>> with tiny messages? > >>>> Anytime I know that an operation is collective, I can > >>> almost guarantee > >>>> I > >>>> can do it better than even a good pt2pt algorithm if I am > >>> utilizing our > >>>> collective network. I think on machines that have remote > completion > >>>> notification an allfenceall() is just a barrier(), and since > >>>> fenceall(); > >>>> barrier(); is going to be replaced by allfenceall(), it > >>> doesn't seem to > >>>> me > >>>> like it is any extra overhead if allfenceall() is just a > >>> barrier() for > >>>> you. > >>>> > >>>> > >>>> My concerns are twofold: 1) we are talking about adding > collective > >>>> completion to passive target when active target was the one > >>> designed to > >>>> have collective completion. That is semantically and API-wise a > >> bit > >>>> ugly. > >>>> 2) I think the allfenceall() as a collective will optimize > >>> to the case > >>>> where you have outstanding requests to everybody and I believe > that > >>>> will be > >>>> slower than the typical case of having outstanding requests to > >> some > >>>> people. I think that users would typically call > >>> allfenceall() rather > >>>> than > >>>> fenceall() + barrier() and then they would see a > >>> performance paradox: > >>>> the > >>>> fenceall() + barrier() could be substantially faster when you have > >> a > >>>> "small" number of peers you are communicating with in this > >>> iteration. > >>>> I am > >>>> not at all worried about the overhead of allfenceall() for > networks > >>>> with > >>>> remote completion. > >>>> Keith > >>>> > >>>> > >>>> > >>>> From: "Underwood, Keith D" <keith.d.underwood(a)intel.com> > >>>> > >>>> To: "MPI 3.0 Remote Memory Access working group" > >>>> <mpi3-rma(a)lists.mpi-forum.org> > >>>> > >>>> Date: 05/20/2010 09:19 AM > >>>> > >>>> Subject Re: [Mpi3-rma] RMA proposal 1 update > >>>> : > >>>> > >>>> Sent mpi3-rma-bounces(a)lists.mpi-forum.org > >>>> by: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>> > >>> > >>> _______________________________________________ > >>> mpi3-rma mailing list > >>> mpi3-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >>> > >> > >> > >> _______________________________________________ > >> mpi3-rma mailing list > >> mpi3-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Jeff Hammond 21 May '10

21 May '10

Please explain how active target gives me ARMCI. Jeff Sent from my iPhone On May 21, 2010, at 3:05 PM, "Barrett, Brian W" <bwbarre(a)sandia.gov> wrote: > I don't know about illegal as much as "doesn't make sense". Keith > brough this up, but I think it got lost in other discussions... > What is the semantic of a collective operation during a point-to- > point epoch? We're constrained by the existing API (as we've > decided not to write a new API) > > I can see a couple of scenarios for allflushall and passive target, > none of which I like. The first is that allflushall only flushes > with peers it currently holds a window open to. This means tracking > such state, which I think is a bad idea due to state required. The > second option is that it's erroneous to call allflushall unless > there's a passive access epoch to every peer in the window. This > seems to encourage a behavior I dont' like (namely, having to have > such an epoch open at all times). > > I could possibly see the benefit of an allflushall in an active > target, where group semantics are a bit more well-defined, but > that's an entirely different discussion. > > Brian > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > ________________________________________ > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mpi3-rma- > bounces(a)lists.mpi-forum.org] On Behalf Of Underwood, Keith D > [keith.d.underwood(a)intel.com] > Sent: Friday, May 21, 2010 1:44 PM > To: MPI 3.0 Remote Memory Access working group > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > That's an interesting point. I would think that would be illegal? > > Keith > >> -----Original Message----- >> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- >> bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur >> Sent: Friday, May 21, 2010 1:38 PM >> To: 'MPI 3.0 Remote Memory Access working group' >> Subject: Re: [Mpi3-rma] RMA proposal 1 update >> >> Lock-unlock is not collective. If we add an allflushall, how would >> the >> processes that don't call lock-unlock call it? Just directly? >> >> Rajeev >> >> >>> -----Original Message----- >>> From: mpi3-rma-bounces(a)lists.mpi-forum.org >>> [mailto:[email protected]] On Behalf Of >>> Underwood, Keith D >>> Sent: Friday, May 21, 2010 11:22 AM >>> To: MPI 3.0 Remote Memory Access working group >>> Subject: Re: [Mpi3-rma] RMA proposal 1 update >>> >>> There is a proposal on the table to add flush(rank) and >>> flushall() as local calls for passive target to allow >>> incremental remote completion without calling unlock(). >>> There is an additional proposal to also include >>> allflushall(), which would be a collective remote completion >>> semantic added to passive target (nominally, we might not >>> want to call it passive target anymore if we did that, but >>> that is a different discussion). The question I had posed >>> was: can you really get a measurable performance advantage >>> in realistic usage scenarios by having allflushall() instead >>> of just doing flushall() + barrier()? I was asking if >>> someone could provide an implementation sketch - preferably >>> along with a real usage scenario where that implementation >>> sketch would yield a performance advantage. A >>> back-of-the-envelope assessment of that would be a really >>> nice to have. >>> >>> My contention was that the number of targets with outstanding >>> requests from a given between one flush/flushall and the next >>> one would frequently not be large enough to justify the cost >>> of a collective. Alternatively, if messages in that window >>> are "large" (for some definition of "large" that is likely >>> less than 4KB and is certainly no larger than the rendezvous >>> threshold), I would contend that generating a software ack >>> for each one would be essentially zero overhead and would >>> allow source side tracking of remote completion such that a >>> flushall could be a local operation. >>> >>> There is a completely separate discussion that needs to occur >>> on whether it is better to add collective completion to >>> passive target or one-sided completion to active target (both >>> of which are likely to meet some resistance because of the >>> impurity the introduce into the model/naming) or whether the >>> two need to be mixed at all. >>> >>> Keith >>> >>>> -----Original Message----- >>>> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- >>>> bounces(a)lists.mpi-forum.org] On Behalf Of Douglas Miller >>>> Sent: Friday, May 21, 2010 7:16 AM >>>> To: MPI 3.0 Remote Memory Access working group >>>> Subject: Re: [Mpi3-rma] RMA proposal 1 update >>>> >>>> It is now obvious that I did not do a sufficient job of explaining >>>> things >>>> before. It's not even clear to me anymore exactly what we're >> talking >>>> about >>>> - is this ARMCI? MPI2? MPI3? Active-target one-sided? >>> Passive target? >>>> My >>>> previous explanation was for BG/P MPI2 active-target one-sided >> using >>>> MPI_Win_fence synchronization. BG/P MPI2 implementations for >>>> MPI_Win_post/start/complete/wait and passive-target >>> MPI_Win_lock/unlock >>>> use >>>> different methods for determining target completion, although >>>> internally >>>> they may use some/all of the same structures. >>>> >>>> What was the original question? >>>> >>>> _______________________________________________ >>>> Douglas Miller BlueGene Messaging Development >>>> IBM Corp., Rochester, MN USA Bldg 030-2 A410 >>>> dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM >>>> >>>> >>>> >>>> "Underwood, Keith >>>> D" >>>> <keith.d.underwoo >>>> To >>>> d(a)intel.com> "MPI 3.0 Remote Memory >> Access >>>> Sent by: working group" >>>> mpi3-rma-bounces@ >>> <mpi3-rma(a)lists.mpi-forum.org> >>>> lists.mpi-forum.o >>>> cc >>>> rg >>>> >>>> Subject >>>> Re: [Mpi3-rma] RMA proposal >> 1 >>>> 05/20/2010 04:56 update >>>> PM >>>> >>>> >>>> Please respond to >>>> "MPI 3.0 Remote >>>> Memory Access >>>> working group" >>>> <mpi3-rma(a)lists.m >>>> pi-forum.org> >>>> >>>> >>>> >>>> >>>> >>>> >>>> My point was, the way Jeff is doing synchronization in >>> NWChem is via a >>>> fenceall(); barrier(); on the equivalent of MPI_COMM_WORLD. >>> If I knew >>>> he >>>> was going to be primarily doing this (ie, that he wanted to >>> know that >>>> all >>>> nodes were synched), I would do something like maintain >>> counts of sent >>>> and >>>> received messages on each node. I could then do something like an >>>> allreduce >>>> of those 2 ints over the tree to determine if everyone is synched. >>>> There >>>> are probably some technical details that would have to be >>> worked out to >>>> ensure this works but it seems good from 10000 feet. >>>> >>>> Right now we do numprocs 0-byte get operations to make sure >>> the torus >>>> is >>>> flushed on each node. A torus operation is ~3us on a >>> 512-way. It grows >>>> slowly with number of midplanes. I'm sure a 72 rack longest >>> Manhattan >>>> distance noncongested pingpong is <10us, but I don't have >>> the data in >>>> front >>>> of me. >>>> >>>> Based on Doug's email, I had assumed you would know who you >>> have sent >>>> messages to. If you knew that in a given fence interval >>> the node had >>>> only >>>> sent distinct messages to 1K other cores, you would only >>> have 1K gets >>>> to >>>> issue. Suck? Yes. Worse than the tree messages? Maybe, >>> maybe not. >>>> There is definitely a cross-over between 1 and np >>> outstanding messages >>>> between fences where on the 1 side of things the tree messages are >>>> worse >>>> and on the np side of things the tree messages are better. There >> is >>>> another spectrum based on request size where getting a response for >>>> every >>>> request becomes an inconsequential overhead. I would have >>> to know the >>>> cost >>>> of processing a message, the size of a response, and the cost of >>>> generating >>>> that response to create a proper graph of that. >>>> >>>> A tree int/sum is roughly 5us on a 512-way and grows >>> similarly. I would >>>> postulate that a 72 rack MPI allreduce int/sum is on the >>> order of 10us. >>>> >>>> So you generate np*np messages vs 1 tree message. Contention and >> all >>>> the >>>> overhead of that many messages will be significantly worse than >> even >>>> several tree messages. >>>> Oh, wait, so, you would sum all sent and sum all received and then >>>> check if >>>> they were equal? And then (presumably) iterate until the answer >> was >>>> yes? >>>> Hrm. That is more interesting. Can you easily separate >>> one-sided and >>>> two >>>> sided messages in your counting while maintaining the performance >> of >>>> one-sided messages? >>>> Doug's earlier answer implied you were going to allreduce a >>> vector of >>>> counts (one per rank) and that would have been ugly. I am >> assuming >>>> you >>>> would do at least 2 tree messages in what I believe you are >>> describing, >>>> so >>>> there is still a crossover between n*np messages and m tree >> messages >>>> (where >>>> n is the number of outstanding requests between fencealls >>> and 2 <= m <= >>>> 10), and the locality of communications impacts that crossover. >>>> BTW, can you actually generate messages fast enough to >>> cause contention >>>> with tiny messages? >>>> Anytime I know that an operation is collective, I can >>> almost guarantee >>>> I >>>> can do it better than even a good pt2pt algorithm if I am >>> utilizing our >>>> collective network. I think on machines that have remote completion >>>> notification an allfenceall() is just a barrier(), and since >>>> fenceall(); >>>> barrier(); is going to be replaced by allfenceall(), it >>> doesn't seem to >>>> me >>>> like it is any extra overhead if allfenceall() is just a >>> barrier() for >>>> you. >>>> >>>> >>>> My concerns are twofold: 1) we are talking about adding collective >>>> completion to passive target when active target was the one >>> designed to >>>> have collective completion. That is semantically and API-wise a >> bit >>>> ugly. >>>> 2) I think the allfenceall() as a collective will optimize >>> to the case >>>> where you have outstanding requests to everybody and I believe that >>>> will be >>>> slower than the typical case of having outstanding requests to >> some >>>> people. I think that users would typically call >>> allfenceall() rather >>>> than >>>> fenceall() + barrier() and then they would see a >>> performance paradox: >>>> the >>>> fenceall() + barrier() could be substantially faster when you have >> a >>>> "small" number of peers you are communicating with in this >>> iteration. >>>> I am >>>> not at all worried about the overhead of allfenceall() for networks >>>> with >>>> remote completion. >>>> Keith >>>> >>>> >>>> >>>> From: "Underwood, Keith D" <keith.d.underwood(a)intel.com> >>>> >>>> To: "MPI 3.0 Remote Memory Access working group" >>>> <mpi3-rma(a)lists.mpi-forum.org> >>>> >>>> Date: 05/20/2010 09:19 AM >>>> >>>> Subject Re: [Mpi3-rma] RMA proposal 1 update >>>> : >>>> >>>> Sent mpi3-rma-bounces(a)lists.mpi-forum.org >>>> by: >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >>> >> >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Barrett, Brian W 21 May '10

21 May '10

I don't know about illegal as much as "doesn't make sense". Keith brough this up, but I think it got lost in other discussions... What is the semantic of a collective operation during a point-to-point epoch? We're constrained by the existing API (as we've decided not to write a new API) I can see a couple of scenarios for allflushall and passive target, none of which I like. The first is that allflushall only flushes with peers it currently holds a window open to. This means tracking such state, which I think is a bad idea due to state required. The second option is that it's erroneous to call allflushall unless there's a passive access epoch to every peer in the window. This seems to encourage a behavior I dont' like (namely, having to have such an epoch open at all times). I could possibly see the benefit of an allflushall in an active target, where group semantics are a bit more well-defined, but that's an entirely different discussion. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories ________________________________________ From: mpi3-rma-bounces(a)lists.mpi-forum.org [mpi3-rma-bounces(a)lists.mpi-forum.org] On Behalf Of Underwood, Keith D [keith.d.underwood(a)intel.com] Sent: Friday, May 21, 2010 1:44 PM To: MPI 3.0 Remote Memory Access working group Subject: Re: [Mpi3-rma] RMA proposal 1 update That's an interesting point. I would think that would be illegal? Keith > -----Original Message----- > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur > Sent: Friday, May 21, 2010 1:38 PM > To: 'MPI 3.0 Remote Memory Access working group' > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > Lock-unlock is not collective. If we add an allflushall, how would the > processes that don't call lock-unlock call it? Just directly? > > Rajeev > > > > -----Original Message----- > > From: mpi3-rma-bounces(a)lists.mpi-forum.org > > [mailto:[email protected]] On Behalf Of > > Underwood, Keith D > > Sent: Friday, May 21, 2010 11:22 AM > > To: MPI 3.0 Remote Memory Access working group > > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > > > There is a proposal on the table to add flush(rank) and > > flushall() as local calls for passive target to allow > > incremental remote completion without calling unlock(). > > There is an additional proposal to also include > > allflushall(), which would be a collective remote completion > > semantic added to passive target (nominally, we might not > > want to call it passive target anymore if we did that, but > > that is a different discussion). The question I had posed > > was: can you really get a measurable performance advantage > > in realistic usage scenarios by having allflushall() instead > > of just doing flushall() + barrier()? I was asking if > > someone could provide an implementation sketch - preferably > > along with a real usage scenario where that implementation > > sketch would yield a performance advantage. A > > back-of-the-envelope assessment of that would be a really > > nice to have. > > > > My contention was that the number of targets with outstanding > > requests from a given between one flush/flushall and the next > > one would frequently not be large enough to justify the cost > > of a collective. Alternatively, if messages in that window > > are "large" (for some definition of "large" that is likely > > less than 4KB and is certainly no larger than the rendezvous > > threshold), I would contend that generating a software ack > > for each one would be essentially zero overhead and would > > allow source side tracking of remote completion such that a > > flushall could be a local operation. > > > > There is a completely separate discussion that needs to occur > > on whether it is better to add collective completion to > > passive target or one-sided completion to active target (both > > of which are likely to meet some resistance because of the > > impurity the introduce into the model/naming) or whether the > > two need to be mixed at all. > > > > Keith > > > > > -----Original Message----- > > > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > > > bounces(a)lists.mpi-forum.org] On Behalf Of Douglas Miller > > > Sent: Friday, May 21, 2010 7:16 AM > > > To: MPI 3.0 Remote Memory Access working group > > > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > > > > > It is now obvious that I did not do a sufficient job of explaining > > > things > > > before. It's not even clear to me anymore exactly what we're > talking > > > about > > > - is this ARMCI? MPI2? MPI3? Active-target one-sided? > > Passive target? > > > My > > > previous explanation was for BG/P MPI2 active-target one-sided > using > > > MPI_Win_fence synchronization. BG/P MPI2 implementations for > > > MPI_Win_post/start/complete/wait and passive-target > > MPI_Win_lock/unlock > > > use > > > different methods for determining target completion, although > > > internally > > > they may use some/all of the same structures. > > > > > > What was the original question? > > > > > > _______________________________________________ > > > Douglas Miller BlueGene Messaging Development > > > IBM Corp., Rochester, MN USA Bldg 030-2 A410 > > > dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > > > > > > > > > > > > "Underwood, Keith > > > D" > > > <keith.d.underwoo > > > To > > > d(a)intel.com> "MPI 3.0 Remote Memory > Access > > > Sent by: working group" > > > mpi3-rma-bounces@ > > <mpi3-rma(a)lists.mpi-forum.org> > > > lists.mpi-forum.o > > > cc > > > rg > > > > > > Subject > > > Re: [Mpi3-rma] RMA proposal > 1 > > > 05/20/2010 04:56 update > > > PM > > > > > > > > > Please respond to > > > "MPI 3.0 Remote > > > Memory Access > > > working group" > > > <mpi3-rma(a)lists.m > > > pi-forum.org> > > > > > > > > > > > > > > > > > > > > > My point was, the way Jeff is doing synchronization in > > NWChem is via a > > > fenceall(); barrier(); on the equivalent of MPI_COMM_WORLD. > > If I knew > > > he > > > was going to be primarily doing this (ie, that he wanted to > > know that > > > all > > > nodes were synched), I would do something like maintain > > counts of sent > > > and > > > received messages on each node. I could then do something like an > > > allreduce > > > of those 2 ints over the tree to determine if everyone is synched. > > > There > > > are probably some technical details that would have to be > > worked out to > > > ensure this works but it seems good from 10000 feet. > > > > > > Right now we do numprocs 0-byte get operations to make sure > > the torus > > > is > > > flushed on each node. A torus operation is ~3us on a > > 512-way. It grows > > > slowly with number of midplanes. I'm sure a 72 rack longest > > Manhattan > > > distance noncongested pingpong is <10us, but I don't have > > the data in > > > front > > > of me. > > > > > > Based on Doug's email, I had assumed you would know who you > > have sent > > > messages to. If you knew that in a given fence interval > > the node had > > > only > > > sent distinct messages to 1K other cores, you would only > > have 1K gets > > > to > > > issue. Suck? Yes. Worse than the tree messages? Maybe, > > maybe not. > > > There is definitely a cross-over between 1 and np > > outstanding messages > > > between fences where on the 1 side of things the tree messages are > > > worse > > > and on the np side of things the tree messages are better. There > is > > > another spectrum based on request size where getting a response for > > > every > > > request becomes an inconsequential overhead. I would have > > to know the > > > cost > > > of processing a message, the size of a response, and the cost of > > > generating > > > that response to create a proper graph of that. > > > > > > A tree int/sum is roughly 5us on a 512-way and grows > > similarly. I would > > > postulate that a 72 rack MPI allreduce int/sum is on the > > order of 10us. > > > > > > So you generate np*np messages vs 1 tree message. Contention and > all > > > the > > > overhead of that many messages will be significantly worse than > even > > > several tree messages. > > > Oh, wait, so, you would sum all sent and sum all received and then > > > check if > > > they were equal? And then (presumably) iterate until the answer > was > > > yes? > > > Hrm. That is more interesting. Can you easily separate > > one-sided and > > > two > > > sided messages in your counting while maintaining the performance > of > > > one-sided messages? > > > Doug's earlier answer implied you were going to allreduce a > > vector of > > > counts (one per rank) and that would have been ugly. I am > assuming > > > you > > > would do at least 2 tree messages in what I believe you are > > describing, > > > so > > > there is still a crossover between n*np messages and m tree > messages > > > (where > > > n is the number of outstanding requests between fencealls > > and 2 <= m <= > > > 10), and the locality of communications impacts that crossover. > > > BTW, can you actually generate messages fast enough to > > cause contention > > > with tiny messages? > > > Anytime I know that an operation is collective, I can > > almost guarantee > > > I > > > can do it better than even a good pt2pt algorithm if I am > > utilizing our > > > collective network. I think on machines that have remote completion > > > notification an allfenceall() is just a barrier(), and since > > > fenceall(); > > > barrier(); is going to be replaced by allfenceall(), it > > doesn't seem to > > > me > > > like it is any extra overhead if allfenceall() is just a > > barrier() for > > > you. > > > > > > > > > My concerns are twofold: 1) we are talking about adding collective > > > completion to passive target when active target was the one > > designed to > > > have collective completion. That is semantically and API-wise a > bit > > > ugly. > > > 2) I think the allfenceall() as a collective will optimize > > to the case > > > where you have outstanding requests to everybody and I believe that > > > will be > > > slower than the typical case of having outstanding requests to > some > > > people. I think that users would typically call > > allfenceall() rather > > > than > > > fenceall() + barrier() and then they would see a > > performance paradox: > > > the > > > fenceall() + barrier() could be substantially faster when you have > a > > > "small" number of peers you are communicating with in this > > iteration. > > > I am > > > not at all worried about the overhead of allfenceall() for networks > > > with > > > remote completion. > > > Keith > > > > > > > > > > > > From: "Underwood, Keith D" <keith.d.underwood(a)intel.com> > > > > > > To: "MPI 3.0 Remote Memory Access working group" > > > <mpi3-rma(a)lists.mpi-forum.org> > > > > > > Date: 05/20/2010 09:19 AM > > > > > > Subject Re: [Mpi3-rma] RMA proposal 1 update > > > : > > > > > > Sent mpi3-rma-bounces(a)lists.mpi-forum.org > > > by: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Underwood, Keith D 21 May '10

21 May '10

That's an interesting point. I would think that would be illegal? Keith > -----Original Message----- > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > bounces(a)lists.mpi-forum.org] On Behalf Of Rajeev Thakur > Sent: Friday, May 21, 2010 1:38 PM > To: 'MPI 3.0 Remote Memory Access working group' > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > Lock-unlock is not collective. If we add an allflushall, how would the > processes that don't call lock-unlock call it? Just directly? > > Rajeev > > > > -----Original Message----- > > From: mpi3-rma-bounces(a)lists.mpi-forum.org > > [mailto:[email protected]] On Behalf Of > > Underwood, Keith D > > Sent: Friday, May 21, 2010 11:22 AM > > To: MPI 3.0 Remote Memory Access working group > > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > > > There is a proposal on the table to add flush(rank) and > > flushall() as local calls for passive target to allow > > incremental remote completion without calling unlock(). > > There is an additional proposal to also include > > allflushall(), which would be a collective remote completion > > semantic added to passive target (nominally, we might not > > want to call it passive target anymore if we did that, but > > that is a different discussion). The question I had posed > > was: can you really get a measurable performance advantage > > in realistic usage scenarios by having allflushall() instead > > of just doing flushall() + barrier()? I was asking if > > someone could provide an implementation sketch - preferably > > along with a real usage scenario where that implementation > > sketch would yield a performance advantage. A > > back-of-the-envelope assessment of that would be a really > > nice to have. > > > > My contention was that the number of targets with outstanding > > requests from a given between one flush/flushall and the next > > one would frequently not be large enough to justify the cost > > of a collective. Alternatively, if messages in that window > > are "large" (for some definition of "large" that is likely > > less than 4KB and is certainly no larger than the rendezvous > > threshold), I would contend that generating a software ack > > for each one would be essentially zero overhead and would > > allow source side tracking of remote completion such that a > > flushall could be a local operation. > > > > There is a completely separate discussion that needs to occur > > on whether it is better to add collective completion to > > passive target or one-sided completion to active target (both > > of which are likely to meet some resistance because of the > > impurity the introduce into the model/naming) or whether the > > two need to be mixed at all. > > > > Keith > > > > > -----Original Message----- > > > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > > > bounces(a)lists.mpi-forum.org] On Behalf Of Douglas Miller > > > Sent: Friday, May 21, 2010 7:16 AM > > > To: MPI 3.0 Remote Memory Access working group > > > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > > > > > It is now obvious that I did not do a sufficient job of explaining > > > things > > > before. It's not even clear to me anymore exactly what we're > talking > > > about > > > - is this ARMCI? MPI2? MPI3? Active-target one-sided? > > Passive target? > > > My > > > previous explanation was for BG/P MPI2 active-target one-sided > using > > > MPI_Win_fence synchronization. BG/P MPI2 implementations for > > > MPI_Win_post/start/complete/wait and passive-target > > MPI_Win_lock/unlock > > > use > > > different methods for determining target completion, although > > > internally > > > they may use some/all of the same structures. > > > > > > What was the original question? > > > > > > _______________________________________________ > > > Douglas Miller BlueGene Messaging Development > > > IBM Corp., Rochester, MN USA Bldg 030-2 A410 > > > dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > > > > > > > > > > > > "Underwood, Keith > > > D" > > > <keith.d.underwoo > > > To > > > d(a)intel.com> "MPI 3.0 Remote Memory > Access > > > Sent by: working group" > > > mpi3-rma-bounces@ > > <mpi3-rma(a)lists.mpi-forum.org> > > > lists.mpi-forum.o > > > cc > > > rg > > > > > > Subject > > > Re: [Mpi3-rma] RMA proposal > 1 > > > 05/20/2010 04:56 update > > > PM > > > > > > > > > Please respond to > > > "MPI 3.0 Remote > > > Memory Access > > > working group" > > > <mpi3-rma(a)lists.m > > > pi-forum.org> > > > > > > > > > > > > > > > > > > > > > My point was, the way Jeff is doing synchronization in > > NWChem is via a > > > fenceall(); barrier(); on the equivalent of MPI_COMM_WORLD. > > If I knew > > > he > > > was going to be primarily doing this (ie, that he wanted to > > know that > > > all > > > nodes were synched), I would do something like maintain > > counts of sent > > > and > > > received messages on each node. I could then do something like an > > > allreduce > > > of those 2 ints over the tree to determine if everyone is synched. > > > There > > > are probably some technical details that would have to be > > worked out to > > > ensure this works but it seems good from 10000 feet. > > > > > > Right now we do numprocs 0-byte get operations to make sure > > the torus > > > is > > > flushed on each node. A torus operation is ~3us on a > > 512-way. It grows > > > slowly with number of midplanes. I'm sure a 72 rack longest > > Manhattan > > > distance noncongested pingpong is <10us, but I don't have > > the data in > > > front > > > of me. > > > > > > Based on Doug's email, I had assumed you would know who you > > have sent > > > messages to.ï¿½ If you knew that in a given fence interval > > the node had > > > only > > > sent distinct messages to 1K other cores, you would only > > have 1K gets > > > to > > > issue.ï¿½ Suck?ï¿½ Yes.ï¿½ Worse than the tree messages?ï¿½ Maybe, > > maybe not. > > > There is definitely a cross-over between 1 and np > > outstanding messages > > > between fences where on the 1 side of things the tree messages are > > > worse > > > and on the np side of things the tree messages are better.ï¿½ There > is > > > another spectrum based on request size where getting a response for > > > every > > > request becomes an inconsequential overhead.ï¿½ I would have > > to know the > > > cost > > > of processing a message, the size of a response, and the cost of > > > generating > > > that response to create a proper graph of that. > > > > > > A tree int/sum is roughly 5us on a 512-way and grows > > similarly. I would > > > postulate that a 72 rack MPI allreduce int/sum is on the > > order of 10us. > > > > > > So you generate np*np messages vs 1 tree message. Contention and > all > > > the > > > overhead of that many messages will be significantly worse than > even > > > several tree messages. > > > Oh, wait, so, you would sum all sent and sum all received and then > > > check if > > > they were equal?ï¿½ And then (presumably) iterate until the answer > was > > > yes? > > > Hrm.ï¿½ That is more interesting.ï¿½ Can you easily separate > > one-sided and > > > two > > > sided messages in your counting while maintaining the performance > of > > > one-sided messages? > > > Doug's earlier answer implied you were going to allreduce a > > vector of > > > counts (one per rank) and that would have been ugly.ï¿½ï¿½ I am > assuming > > > you > > > would do at least 2 tree messages in what I believe you are > > describing, > > > so > > > there is still a crossover between n*np messages and m tree > messages > > > (where > > > n is the number of outstanding requests between fencealls > > and 2 <= m <= > > > 10), and the locality of communications impacts that crossover. > > > BTW, can you actually generate messages fast enough to > > cause contention > > > with tiny messages? > > > Anytime I know that an operation is collective, I can > > almost guarantee > > > I > > > can do it better than even a good pt2pt algorithm if I am > > utilizing our > > > collective network. I think on machines that have remote completion > > > notification an allfenceall() is just a barrier(), and since > > > fenceall(); > > > barrier(); is going to be replaced by allfenceall(), it > > doesn't seem to > > > me > > > like it is any extra overhead if allfenceall() is just a > > barrier() for > > > you. > > > > > > > > > My concerns are twofold:ï¿½ 1) we are talking about adding collective > > > completion to passive target when active target was the one > > designed to > > > have collective completion.ï¿½ That is semantically and API-wise a > bit > > > ugly. > > > 2) I think the allfenceall() as a collective will optimize > > to the case > > > where you have outstanding requests to everybody and I believe that > > > will be > > > slower than the typical ï¿½case of having outstanding requests to > some > > > people.ï¿½ I think that users would typically call > > allfenceall() rather > > > than > > > fenceall() + barrier() and then they would see a > > performance paradox: > > > the > > > fenceall() + barrier() could be substantially faster when you have > a > > > "small" number of peers you are communicating with in this > > iteration. > > > I am > > > not at all worried about the overhead of allfenceall() for networks > > > with > > > remote completion. > > > Keith > > > > > > > > > > > > From: "Underwood, Keith D" <keith.d.underwood(a)intel.com> > > > > > > To: "MPI 3.0 Remote Memory Access working group" > > > <mpi3-rma(a)lists.mpi-forum.org> > > > > > > Date: 05/20/2010 09:19 AM > > > > > > Subject Re: [Mpi3-rma] RMA proposal 1 update > > > : > > > > > > Sent mpi3-rma-bounces(a)lists.mpi-forum.org > > > by: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] RMA proposal 1 update
by Rajeev Thakur 21 May '10

21 May '10

Lock-unlock is not collective. If we add an allflushall, how would the processes that don't call lock-unlock call it? Just directly? Rajeev > -----Original Message----- > From: mpi3-rma-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of > Underwood, Keith D > Sent: Friday, May 21, 2010 11:22 AM > To: MPI 3.0 Remote Memory Access working group > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > There is a proposal on the table to add flush(rank) and > flushall() as local calls for passive target to allow > incremental remote completion without calling unlock(). > There is an additional proposal to also include > allflushall(), which would be a collective remote completion > semantic added to passive target (nominally, we might not > want to call it passive target anymore if we did that, but > that is a different discussion). The question I had posed > was: can you really get a measurable performance advantage > in realistic usage scenarios by having allflushall() instead > of just doing flushall() + barrier()? I was asking if > someone could provide an implementation sketch - preferably > along with a real usage scenario where that implementation > sketch would yield a performance advantage. A > back-of-the-envelope assessment of that would be a really > nice to have. > > My contention was that the number of targets with outstanding > requests from a given between one flush/flushall and the next > one would frequently not be large enough to justify the cost > of a collective. Alternatively, if messages in that window > are "large" (for some definition of "large" that is likely > less than 4KB and is certainly no larger than the rendezvous > threshold), I would contend that generating a software ack > for each one would be essentially zero overhead and would > allow source side tracking of remote completion such that a > flushall could be a local operation. > > There is a completely separate discussion that needs to occur > on whether it is better to add collective completion to > passive target or one-sided completion to active target (both > of which are likely to meet some resistance because of the > impurity the introduce into the model/naming) or whether the > two need to be mixed at all. > > Keith > > > -----Original Message----- > > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- > > bounces(a)lists.mpi-forum.org] On Behalf Of Douglas Miller > > Sent: Friday, May 21, 2010 7:16 AM > > To: MPI 3.0 Remote Memory Access working group > > Subject: Re: [Mpi3-rma] RMA proposal 1 update > > > > It is now obvious that I did not do a sufficient job of explaining > > things > > before. It's not even clear to me anymore exactly what we're talking > > about > > - is this ARMCI? MPI2? MPI3? Active-target one-sided? > Passive target? > > My > > previous explanation was for BG/P MPI2 active-target one-sided using > > MPI_Win_fence synchronization. BG/P MPI2 implementations for > > MPI_Win_post/start/complete/wait and passive-target > MPI_Win_lock/unlock > > use > > different methods for determining target completion, although > > internally > > they may use some/all of the same structures. > > > > What was the original question? > > > > _______________________________________________ > > Douglas Miller BlueGene Messaging Development > > IBM Corp., Rochester, MN USA Bldg 030-2 A410 > > dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > > > > > > > > "Underwood, Keith > > D" > > <keith.d.underwoo > > To > > d(a)intel.com> "MPI 3.0 Remote Memory Access > > Sent by: working group" > > mpi3-rma-bounces@ > <mpi3-rma(a)lists.mpi-forum.org> > > lists.mpi-forum.o > > cc > > rg > > > > Subject > > Re: [Mpi3-rma] RMA proposal 1 > > 05/20/2010 04:56 update > > PM > > > > > > Please respond to > > "MPI 3.0 Remote > > Memory Access > > working group" > > <mpi3-rma(a)lists.m > > pi-forum.org> > > > > > > > > > > > > > > My point was, the way Jeff is doing synchronization in > NWChem is via a > > fenceall(); barrier(); on the equivalent of MPI_COMM_WORLD. > If I knew > > he > > was going to be primarily doing this (ie, that he wanted to > know that > > all > > nodes were synched), I would do something like maintain > counts of sent > > and > > received messages on each node. I could then do something like an > > allreduce > > of those 2 ints over the tree to determine if everyone is synched. > > There > > are probably some technical details that would have to be > worked out to > > ensure this works but it seems good from 10000 feet. > > > > Right now we do numprocs 0-byte get operations to make sure > the torus > > is > > flushed on each node. A torus operation is ~3us on a > 512-way. It grows > > slowly with number of midplanes. I'm sure a 72 rack longest > Manhattan > > distance noncongested pingpong is <10us, but I don't have > the data in > > front > > of me. > > > > Based on Dougï¿½s email, I had assumed you would know who you > have sent > > messages toï¿½ï¿½ If you knew that in a given fence interval > the node had > > only > > sent distinct messages to 1K other cores, you would only > have 1K gets > > to > > issue.ï¿½ Suck?ï¿½ Yes.ï¿½ Worse than the tree messages?ï¿½ Maybe, > maybe not. > > There is definitely a cross-over between 1 and np > outstanding messages > > between fences where on the 1 side of things the tree messages are > > worse > > and on the np side of things the tree messages are better.ï¿½ There is > > another spectrum based on request size where getting a response for > > every > > request becomes an inconsequential overhead.ï¿½ I would have > to know the > > cost > > of processing a message, the size of a response, and the cost of > > generating > > that response to create a proper graph of that. > > > > A tree int/sum is roughly 5us on a 512-way and grows > similarly. I would > > postulate that a 72 rack MPI allreduce int/sum is on the > order of 10us. > > > > So you generate np*np messages vs 1 tree message. Contention and all > > the > > overhead of that many messages will be significantly worse than even > > several tree messages. > > Oh, wait, so, you would sum all sent and sum all received and then > > check if > > they were equal?ï¿½ And then (presumably) iterate until the answer was > > yes? > > Hrm.ï¿½ That is more interesting.ï¿½ Can you easily separate > one-sided and > > two > > sided messages in your counting while maintaining the performance of > > one-sided messages? > > Dougï¿½s earlier answer implied you were going to allreduce a > vector of > > counts (one per rank) and that would have been ugly.ï¿½ï¿½ I am assuming > > you > > would do at least 2 tree messages in what I believe you are > describing, > > so > > there is still a crossover between n*np messages and m tree messages > > (where > > n is the number of outstanding requests between fencealls > and 2 <= m <= > > 10), and the locality of communications impacts that crossoverï¿½ > > BTW, can you actually generate messages fast enough to > cause contention > > with tiny messages? > > Anytime I know that an operation is collective, I can > almost guarantee > > I > > can do it better than even a good pt2pt algorithm if I am > utilizing our > > collective network. I think on machines that have remote completion > > notification an allfenceall() is just a barrier(), and since > > fenceall(); > > barrier(); is going to be replaced by allfenceall(), it > doesn't seem to > > me > > like it is any extra overhead if allfenceall() is just a > barrier() for > > you. > > > > > > My concerns are twofold:ï¿½ 1) we are talking about adding collective > > completion to passive target when active target was the one > designed to > > have collective completion.ï¿½ That is semantically and API-wise a bit > > ugly. > > 2) I think the allfenceall() as a collective will optimize > to the case > > where you have outstanding requests to everybody and I believe that > > will be > > slower than the typical ï¿½case of having outstanding requests to some > > people.ï¿½ I think that users would typically call > allfenceall() rather > > than > > fenceall() + barrier() and then they would see a > performance paradox: > > the > > fenceall() + barrier() could be substantially faster when you have a > > ï¿½smallï¿½ number of peers you are communicating with in this > iteration. > > I am > > not at all worried about the overhead of allfenceall() for networks > > with > > remote completion. > > Keith > > > > > > > > From: "Underwood, Keith D" <keith.d.underwood(a)intel.com> > > > > To: "MPI 3.0 Remote Memory Access working group" > > <mpi3-rma(a)lists.mpi-forum.org> > > > > Date: 05/20/2010 09:19 AM > > > > Subject Re: [Mpi3-rma] RMA proposal 1 update > > : > > > > Sent mpi3-rma-bounces(a)lists.mpi-forum.org > > by: > > > > > > > > > > > > > > > > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0