April 2009 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] MPI One-Sided Communication
by Underwood, Keith D 27 Apr '09

27 Apr '09

<sigh> I meant the copy in the application that comes before MPI is called. A lot of apps don't have all of the data they want to send in a contiguous buffer. And, yes, derived datatypes can help this, but aren't commonly used and still involve a copy on a lot of hardware. Keith ----- Original Message ----- From: mpi-forum-bounces(a)lists.mpi-forum.org <mpi-forum-bounces(a)lists.mpi-forum.org> To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> Sent: Mon Apr 27 06:32:53 2009 Subject: Re: [Mpi-forum] MPI One-Sided Communication On Apr 27, 2009, at 9:14 AM, Underwood, Keith D wrote: > For many (most?) science and engineering codes, the trade-off > between matching for two sided and synchronization for one sided is > at best a wash, and often would fall in favor of two sided. There > is, of course, an exception.... > > If one sided delivered truly stellar performance in terms of message > rate and latency, you could (in some cases) eliminate the cost of > the copy at the sender that is tyically done to send long messages > to cover the overhead. The hardware to deliver that level of > performance is truly rare... > > Keith "cost of the copy at the sender that is tyically done to send long messages to cover the overhead" ? Typically, any HPC NIC uses zero-copy for large messages. Copy is only used for small messages (where small depends on the fabric/NIC/ software stack). It is even possible to avoid copy on send for large messages over standard Ethernet using sendfile() and other options. Scott _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Hubert Ritzdorf 27 Apr '09

27 Apr '09

Underwood, Keith D wrote: >> On the Earth Simulator, there are/were several application codes which are >> using one-sided communication (instead of 2-sided). They used one-sided >> communication especially to overlap communication and computation. >> When I remember correctly, at least one of this applications won a Gordon >> Bell Award of SC. >> > > The ambiguity of the progress rule notwithstanding, there is no particular reason that one-sided should give you better overlap than two-sided. If this is the reason that people use one-sided, maybe we should revisit the progress rule ;-) > The application codes can overlap communication and computation also for pt2pt on the Earth Simulator. The raw performance of MPI pt2pt and one-sided communication is similar. Which communication is selected, should depend on the application code. And there are application codes which benefit on the one-sided communication model. But the usage of the MPI-2 one-sided library is not zero and one-sided comm is not "always" slower than 2-sided. Hubert PS: Nevertheless, also the Earth Simulator application programmers have asked for simpler synchronization functions for one-sided comms.

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Scott Atchley 27 Apr '09

27 Apr '09

On Apr 27, 2009, at 9:02 AM, Vinod tipparaju wrote: > Don't forget matching. The model depends on a relation between send > and receive. This is the fundamental reason for potential difference > in overlap. If you talk implementation, which we technically > shouldn't for this argument, eventually the fact that a matching > receiver is required for a send does impact over multiple sends. > > the one-sided model > > a->b > > is independent of b. > > the two sided model > > a<->b > > because of its dependance implies validity -- you can hide the cost > of validity but can't eliminate it. > > Vinod. Ok, I am stepping in over my head here and Patrick may slap me for doing so. :-) Doesn't one-sided also require some form of matching in the implementation of A->B to determine where the bytes go in B? Call it a rkey or whatever you want, there is some form of address hint that requires translation to place the data in the right buffer. It can be done on NIC or elsewhere, but it is not free. My view of one-sided and two-sided is simply one from a library user (to implement an application). Originally, I drank from the cup of the One-Sided and helped implement a wide-area, capability-based, file transfer service. Our users, however, always wanted to know when an operation completed, which required an extra notification message. It is that notification that always leads me back to two-sided. Surely I am missing something obvious (and please let me know), but I always thought that the biggest benefit to one sided is the case where A->B multiple times and B does not have to reregister/repost the buffer. But if A has to notify B that the data is there (or B has to poll on a byte or doorbell), how is that different than checking the completion of a receive? Overlap can be accomplished in either method (don't use blocking send/ recv). Scott

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Vinod tipparaju 27 Apr '09

27 Apr '09

Hoping not to sound like a fool who preaches to the choir (portals 4.0, denial?) but still: we are moving forward, towards truly stellar performance. You look at one-sided to achieve this, for a good reason. A lot is changing in application space. As applications move forward to and beyond 100k processors, the complexity of coding for expected receives will become harder. Applications that think of limited receivers (small group of expected senders) and never do collective communication, they are a different category. They certainly have many merits but I don't want to think of them when designing large systems. Also, many applications are restarting their thought process for petascale and beyond. New applications are being designed and more of them are using and looking at GAS, hence one-sided, model. I don't want to be the culprit who starts the manycore discussion too, but manycore naturally and fundamentally fit PGAS because one may no longer want to have a core\process level address space but a socket level address space in the OS to make it easier to run programs. From: keith.d.underwood(a)intel.com To: mpi-forum(a)lists.mpi-forum.org Date: Mon, 27 Apr 2009 07:14:12 -0600 Subject: Re: [Mpi-forum] MPI One-Sided Communication For many (most?) science and engineering codes, the trade-off between matching for two sided and synchronization for one sided is at best a wash, and often would fall in favor of two sided. There is, of course, an exception.... If one sided delivered truly stellar performance in terms of message rate and latency, you could (in some cases) eliminate the cost of the copy at the sender that is tyically done to send long messages to cover the overhead. The hardware to deliver that level of performance is truly rare... Keith From: mpi-forum-bounces(a)lists.mpi-forum.org <mpi-forum-bounces(a)lists.mpi-forum.org> To: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Sent: Mon Apr 27 06:02:29 2009 Subject: Re: [Mpi-forum] MPI One-Sided Communication Don't forget matching. The model depends on a relation between send and receive. This is the fundamental reason for potential difference in overlap. If you talk implementation, which we technically shouldn't for this argument, eventually the fact that a matching receiver is required for a send does impact over multiple sends. the one-sided model a->b is independent of b. the two sided model a<->b because of its dependance implies validity -- you can hide the cost of validity but can't eliminate it. Vinod. > From: keith.d.underwood(a)intel.com > To: mpi-forum(a)lists.mpi-forum.org > Date: Mon, 27 Apr 2009 06:48:57 -0600 > Subject: Re: [Mpi-forum] MPI One-Sided Communication > > >>On the Earth Simulator, there are/were several application codes which are >>using one-sided communication (instead of 2-sided). They used one-sided >>communication especially to overlap communication and computation. >>When I remember correctly, at least one of this applications won a Gordon >>Bell Award of SC. > > The ambiguity of the progress rule notwithstanding, there is no particular reason that one-sided should give you better overlap than two-sided. If this is the reason that people use one-sided, maybe we should revisit the progress rule ;-) > > Keith > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Scott Atchley 27 Apr '09

27 Apr '09

On Apr 27, 2009, at 9:14 AM, Underwood, Keith D wrote: > For many (most?) science and engineering codes, the trade-off > between matching for two sided and synchronization for one sided is > at best a wash, and often would fall in favor of two sided. There > is, of course, an exception.... > > If one sided delivered truly stellar performance in terms of message > rate and latency, you could (in some cases) eliminate the cost of > the copy at the sender that is tyically done to send long messages > to cover the overhead. The hardware to deliver that level of > performance is truly rare... > > Keith "cost of the copy at the sender that is tyically done to send long messages to cover the overhead" ? Typically, any HPC NIC uses zero-copy for large messages. Copy is only used for small messages (where small depends on the fabric/NIC/ software stack). It is even possible to avoid copy on send for large messages over standard Ethernet using sendfile() and other options. Scott

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Underwood, Keith D 27 Apr '09

27 Apr '09

For many (most?) science and engineering codes, the trade-off between matching for two sided and synchronization for one sided is at best a wash, and often would fall in favor of two sided. There is, of course, an exception.... If one sided delivered truly stellar performance in terms of message rate and latency, you could (in some cases) eliminate the cost of the copy at the sender that is tyically done to send long messages to cover the overhead. The hardware to deliver that level of performance is truly rare... Keith ________________________________ From: mpi-forum-bounces(a)lists.mpi-forum.org <mpi-forum-bounces(a)lists.mpi-forum.org> To: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Sent: Mon Apr 27 06:02:29 2009 Subject: Re: [Mpi-forum] MPI One-Sided Communication Don't forget matching. The model depends on a relation between send and receive. This is the fundamental reason for potential difference in overlap. If you talk implementation, which we technically shouldn't for this argument, eventually the fact that a matching receiver is required for a send does impact over multiple sends. the one-sided model a->b is independent of b. the two sided model a<->b because of its dependance implies validity -- you can hide the cost of validity but can't eliminate it. Vinod. > From: keith.d.underwood(a)intel.com > To: mpi-forum(a)lists.mpi-forum.org > Date: Mon, 27 Apr 2009 06:48:57 -0600 > Subject: Re: [Mpi-forum] MPI One-Sided Communication > > >>On the Earth Simulator, there are/were several application codes which are >>using one-sided communication (instead of 2-sided). They used one-sided >>communication especially to overlap communication and computation. >>When I remember correctly, at least one of this applications won a Gordon >>Bell Award of SC. > > The ambiguity of the progress rule notwithstanding, there is no particular reason that one-sided should give you better overlap than two-sided. If this is the reason that people use one-sided, maybe we should revisit the progress rule ;-) > > Keith > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Vinod tipparaju 27 Apr '09

27 Apr '09

Don't forget matching. The model depends on a relation between send and receive. This is the fundamental reason for potential difference in overlap. If you talk implementation, which we technically shouldn't for this argument, eventually the fact that a matching receiver is required for a send does impact over multiple sends. the one-sided model a->b is independent of b. the two sided model a<->b because of its dependance implies validity -- you can hide the cost of validity but can't eliminate it. Vinod. > From: keith.d.underwood(a)intel.com > To: mpi-forum(a)lists.mpi-forum.org > Date: Mon, 27 Apr 2009 06:48:57 -0600 > Subject: Re: [Mpi-forum] MPI One-Sided Communication > > >>On the Earth Simulator, there are/were several application codes which are >>using one-sided communication (instead of 2-sided). They used one-sided >>communication especially to overlap communication and computation. >>When I remember correctly, at least one of this applications won a Gordon >>Bell Award of SC. > > The ambiguity of the progress rule notwithstanding, there is no particular reason that one-sided should give you better overlap than two-sided. If this is the reason that people use one-sided, maybe we should revisit the progress rule ;-) > > Keith > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Underwood, Keith D 27 Apr '09

27 Apr '09

>On the Earth Simulator, there are/were several application codes which are >using one-sided communication (instead of 2-sided). They used one-sided >communication especially to overlap communication and computation. >When I remember correctly, at least one of this applications won a Gordon >Bell Award of SC. The ambiguity of the progress rule notwithstanding, there is no particular reason that one-sided should give you better overlap than two-sided. If this is the reason that people use one-sided, maybe we should revisit the progress rule ;-) Keith

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Hubert Ritzdorf 27 Apr '09

27 Apr '09

Greg Lindahl wrote: > On Thu, Apr 23, 2009 at 01:28:22PM -0600, Simone Atzeni wrote: > > >> Hi, I'm an italian student and I'm doing a thesis about Formal >> Verification. Now, I'm looking for information about mpi one-sided >> communication to see who uses one-sided and for what (user applications >> or implementing library). >> > > As far as I can tell, usage of the MPI-2 one-sided library is about > zero. People tried it on various interconnects, and when it turned out > to be slower than 2-sided, stopped trying. > > > On the Earth Simulator, there are/were several application codes which are using one-sided communication (instead of 2-sided). They used one-sided communication especially to overlap communication and computation. When I remember correctly, at least one of this applications won a Gordon Bell Award of SC. There are performance advantages for one-sided communication on the Earth Simulator (and SX systems) (nevertheless, it is not clear to me how you define latency for small messages together with the synchronization functions of MPI-2). Hubert NEC Europe

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Rajeev Thakur 25 Apr '09

25 Apr '09

Jeff, Thanks for your comments. A good goal for MPI-3 RMA would be to define an interface that NWCHEM would want to use directly, instead of having to rely on ARMCI. Your input in defining such an interface would be valuable. Rajeev > -----Original Message----- > From: mpi-forum-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of > Jeff Hammond > Sent: Saturday, April 25, 2009 7:16 AM > To: Main MPI Forum mailing list > Subject: Re: [Mpi-forum] MPI One-Sided Communication > > Patrick and Keith, > > You both make very helpful points. Yes, I am from the Church > of One-sided, but only because I've got hundreds of papers > showing that one-sided implementations scale very well for my > application and zero papers showing scaling for two-sided. > You argue against one-sided using the prevalence of MPI-1 > use, but if you look at my field, the exact same argument > goes against MPI-1. > > The reason I am trying to participate in the MPI-3 discussion > is because whenever you all finish the standard and the > various implementations, I am the one who has to develop > applications with it. > I don't know all the low-level issues, but what I want is > something that quacks like ARMCI. If MPI-3 RMA doesn't do > that, I will continue not to use MPI. The reason I wanted > MPI-3 RMA to take over for ARMCI is that vendors port and > tune MPI first and I don't like having to wait a year to run > NWChem, etc. on the leadership-class hardware. I don't care > what happens under the hood (active-messages versus RDMA, > etc.) as long as I can Put, Get and Accumulate without > interrupting the remote processor. > > Why do almost everyone use MPI-1? Find me a syllabus for > course on parallel computing which teaches one-sided > anything. Find me on that doesn't teach MPI-1 exclusively. > Yes, there will be a few out Berkeley for UPC, UIUC for > Charm++, etc. but the overwhelming majority of parallel > programming education today focuses exclusively on MPI-1. > If you teach a man to fish, he will fish, not farm wheat. > > I often confuse RDMA and one-sided, among other things. I am > a HPC code developer not a hardware designer or NIC-protocol > writer. I'd love it if someone can tell me the (accessible) > book I can read to figure these things out. I'm willing to > plead ignorance on some of the low-level issues here if, in > return, try not to claim that everything in my field can > easily be done with MPI-1, since 20 years of papers say it > can't (at least easily). All I'm trying to say telling me I > should be using MPI-1 instead is not an argument. > > Jeff > > -- > Jeff Hammond > The University of Chicago > http://home.uchicago.edu/~jhammond/ > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >

1 0