April 2009 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] MPI One-Sided Communication
by Jeff Hammond 25 Apr '09

25 Apr '09

Patrick and Keith, You both make very helpful points. Yes, I am from the Church of One-sided, but only because I've got hundreds of papers showing that one-sided implementations scale very well for my application and zero papers showing scaling for two-sided. You argue against one-sided using the prevalence of MPI-1 use, but if you look at my field, the exact same argument goes against MPI-1. The reason I am trying to participate in the MPI-3 discussion is because whenever you all finish the standard and the various implementations, I am the one who has to develop applications with it. I don't know all the low-level issues, but what I want is something that quacks like ARMCI. If MPI-3 RMA doesn't do that, I will continue not to use MPI. The reason I wanted MPI-3 RMA to take over for ARMCI is that vendors port and tune MPI first and I don't like having to wait a year to run NWChem, etc. on the leadership-class hardware. I don't care what happens under the hood (active-messages versus RDMA, etc.) as long as I can Put, Get and Accumulate without interrupting the remote processor. Why do almost everyone use MPI-1? Find me a syllabus for course on parallel computing which teaches one-sided anything. Find me on that doesn't teach MPI-1 exclusively. Yes, there will be a few out Berkeley for UPC, UIUC for Charm++, etc. but the overwhelming majority of parallel programming education today focuses exclusively on MPI-1. If you teach a man to fish, he will fish, not farm wheat. I often confuse RDMA and one-sided, among other things. I am a HPC code developer not a hardware designer or NIC-protocol writer. I'd love it if someone can tell me the (accessible) book I can read to figure these things out. I'm willing to plead ignorance on some of the low-level issues here if, in return, try not to claim that everything in my field can easily be done with MPI-1, since 20 years of papers say it can't (at least easily). All I'm trying to say telling me I should be using MPI-1 instead is not an argument. Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Patrick Geoffray 24 Apr '09

24 Apr '09

Dear Jeff, Jeff Hammond wrote: > Frankly, this is just a religious war already waged in HPCWire > (Myricom versus IBM). It appears you're saying that IBM and Cray Ok, this is my invitation to participate in this sterile discussion. First of all, it was not Myricom versus IBM, it was me, as someone who implemented MPI-1 and one-sided interfaces such as VIA, versus a guy who sat on IB/iWarp/iSCSI/FC committees knowing zip about communication middlewares. It looks like a religion war for people who do not understand the issues. Fortunately, a lot of people in this forum do, as least the implementors. From your background and comments, you are a product of the PNNL Church of One-Siding. Nothing wrong with that, free country. However, you should step outside of the NWchem/GA/ARMCI community and see what the rest of the world is doing. How many apps use GA/ARMCI except NWchem and Molpro ? I happen to believe that there is no difference in performance between one and two-sided interfaces on most apps, as long as the app and middleware codes are optimized. I had this discussion with Vinod a few years ago, when he was presenting an ARMCI vs MPI paper: if you spend the same care when writing the MPI version of the code, it would perform the same as the freshly written ARMCI version. It should not be a performance discussion, it should be about ease of use. Ease of use is subjective of course, but the fact that there are a gazillions more MPI codes than one-sided codes out there could be a hint. > supporting powerful one-sided hardware is a waste of money. Is that > why their machines, SGI's and those running Infiniband (excellent > RDMA) compose the first 26 slots on the Top500 list? Anti-one-sided > Myricom's best showing is #27. If you think that people buy machines because of one-sided interface, you are greatly mistaken. If you think the large IB machines use one-sided operations for MPI small/medium messages at scale, you are wrong again. Finally, if you think that Myricom is anti-one-sided, you don't know the differences between RDMA and one-sided. We happen to have ported ARMCI on top of asynchronous Active Messages in MX, like Pathscale did. It walks like one-sided, it quacks like one-sided, but it scales well. This is essentially Greg's point: you can implement one-sided semantics on Active Messages and it does scale. Now, it doesn't help in the discussion about MPI-3 one-sided interface. There is simply no incentive for implementors to spend time on it if nobody uses it. Today, almost nobody does. Why ? Patrick

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Underwood, Keith D 24 Apr '09

24 Apr '09

For the record, I'm an advocate of providing one-sided semantics and one-sided hardware support. Heck, I help work on the Portals specification. I just don't want to see that support to the exclusion of MPI and MPI hardware support. I also want good arguments and examples to use to justify that support to get it in more hardware. Scalable, fast (low latency, high message rate) one-sided hardware is not so prevalent as you seem to think. >> Ahem. ï¿½I think you just proved Greg's point. ï¿½Jaguar is a Cray XT >> series. ï¿½While I rather like that machine (I was part of the team at >> Sandia at the time), it has atrocious one-sided performance for small >> messages. ï¿½If you can do it with big messages, you can do it with MPI-1 >> two sided. > >I repeat: show me the code. By claiming that I could do it with >MPI-1, you're saying that $100 million worth of one-sided-based >quantum chemistry codes were a waste of time and money and that some >very well regarded people are, in fact, quite stupid. I'm claiming nothing of the sort. I'm simply pointing out that if you set the one-sided performance of the XT as your bar, you are setting an awfully low bar. It's got nice one-sided SEMANTICS (I like Portals, too), but the performance... >>>See the attached papers for the details of the algorithm. ï¿½The >>>GA/ARMCI implementation is less than 1000 lines of code. ï¿½Please let >>>me know when you can match this with MPI-1. >> >> At this point, we are just bickering about where the work is, right? ï¿½So, >the app wants to program in a GA model for ease of use. ï¿½Great. ï¿½That isn't >a performance argument. > >If programmability isn't part of the equation, why have MPI? Why have >C? We can all just write in assembly, right? > >We need to have an optimal balance between programmability and >performance. For the HPC applications I work on, one-sided beats >two-sided hands down. That's not a philosophy, it's 20 years of >results on machines from the IBM SP-2 to the Cray XT5. Where did I say I was opposed to programmability?!? If you want to work in GA for ease of use, Great! If you need it for performance, Great! However, don't tell me that you need it for performance and show me a system that takes an interrupt for EVERY SINGLE MESSAGE as an example. >>>Frankly, this is just a religious war already waged in HPCWire >>>(Myricom versus IBM). ï¿½It appears you're saying that IBM and Cray >>>supporting powerful one-sided hardware is a waste of money. ï¿½Is that >>>why their machines, SGI's and those running Infiniband (excellent >>>RDMA) compose the first 26 slots on the Top500 list? ï¿½Anti-one-sided >>>Myricom's best showing is #27. >> >> Um, yeah, Infiniband isn't up there because of its one-sided >> support. ï¿½Neither is the Cray machine. ï¿½Oh, wait, neither are the IBM >> machines. > >So good hardware support for one-sided correlates strongly with Top500 >performance? Why then argue against one-sided hardware? No, it doesn't. And, no, I'm not. The XT series is a great product with great semantic support for one-sided and relatively poor one-sided performance. I've never worked with a BG machine, so can't say. >Honestly, I can't follow this moving target argument against >one-sided. What is bad about one-sided, the hardware support for it, >the programming model itself, or the use of implementation layers for >programming within the model? It seems you guys want to pick and >choose what aspect of one-sided you want to trash whenever I attempt >to pin down what your trying to say. I'm not arguing against one-sided hardware, software, or implementations. I was simply frustrated by your arguments against Greg. It is very hard to create new one-sided/RMA semantics for MPI when we can't zero in on what the semantic and performance requirements are. I would love to see a quantitative study of the one-sided performance needs of an application. Telling me that Jaguar is great for the purpose would lead me to believe that those requirements aren't particularly strict. Keith

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Jeff Hammond 24 Apr '09

24 Apr '09

On Fri, Apr 24, 2009 at 9:22 PM, Underwood, Keith D <keith.d.underwood(a)intel.com> wrote: >>No, that's not true in the slightest for what Vinod is referring to. >>NWChem's CCSD(T) code hit 0.357 petaflops at 0.55% efficiency on >>Jaguar and this method is anything but simple. ï¿½The most scalable >>portion is a 7-loop accumulation wherein the first loop computes very >>large intermediates (by design, it fills up the memory) which must be >>communicated all over the place in a non-trivial way along with other >>intermediates and permanent data inside of another set of do loops. >>Once all the buffers get pushed around, the inner loops are just a >>bunch of DGEMMs. > > Ahem. ï¿½I think you just proved Greg's point. ï¿½Jaguar is a Cray XT series. ï¿½While I rather like that machine (I was part of the team at Sandia at the time), it has atrocious one-sided performance for small messages. ï¿½If you can do it with big messages, you can do it with MPI-1 two sided. I repeat: show me the code. By claiming that I could do it with MPI-1, you're saying that $100 million worth of one-sided-based quantum chemistry codes were a waste of time and money and that some very well regarded people are, in fact, quite stupid. >>See the attached papers for the details of the algorithm. ï¿½The >>GA/ARMCI implementation is less than 1000 lines of code. ï¿½Please let >>me know when you can match this with MPI-1. > > At this point, we are just bickering about where the work is, right? ï¿½So, the app wants to program in a GA model for ease of use. ï¿½Great. ï¿½That isn't a performance argument. If programmability isn't part of the equation, why have MPI? Why have C? We can all just write in assembly, right? We need to have an optimal balance between programmability and performance. For the HPC applications I work on, one-sided beats two-sided hands down. That's not a philosophy, it's 20 years of results on machines from the IBM SP-2 to the Cray XT5. >>> I think this nicely illustrates my point that I don't know of any >>> existing HPC application that really needs one-sided hardware to get >>> great performance.* >> >>Frankly, this is just a religious war already waged in HPCWire >>(Myricom versus IBM). ï¿½It appears you're saying that IBM and Cray >>supporting powerful one-sided hardware is a waste of money. ï¿½Is that >>why their machines, SGI's and those running Infiniband (excellent >>RDMA) compose the first 26 slots on the Top500 list? ï¿½Anti-one-sided >>Myricom's best showing is #27. > > Um, yeah, Infiniband isn't up there because of its one-sided support. ï¿½Neither is the Cray machine. ï¿½Oh, wait, neither are the IBM machines. So good hardware support for one-sided correlates strongly with Top500 performance? Why then argue against one-sided hardware? Honestly, I can't follow this moving target argument against one-sided. What is bad about one-sided, the hardware support for it, the programming model itself, or the use of implementation layers for programming within the model? It seems you guys want to pick and choose what aspect of one-sided you want to trash whenever I attempt to pin down what your trying to say. I've heard that when MPI-1 was being developed, having an implementation settled a lot of religious debates immediately. I'm asking you now to follow that pattern and show me the two-sided implementation that beats one-sided for a specific HPC application: CCSD(T). Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Greg Lindahl 24 Apr '09

24 Apr '09

On Fri, Apr 24, 2009 at 09:10:48PM -0500, Jeff Hammond wrote: > Frankly, this is just a religious war already waged in HPCWire > (Myricom versus IBM). It appears you're saying that IBM and Cray > supporting powerful one-sided hardware is a waste of money. No, I wasn't really saying that. I did apparently gore your ox, though... I didn't realize that a discussion of a high level interface for MPI was a stalking horse for arguing about the benefits of RDMA. > Is that > why their machines, SGI's and those running Infiniband (excellent > RDMA) compose the first 26 slots on the Top500 list? I believe that many of those systems spend very little time runing one-sided code. And that the ability to run one-sided code was probably not the biggest factor in picking interconnect for them. I was hoping this discussion would stick to technical things. Market share is by its very nature not a very technical discussion. The philosophy of whether RDMA is a good idea is by its very nature not a very technical discussion. I will now bow out of this discussion. -- greg

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Underwood, Keith D 24 Apr '09

24 Apr '09

>No, that's not true in the slightest for what Vinod is referring to. >NWChem's CCSD(T) code hit 0.357 petaflops at 0.55% efficiency on >Jaguar and this method is anything but simple. The most scalable >portion is a 7-loop accumulation wherein the first loop computes very >large intermediates (by design, it fills up the memory) which must be >communicated all over the place in a non-trivial way along with other >intermediates and permanent data inside of another set of do loops. >Once all the buffers get pushed around, the inner loops are just a >bunch of DGEMMs. Ahem. I think you just proved Greg's point. Jaguar is a Cray XT series. While I rather like that machine (I was part of the team at Sandia at the time), it has atrocious one-sided performance for small messages. If you can do it with big messages, you can do it with MPI-1 two sided. >See the attached papers for the details of the algorithm. The >GA/ARMCI implementation is less than 1000 lines of code. Please let >me know when you can match this with MPI-1. At this point, we are just bickering about where the work is, right? So, the app wants to program in a GA model for ease of use. Great. That isn't a performance argument. >> I think this nicely illustrates my point that I don't know of any >> existing HPC application that really needs one-sided hardware to get >> great performance.* > >Frankly, this is just a religious war already waged in HPCWire >(Myricom versus IBM). It appears you're saying that IBM and Cray >supporting powerful one-sided hardware is a waste of money. Is that >why their machines, SGI's and those running Infiniband (excellent >RDMA) compose the first 26 slots on the Top500 list? Anti-one-sided >Myricom's best showing is #27. Um, yeah, Infiniband isn't up there because of its one-sided support. Neither is the Cray machine. Oh, wait, neither are the IBM machines. Keith

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Jeff Hammond 24 Apr '09

24 Apr '09

>> Hmm. There are modules of NWChem that have datasets capable of >> scaling from 1s to 100's of thousands of cores. > > Right. Those don't need a fast interconnect. They probably also would > run fine with MPI-1. That's a solved problem; we don't need to think > much about it. No, that's not true in the slightest for what Vinod is referring to. NWChem's CCSD(T) code hit 0.357 petaflops at 0.55% efficiency on Jaguar and this method is anything but simple. The most scalable portion is a 7-loop accumulation wherein the first loop computes very large intermediates (by design, it fills up the memory) which must be communicated all over the place in a non-trivial way along with other intermediates and permanent data inside of another set of do loops. Once all the buffers get pushed around, the inner loops are just a bunch of DGEMMs. See the attached papers for the details of the algorithm. The GA/ARMCI implementation is less than 1000 lines of code. Please let me know when you can match this with MPI-1. > I think this nicely illustrates my point that I don't know of any > existing HPC application that really needs one-sided hardware to get > great performance.* Frankly, this is just a religious war already waged in HPCWire (Myricom versus IBM). It appears you're saying that IBM and Cray supporting powerful one-sided hardware is a waste of money. Is that why their machines, SGI's and those running Infiniband (excellent RDMA) compose the first 26 slots on the Top500 list? Anti-one-sided Myricom's best showing is #27. If you want to make a real argument against one-sided communication, show me an MPI-1-based CCSD(T) code that hits a third of a petaflop. Show me the code. Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Greg Lindahl 24 Apr '09

24 Apr '09

On Fri, Apr 24, 2009 at 08:43:14PM -0400, Vinod tipparaju wrote: > Hmm. There are modules of NWChem that have datasets capable of > scaling from 1s to 100's of thousands of cores. Right. Those don't need a fast interconnect. They probably also would run fine with MPI-1. That's a solved problem; we don't need to think much about it. > You take a module that doesn't scale by its nature, say you prove > that you can scale it with an incompatible model, I was making a comment about how a difficult-to-scale but still interesting module could scale just fine with MPI-1. If Siosi6 isn't an interesting molecule and module then why is PNNL publishing benchmarks with it? http://www.emsl.pnl.gov/docs/nwchem/benchmarks/index.html Also, I don't see what you think is "incompatible." A PathScale employee wrote a modest patch to NWChem to use standard MPI-1 2-sided calls underneath the hood. I forget if the changes were at the GA level or the ARMCI level, but they didn't change the application at all. > and allowing NWChem to use 75% of compute power > (try seeing your utilized flop rate) As you can see, thanks to the better scaling, the utilized flop rate at large scale was better than the machine with real one-sided hardware. The graph is absolute performance, not scaling. The label on the Y axis makes that clear. Of course the cpus aren't exactly the same, but they are reasonably comparable, and that's the only data that was available to compare to. (BTW, that 75% could probably be improved, we never tested with other ratios.) I think this nicely illustrates my point that I don't know of any existing HPC application that really needs one-sided hardware to get great performance.* -- greg (* I'm not counting 1-sided-only apps from the intelligence community, of course.)

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Vinod tipparaju 24 Apr '09

24 Apr '09

Hmm. There are modules of NWChem that have datasets capable of scaling from 1s to 100's of thousands of cores. You take a module that doesn't scale by its nature, say you prove that you can scale it with an incompatible model, and allowing NWChem to use 75% of compute power (try seeing your utilized flop rate) make a statement about it ... and this doesn't have anything to do with anything? The particular hardware did not have good support for one-sided communication. It was a thin card that relied on host a lot and in this case, using the host gave some performance. This in my opinion is a bad way of making a point about comparing two different models. Bills point about implementations hits its sweet spot here. Please excuse my understanding if I am missing your point here. > Date: Fri, 24 Apr 2009 17:18:27 -0700 > From: lindahl(a)pbm.com > To: mpi-forum(a)lists.mpi-forum.org > Subject: Re: [Mpi-forum] MPI One-Sided Communication > > On Fri, Apr 24, 2009 at 05:50:23PM -0400, Vinod Tipparaju wrote: > >> The scaling in the chart discussed below is immaterial because the >> comparision point is flawed. I tried to convey this to some of the >> developers on the phone a couple of years ago but invain. The fact >> this claim of performance went no where beyond the white paper >> should convey something. Did you know dft (siosi6) doesn't scale >> (for various reasons) beyond 1k? > > Pretty much every code and dataset have a maximum scalability point. I > don't see what that has to do with anything. > > The point of the chart is that a pure MPI-1 implementation scaled > great and had great absolute performance on particular hardware/MPI, > even though the developers were sure that specialized hardware & > non-MPI underlying messaging software was needed. > > -- greg > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Greg Lindahl 24 Apr '09

24 Apr '09

On Fri, Apr 24, 2009 at 04:37:46PM -0500, William Gropp wrote: > One other thing that I'd like to remind everyone is that there's a > difference between "MPI (the standard)" and "the performance of a > particular MPI implementation". These are all too often confused, and > there are a lot of suboptimal MPI implementations of the MPI RMA > routines out there. I would certainly agree, and it would be a good thing if someone could point out an especially good implementation. That would show everyone else how well it could be done, and probably lead to improved RMA implementations. It might take a new generation of hardware after that before users begin to care, though. A very slow process. -- greg

1 0