April 2009 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] MPI One-Sided Communication
by Greg Lindahl 24 Apr '09

24 Apr '09

On Fri, Apr 24, 2009 at 05:50:23PM -0400, Vinod Tipparaju wrote: > The scaling in the chart discussed below is immaterial because the > comparision point is flawed. I tried to convey this to some of the > developers on the phone a couple of years ago but invain. The fact > this claim of performance went no where beyond the white paper > should convey something. Did you know dft (siosi6) doesn't scale > (for various reasons) beyond 1k? Pretty much every code and dataset have a maximum scalability point. I don't see what that has to do with anything. The point of the chart is that a pure MPI-1 implementation scaled great and had great absolute performance on particular hardware/MPI, even though the developers were sure that specialized hardware & non-MPI underlying messaging software was needed. -- greg

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Vinod Tipparaju 24 Apr '09

24 Apr '09

Greg, The scaling in the chart discussed below is immaterial because the comparision point is flawed. I tried to convey this to some of the developers on the phone a couple of years ago but invain. The fact this claim of performance went no where beyond the white paper should convey something. Did you know dft (siosi6) doesn't scale (for various reasons) beyond 1k? -- typing from phone..small keyboard big thumbs. ________________________ On 2009-04-24 13:33:56 -0400 Greg Lindahl <lindahl(a)pbm.com> wrote: > > On Fri, Apr 24, 2009 at 07:38:39AM -0500, Jeff Hammond wrote: > >> You're comparing ("scaled better than") apples ("an implementation on >> top of 2-sided MPI-1 send/recv") and oranges ("real one-sided >> hardware"). Please post the whitepaper (I cannot find it online) and >> clarify what you mean here. > > QLogic seems to have hidden it here: > > http://www.qlogic.com/SiteCollectionDocuments/Education_and_Resource/whitep… > > I used the Siosi6 benchmark, since I needed something that had public > benchmark numbers available. > > Our implementation used 1/4 of the cores as "memory servers" which > held the global memory and served as the counterparty to one-sided > operations. Despite wasting 25% of the cores, it scaled great. > > Now this result won't apply to all one-sided programs; GA's model and > usage is somewhat different from most other one-sided models. > >

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by William Gropp 24 Apr '09

24 Apr '09

Thanks, Greg. One other thing that I'd like to remind everyone is that there's a difference between "MPI (the standard)" and "the performance of a particular MPI implementation". These are all too often confused, and there are a lot of suboptimal MPI implementations of the MPI RMA routines out there. In particular, in most cases, if the asymptotic bandwidth of MPI (whether it is send/recv or put) isn't as high as some other approach, that almost always means that the MPI implementation is doing something inefficient, not that "MPI" is inefficient. There was one vendor implementation, for example, that made unnecessary buffer copies - that's not the fault of MPI, just an unoptimized implementation. Bill On Apr 24, 2009, at 12:01 PM, Greg Lindahl wrote: > On Fri, Apr 24, 2009 at 07:38:39AM -0500, Jeff Hammond wrote: > >> You're comparing ("scaled better than") apples ("an implementation on >> top of 2-sided MPI-1 send/recv") and oranges ("real one-sided >> hardware"). Please post the whitepaper (I cannot find it online) and >> clarify what you mean here. > > QLogic seems to have hidden it here: > > http://www.qlogic.com/SiteCollectionDocuments/Education_and_Resource/whitep… > > I used the Siosi6 benchmark, since I needed something that had public > benchmark numbers available. > > Our implementation used 1/4 of the cores as "memory servers" which > held the global memory and served as the counterparty to one-sided > operations. Despite wasting 25% of the cores, it scaled great. > > Now this result won't apply to all one-sided programs; GA's model and > usage is somewhat different from most other one-sided models. > > -- greg > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Chulho Kim 24 Apr '09

24 Apr '09

Hi, Jeff Hammond wrote: Active messages are very appealing to me but I'm not aware of a library implementation (F77 and C compatible) that runs on BlueGene/P, Cray XT, Infiniband and Myrinet. Can you point me to one? IBM has since 1997 an Active Messaging library for the SP/Power4,5&6 clusters (SP switch, SP switch2, HPS and Infiniband networked clusters). It is called Low Level API (LAPI for short). It supports F77 (old interfaces), F95 (newer interfaces), C and C++ bindings. This library was initially written to support NWChem/GA/ARMCI on SP cluster for PNNL. http://publib.boulder.ibm.com/epubs/pdf/a2279368.pdf (latest document). GA/ARMCI, GASNet and IBM's Parallel Environment MPI are coded to LAPI on IBM's Power clusters (AIX/Linux). They all use LAPI's Reliable Active Messaging layer and and some of LAPI's one-sided Put/Get/RMW functions. LAPI library exploits RDMA, striping across multiple adapters, failover/recovery for the upper protocols plus other things. On BG/P there is DCMF (Deep Computing Messaging Framework) library that supports active messaging. I believe it is limited to C and C++ bindings. There also, GA/ARMCI, GASNet and MPICH uses DCMF to do the messaging on BG/P. best regards, -- chulho PE & Communication Protocols Architect (STSM) 2455 South Road, P963/7T, Poughkeepsie, NY 12601 Poughkeepsie HPC Development Lab (845) 433-7863 T/L 8 - 293-7863 chulho(a)us.ibm.com From: Jeff Hammond <jeff.science(a)gmail.com> To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> Date: 04/24/2009 08:43 AM Subject: Re: [Mpi-forum] MPI One-Sided Communication Sent by: mpi-forum-bounces(a)lists.mpi-forum.org Greg, > The only significant public one-sided app that I've run into is > NWCHEM, which uses a non-MPI one-sided library. And in that case, > PathScale did an implementation on top of 2-sided MPI-1 send/recv, and > it scaled better than competing real one-sided hardware. (Check out my > old whitepaper for numbers.) You're comparing ("scaled better than") apples ("an implementation on top of 2-sided MPI-1 send/recv") and oranges ("real one-sided hardware"). Please post the whitepaper (I cannot find it online) and clarify what you mean here. Did Pathscale implement Global Arrays on top of MPI-1 instead of ARMCI and found that NWChem scaled better in this context? NWChem has many different modules, some of which can be scaled fine with a message-passing (MP) model whereas others cannot. It took a decade after NWChem first did it with GA for anyone to demonstrate scalable quantum chemical many-body theories using MPI and then only with a pseudo-compiler and new programming language (ACES3 + SIAL if anyone cares). One-sided provides a much better programming model than MP for complex algorithms that show up in quantum chemistry, which is why GA not MPI is the basis for every existing large-scale code of this kind. If MPI-1 was better than ARMCI for the "meat and potatoes" in NWChem, NWChem would be using it instead. > Now that I'm not an HPC guy anymore, I'll note that the distributed > database that I'm doing is completely implemented in terms of active messages. Active messages are very appealing to me but I'm not aware of a library implementation (F77 and C compatible) that runs on BlueGene/P, Cray XT, Infiniband and Myrinet. Can you point me to one? > They're easier to use corrrectly than real one-sided > communications, which are hard because programmers screw up > synchronization & double-buffering. If one-sided is evaluated at the level of Global Arrays rather than ARMCI by itself, those issues disappear. Perhaps that's an unfair comparison but direct use of ARMCI is rare whereas GA is relatively popular. Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/ _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Greg Lindahl 24 Apr '09

24 Apr '09

On Fri, Apr 24, 2009 at 07:38:39AM -0500, Jeff Hammond wrote: > You're comparing ("scaled better than") apples ("an implementation on > top of 2-sided MPI-1 send/recv") and oranges ("real one-sided > hardware"). Please post the whitepaper (I cannot find it online) and > clarify what you mean here. QLogic seems to have hidden it here: http://www.qlogic.com/SiteCollectionDocuments/Education_and_Resource/whitep… I used the Siosi6 benchmark, since I needed something that had public benchmark numbers available. Our implementation used 1/4 of the cores as "memory servers" which held the global memory and served as the counterparty to one-sided operations. Despite wasting 25% of the cores, it scaled great. Now this result won't apply to all one-sided programs; GA's model and usage is somewhat different from most other one-sided models. -- greg

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Scott Atchley 24 Apr '09

24 Apr '09

On Apr 24, 2009, at 8:38 AM, Jeff Hammond wrote: > Active messages are very appealing to me but I'm not aware of a > library implementation (F77 and C compatible) that runs on BlueGene/P, > Cray XT, Infiniband and Myrinet. Can you point me to one? MX does support a limited active message model. The application can register an unexpected handler that is called whenever MX receives a message for which the application has not pre-posted a receive. The handler has access to the match tag, the sender info, the length of the message, and the actual data if the length is 1 KB or less (small but useable). It is useful in filesystems where there are a lot of unexpected messages. It avoids having to pre-post a lot of generic receives. I believe Open-MX supports it as well. Scott

1 0

Re: [Mpi-forum] MPI-3 One-Sided Communications
by Richard Graham 24 Apr '09

24 Apr '09

Please move this discussion to the RMA mailing list ( mpi3-rma(a)lists.mpi-forum.org) so that people looking through the RMA e-mail archives can see this. Thanks, Rich On 4/24/09 9:20 AM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: > Tony, > > On Thu, Apr 23, 2009 at 8:17 PM, Anthony Skjellum <tony(a)verarisoft.com> wrote: >> Hi, I am not familiar with you project. ï¿½Tell us more. > > Vinod should be the one to tell you about ARMCI in detail since he > wrote a good portion of it. I just use it. There are also a number > of papers available (see below). > > http://www.emsl.pnl.gov/docs/parsoft/armci/armci/armci.pdf > http://www.springerlink.com/content/p581340602373484/ > http://hpc.sagepub.com/cgi/content/abstract/20/2/233 > >> In order to achieve lowest latency (or overhead, depending on your >> optimization, or a blend thereof), a protocol designer might not want to pay >> for fixed costs that once amortized over long transfers yields more long >> transfer performance. ï¿½Classic fixed vs variable cost situation. The classic >> trades in two-sided are zero copy for long and two copy for short. If you >> have evidence to the contrary, great... Lots of people reported in the past >> the effect of a tradeoff as I described coming from the complex semantics of >> the mpi one sided api. > > I think the trade-off comes from the implementation not the one-sided > model itself. I agree achieving low-latency with a clunky > implementation requires a compromise elsewhere. > >> One also has to ask if we can get even lower latency with less complex >> protocol too imho. > > ARMCI is quite minimal and sits very close the hardware layer. GASNet > is also very lightweight, perhaps as much as can be achieved without > compromising portability. > > Einstein's "Make everything as simple as possible, but not simpler" is > the right guiding principle for HPC, in contrast to the standard model > of "make everything as general as possible." > >> Do you get both lower latency than two sided short msgs and higher bandwidth >> than two sided? > > Someone else should have data on this. If it's out-dated, I'll do > fresh timings on current hardware this summer. > >> What is the baseline of performance for two sided? Is it optimized? :-) > > The ARMCI performance page notes that the vendor MPI implementation > was used in many of the cases and it can be assumed in others. > > Jeff

1 0

Re: [Mpi-forum] MPI-3 One-Sided Communications
by Jeff Hammond 24 Apr '09

24 Apr '09

Tony, On Thu, Apr 23, 2009 at 8:17 PM, Anthony Skjellum <tony(a)verarisoft.com> wrote: > Hi, I am not familiar with you project. ï¿½Tell us more. Vinod should be the one to tell you about ARMCI in detail since he wrote a good portion of it. I just use it. There are also a number of papers available (see below). http://www.emsl.pnl.gov/docs/parsoft/armci/armci/armci.pdf http://www.springerlink.com/content/p581340602373484/ http://hpc.sagepub.com/cgi/content/abstract/20/2/233 > In order to achieve lowest latency (or overhead, depending on your optimization, or a blend thereof), a protocol designer might not want to pay for fixed costs that once amortized over long transfers yields more long transfer performance. ï¿½Classic fixed vs variable cost situation. The classic trades in two-sided are zero copy for long and two copy for short. If you have evidence to the contrary, great... Lots of people reported in the past the effect of a tradeoff as I described coming from the complex semantics of the mpi one sided api. I think the trade-off comes from the implementation not the one-sided model itself. I agree achieving low-latency with a clunky implementation requires a compromise elsewhere. > One also has to ask if we can get even lower latency with less complex protocol too imho. ARMCI is quite minimal and sits very close the hardware layer. GASNet is also very lightweight, perhaps as much as can be achieved without compromising portability. Einstein's "Make everything as simple as possible, but not simpler" is the right guiding principle for HPC, in contrast to the standard model of "make everything as general as possible." > Do you get both lower latency than two sided short msgs and higher bandwidth than two sided? Someone else should have data on this. If it's out-dated, I'll do fresh timings on current hardware this summer. > What is the baseline of performance for two sided? Is it optimized? :-) The ARMCI performance page notes that the vendor MPI implementation was used in many of the cases and it can be assumed in others. Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi-forum] MPI-3 One-Sided Communications
by Jeff Hammond 24 Apr '09

24 Apr '09

On Thu, Apr 23, 2009 at 10:39 PM, Anthony Skjellum <tony(a)verarisoft.com> wrote: > Can you point me at the proposed solutions? :-) Is what you're looking for? https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi-forum] MPI One-Sided Communication
by Jeff Hammond 24 Apr '09

24 Apr '09

Greg, > The only significant public one-sided app that I've run into is > NWCHEM, which uses a non-MPI one-sided library. And in that case, > PathScale did an implementation on top of 2-sided MPI-1 send/recv, and > it scaled better than competing real one-sided hardware. (Check out my > old whitepaper for numbers.) You're comparing ("scaled better than") apples ("an implementation on top of 2-sided MPI-1 send/recv") and oranges ("real one-sided hardware"). Please post the whitepaper (I cannot find it online) and clarify what you mean here. Did Pathscale implement Global Arrays on top of MPI-1 instead of ARMCI and found that NWChem scaled better in this context? NWChem has many different modules, some of which can be scaled fine with a message-passing (MP) model whereas others cannot. It took a decade after NWChem first did it with GA for anyone to demonstrate scalable quantum chemical many-body theories using MPI and then only with a pseudo-compiler and new programming language (ACES3 + SIAL if anyone cares). One-sided provides a much better programming model than MP for complex algorithms that show up in quantum chemistry, which is why GA not MPI is the basis for every existing large-scale code of this kind. If MPI-1 was better than ARMCI for the "meat and potatoes" in NWChem, NWChem would be using it instead. > Now that I'm not an HPC guy anymore, I'll note that the distributed > database that I'm doing is completely implemented in terms of active messages. Active messages are very appealing to me but I'm not aware of a library implementation (F77 and C compatible) that runs on BlueGene/P, Cray XT, Infiniband and Myrinet. Can you point me to one? > They're easier to use corrrectly than real one-sided > communications, which are hard because programmers screw up > synchronization & double-buffering. If one-sided is evaluated at the level of Global Arrays rather than ARMCI by itself, those issues disappear. Perhaps that's an unfair comparison but direct use of ARMCI is rare whereas GA is relatively popular. Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0