Hi, Jeff Hammond wrote: Active messages are very appealing to me but I'm not aware of a library implementation (F77 and C compatible) that runs on BlueGene/P, Cray XT, Infiniband and Myrinet. Can you point me to one? IBM has since 1997 an Active Messaging library for the SP/Power4,5&6 clusters (SP switch, SP switch2, HPS and Infiniband networked clusters). It is called Low Level API (LAPI for short). It supports F77 (old interfaces), F95 (newer interfaces), C and C++ bindings. This library was initially written to support NWChem/GA/ARMCI on SP cluster for PNNL. http://publib.boulder.ibm.com/epubs/pdf/a2279368.pdf (latest document). GA/ARMCI, GASNet and IBM's Parallel Environment MPI are coded to LAPI on IBM's Power clusters (AIX/Linux). They all use LAPI's Reliable Active Messaging layer and and some of LAPI's one-sided Put/Get/RMW functions. LAPI library exploits RDMA, striping across multiple adapters, failover/recovery for the upper protocols plus other things. On BG/P there is DCMF (Deep Computing Messaging Framework) library that supports active messaging. I believe it is limited to C and C++ bindings. There also, GA/ARMCI, GASNet and MPICH uses DCMF to do the messaging on BG/P. best regards, -- chulho PE & Communication Protocols Architect (STSM) 2455 South Road, P963/7T, Poughkeepsie, NY 12601 Poughkeepsie HPC Development Lab (845) 433-7863 T/L 8 - 293-7863 [email protected] From: Jeff Hammond <[email protected]> To: Main MPI Forum mailing list <[email protected]> Date: 04/24/2009 08:43 AM Subject: Re: [Mpi-forum] MPI One-Sided Communication Sent by: [email protected] Greg,

The only significant public one-sided app that I've run into is NWCHEM, which uses a non-MPI one-sided library. And in that case, PathScale did an implementation on top of 2-sided MPI-1 send/recv, and it scaled better than competing real one-sided hardware. (Check out my old whitepaper for numbers.)

You're comparing ("scaled better than") apples ("an implementation on top of 2-sided MPI-1 send/recv") and oranges ("real one-sided hardware"). Please post the whitepaper (I cannot find it online) and clarify what you mean here. Did Pathscale implement Global Arrays on top of MPI-1 instead of ARMCI and found that NWChem scaled better in this context? NWChem has many different modules, some of which can be scaled fine with a message-passing (MP) model whereas others cannot. It took a decade after NWChem first did it with GA for anyone to demonstrate scalable quantum chemical many-body theories using MPI and then only with a pseudo-compiler and new programming language (ACES3 + SIAL if anyone cares). One-sided provides a much better programming model than MP for complex algorithms that show up in quantum chemistry, which is why GA not MPI is the basis for every existing large-scale code of this kind. If MPI-1 was better than ARMCI for the "meat and potatoes" in NWChem, NWChem would be using it instead.

Now that I'm not an HPC guy anymore, I'll note that the distributed database that I'm doing is completely implemented in terms of active messages.

Active messages are very appealing to me but I'm not aware of a library implementation (F77 and C compatible) that runs on BlueGene/P, Cray XT, Infiniband and Myrinet. Can you point me to one?

They're easier to use corrrectly than real one-sided communications, which are hard because programmers screw up synchronization & double-buffering.

If one-sided is evaluated at the level of Global Arrays rather than ARMCI by itself, those issues disappear. Perhaps that's an unfair comparison but direct use of ARMCI is rare whereas GA is relatively popular. Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/ _______________________________________________ mpi-forum mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum