February 2010 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] FW: MPI 3 RMA Examples needed
by Jeff Hammond 01 Feb '10

01 Feb '10

To add to Robert's point about non-scalable metadata (remote addresses etc.), it will prohibitively expensive to use Global Arrays (GA) on our next machine without remote method invocation (RMI). According to the Internet [1], that machine will have ~0.75M cores. In the most minimal scenario (8 bytes per remote address), that's 6 MB to index the whole machine, if one is going to run one process per core. If one estimates using IB metadata with static all-connectivity (stupid, of course), 44 kB * 0.75M = 33 GB, which is impossible. While globally-scoped arrays are probably a poor idea to start with, some applications will try to use them. If one stores only one copy of metadata per node, O(50K)*sizeof(metadata) = 50 MB per array if one is fairly stingy. A feature I hope to use to ameliorate this problem is to run RMA through a metadata server. One can do this right now with put/get, but the latency will be worse than with RMI. At present, one would have to do two remote gets, a blocking one to the metadata server, and then the acquisition of the actual data. With RMI, the metadata operation need not block on the initiator and requires two active-messages and the put from the remote location to the initiator. Assuming we cross the 2M node barrier in ~2015, the only practical way to have globally-scoped data will be to use hierarchical metadata servers, where the lack of RMI will lead to even less efficient remote metadata acquisition. It would not surprise me if the get(+get)^tiers method for RMA will be an order-of-magnitude slower than the (RMI)^tiers+put method, given the likelihood of network contention across a hierarchical network at this scale. While we can debate the sanity of globally-scoped data at exascale, there will always be science which is not amenable to simple domain decomposition and thus requires some type of global view. Whether the data structures are simple ala GA or complex ala Madness, RMI is the only sensible ways to implement these involve RMI. Jeff On Sun, Jan 31, 2010 at 2:00 PM, Robert Harrison <...> wrote: > Bill, > > A compelling and near universal example is maintaining > and updating remote data structures. ï¿½Only arrays or simple static > data structures can be treated efficiently with RMA, and even this assumes > a non-scalable replication of addresses across all nodes. ï¿½More complex > structures (trees, hash tables, ...) and scalable solutions even for arrays > require remote method invocation to access state and to perform some > computation. > > This example also motivates considerations of optional message ordering > (for sequential consistency), optional atomicity (for correctness with multiple > updaters), etc. ï¿½Even within a single application there is utility for different > variants. > > Some remote operations will return a value and how to do so efficiently > without running into network/NIC flow problems seems to be an issue. > > A concrete example is the sparse tree (in dimensions 1,2,..,6) in madness. > These are stored in distributed hash tables. ï¿½Minimally, we must be able to insert, > erase, replace, and read entries. ï¿½More generally the entries are objects and we > wish to invoke their methods so that applications can be composed in the spirit > of Charm++ ... messaging between objects addressed by their name in a > namespace (container). > > Best wishes > > ï¿½ Robert -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] MPI 3 RMA Examples needed
by Stephen Poole 31 Jan '10

31 Jan '10

That would be the goal, of course. My overall goal is to allow PGAS/APGAS to ride on the good work being done by the community, which in turn allows for the greatest impact (positive) from a vendor perspective. In this case a "vendor" is anyone delivering MPI/APGAS/PGAS implementations. I will read what is posted, and thank you for the info. Best Steve... Jeff Hammond wrote: > Steve, > > See https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage . > > I haven't seen anything in the proposed additions to standard that > interferes with PGAS. However, a strong passive progress would burn a > thread on many architectures, so implementations would have to be > mindful of what a PGAS run-time was doing. Of course, vendors > implementing both MPI and UPC, for example, could "do the right thing" > and ensure interoperability and non-redundant progress threads. > > Best, > > Jeff > > On Sun, Jan 31, 2010 at 5:44 PM, Stephen Poole <spoole(a)ornl.gov> wrote: >> Keith, >> >> Where can I read all of the proposal ? I want to make sure this does not >> have a negative impact on what needs to be done for PGAS/APGAS. >> >> Best >> Steve... >> >> Underwood, Keith D wrote: >>> This is an excellent framing of one of the things that will help the RMA >>> group move forward; however, I do believe we need to get performance into >>> the process. What I do not know is how we separate performance of the >>> interface from performance of an implementation. Just because you are able >>> to implement something without unbearably awkward code doesn't mean that >>> code will perform. How do we account for what is a semantic problem (it >>> isn't POSSIBLE to implement this piece of code and make it go fast) and what >>> is an implementation problem (nobody has bothered to make this piece of code >>> fast)? That input is just as important as what is syntactically awkward. >>> >>> Keith >>> >>>> -----Original Message----- >>>> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- >>>> bounces(a)lists.mpi-forum.org] On Behalf Of William Gropp >>>> Sent: Sunday, January 31, 2010 7:17 AM >>>> To: MPI 3.0 Remote Memory Access working group >>>> Subject: [Mpi3-rma] MPI 3 RMA Examples needed >>>> >>>> Dear MPI RMA Group, >>>> >>>> We have several partial MPI RMA proposals. To move forward, we need >>>> to have a better understanding of the real needs by users, and we will >>>> probably need to make some tough decisions about what we will support >>>> and what we won't (as Marc has noted, some fairly obvious shared >>>> memory operations are very tough in OpenMP, so its clear that being >>>> universal isn't requred). >>>> >>>> What we'd like by this *Friday* are some specific examples of >>>> operations that are hard to achieve in MPI RMA *and* that have a clear >>>> application need. What we *don't* want is simply "we should have a >>>> better Put", "we need active messages", or "the implementations I've >>>> used are >>>> too slow". What we do want is something like the following: >>>> >>>> We've implemented a halo exchange with MPI-RMA, and the construction >>>> of the Memory windows is awkward and limiting, particularly if the >>>> domains are created dynamically, making it hard to create the memory >>>> windows collectively. We need either a method that lets us export a >>>> local window or a way to allow all processes to refer to one single >>>> window (something like the MPI_WIN_WORLD proposal). Example code can >>>> be found at <url here> (or post on wiki). >>>> >>>> or >>>> >>>> We need a fetch and increment (or something similar) to implement a >>>> remote lock that will allow us to make a complex series of remote >>>> updates (and accesses) atomically that are needed for <specific >>>> application description here>. As shown in Using MPI-2, while a fetch >>>> and increment is possible in MPI-RMA, it is extremely complex and >>>> awkward. >>>> >>>> We'll take these examples and compare them to the current proposals >>>> and the original MPI RMA in order to evaluate where we are. >>>> >>>> Again, please send us your concrete requirements by Friday, Feb 5th. >>>> Thanks! >>>> >>>> Bill and Rajeev >>>> >>>> William Gropp >>>> Deputy Director for Research >>>> Institute for Advanced Computing Applications and Technologies >>>> Paul and Cynthia Saylor Professor of Computer Science >>>> University of Illinois Urbana-Champaign >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpi3-rma mailing list >>>> mpi3-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> -- >> >> ======================> >> >> Steve Poole >> Computer Science and Mathematics Division >> Chief Scientist / Director of Special Programs >> National Center for Computational Sciences Division (OLCF) >> Chief Architect >> Oak Ridge National Laboratory >> 865.574.9008 >> "Wisdom is not a product of schooling, but of the lifelong attempt to >> acquire it" Albert Einstein >> >> =====================> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> > > > -- ======================> Steve Poole Computer Science and Mathematics Division Chief Scientist / Director of Special Programs National Center for Computational Sciences Division (OLCF) Chief Architect Oak Ridge National Laboratory 865.574.9008 "Wisdom is not a product of schooling, but of the lifelong attempt to acquire it" Albert Einstein =====================>

1 0

Re: [Mpi3-rma] MPI 3 RMA Examples needed
by Jeff Hammond 31 Jan '10

31 Jan '10

Keith, Pavan and I are trying to separate implementation versus semantic shortcomings by comparing ARMCI and MVAPICH over IB, and I will try to also include OFED in an apples-to-apples-to-apples comparison of the simplest operations. Both ARMCI and MVAPICH are heavily tuned for IB and so I expect the implementation quality to be fairly similar; hence, performance differences are probably due to semantics. We are also working going to see how close we can get to ARMCI with MPI-2 RMA in an attempt to elucidate the semantic limitations of the MPI standard. Obviously, ARMCI is not the end-all-be-all in one-sided APIs, but I cannot get GASNet to function and OpenSHMEM is vaporware (i.e. I cannot download it nor is it available on any machine I have access to) so I choose to focus exclusively on ARMCI. Jeff On Sun, Jan 31, 2010 at 5:47 PM, William Gropp <wgropp(a)illinois.edu> wrote: > I agree that performance is an important issue, and we will need to discuss > it. ï¿½However, all too many of the current implementations are unnecessarily > poor in performance, so to first order, I believe we should concentrate on > what our users need (and believe can be implemented efficiently on some > platform) rather than the experience people have had with slow and > unoptimized implementations. > > Bill > > On Jan 31, 2010, at 5:34 PM, Underwood, Keith D wrote: > >> This is an excellent framing of one of the things that will help the RMA >> group move forward; however, I do believe we need to get performance into >> the process. ï¿½What I do not know is how we separate performance of the >> interface from performance of an implementation. ï¿½Just because you are able >> to implement something without unbearably awkward code doesn't mean that >> code will perform. ï¿½How do we account for what is a semantic problem (it >> isn't POSSIBLE to implement this piece of code and make it go fast) and what >> is an implementation problem (nobody has bothered to make this piece of code >> fast)? ï¿½That input is just as important as what is syntactically awkward. >> >> Keith >> >>> -----Original Message----- >>> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- >>> bounces(a)lists.mpi-forum.org] On Behalf Of William Gropp >>> Sent: Sunday, January 31, 2010 7:17 AM >>> To: MPI 3.0 Remote Memory Access working group >>> Subject: [Mpi3-rma] MPI 3 RMA Examples needed >>> >>> Dear MPI RMA Group, >>> >>> We have several partial MPI RMA proposals. ï¿½To move forward, we need >>> to have a better understanding of the real needs by users, and we will >>> probably need to make some tough decisions about what we will support >>> and what we won't (as Marc has noted, some fairly obvious shared >>> memory operations are very tough in OpenMP, so its clear that being >>> universal isn't requred). >>> >>> What we'd like by this *Friday* are some specific examples of >>> operations that are hard to achieve in MPI RMA *and* that have a clear >>> application need. ï¿½What we *don't* want is simply "we should have a >>> better ï¿½Put", "we need active messages", or "the implementations I've >>> used are >>> too slow". ï¿½What we do want is something like the following: >>> >>> We've implemented a halo exchange with MPI-RMA, and the construction >>> of the Memory windows is awkward and limiting, particularly if the >>> domains are created dynamically, making it hard to create the memory >>> windows collectively. ï¿½We need either a method that lets us export a >>> local window or a way to allow all processes to refer to one single >>> window (something like the MPI_WIN_WORLD proposal). ï¿½Example code can >>> be found at <url here> ï¿½(or post on wiki). >>> >>> or >>> >>> We need a fetch and increment (or something similar) to implement a >>> remote lock that will allow us to make a complex series of remote >>> updates (and accesses) atomically that are needed for <specific >>> application description here>. ï¿½As shown in Using MPI-2, while a fetch >>> and increment is possible in MPI-RMA, it is extremely complex and >>> awkward. >>> >>> We'll take these examples and compare them to the current proposals >>> and the original MPI RMA in order to evaluate where we are. >>> >>> Again, please send us your concrete requirements by Friday, Feb 5th. >>> Thanks! >>> >>> Bill and Rajeev >>> >>> William Gropp >>> Deputy Director for Research >>> Institute for Advanced Computing Applications and Technologies >>> Paul and Cynthia Saylor Professor of Computer Science >>> University of Illinois Urbana-Champaign >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > William Gropp > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] MPI 3 RMA Examples needed
by Jeff Hammond 31 Jan '10

31 Jan '10

Steve, See https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage . I haven't seen anything in the proposed additions to standard that interferes with PGAS. However, a strong passive progress would burn a thread on many architectures, so implementations would have to be mindful of what a PGAS run-time was doing. Of course, vendors implementing both MPI and UPC, for example, could "do the right thing" and ensure interoperability and non-redundant progress threads. Best, Jeff On Sun, Jan 31, 2010 at 5:44 PM, Stephen Poole <spoole(a)ornl.gov> wrote: > Keith, > > ï¿½Where can I read all of the proposal ? I want to make sure this does not > have a negative impact on what needs to be done for PGAS/APGAS. > > Best > Steve... > > Underwood, Keith D wrote: >> >> This is an excellent framing of one of the things that will help the RMA >> group move forward; however, I do believe we need to get performance into >> the process. ï¿½What I do not know is how we separate performance of the >> interface from performance of an implementation. ï¿½Just because you are able >> to implement something without unbearably awkward code doesn't mean that >> code will perform. ï¿½How do we account for what is a semantic problem (it >> isn't POSSIBLE to implement this piece of code and make it go fast) and what >> is an implementation problem (nobody has bothered to make this piece of code >> fast)? ï¿½That input is just as important as what is syntactically awkward. >> >> Keith >> >>> -----Original Message----- >>> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma- >>> bounces(a)lists.mpi-forum.org] On Behalf Of William Gropp >>> Sent: Sunday, January 31, 2010 7:17 AM >>> To: MPI 3.0 Remote Memory Access working group >>> Subject: [Mpi3-rma] MPI 3 RMA Examples needed >>> >>> Dear MPI RMA Group, >>> >>> We have several partial MPI RMA proposals. ï¿½To move forward, we need >>> to have a better understanding of the real needs by users, and we will >>> probably need to make some tough decisions about what we will support >>> and what we won't (as Marc has noted, some fairly obvious shared >>> memory operations are very tough in OpenMP, so its clear that being >>> universal isn't requred). >>> >>> What we'd like by this *Friday* are some specific examples of >>> operations that are hard to achieve in MPI RMA *and* that have a clear >>> application need. ï¿½What we *don't* want is simply "we should have a >>> better ï¿½Put", "we need active messages", or "the implementations I've >>> used are >>> too slow". ï¿½What we do want is something like the following: >>> >>> We've implemented a halo exchange with MPI-RMA, and the construction >>> of the Memory windows is awkward and limiting, particularly if the >>> domains are created dynamically, making it hard to create the memory >>> windows collectively. ï¿½We need either a method that lets us export a >>> local window or a way to allow all processes to refer to one single >>> window (something like the MPI_WIN_WORLD proposal). ï¿½Example code can >>> be found at <url here> ï¿½(or post on wiki). >>> >>> or >>> >>> We need a fetch and increment (or something similar) to implement a >>> remote lock that will allow us to make a complex series of remote >>> updates (and accesses) atomically that are needed for <specific >>> application description here>. ï¿½As shown in Using MPI-2, while a fetch >>> and increment is possible in MPI-RMA, it is extremely complex and >>> awkward. >>> >>> We'll take these examples and compare them to the current proposals >>> and the original MPI RMA in order to evaluate where we are. >>> >>> Again, please send us your concrete requirements by Friday, Feb 5th. >>> Thanks! >>> >>> Bill and Rajeev >>> >>> William Gropp >>> Deputy Director for Research >>> Institute for Advanced Computing Applications and Technologies >>> Paul and Cynthia Saylor Professor of Computer Science >>> University of Illinois Urbana-Champaign >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > -- > > ======================> > > Steve Poole > Computer Science and Mathematics Division > Chief Scientist / Director of Special Programs > National Center for Computational Sciences Division (OLCF) > Chief Architect > Oak Ridge National Laboratory > 865.574.9008 > "Wisdom is not a product of schooling, but of the lifelong attempt to > acquire it" Albert Einstein > > =====================> > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0