December 2013 - mpiwg-rma - mailman3.cels.anl.gov

summary of ticket #398 discussion
by Jeff Hammond 10 Dec '13

10 Dec '13

Jim wanted this to go on the Wiki but I decided to inline it on the ticket. Feel free to copy from or link to the ticket on the RMA WG Wiki. Jeff ---------- Forwarded message ---------- From: MPI Forum <mpi-forum(a)lists.mpi-forum.org> Date: Tue, Dec 10, 2013 at 3:56 PM Subject: Re: [MPI Forum] #398: request-based remote completion for RMA To: #398: request-based remote completion for RMA -------------------------------------+------------------------------------- Reporter: | Owner: jhammond | jhammond Type: New | Status: new routine(s) | Milestone: Priority: Not | 2013/03/11 Chicago, USA ready / author rework | Resolution: Version: MPI | Implementation status: <next> | Waiting Keywords: RMA | Author: Rich Graham: 0 Author: Bill Gropp: 0 | Author: Torsten Hoefler: 0 Author: Adam Moody: 0 | Author: Jesper Larsson Traeff: 0 Author: Dick Treumann: 0 | Author: David Solt: 0 Author: George Bosilca: 0 | Author: Rajeev Thakur: 0 Author: Bronis de Supinski: 0 | Author: Alexander Supalov: 0 Author: Jeff Squyres: 0 | Author: Rolf Rabenseifner: 0 | -------------------------------------+------------------------------------- Comment (by jhammond): Summary of working group (WG) discussion: * The justification of fine-grain remote completion was determined to be weak by multiple members of the WG. The two uses were stated to be CAF 2.0 and Charm++. Members of the WG could not come up with a strong performance motivation i.e. could not come up with a hardware scenario where separation of remote completion would measurably improve performance. * Brian did not like multiple requests and proposed instead to have a single request for a new function that would have end-to-end completion semantics, i.e. the completion of the request would correspond to '''both''' local and remote completion - no separation of local and remote completion is available. * Pavan and others believe that nonblocking flush is sufficient to address the major issue at hand, which is that blocking flush causes problems. * Jim and others believe that returning a request from {{{IWIN_FLUSH} and then testing on it is the wrong way to do this and proposes instead to add {{{WIN_TEST}}}, which has different semantics than {{{IWIN_FLUSH}}}. In particular, there is no request to be managed. * Jeff, Brian and possibly Jim thought that there might be some value in adding a SHMEM-like {{{PUT}}} to the MPI standard, which is a Put that blocks on local completion so as to eliminate the need for two function calls to achieve this effect. This feature was determined to be out-of- scope w.r.t. this ticket and in need of its own ticket if the WG wishes to pursue it. -- Ticket URL: <https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398#comment:4> MPI Forum <https://svn.mpi-forum.org/> MPI Forum -- Jeff Hammond jeff.science(a)gmail.com

1 0

MPI_Aint_add
by Jim Dinan 03 Dec '13

03 Dec '13

Hi All, Our ticket #349, which adds MPI_Aint_add, is slated for a formal reading at the upcoming meeting: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/349 I created a new ticket that uses MPI_Aint_add to clean up the linked list example: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/402 This removes horrible casts from an MPI_Aint back to a pointer to do pointer arithmetic. An updated runnable example from the MPICH test suite is attached. Could we please add #402 to the WG agenda? Thanks, ~Jim.

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: making dynamic windows suck less
by Jeff Hammond 02 Dec '13

02 Dec '13

PAMI can do RDMA against both registered memory segments and virtual addresses today, but the performance difference is noticeable on BGQ. Other implementations have different performance characteristics, so it's a solved problem in some sense. I just do not like having to assume the existence of the necessary hardware features that enable virtual-to-physical translation in the network. I'm not too worried about Cray solving these problems in the future, so if you're sure that OFED and Portals4 are now or will soon be able to deal with the issues created by dynamic windows, then I am less concerned about this issue. I think Pavan's proposal is the right one to focus on at this point since it's a simple concept and addresses the common case anyways. Best, Jeff On Mon, Dec 2, 2013 at 9:50 AM, Barrett, Brian W <bwbarre(a)sandia.gov> wrote: > The idea behind dynamic windows was to make them possible to implement today > and push the vendors to support a more rational programming model tomorrow. > Like Jim, I'm not sure things are bleak enough that we should change. > > In OFED, I think it would be very difficult today. But with Mellanox's work > to support dynamic registration cache in hardware, I think things are > trending in the right direction there. > > For Portals 4, if an implementation supports the BIND_INACCESSIBLE feature, > and I think most implementations will, supporting dynamic windows is > straight-forward. > > PAMI and DMAPP are probably more difficult, but a firehose-like setup should > be possible until hardware catches up. > > Brian > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > ________________________________ > From: mpiwg-rma [mpiwg-rma-bounces(a)lists.mpi-forum.org] on behalf of Jeff > Hammond [jeff.science(a)gmail.com] > Sent: Sunday, December 01, 2013 10:16 AM > To: MPI WG Remote Memory Access working group > Cc: MPI WG Remote Memory Access working group > Subject: [EXTERNAL] Re: [mpiwg-rma] making dynamic windows suck less > > How do you do pure (i.e. active-message and rendezvous-free) RDMA with just > a virtual address (MPI_Aint) in (1) PAMI, (2) OFED, (3) DMAPP, (4) Portals4? > > Jeff > > Sent from my iPhone > > On Dec 1, 2013, at 10:40 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Can the MPI_Win_memobj be an MPI_Aint? > > I don't share this pessimism with respect to dynamic windows; I think we > need to prove that these issues exist before we try to fix them. > > ~Jim. > > > On Wed, Nov 27, 2013 at 5:46 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: >> >> Lots of people think - I take position on their correctness - that >> dynamic windows are doomed to substandard performance. I can >> certainly see some potential for this on networks that require memory >> registration. >> >> Can we try to add an opaque object that can encapsulate memory >> registration such that dynamic windows 2.0 would not have so many >> problems? If one were to send-recv not just the virtual address but >> MPI_Win_memobj (to be named properly later), then maybe we could deal >> with the shortcomings of IB, etc. w.r.t. RDMA and memory registration. >> >> This is really just a thought. I have no concrete proposal. Maybe >> people will have ideas in December. >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: making dynamic windows suck less
by Barrett, Brian W 02 Dec '13

02 Dec '13

The idea behind dynamic windows was to make them possible to implement today and push the vendors to support a more rational programming model tomorrow. Like Jim, I'm not sure things are bleak enough that we should change. In OFED, I think it would be very difficult today. But with Mellanox's work to support dynamic registration cache in hardware, I think things are trending in the right direction there. For Portals 4, if an implementation supports the BIND_INACCESSIBLE feature, and I think most implementations will, supporting dynamic windows is straight-forward. PAMI and DMAPP are probably more difficult, but a firehose-like setup should be possible until hardware catches up. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories ________________________________ From: mpiwg-rma [mpiwg-rma-bounces(a)lists.mpi-forum.org] on behalf of Jeff Hammond [jeff.science(a)gmail.com] Sent: Sunday, December 01, 2013 10:16 AM To: MPI WG Remote Memory Access working group Cc: MPI WG Remote Memory Access working group Subject: [EXTERNAL] Re: [mpiwg-rma] making dynamic windows suck less How do you do pure (i.e. active-message and rendezvous-free) RDMA with just a virtual address (MPI_Aint) in (1) PAMI, (2) OFED, (3) DMAPP, (4) Portals4? Jeff Sent from my iPhone On Dec 1, 2013, at 10:40 AM, Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> wrote: Can the MPI_Win_memobj be an MPI_Aint? I don't share this pessimism with respect to dynamic windows; I think we need to prove that these issues exist before we try to fix them. ~Jim. On Wed, Nov 27, 2013 at 5:46 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: Lots of people think - I take position on their correctness - that dynamic windows are doomed to substandard performance. I can certainly see some potential for this on networks that require memory registration. Can we try to add an opaque object that can encapsulate memory registration such that dynamic windows 2.0 would not have so many problems? If one were to send-recv not just the virtual address but MPI_Win_memobj (to be named properly later), then maybe we could deal with the shortcomings of IB, etc. w.r.t. RDMA and memory registration. This is really just a thought. I have no concrete proposal. Maybe people will have ideas in December. Jeff -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] making dynamic windows suck less
by Pavan Balaji 01 Dec '13

01 Dec '13

What you are suggesting is a valid improvement. With the current MPI-3 standard dynamic windows are hard to implement efficiently on networks that require registration. However, in the list of items to discuss that I had sent, I had a different proposal for the same problem. The idea is to use an info hint where PUT/GETs and ATTACH/DETACHs are not interleaved. This means that the MPI implementation will know that once I do a PUT to a location, it won’t be DETACHED and ATTACHED back again, thus potentially changing memory registration information. With this info hint, the first PUT operation would be an active message that would fetch the registration information and the later PUT operations can use it. — Pavan On Dec 1, 2013, at 11:16 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > How do you do pure (i.e. active-message and rendezvous-free) RDMA with just a virtual address (MPI_Aint) in (1) PAMI, (2) OFED, (3) DMAPP, (4) Portals4? > > Jeff > > Sent from my iPhone > > On Dec 1, 2013, at 10:40 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> Can the MPI_Win_memobj be an MPI_Aint? >> >> I don't share this pessimism with respect to dynamic windows; I think we need to prove that these issues exist before we try to fix them. >> >> ~Jim. >> >> >> On Wed, Nov 27, 2013 at 5:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> Lots of people think - I take position on their correctness - that >> dynamic windows are doomed to substandard performance. I can >> certainly see some potential for this on networks that require memory >> registration. >> >> Can we try to add an opaque object that can encapsulate memory >> registration such that dynamic windows 2.0 would not have so many >> problems? If one were to send-recv not just the virtual address but >> MPI_Win_memobj (to be named properly later), then maybe we could deal >> with the shortcomings of IB, etc. w.r.t. RDMA and memory registration. >> >> This is really just a thought. I have no concrete proposal. Maybe >> people will have ideas in December. >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] making dynamic windows suck less
by Jeff Hammond 01 Dec '13

01 Dec '13

How do you do pure (i.e. active-message and rendezvous-free) RDMA with just a virtual address (MPI_Aint) in (1) PAMI, (2) OFED, (3) DMAPP, (4) Portals4? Jeff Sent from my iPhone > On Dec 1, 2013, at 10:40 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Can the MPI_Win_memobj be an MPI_Aint? > > I don't share this pessimism with respect to dynamic windows; I think we need to prove that these issues exist before we try to fix them. > > ~Jim. > > >> On Wed, Nov 27, 2013 at 5:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> Lots of people think - I take position on their correctness - that >> dynamic windows are doomed to substandard performance. I can >> certainly see some potential for this on networks that require memory >> registration. >> >> Can we try to add an opaque object that can encapsulate memory >> registration such that dynamic windows 2.0 would not have so many >> problems? If one were to send-recv not just the virtual address but >> MPI_Win_memobj (to be named properly later), then maybe we could deal >> with the shortcomings of IB, etc. w.r.t. RDMA and memory registration. >> >> This is really just a thought. I have no concrete proposal. Maybe >> people will have ideas in December. >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] making dynamic windows suck less
by Jim Dinan 01 Dec '13

01 Dec '13

Can the MPI_Win_memobj be an MPI_Aint? I don't share this pessimism with respect to dynamic windows; I think we need to prove that these issues exist before we try to fix them. ~Jim. On Wed, Nov 27, 2013 at 5:46 PM, Jeff Hammond <jeff.science(a)gmail.com>wrote: > Lots of people think - I take position on their correctness - that > dynamic windows are doomed to substandard performance. I can > certainly see some potential for this on networks that require memory > registration. > > Can we try to add an opaque object that can encapsulate memory > registration such that dynamic windows 2.0 would not have so many > problems? If one were to send-recv not just the virtual address but > MPI_Win_memobj (to be named properly later), then maybe we could deal > with the shortcomings of IB, etc. w.r.t. RDMA and memory registration. > > This is really just a thought. I have no concrete proposal. Maybe > people will have ideas in December. > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0