August 2013 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Jeff Hammond 26 Aug '13

26 Aug '13

When does this actually improve performance? If memory is a limiting factor, it merely eliminates a function call. If memory is limiting, I don't see how doing extra ping-pong to keep from hitting OOM errors is noticeable. JeffS and Dave: What's the use case? So far, I fail to see how this is more than icing. Jeff On Mon, Aug 26, 2013 at 12:55 PM, Sur, Sayantan <sayantan.sur(a)intel.com>wrote: > This is a good feature to have in general that improves performance, > although the devil is in the details. > > What is the app trying to do when it finds itself in the quandary of > having to accept any sized message (probably from any source as well?). It > appears to me that the kind of app that is looking for this feature, or > could benefit from this feature is really looking for a many-to-one (or > one-to-one, many-to-many) reliably connected stream. It is likely that the > app that benefits significantly from this feature will be repeatedly using > this call to fetch differing amounts of data each time. > > A few meetings ago, Bill mentioned that it would be good to think of the > standard as a whole and not get bogged down in narrow use-cases. He > proposed that a streaming model be added to MPI P2P communications. Can > this use-case be included in that model? > > Thanks, > Sayantan > > > > -----Original Message----- > > From: mpi-forum [mailto:[email protected]] On > > Behalf Of Jeff Squyres (jsquyres) > > Sent: Monday, August 26, 2013 8:11 AM > > To: MPI Forum list > > Subject: [Mpi-forum] MPI "Allocate receive" proposal > > > > Dave Goodell and I have a proposal that we've socialized a bit around > with > > other Forum members, and would now like larger Forum feedback. I'll be > > presenting the attached slides on the concept of an "allocate receive" in > > Madrid (3:30-4pm on Thursday). > > > > There's no text or ticket yet; this is an idea that we want to get > feedback on > > before working up a full proposal. > > > > -- > > Jeff Squyres > > jsquyres(a)cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Sur, Sayantan 26 Aug '13

26 Aug '13

This is a good feature to have in general that improves performance, although the devil is in the details. What is the app trying to do when it finds itself in the quandary of having to accept any sized message (probably from any source as well?). It appears to me that the kind of app that is looking for this feature, or could benefit from this feature is really looking for a many-to-one (or one-to-one, many-to-many) reliably connected stream. It is likely that the app that benefits significantly from this feature will be repeatedly using this call to fetch differing amounts of data each time. A few meetings ago, Bill mentioned that it would be good to think of the standard as a whole and not get bogged down in narrow use-cases. He proposed that a streaming model be added to MPI P2P communications. Can this use-case be included in that model? Thanks, Sayantan > -----Original Message----- > From: mpi-forum [mailto:[email protected]] On > Behalf Of Jeff Squyres (jsquyres) > Sent: Monday, August 26, 2013 8:11 AM > To: MPI Forum list > Subject: [Mpi-forum] MPI "Allocate receive" proposal > > Dave Goodell and I have a proposal that we've socialized a bit around with > other Forum members, and would now like larger Forum feedback. I'll be > presenting the attached slides on the concept of an "allocate receive" in > Madrid (3:30-4pm on Thursday). > > There's no text or ticket yet; this is an idea that we want to get feedback on > before working up a full proposal. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Madrid hybrid slides: init/finalize
by Jeff Squyres (jsquyres) 26 Aug '13

26 Aug '13

The hybrid working group will be hosting a plenary on Friday afternoon in Madrid. We will be presenting a proposal on MPI_INIT/MPI_FINALIZE regarding interaction with threads (going back to the use cases Jeff Hammond raised at a prior Forum meeting), and nested MPI_INIT/MPI_FINALIZE. This is not a formal reading; we're requesting Forum feedback on the overall ideas. The slides are attached. See you in Madrid. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] [EXTERNAL] Re: MPI "Allocate receive" proposal
by Dries Kimpe 26 Aug '13

26 Aug '13

* Barrett, Brian W <bwbarre(a)sandia.gov> [2013-08-26 16:35:57]: > On 8/26/13 10:31 AM, "Dries Kimpe" <dkimpe(a)mcs.anl.gov> wrote: > >* Barrett, Brian W <bwbarre(a)sandia.gov> [2013-08-26 16:06:00]: > >[...] > >> That being said, I don't think resource exhaustion corner cases are a > >>deal > >> breaker for me. I think some implementation-dependent phrasing might be > >> acceptable. It might be useful to define the message > >> transmission/matching semantics for these corner cases. For example, if > >> MPI_Arecv returns an error because of resource exhaustion, is the > >>message > >> lost (my preference) or left in the receive queue? If the message is > >> lost, what happens to the sender if the send was a synchronous send? > >I think allowing the message to be 'lost' should be last resort and not > >taken lightly, since as far as I know, a message cannot be lost using the > >existing MPI functions (ignoring fault tolerance/resilience here) and, as > >you pointout in your example, might cause many complexities at other > >places. > Sure they can. The standard is totally silent on what happens when > MPI_RECV returns an error. You're firmly in the realm of undefined > behavior. Yes, but you were talking about *defining* semantics which include dropping a message. Dries

1 0

Re: [Mpi-forum] [EXTERNAL] Re: MPI "Allocate receive" proposal
by Barrett, Brian W 26 Aug '13

26 Aug '13

On 8/26/13 10:31 AM, "Dries Kimpe" <dkimpe(a)mcs.anl.gov> wrote: >* Barrett, Brian W <bwbarre(a)sandia.gov> [2013-08-26 16:06:00]: > >[...] > >> That being said, I don't think resource exhaustion corner cases are a >>deal >> breaker for me. I think some implementation-dependent phrasing might be >> acceptable. It might be useful to define the message >> transmission/matching semantics for these corner cases. For example, if >> MPI_Arecv returns an error because of resource exhaustion, is the >>message >> lost (my preference) or left in the receive queue? If the message is >> lost, what happens to the sender if the send was a synchronous send? > >I think allowing the message to be 'lost' should be last resort and not >taken lightly, since as far as I know, a message cannot be lost using the >existing MPI functions (ignoring fault tolerance/resilience here) and, as >you pointout in your example, might cause many complexities at other >places. Sure they can. The standard is totally silent on what happens when MPI_RECV returns an error. You're firmly in the realm of undefined behavior. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [Mpi-forum] [EXTERNAL] Re: MPI "Allocate receive" proposal
by Dries Kimpe 26 Aug '13

26 Aug '13

* Barrett, Brian W <bwbarre(a)sandia.gov> [2013-08-26 16:06:00]: [...] > That being said, I don't think resource exhaustion corner cases are a deal > breaker for me. I think some implementation-dependent phrasing might be > acceptable. It might be useful to define the message > transmission/matching semantics for these corner cases. For example, if > MPI_Arecv returns an error because of resource exhaustion, is the message > lost (my preference) or left in the receive queue? If the message is > lost, what happens to the sender if the send was a synchronous send? I think allowing the message to be 'lost' should be last resort and not taken lightly, since as far as I know, a message cannot be lost using the existing MPI functions (ignoring fault tolerance/resilience here) and, as you pointout in your example, might cause many complexities at other places. Dries

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Rolf Rabenseifner 26 Aug '13

26 Aug '13

I've three other question/comments. 1) Your last slide 7 says - ARECV scenario: unexpected ... T=2: buffers unexpected message, puts on unexpected list At this time, it is absolutely unknown, whether a following MPI_(I)recv or MPI_(I)arecv will be called. The message may be sent with MPI_Send / Bsend / Ssend (not Rsend because here scenario is "unexpected). Please, can you clarify, what may be the expected inner behavior for all three forms of sending and both forms of receiving. 2) I do not understand the datatype concept. There should be something like an etype and the MPI_Arecv(...etype...) receives a multiple etypes and a multiple of etypes must match "count, datatype" in the MPI_Send call. one may use the same restrictions on this etype that exist for the MPI-I/O etype. 3) Small notice: you may use BASEPTR instead of buffer. Your MPI_Status_get_buffer(status, &buffer) Returns a baseptr in the same way as MPI_ALLOC_MEM. Your MPI_Free_mem(buffer) frees the "base". Please note that this baseptr/base interface is asymetric. Best regards Rolf ----- Original Message ----- > From: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> > To: "MPI Forum list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Monday, August 26, 2013 5:10:30 PM > Subject: [Mpi-forum] MPI "Allocate receive" proposal > Dave Goodell and I have a proposal that we've socialized a bit around > with other Forum members, and would now like larger Forum feedback. > I'll be presenting the attached slides on the concept of an "allocate > receive" in Madrid (3:30-4pm on Thursday). > > There's no text or ticket yet; this is an idea that we want to get > feedback on before working up a full proposal. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [Mpi-forum] [EXTERNAL] Re: MPI "Allocate receive" proposal
by Barrett, Brian W 26 Aug '13

26 Aug '13

Jeff - I think it is different from a standards point of view, and has been different for some implementations. Take the Portals 3 implementation on Red Storm; the implementation did not allocate memory internally after MPI_INIT. So there was never a point at which the implementation would receive a message and try to allocate memory to land it. The user might be faced with that decision, but she can also free other user memory to make space for the incoming message before posting the receive. So I think it is, semantically, different. That being said, I don't think resource exhaustion corner cases are a deal breaker for me. I think some implementation-dependent phrasing might be acceptable. It might be useful to define the message transmission/matching semantics for these corner cases. For example, if MPI_Arecv returns an error because of resource exhaustion, is the message lost (my preference) or left in the receive queue? If the message is lost, what happens to the sender if the send was a synchronous send? Brian On 8/26/13 9:54 AM, "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> wrote: >FWIW, I'm not really sure I see how the failure for MPI_ARECV to get a >buffer (via malloc or whatever mechanism it wants) is substantially >different than failing to get a buffer for any incoming message. > >Indeed, for ARECV, an implementation may well try to use a freelist to >allocate a buffer, or fully-lazily allocate a buffer (because it won't >know the size until the message is received). Meaning that the >allocation will (potentially) occur on the same/similar path as all >receive messages. > >Put that together with the fact that normal receives can also fail to get >an eager or unexpected buffer when a new message arrives, for example. > >Therefore the error handling could well be the same between the ARECV and >"regular" receive cases. > > >On Aug 26, 2013, at 11:27 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> >> Can you clarify a bit more? >> >> Is that a fatal failure at that point? Or is it more like >>MPI_Alloc_mem failing where I can retry later or just do a regular >>malloc (in this case, a regular MPI_RECV). >> >> -- Pavan >> >> On 08/26/2013 10:19 AM, David Goodell (dgoodell) wrote: >>> I'd say that's implementation defined behavior, no different than >>>hitting any other system resource limitation right now. >>> >>> -Dave >>> >>> On Aug 26, 2013, at 10:14 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>> >>>> Jeff, >>>> >>>> This should probably be moved to the point-to-point WG, but I'll >>>>comment on this thread for now. >>>> >>>> Such a proposal was brought up at the very early stage of the MPI-2.1 >>>>process (I don't remember by whom). The main concern at that point >>>>was what happens when the memory allocation fails. Will the receive >>>>fail? What happens to the sent message? >>>> >>>> Can you work these details into your slides? >>>> >>>> -- Pavan >>>> >>>> On 08/26/2013 10:10 AM, Jeff Squyres (jsquyres) wrote: >>>>> Dave Goodell and I have a proposal that we've socialized a bit >>>>>around with other Forum members, and would now like larger Forum >>>>>feedback. I'll be presenting the attached slides on the concept of >>>>>an "allocate receive" in Madrid (3:30-4pm on Thursday). >>>>> >>>>> There's no text or ticket yet; this is an idea that we want to get >>>>>feedback on before working up a full proposal. >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi-forum mailing list >>>>> mpi-forum(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >>>>> >>>> >>>> -- >>>> Pavan Balaji >>>> http://www.mcs.anl.gov/~balaji >>>> _______________________________________________ >>>> mpi-forum mailing list >>>> mpi-forum(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >>> >>> _______________________________________________ >>> mpi-forum mailing list >>> mpi-forum(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > >-- >Jeff Squyres >jsquyres(a)cisco.com >For corporate legal information go to: >http://www.cisco.com/web/about/doing_business/legal/cri/ > >_______________________________________________ >mpi-forum mailing list >mpi-forum(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Jeff Squyres (jsquyres) 26 Aug '13

26 Aug '13

FWIW, I'm not really sure I see how the failure for MPI_ARECV to get a buffer (via malloc or whatever mechanism it wants) is substantially different than failing to get a buffer for any incoming message. Indeed, for ARECV, an implementation may well try to use a freelist to allocate a buffer, or fully-lazily allocate a buffer (because it won't know the size until the message is received). Meaning that the allocation will (potentially) occur on the same/similar path as all receive messages. Put that together with the fact that normal receives can also fail to get an eager or unexpected buffer when a new message arrives, for example. Therefore the error handling could well be the same between the ARECV and "regular" receive cases. On Aug 26, 2013, at 11:27 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > Can you clarify a bit more? > > Is that a fatal failure at that point? Or is it more like MPI_Alloc_mem failing where I can retry later or just do a regular malloc (in this case, a regular MPI_RECV). > > -- Pavan > > On 08/26/2013 10:19 AM, David Goodell (dgoodell) wrote: >> I'd say that's implementation defined behavior, no different than hitting any other system resource limitation right now. >> >> -Dave >> >> On Aug 26, 2013, at 10:14 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >>> Jeff, >>> >>> This should probably be moved to the point-to-point WG, but I'll comment on this thread for now. >>> >>> Such a proposal was brought up at the very early stage of the MPI-2.1 process (I don't remember by whom). The main concern at that point was what happens when the memory allocation fails. Will the receive fail? What happens to the sent message? >>> >>> Can you work these details into your slides? >>> >>> -- Pavan >>> >>> On 08/26/2013 10:10 AM, Jeff Squyres (jsquyres) wrote: >>>> Dave Goodell and I have a proposal that we've socialized a bit around with other Forum members, and would now like larger Forum feedback. I'll be presenting the attached slides on the concept of an "allocate receive" in Madrid (3:30-4pm on Thursday). >>>> >>>> There's no text or ticket yet; this is an idea that we want to get feedback on before working up a full proposal. >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpi-forum mailing list >>>> mpi-forum(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >>>> >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >>> _______________________________________________ >>> mpi-forum mailing list >>> mpi-forum(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] [EXTERNAL] Re: MPI "Allocate receive" proposal
by Jeff Squyres (jsquyres) 26 Aug '13

26 Aug '13

On Aug 26, 2013, at 11:39 AM, "Barrett, Brian W" <bwbarre(a)sandia.gov> wrote: > The working group mailing list is active and people read them. I think > we've struggled for a proposal that causes enough interest that we can > escape our current (inert) state. Ok. There's been some discussion here on the mpi-forum list, so if people could indulge us, I'd like to let it play out here. We'll do future posts about MPI_ARECV to the mpiwg-p2p list. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0