August 2013 - mpi-forum - mailman3.cels.anl.gov

2 week cutoff: tonight
by Jeff Squyres (jsquyres) 27 Aug '13

27 Aug '13

According to http://meetings.mpi-forum.org/secretary/2013/09/agenda.php, the Madrid meeting starts at 9:30am on Wednesday, Sep. 11, 2013. If you want a formal reading of something at the Madrid meeting, you need to send it to this list before 9:30am tomorrow morning (Madrid time -- i.e., effectively tonight :-) ). -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Rolf Rabenseifner 27 Aug '13

27 Aug '13

> Note that the type matching rules for send-receive say that MPI_BYTE > matches only MPI_BYTE, not MPI_INT/FLOAT, or other type (pg 33-34). So > this function would have to take a datatype argument, otherwise it > would match only sends of bytes. See 2) in my previous email. I would propose to add an etype. - Rolf ----- Original Message ----- > From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Monday, August 26, 2013 6:17:05 PM > Subject: Re: [Mpi-forum] MPI "Allocate receive" proposal > I've three other question/comments. > > 1) Your last slide 7 says > > - ARECV scenario: unexpected > ... > T=2: buffers unexpected message, > puts on unexpected list > > At this time, it is absolutely unknown, whether a > following MPI_(I)recv or MPI_(I)arecv will be called. > > The message may be sent with MPI_Send / Bsend / Ssend > (not Rsend because here scenario is "unexpected). > > Please, can you clarify, what may be the expected inner > behavior for all three forms of sending and both forms > of receiving. > > 2) I do not understand the datatype concept. > There should be something like an etype > and the MPI_Arecv(...etype...) receives a multiple > etypes and a multiple of etypes must match > "count, datatype" in the MPI_Send call. > one may use the same restrictions on this etype > that exist for the MPI-I/O etype. > > 3) Small notice: you may use BASEPTR instead of buffer. > Your MPI_Status_get_buffer(status, &buffer) > Returns a baseptr in the same way as MPI_ALLOC_MEM. > Your MPI_Free_mem(buffer) frees the "base". > Please note that this baseptr/base interface is asymetric. > > Best regards > Rolf ----- Original Message ----- > From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> > To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Tuesday, August 27, 2013 5:20:33 AM > Subject: Re: [Mpi-forum] MPI "Allocate receive" proposal > > Note that the type matching rules for send-receive say that MPI_BYTE > matches only MPI_BYTE, not MPI_INT/FLOAT, or other type (pg 33-34). So > this function would have to take a datatype argument, otherwise it > would match only sends of bytes. > > Rajeev > > > On Aug 26, 2013, at 10:10 AM, Jeff Squyres (jsquyres) wrote: > > > Dave Goodell and I have a proposal that we've socialized a bit > > around with other Forum members, and would now like larger Forum > > feedback. I'll be presenting the attached slides on the concept of > > an "allocate receive" in Madrid (3:30-4pm on Thursday). > > > > There's no text or ticket yet; this is an idea that we want to get > > feedback on before working up a full proposal. > > > > -- > > Jeff Squyres > > jsquyres(a)cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > <arecv.pptx>_______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Rajeev Thakur 26 Aug '13

26 Aug '13

Note that the type matching rules for send-receive say that MPI_BYTE matches only MPI_BYTE, not MPI_INT/FLOAT, or other type (pg 33-34). So this function would have to take a datatype argument, otherwise it would match only sends of bytes. Rajeev On Aug 26, 2013, at 10:10 AM, Jeff Squyres (jsquyres) wrote: > Dave Goodell and I have a proposal that we've socialized a bit around with other Forum members, and would now like larger Forum feedback. I'll be presenting the attached slides on the concept of an "allocate receive" in Madrid (3:30-4pm on Thursday). > > There's no text or ticket yet; this is an idea that we want to get feedback on before working up a full proposal. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > <arecv.pptx>_______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Jed Brown 26 Aug '13

26 Aug '13

Dries Kimpe <dkimpe(a)mcs.anl.gov> writes: > However, think about the cost of the straightforward implementation > without MPI_Arecv. > > MPI_Send (count) > MPI_Send (vector) for i in target_ranks: MPI_Isend(tcount[i],...,&sends[m++]); MPI_Isend(tdata[i],...,&sends[m++]); > Other side: > > MPI_Recv (count) > malloc (count) > MPI_Recv (...) The control flow is more like: for i in source_ranks: MPI_Irecv(&scount[i],...,&recvs[n++]); while MPI_Waitsome(n,recvs): for s in first_round: MPI_Irecv(sdata[s.rank],...,&recvs[n++]); for s in second_round: do the processing you would have done MPI_Waitall(m,sends); > In the overall picture, the cost of the extra recv/send/MProbe call is > going to be very minimal. Agreed, my point was that MPI_Arecv may not address a lot of the cases where it would initially seem to apply. While it might be nice to simplify the control flow, the proposal would be a lot stronger if it included demonstration of a performance advantage.

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Jed Brown 26 Aug '13

26 Aug '13

"Underwood, Keith D" <keith.d.underwood(a)intel.com> writes: > That's just it. I find it extremely unlikely that the user really has > a contiguous buffer that they need to receive this stuff into > (particularly in these examples). Hmm, I think the desire for a common receive buffer is common and the proposed MPI_Arecv would prevent it from being used without both dynamic allocation and another copy. > Both of these examples might actually be done *better* with pipelined > transfers from the peers. First, you send the count and then you send > the data in chunks. Because you cap the size of a chunk, you can > prepost several receives (ANY_SOURCE) without major fragmentation. > Pipelining actually allows you to overlap the data transfer with the > memory copy that you are inevitably doing on both ends. It's more complicated code, but I agree that appending the second round of receives to the request array given to MPI_Waitsome and then acting on completion of either (creating a second receive when the first completes or starting processing the data when the second completes) may be as good as one can hope for. > In fact, these could be an example of a place where providing a > different abstraction is going to cause the user to code it in a way > that is ultimately worse for them (though maybe easier to code). I'm not arguing with that. I use constant-size receives in my dynamic exchange code so that any larger messages will be expected and delivered to their ultimate resting place.

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Dries Kimpe 26 Aug '13

26 Aug '13

* Jed Brown <jedbrown(a)mcs.anl.gov> [2013-08-26 15:11:11]: > "Underwood, Keith D" <keith.d.underwood(a)intel.com> writes: > > But, do they know *approximately* the size of the message that is > > expected? Because, if they do, then most of the advantage isn't > > there. I struggle a little bit with the idea of well coded apps that > > have little enough idea about the size of the message that they need > > this. > Two example scenarios: > 1. After graph partitioning, we know who will receive our vertices, but > we don't know how many we will receive or from whom. In incremental > load balancing, we might know that we only receive from our > neighbors, and we have a bound on the total amount of data that we'll > receive, but may not have enough memory to post maximal receives from > all neighbors. (Only the incremental case is relevant for > performance because non-incremental partitioning is way expensive, > thus workaround 2 is fine.) > 2. In particle simulations, the physics may provide an upper bound on > total data received, but we don't know in advance from whom. > I think that in both of these cases, the user ultimately wants to > receive into a single buffer in some way. They might in fact have > allocated the buffer in advance and they'd be happy if they could decide > on a starting point and increment a counter each time a message appears. > Neither MPI_Mprobe with ANY_SOURCE or looping over MPI_Iprobe are > attractive compared to MPI_Waitsome, but the latter currently cannot be > used in the scenario above. I'm not disagreeing with the use case (in fact, I've dealt with this exact scenario in the past and confronted the same issue). However, think about the cost of the straightforward implementation without MPI_Arecv. MPI_Send (count) MPI_Send (vector) Other side: MPI_Recv (count) malloc (count) MPI_Recv (...) Or, even MPI_Mprobe (which didn't exist back when I was working on load balancing). In the overall picture, the cost of the extra recv/send/MProbe call is going to be very minimal. Also, certain implementations (apps) might not use this anyway, since they'll be receiving data from multiple ranks and so a collective (neighbourhood or regular) might be done to collect some more data first (like sizes) and create an optimized data movement schedule or properly allocate space in internal data structures. Dries

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Underwood, Keith D 26 Aug '13

26 Aug '13

> Two example scenarios: > > 1. After graph partitioning, we know who will receive our vertices, but > we don't know how many we will receive or from whom. In incremental > load balancing, we might know that we only receive from our > neighbors, and we have a bound on the total amount of data that we'll > receive, but may not have enough memory to post maximal receives from > all neighbors. (Only the incremental case is relevant for > performance because non-incremental partitioning is way expensive, > thus workaround 2 is fine.) > > 2. In particle simulations, the physics may provide an upper bound on > total data received, but we don't know in advance from whom. > > > I think that in both of these cases, the user ultimately wants to receive into a > single buffer in some way. They might in fact have allocated the buffer in > advance and they'd be happy if they could decide on a starting point and > increment a counter each time a message appears. > Neither MPI_Mprobe with ANY_SOURCE or looping over MPI_Iprobe are > attractive compared to MPI_Waitsome, but the latter currently cannot be > used in the scenario above. That's just it. I find it extremely unlikely that the user really has a contiguous buffer that they need to receive this stuff into (particularly in these examples). Both of these examples might actually be done *better* with pipelined transfers from the peers. First, you send the count and then you send the data in chunks. Because you cap the size of a chunk, you can prepost several receives (ANY_SOURCE) without major fragmentation. Pipelining actually allows you to overlap the data transfer with the memory copy that you are inevitably doing on both ends. In fact, these could be an example of a place where providing a different abstraction is going to cause the user to code it in a way that is ultimately worse for them (though maybe easier to code).

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Jed Brown 26 Aug '13

26 Aug '13

"Underwood, Keith D" <keith.d.underwood(a)intel.com> writes: > But, do they know *approximately* the size of the message that is > expected? Because, if they do, then most of the advantage isn't > there. I struggle a little bit with the idea of well coded apps that > have little enough idea about the size of the message that they need > this. Two example scenarios: 1. After graph partitioning, we know who will receive our vertices, but we don't know how many we will receive or from whom. In incremental load balancing, we might know that we only receive from our neighbors, and we have a bound on the total amount of data that we'll receive, but may not have enough memory to post maximal receives from all neighbors. (Only the incremental case is relevant for performance because non-incremental partitioning is way expensive, thus workaround 2 is fine.) 2. In particle simulations, the physics may provide an upper bound on total data received, but we don't know in advance from whom. I think that in both of these cases, the user ultimately wants to receive into a single buffer in some way. They might in fact have allocated the buffer in advance and they'd be happy if they could decide on a starting point and increment a counter each time a message appears. Neither MPI_Mprobe with ANY_SOURCE or looping over MPI_Iprobe are attractive compared to MPI_Waitsome, but the latter currently cannot be used in the scenario above.

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by Dries Kimpe 26 Aug '13

26 Aug '13

* Anthony Skjellum <tony(a)cis.uab.edu> [2013-08-26 14:29:01]: > If I understand this correctly, some of us suggested allocate-on-receive, free-on-send mode, > but left this out in MPI-1. I advocated this type of reactive semantics ... That's how > the Reactive Kernel worked at Caltech for instance... Allowing for MPI to choose efficient memory for messages of some sizes > can provide nice properties, provided the memory is good to compute in. There are always concerns about starvation from holding > the messages too long :-) > I'd like to see this get full consideration. It could be a big help in exascale to have fine-grain computing. > For instance, could we not get zero-copy unexpected short messages in this way? Don't forget to consider one-sided communication. Many of the proposed uses for MPI_Arecv could be implemented using one-sided, given some more application knowledge (which the application programmer has). For example, implementing zero-copy unexpected short messages is pretty trivial using the newer one-sided functions. Dries

1 0

Re: [Mpi-forum] MPI "Allocate receive" proposal
by David Goodell (dgoodell) 26 Aug '13

26 Aug '13

On Aug 26, 2013, at 2:25 PM, "Underwood, Keith D" <keith.d.underwood(a)intel.com> wrote: >> As per the use case, at least ADLB and Charm++ will benefit from this. >> They don't know what message size will come in. But I'm sure there are >> more. > > But, do they know *approximately* the size of the message that is expected? Because, if they do, then most of the advantage isn't there. I struggle a little bit with the idea of well coded apps that have little enough idea about the size of the message that they need this. Remember that the proposed Arecv interface solves a few problems: 1) There is currently no good way to receive totally unknown-size messages and known-size (or at least reasonably bounded-size) messages and still drive your messaging from a single MPI_Wait*/MPI_Test* call. The Arecv model fits nicely with these existing completion routines. 2) Unexpected messages tend to introduced an extra copy of some or all of the data, depending on the message size. This can be eliminated with a pre-posted Arecv. 3) For unexpected-size larger messages we go all the way back up the stack to wait for the user to give us a buffer, then go back down to send a CTS back (possibly with buffer/key/etc. info) to the sender. The Arecv eliminates the need to rendezvous all the way up to the application. The primary motivation for this proposal was some code I was writing that had client-server semantics where the requests from the client could not be neatly bounded. Furthermore, I wanted to drive the application from an MPI_Waitsome loop instead of an {MPI_Testsome;MPI_Improbe} loop (i.e., problem #1). The fact that #2 & #3 happen to be solved by this solution are pure gravy from my point of view. >> My concern is not whether someone can use it, but rather with respect to >> how much the MPI implementation can do without the sender knowing if >> the receiver is going to receive the message using RECV or ARECV. But >> maybe with Iarecv, it'll not be a problem. > > I'm more concerned about calling malloc() in the library. At the end of the day, even with the measly int-limited max message size, you will *have* to malloc in that path. > > I am also wondering if there is a better way to do this by exposing the traditional rendezvous? Sure, if you are under the rendezvous threshold, toss the eager receive buffer up. If you are above the threshold, toss it up to the library and provide a way to do a "pull" of the rest of the data. You could even go crazy and push an opaque handle out at the receiver to use in the pull. Given the implementation implications of this concept, my bias would be to not hide reality from the user. We should really only abstract the things for which the MPI library can do optimizations on the user's behalf. If we exposed and efficient way for the user to do rendezvous themselves, that may be better. I welcome your specific proposal :) Some sort of Mprobe/Improbe which can go into an existing MPI completion routine would be fine, but I couldn't come up with semantics that I liked for that. There are also all sorts of active-message-like solutions that you could come up with, but I don't personally want to fight to standardize what you can or cannot do in such a callback just so that I can get the more modest functionality that I actually desire. -Dave

1 0