July 2011 - mpiwg-rma - mailman3.cels.anl.gov

[Mpi3-rma] Alternative Proposal for Shared Memory Support [act 2]
by Torsten Hoefler 15 Jul '11

15 Jul '11

Hello all, I apologize for the cross-post but this is of interest to both groups. This email is logically following my previous email from March 15th on the same topic. Jim and I revamped the originally proposed interface and are ready to present it at the Forum. We also have a proposal draft (as patch to the proposed MPI-3.0 One Sided Chapter). We will present the proposal at next week's meeting. However, it would be great if folks would prepare and look at the proposed changes (they're in draft stadium) attached to ticket #284. https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/284 See you next week! All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] RMA communication with single network messages
by Oliver Mangold 14 Jul '11

14 Jul '11

On 14.07.2011 17:40, Pavan Balaji wrote: > > MPI_Rsend still has matching complexity which RMA operations don't. > Yes, and I also guess preposting the receives for each Rsend is an overhead. > I think what Oliver wants can be achieved by passing the > MPI_MODE_NOCHECK assertion to the post/start calls. > That looks very interesting. I will check if it actually makes a performance difference.

1 0

Re: [Mpi3-rma] RMA communication with single network messages
by Pavan Balaji 14 Jul '11

14 Jul '11

MPI_Rsend still has matching complexity which RMA operations don't. I think what Oliver wants can be achieved by passing the MPI_MODE_NOCHECK assertion to the post/start calls. -- Pavan On 07/14/2011 08:55 AM, James Dinan wrote: > Hi Oliver, > > If you are certain that the receive has already been posted, you could > use MPI_Rsend, a "ready" send. This should be able to give you both (a) > and (b). > > ~Jim. > > On 07/11/2011 10:59 AM, Oliver Mangold wrote: >> What I would like to do with MPI is point-to-point communication with >> minimal overhead/latency, which means the communication >> >> (a) does not need storage of data in intermediate buffers at the >> destination >> (b) does not need more than 1 data transfer (network message) per >> communication >> >> Using MPI point-to-point has the problem that it cannot abide both (a) >> and (b) at the same time, as the sent data might receive the destination >> before MPI_Recv is called. >> >> So I am wondering if this will be possible with the improved RMA >> features of MPI-3. The draft paper (as the 2.2 standard also does) says: >>> RMA communications fall in two categories: >>> * active target communication, where data is moved from the memory of >>> one process >>> to the memory of another, and both are explicitly involved in the >>> communication. This >>> communication pattern is similar to message passing, except that all >>> the data transfer >>> arguments are provided by one process, and the second process only >>> participates in >>> the synchronization. >> Actually this would be exactly what I wanted. The sender should provide >> all the transfer arguments (including destination memory location) and >> the receiver only waits for the message to arrive. >> >> As I understand MPI-2.2 RMA, the sender has to do: >> >> MPI_Win_start(group, flag, win); >> MPI_Put(...,win); >> MPI_Win_complete(win); >> >> While the receiver does: >> >> MPI_Win_post(group, flag, win); >> MPI_Win_wait(win); >> >> As I understand things, the semantics of MPI RMA require that >> destination memory is not written before the call to MPI_Win_post(). >> This means either must the receiver signal the sender that he has >> reached MPI_Win_post() or the data must be buffered at the receiver, >> resulting in the same problem as with MPI point-to-point. Please correct >> me if I'm wrong. >> >> The problem could be solved, if there were windows that are always >> 'open', this means no MPI_Win_start() and MPI_Win_post() necessary (only >> MPI_Win_complete() and MPI_Win_wait() to inform the receiver that the >> data has arrived). If the framework merges the data transfers needed for >> MPI_Put and MPI_Win_complete, a single message would be sufficient. But >> I couldn't find a feature in the draft standard that helps here. So the >> question is, is there a way to do what I want with MPI-3? >> >> Maybe I should note that windows which are always open are useful, as >> with pair-wise-communication (with data always going both ways) double >> buffering would fix race conditions. >> >> Example (assuming MPI_Win_start() and MPI_Win_post() are not needed): >> >> Process 0: >> while () { >> MPI_Put(...,win1a); >> MPI_Win_complete(win1a); >> MPI_Win_wait(win0a); >> ... do computation on data from win0a and write data for win1b ... >> MPI_Put(...,win1b); >> MPI_Win_complete(win1b); >> MPI_Win_wait(win0b); >> ... do computation on data from win0b and write data for win1a ... >> } >> >> Process 1: >> while () { >> MPI_Put(...,win0a); >> MPI_Win_complete(win0a); >> MPI_Win_wait(win1a); >> ... do computation on data from win1a and write data for win0b ... >> MPI_Put(...,win0b); >> MPI_Win_complete(win0b); >> MPI_Win_wait(win1b); >> ... do computation on data from win1b and write data for 0a ... >> } >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] RMA communication with single network messages
by James Dinan 14 Jul '11

14 Jul '11

Hi Oliver, If you are certain that the receive has already been posted, you could use MPI_Rsend, a "ready" send. This should be able to give you both (a) and (b). ~Jim. On 07/11/2011 10:59 AM, Oliver Mangold wrote: > What I would like to do with MPI is point-to-point communication with > minimal overhead/latency, which means the communication > > (a) does not need storage of data in intermediate buffers at the > destination > (b) does not need more than 1 data transfer (network message) per > communication > > Using MPI point-to-point has the problem that it cannot abide both (a) > and (b) at the same time, as the sent data might receive the destination > before MPI_Recv is called. > > So I am wondering if this will be possible with the improved RMA > features of MPI-3. The draft paper (as the 2.2 standard also does) says: >> RMA communications fall in two categories: >> * active target communication, where data is moved from the memory of >> one process >> to the memory of another, and both are explicitly involved in the >> communication. This >> communication pattern is similar to message passing, except that all >> the data transfer >> arguments are provided by one process, and the second process only >> participates in >> the synchronization. > Actually this would be exactly what I wanted. The sender should provide > all the transfer arguments (including destination memory location) and > the receiver only waits for the message to arrive. > > As I understand MPI-2.2 RMA, the sender has to do: > > MPI_Win_start(group, flag, win); > MPI_Put(...,win); > MPI_Win_complete(win); > > While the receiver does: > > MPI_Win_post(group, flag, win); > MPI_Win_wait(win); > > As I understand things, the semantics of MPI RMA require that > destination memory is not written before the call to MPI_Win_post(). > This means either must the receiver signal the sender that he has > reached MPI_Win_post() or the data must be buffered at the receiver, > resulting in the same problem as with MPI point-to-point. Please correct > me if I'm wrong. > > The problem could be solved, if there were windows that are always > 'open', this means no MPI_Win_start() and MPI_Win_post() necessary (only > MPI_Win_complete() and MPI_Win_wait() to inform the receiver that the > data has arrived). If the framework merges the data transfers needed for > MPI_Put and MPI_Win_complete, a single message would be sufficient. But > I couldn't find a feature in the draft standard that helps here. So the > question is, is there a way to do what I want with MPI-3? > > Maybe I should note that windows which are always open are useful, as > with pair-wise-communication (with data always going both ways) double > buffering would fix race conditions. > > Example (assuming MPI_Win_start() and MPI_Win_post() are not needed): > > Process 0: > while () { > MPI_Put(...,win1a); > MPI_Win_complete(win1a); > MPI_Win_wait(win0a); > ... do computation on data from win0a and write data for win1b ... > MPI_Put(...,win1b); > MPI_Win_complete(win1b); > MPI_Win_wait(win0b); > ... do computation on data from win0b and write data for win1a ... > } > > Process 1: > while () { > MPI_Put(...,win0a); > MPI_Win_complete(win0a); > MPI_Win_wait(win1a); > ... do computation on data from win1a and write data for win0b ... > MPI_Put(...,win0b); > MPI_Win_complete(win0b); > MPI_Win_wait(win1b); > ... do computation on data from win1b and write data for 0a ... > } > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] RMA communication with single network messages
by Oliver Mangold 14 Jul '11

14 Jul '11

On 13.07.2011 23:00, Rajeev Thakur wrote: > There isn't a way to do exactly what you want, but you could do either of the following. > > 1. Call MPI_Win_post immediately after MPI_Win_create and immediately after every call to MPI_Win_complete. > This would necessarily result in an extra message for each MPI_Win_post or buffering on the receiver side, right? I do not see a way around as the framework has no way to know, that post will always happen after complete. Is there a reason why the new MPI will not allow for this? From the viewpoint of performance it's so obviously desirable. I assume I'm not the first one thinking about it. I guess it would be possible for implementations to define a key for the info of MPI_Win_Create to specify that the windows should be always open and MPI_Win_post and MPI_Win_start do exactly nothing for this window. Would be implementation-specific, of course.

1 0

Re: [Mpi3-rma] RMA communication with single network messages
by Rajeev Thakur 13 Jul '11

13 Jul '11

There isn't a way to do exactly what you want, but you could do either of the following. 1. Call MPI_Win_post immediately after MPI_Win_create and immediately after every call to MPI_Win_complete. 2. Use active-target synchronization (lock-unlock), in which the target window is always open. However, the target doesn't know when the data transfer has completed, and you have to use some other means to know that. Rajeev On Jul 11, 2011, at 10:59 AM, Oliver Mangold wrote: > What I would like to do with MPI is point-to-point communication with minimal overhead/latency, which means the communication > > (a) does not need storage of data in intermediate buffers at the destination > (b) does not need more than 1 data transfer (network message) per communication > > Using MPI point-to-point has the problem that it cannot abide both (a) and (b) at the same time, as the sent data might receive the destination before MPI_Recv is called. > > So I am wondering if this will be possible with the improved RMA features of MPI-3. The draft paper (as the 2.2 standard also does) says: >> RMA communications fall in two categories: >> * active target communication, where data is moved from the memory of one process >> to the memory of another, and both are explicitly involved in the communication. This >> communication pattern is similar to message passing, except that all the data transfer >> arguments are provided by one process, and the second process only participates in >> the synchronization. > Actually this would be exactly what I wanted. The sender should provide all the transfer arguments (including destination memory location) and the receiver only waits for the message to arrive. > > As I understand MPI-2.2 RMA, the sender has to do: > > MPI_Win_start(group, flag, win); > MPI_Put(...,win); > MPI_Win_complete(win); > > While the receiver does: > > MPI_Win_post(group, flag, win); > MPI_Win_wait(win); > > As I understand things, the semantics of MPI RMA require that destination memory is not written before the call to MPI_Win_post(). This means either must the receiver signal the sender that he has reached MPI_Win_post() or the data must be buffered at the receiver, resulting in the same problem as with MPI point-to-point. Please correct me if I'm wrong. > > The problem could be solved, if there were windows that are always 'open', this means no MPI_Win_start() and MPI_Win_post() necessary (only MPI_Win_complete() and MPI_Win_wait() to inform the receiver that the data has arrived). If the framework merges the data transfers needed for MPI_Put and MPI_Win_complete, a single message would be sufficient. But I couldn't find a feature in the draft standard that helps here. So the question is, is there a way to do what I want with MPI-3? > > Maybe I should note that windows which are always open are useful, as with pair-wise-communication (with data always going both ways) double buffering would fix race conditions. > > Example (assuming MPI_Win_start() and MPI_Win_post() are not needed): > > Process 0: > while () { > MPI_Put(...,win1a); > MPI_Win_complete(win1a); > MPI_Win_wait(win0a); > ... do computation on data from win0a and write data for win1b ... > MPI_Put(...,win1b); > MPI_Win_complete(win1b); > MPI_Win_wait(win0b); > ... do computation on data from win0b and write data for win1a ... > } > > Process 1: > while () { > MPI_Put(...,win0a); > MPI_Win_complete(win0a); > MPI_Win_wait(win1a); > ... do computation on data from win1a and write data for win0b ... > MPI_Put(...,win0b); > MPI_Win_complete(win0b); > MPI_Win_wait(win1b); > ... do computation on data from win1b and write data for 0a ... > } > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

[Mpi3-rma] RMA communication with single network messages
by Oliver Mangold 11 Jul '11

11 Jul '11

What I would like to do with MPI is point-to-point communication with minimal overhead/latency, which means the communication (a) does not need storage of data in intermediate buffers at the destination (b) does not need more than 1 data transfer (network message) per communication Using MPI point-to-point has the problem that it cannot abide both (a) and (b) at the same time, as the sent data might receive the destination before MPI_Recv is called. So I am wondering if this will be possible with the improved RMA features of MPI-3. The draft paper (as the 2.2 standard also does) says: > RMA communications fall in two categories: > * active target communication, where data is moved from the memory of > one process > to the memory of another, and both are explicitly involved in the > communication. This > communication pattern is similar to message passing, except that all > the data transfer > arguments are provided by one process, and the second process only > participates in > the synchronization. Actually this would be exactly what I wanted. The sender should provide all the transfer arguments (including destination memory location) and the receiver only waits for the message to arrive. As I understand MPI-2.2 RMA, the sender has to do: MPI_Win_start(group, flag, win); MPI_Put(...,win); MPI_Win_complete(win); While the receiver does: MPI_Win_post(group, flag, win); MPI_Win_wait(win); As I understand things, the semantics of MPI RMA require that destination memory is not written before the call to MPI_Win_post(). This means either must the receiver signal the sender that he has reached MPI_Win_post() or the data must be buffered at the receiver, resulting in the same problem as with MPI point-to-point. Please correct me if I'm wrong. The problem could be solved, if there were windows that are always 'open', this means no MPI_Win_start() and MPI_Win_post() necessary (only MPI_Win_complete() and MPI_Win_wait() to inform the receiver that the data has arrived). If the framework merges the data transfers needed for MPI_Put and MPI_Win_complete, a single message would be sufficient. But I couldn't find a feature in the draft standard that helps here. So the question is, is there a way to do what I want with MPI-3? Maybe I should note that windows which are always open are useful, as with pair-wise-communication (with data always going both ways) double buffering would fix race conditions. Example (assuming MPI_Win_start() and MPI_Win_post() are not needed): Process 0: while () { MPI_Put(...,win1a); MPI_Win_complete(win1a); MPI_Win_wait(win0a); ... do computation on data from win0a and write data for win1b ... MPI_Put(...,win1b); MPI_Win_complete(win1b); MPI_Win_wait(win0b); ... do computation on data from win0b and write data for win1a ... } Process 1: while () { MPI_Put(...,win0a); MPI_Win_complete(win0a); MPI_Win_wait(win1a); ... do computation on data from win1a and write data for win0b ... MPI_Put(...,win0b); MPI_Win_complete(win0b); MPI_Win_wait(win1b); ... do computation on data from win1b and write data for 0a ... }

1 0

[Mpi3-rma] Updated RMA chapter
by William Gropp 02 Jul '11

02 Jul '11

I've uploaded a new version of the RMA chapter to https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/mpi3-rma-propo… This follows the ground rules that we discussed at the forum meeting after the first reading - only "ticket 0" changes were made, though these included some changes to the examples. Note that the issue of the type of the values returned by the attributes will be handled by a separate ticket; this is significant to require a reset to the clock if made to the chapter and is likely to require changes in some applications and implementations, and thus is best handled separately. Let me know if there are either additional and important ticket 0 changes or if there is something more major. Minor ticket 0 changes should be accumulated and applied separately - the goal is to bring a complete RMA chapter to a final vote. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0