February 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by Jim Dinan 11 Feb '14

11 Feb '14

Being on this list should be good. If you create an account on the trac page, you can also add yourself to the cc list for the ticket. You're also welcome to join meetings; right now we are focused on the March face-to-face meeting deadlines so we won't be able to work in any new business until after that. Changing the standard is a long process, so this could take a while, but staying involved to provide motivating use cases would be very helpful. :) On Tue, Feb 11, 2014 at 11:53 AM, Thomas Jahns <jahns(a)dkrz.de> wrote: > Hello Jim, > > On 02/11/14 16:59, Jim Dinan wrote: > > This sounds like a reasonable proposal to me; it should be possible to > create a > > sane definition for windows on intercommunicators. The RMA interface has > > focused on intracommunicators, so it could require several additions to > e.g. > > make it possible to query both local and remote groups from the window. > I think > > the main argument against adding this would be the increased complexity > in the > > standard and to implementations. Given that there is a workaround > (flattening > > the communicator to make the window, and using post/start/complete/wait > to > > restrict the set of communication peers in active target), we would need > to > > motivate such a change. > > > > I captured your proposal in a ticket, > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/412, so we remember > to > > discuss this further in the RMA working group. > > thank you. Will remaining subscribed to this list be sufficient to receive > notice of updates or should I poll above issue every few weeks? > > Regards, Thomas > -- > Thomas Jahns > DKRZ GmbH, Department: Application software > > Deutsches Klimarechenzentrum > Bundesstraße 45a > D-20146 Hamburg > > Phone: +49-40-460094-151 > Fax: +49-40-460094-270 > Email: Thomas Jahns <jahns(a)dkrz.de> > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by Jeff Hammond 11 Feb '14

11 Feb '14

On Tue, Feb 11, 2014 at 10:53 AM, Thomas Jahns <jahns(a)dkrz.de> wrote: > Hello Jim, > > On 02/11/14 16:59, Jim Dinan wrote: >> This sounds like a reasonable proposal to me; it should be possible to create a >> sane definition for windows on intercommunicators. The RMA interface has >> focused on intracommunicators, so it could require several additions to e.g. >> make it possible to query both local and remote groups from the window. I think >> the main argument against adding this would be the increased complexity in the >> standard and to implementations. Given that there is a workaround (flattening >> the communicator to make the window, and using post/start/complete/wait to >> restrict the set of communication peers in active target), we would need to >> motivate such a change. >> >> I captured your proposal in a ticket, >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/412, so we remember to >> discuss this further in the RMA working group. > > thank you. Will remaining subscribed to this list be sufficient to receive > notice of updates or should I poll above issue every few weeks? Polling is bad :-) I can cc you on the ticket as well if you like. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] same_disp_unit info key
by Jeff Hammond 11 Feb '14

11 Feb '14

I assume this is trivial so I can do it. Particularly since I reported it. I am attending the Forum on M and T only. Jeff On Tue, Feb 11, 2014 at 10:53 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi All, > > I have prepared a formal proposal for the same_disp_unit info key ticket. > Please review: > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/369 > > I won't be able to attend this meeting, is there a representative from the > RMA WG that can present this? > > Thanks, > ~Jim. > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

same_disp_unit info key
by Jim Dinan 11 Feb '14

11 Feb '14

Hi All, I have prepared a formal proposal for the same_disp_unit info key ticket. Please review: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/369 I won't be able to attend this meeting, is there a representative from the RMA WG that can present this? Thanks, ~Jim.

1 0

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by Thomas Jahns 11 Feb '14

11 Feb '14

Hello Jim, On 02/11/14 16:59, Jim Dinan wrote: > This sounds like a reasonable proposal to me; it should be possible to create a > sane definition for windows on intercommunicators. The RMA interface has > focused on intracommunicators, so it could require several additions to e.g. > make it possible to query both local and remote groups from the window. I think > the main argument against adding this would be the increased complexity in the > standard and to implementations. Given that there is a workaround (flattening > the communicator to make the window, and using post/start/complete/wait to > restrict the set of communication peers in active target), we would need to > motivate such a change. > > I captured your proposal in a ticket, > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/412, so we remember to > discuss this further in the RMA working group. thank you. Will remaining subscribed to this list be sufficient to receive notice of updates or should I poll above issue every few weeks? Regards, Thomas -- Thomas Jahns DKRZ GmbH, Department: Application software Deutsches Klimarechenzentrum Bundesstraße 45a D-20146 Hamburg Phone: +49-40-460094-151 Fax: +49-40-460094-270 Email: Thomas Jahns <jahns(a)dkrz.de>

1 0

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by Jim Dinan 11 Feb '14

11 Feb '14

Hi Thomas, This sounds like a reasonable proposal to me; it should be possible to create a sane definition for windows on intercommunicators. The RMA interface has focused on intracommunicators, so it could require several additions to e.g. make it possible to query both local and remote groups from the window. I think the main argument against adding this would be the increased complexity in the standard and to implementations. Given that there is a workaround (flattening the communicator to make the window, and using post/start/complete/wait to restrict the set of communication peers in active target), we would need to motivate such a change. I captured your proposal in a ticket, https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/412, so we remember to discuss this further in the RMA working group. Best, ~Jim. On Wed, Jan 22, 2014 at 5:48 AM, Thomas Jahns <jahns(a)dkrz.de> wrote: > Hello Jeff, > > On 01/21/14 20:03, Jeff Hammond wrote: > > It makes my brain hurt just to think about this. > > I certainly intended no harm, however mild ;-). > > > First, one should recognize that RMA communication occurs using the > > window object, not the communicator as in p2p and collectives. The > > reason for a comm object in WIN_CREATE et al. is to enable MPI to > > communicate between participating processes in order to create the > > window. For example, many implementations will do an ALLGATHER inside > > of WIN_CREATE. That operation needs a comm object. > > So there is no problem here with using an intercommunicator? > > > The second purpose of the comm is to generate the group of the window > > in order to denote which ranks know about the window. It bothers me a > > negligible amount that the exact definition of the rank arguments for > > RMA is not defined explicitly, but one can reasonably assume that > > those are ranks on the group of the window. I suppose one could talk > > about local and remote groups for a window but again, brain hurt. > > I think what's defined in lines 20-25 of page 420 of MPI 3.0 already makes > perfect sense if comm were an intercommunicator since it's specified in > terms of > what would happen for point-to-point communications. So no problem either. > > > I do not believe that the communicator has any purpose beyond this and > > thus do not know what the consequences of allowing intercomms would > > be. Perhaps we can use - only for the sake of argument since it is > > not required to have these semantics - that WIN_CREATE is like > > ALLGATHER and use the semantics of ALLGATHER on intercomms to define > > the semantics of win objects created using intercomms. > > For e.g. MPI_Win_post one needs a group, based on the communicator > originally > passed to MPI_Win_create but that's already solved for intracommunicators > and no > new problem. > > > In any case, I don't see any real value in this, honestly. Efficiency > > arguments assume implementation details not specified by the standard. > > Making a promise to not communicate with some ranks has no effect on > > some implementations. Of course we know ones where it does, but > > that's not germane. > > Still it's something I cannot express currently. > > > Honestly, I think any sane implementer will just MPI_COMM_TEST_INTER > > -> INTERCOMM_MERGE inside of of WIN_CREATE (et al.), do everything as > > before and then free the temp intracomm at the end. Then I would > > never, ever think about this issue again until someone paid me a huge > > sum of money to optimize for the intercomm case. Thus, the result of > > changing the standard will be almost nothing other than people will be > > able to avoid doing the following explicitly: > > For the above scenario the original communicator needs to be retained, > i.e. to > make use of for a later MPI_Group_translate, but otherwise you are right > unless > mpi_win_create were to be significantly less expensive with less potential > paths. > > > int MPE_Win_create(void *base, MPI_Aint size, int disp_unit, MPI_Info > > info, MPI_Comm comm, MPI_Win *win) > > { > > int is_intercomm; > > MPI_Comm intracomm; > > MPI_Comm_test_inter(comm, &is_intercomm); > > if (is_intercomm) > > MPI_Intercomm_merge(comm, 0, &intracomm); > > else > > intracomm = comm; > > MPI_Win_create(base, size, disp_unit, info, intracomm, win); > > if (is_intercomm) > > MPI_Comm_free(&intracomm); > > return MPI_SUCCESS; > > } > > > > In short, I don't see any value in trying to define the perverse > > semantics of intercomm-based windows to avoid the ~12 lines of code > > above. > > In my case, where I have a client-server setup with point-to-point > communication > happening via an intercommunicator, adding bulk communication via RMA the > current state of affairs means I have to use an additional communicator and > translate IDs. But sure, it can be done with something similar to the > above, > it's just more code on the user side. > > > In the event we try to act on this proposal, I suppose we use the very > > limited text regarding send-recv on intercomms as our guide: "A target > > process is addressed by its rank in the remote group, both for sends > > and for receives." [pg. 258] > > That's what seemed intuitive to me too (see above). > > Regards, Thomas > -- > Thomas Jahns > DKRZ GmbH, Department: Application software > > Deutsches Klimarechenzentrum > Bundesstraße 45a > D-20146 Hamburg > > Phone: +49-40-460094-151 > Fax: +49-40-460094-270 > Email: Thomas Jahns <jahns(a)dkrz.de> > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Balaji, Pavan 05 Feb '14

05 Feb '14

On Feb 5, 2014, at 12:25 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Does load-store against a shared memory window have to be inside of an > epoch? Yes.

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Rolf Rabenseifner 05 Feb '14

05 Feb '14

MPI-3.0 p449:9 requires a passive target epoch when you want to call MPI_WIN_SYNC. I do not see need to change the rules of MPI-3, because the shared lock does not hurt. I would only add the following example at the end of 11.7: -------------------- Example 11.13 The following example demonstrates the rules for the unified memory model if the data transfer is implemented only with load and store (instead of MPI_PUT or MPI_GET) and the synchronization between the processes is done with MPI communication (instead of RMA synchronization routines). The used synchronization must be supplemented with MPI_WIN_SYNC which acts only local as a processor-memory-barrier. For MPI_WIN_SYNC, a passive target epoch is established with MPI_WIN_LOCK_ALL. X is part of a shared memory window and should be the same memory location in both processes. Process A Process B MPI_WIN_LOCK_ALL( MPI_WIN_LOCK_ALL( MPI_MODE_NOCHECK,win) MPI_MODE_NOCHECK,win) DO ... DO ... X=... MPI_F_SYNC_REG(X) MPI_WIN_SYNC(win) MPI_Barrier MPI_Barrier MPI_WIN_SYNC(win) MPI_F_SYNC_REG(X) print X END DO END DO MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win) -------------------- Is the text okay? Any text correctons needed? Rolf ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Wednesday, February 5, 2014 7:25:19 PM > Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows > > On Wed, Feb 5, 2014 at 12:14 PM, Balaji, Pavan <balaji(a)anl.gov> > wrote: > > > > OK, I thought you were referring to a generic memory barrier call, > > not > > related to the window. > > > > In what case will you need this call outside the epoch, given that > > you can > > only access the window content in an epoch? > > > > On Feb 5, 2014 12:06 PM, Jeff Hammond <jeff.science(a)gmail.com> > > wrote: > > On Wed, Feb 5, 2014 at 11:57 AM, Balaji, Pavan <balaji(a)anl.gov> > > wrote: > >> > >> On Feb 5, 2014, at 11:54 AM, Jeff Hammond <jeff.science(a)gmail.com> > >> wrote: > >>> I think advice to users regarding WIN_SYNC wouldn't hurt given > >>> that > >>> some of us are talking about it as if it is a portable > >>> abstraction for > >>> a memory barrier when it is now only correct to use it within an > >>> RMA > >>> sync epoch. Or we need to relax this restriction. > >> > >> Without this restriction, there’s no connection to what memory is > >> being > >> made consistent with the WIN_SYNC call. I’d be against removing > >> it. > > > > The memory being made consistent is obviously the public and > > private > > window. In the case of UNIFIED, it's just a memory barrier. > > Right? > > > > I suppose someone could always WIN_LOCK+WIN_SYNC+WIN_UNLOCK as a > > portable abstraction for a memory barrier in the case where an > > epoch > > is not already established. > > Does load-store against a shared memory window have to be inside of > an > epoch? I can't remember but if it does, then shared memory windows > are not the magic abstraction for POSIX shm that I though they were. > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Jeff Hammond 05 Feb '14

05 Feb '14

On Wed, Feb 5, 2014 at 12:14 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > OK, I thought you were referring to a generic memory barrier call, not > related to the window. > > In what case will you need this call outside the epoch, given that you can > only access the window content in an epoch? > > On Feb 5, 2014 12:06 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > On Wed, Feb 5, 2014 at 11:57 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >> On Feb 5, 2014, at 11:54 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> I think advice to users regarding WIN_SYNC wouldn't hurt given that >>> some of us are talking about it as if it is a portable abstraction for >>> a memory barrier when it is now only correct to use it within an RMA >>> sync epoch. Or we need to relax this restriction. >> >> Without this restriction, there’s no connection to what memory is being >> made consistent with the WIN_SYNC call. I’d be against removing it. > > The memory being made consistent is obviously the public and private > window. In the case of UNIFIED, it's just a memory barrier. Right? > > I suppose someone could always WIN_LOCK+WIN_SYNC+WIN_UNLOCK as a > portable abstraction for a memory barrier in the case where an epoch > is not already established. Does load-store against a shared memory window have to be inside of an epoch? I can't remember but if it does, then shared memory windows are not the magic abstraction for POSIX shm that I though they were. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows
by Balaji, Pavan 05 Feb '14

05 Feb '14

OK, I thought you were referring to a generic memory barrier call, not related to the window. In what case will you need this call outside the epoch, given that you can only access the window content in an epoch? On Feb 5, 2014 12:06 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: On Wed, Feb 5, 2014 at 11:57 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > On Feb 5, 2014, at 11:54 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> I think advice to users regarding WIN_SYNC wouldn't hurt given that >> some of us are talking about it as if it is a portable abstraction for >> a memory barrier when it is now only correct to use it within an RMA >> sync epoch. Or we need to relax this restriction. > > Without this restriction, there’s no connection to what memory is being made consistent with the WIN_SYNC call. I’d be against removing it. The memory being made consistent is obviously the public and private window. In the case of UNIFIED, it's just a memory barrier. Right? I suppose someone could always WIN_LOCK+WIN_SYNC+WIN_UNLOCK as a portable abstraction for a memory barrier in the case where an epoch is not already established. Jeff -- Jeff Hammond jeff.science(a)gmail.com _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0