July 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions
by Rolf Rabenseifner 11 Jul '14

11 Jul '14

Bill, > In general I am opposed to decreasing the performance of applications > by mandating strong synchronization semantics. I tried to put all known pieces together in the new ticket https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437 The ticket does not add additional synchronization rather than trying a proper meaning for shared memory windows. If an application does not want to synchronize between a sending process and a receiving process, it can use MPI_WIN_SYNC as shown in example added to MPI-3.0 errata in https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/413 and based on the Erratum in the new ticket https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/436 I hope we can discuss this as soon as possible. MPI-3.1 should not be released before the meaning of shared memory windows is not clear. This discussion is already delayed for about 21 month because Hubert asked for clarification already on Sep 11, 2012 with subject "[Mpi3-rma] MPI_Win_allocate_shared and synchronization functions". Best regards Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "Bill Gropp" <wgropp(a)uiuc.edu>, "Rajeev Thakur" <thakur(a)anl.gov>, "Hubert Ritzdorf" > <Hubert.Ritzdorf(a)EMEA.NEC.COM>, "Pavan Balaji" <balaji(a)anl.gov>, "Torsten Hoefler" <htor(a)inf.ethz.ch>, "MPI WG > Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Wednesday, July 2, 2014 4:46:40 PM > Subject: Re: Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions > > In general I am opposed to decreasing the performance of applications > by mandating strong synchronization semantics. > > > I will have to study this particular instance carefully. > > > Bill > > > > > > > William Gropp > Director, Parallel Computing Institute Thomas M. Siebel Chair in > Computer Science > > > University of Illinois Urbana-Champaign > > > > > > > > On Jul 2, 2014, at 9:38 AM, Rolf Rabenseifner wrote: > > > > Bill, Rajeev, and all other RMA WG members, > > Hubert and Torsten discussed already in 2012 the meaning of > the MPI one-sided synchronization routines for MPI-3.0 shared memory. > > This question is still unresolved in the MPI-3.0 + errata. > > Does the term "RMA Operation" include "a remote load/store > from an origin process to the window Memory on a target"? > > Or not? > > The ticket #435 expects "not". > > In this case, MPI_Win_fence and post-start-complete-wait > cannot be used for synchronizing the sending process of data > with the receiving process of data that use only > local and remote load/stores on shared memory windows. > > Ticket 435 extends the meaning of > MPI_Win_fence and post-start-complete-wait that they > provide sender-Receiver synchronization between processes > that use local and remote load/stores > on shared memory windows. > > I hope that all RMA working group members, agree that > - currently the behavior of these sync-routines for > shared memory remote load/stores is undefined due to > the undefined definition of "RMA Operation" > - and that we need an errata that resolves this problem. > > What is your opinion about the solution provided in > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/435 ? > > Best regards > Rolf > > PS: Ticket 435 is the result of a discussion of Pavan, Hubert and I > at ISC2014. > > ----- Original Message ----- > > > From: Hubert Ritzdorf > > > Sent: Tuesday, September 11, 2012 7:26 PM > > > To: mpi3-rma(a)lists.mpi-forum.org > > > Subject: MPI_Win_allocate_shared and synchronization functions > > > > > > Hi, > > > > > > it's quite unclear what Page 410, Lines 17-19 > > > > > > A consistent view can be created in the uni?fied > > > memory model (see Section 11.4) by utilizing the window > > > synchronization functions (see > > > Section 11.5) > > > > > > really means. Section 11.5 doesn't mention any (load/store) access to > > > shared memory. > > > Thus, must > > > > > > (*) RMA communication calls and RMA operations > > > be interpreted as RMA communication calls (MPI_GET, MPI_PUT, > > > ...) and > > > ANY load/store access to shared > > > window > > > (*) put call as put call and any store to shared memory > > > (*) get call as get call and any load from shared memory > > > (*) accumulate call as accumulate call and any load or store access > > > to shared window ? > > > > > > Example: Assertion MPI_MODE_NOPRECEDE > > > > > > Does > > > > > > the fence does not complete any sequence of locally issued RMA calls > > > > > > mean for windows created by MPI_Win_Allocate_shared () > > > > > > the fence does not complete any sequence of locally issued RMA calls > > > or > > > any load/store access to the window memory ? > > > > > > It's not clear to me. I will be probably not clear for the standard > > > MPI user. > > > RMA operations are defined only MPI functions for window objects > > > (as far as I can see). > > > But possibly I'm totally wrong and the synchronization functions > > > synchronize > > > only the RMA communication calls (MPI_GET, MPI_PUT, ...). > > > > > > Hubert > > > > > > ----------------------------------------------------------------------------- > > > > > > Wednesday, September 12, 2012 11:37 AM > > > > > > Hubert, > > > > > > This is what I was referring to. I'm in favor of this proposal. > > > > > > Torsten > > > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Fence + threads
by William Gropp 10 Jul '14

10 Jul '14

I some sense, it would be the collective group, since we're talking about collective semantics. Another option would be to make a list of cross-group issues and make the editor responsible for getting the groups together for a consistent answer. That's probably the most practical approach. Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Jul 5, 2014, at 1:32 AM, Thakur, Rajeev wrote: > BTW, which working group does this fall under? It needs to make sure that MPI_Win_fence, MPI_File_sync, and anything else for which this has not been specified are covered. > > Rajeev > > On Jul 2, 2014, at 3:12 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> I agree; I think this was just an oversight. >> >> >> On Sat, Jun 21, 2014 at 11:24 AM, William Gropp <wgropp(a)illinois.edu> wrote: >> I agree with Rajeev, and I suspect the text that only mentions communicators was an oversight - it probably came from the language in MPI-1.1 where only communicators existed. I do think that this needs to be clarified. After all, it is (as it should be) incorrect for two threads in the same process to call MPI_Barrier on the same communicator, even though we could define the semantics in this specific situation. >> >> Bill >> >> William Gropp >> Director, Parallel Computing Institute >> Thomas M. Siebel Chair in Computer Science >> University of Illinois Urbana-Champaign >> >> >> >> >> >> On Jun 20, 2014, at 1:36 PM, Rajeev Thakur wrote: >> >>> I always assumed that concurrent calls from multiple threads to collective functions on the same comm, win, or fh are not allowed. For example, there is also a collective MPI_File_sync(fh). >>> >>> Rajeev >>> >>> >>> On Jun 20, 2014, at 1:32 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>> >>>> >>>> Right. We don’t say something similar for windows. Was it intentional or just an accidental ommission? >>>> >>>> — Pavan >>>> >>>> On Jun 20, 2014, at 1:29 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>>> >>>>> FWIW, the collective communication chapter says this on 217:24-27. >>>>> >>>>> "Finally, in multithreaded implementations, one can have more than one, concurrently >>>>> executing, collective communication call at a process. In these situations, it is the user's responsibility >>>>> to ensure that the same communicator is not used concurrently by two di erent >>>>> collective communication calls at the same process." >>>>> >>>>> >>>>> >>>>> On Jun 20, 2014, at 1:18 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>>> >>>>>> >>>>>> That doesn’t disallow it. It just says the ordering is arbitrary, which is OK in my example. >>>>>> >>>>>> — Pavan >>>>>> >>>>>> On Jun 20, 2014, at 1:16 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>>>>> >>>>>>> There is this text on pg 484, ln 18-22 >>>>>>> >>>>>>> "Collective calls: Matching of collective calls on a communicator, window, or file handle is done according to the order in which the calls are issued at each process. If concurrent threads issue such calls on the same communicator, window or file handle, it is up to the user to make sure the calls are correctly ordered, using interthread synchronization." >>>>>>> >>>>>>> Rajeev >>>>>>> >>>>>>> On Jun 20, 2014, at 1:11 PM, "Balaji, Pavan" <balaji(a)anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> The standard itself doesn’t seem to disallow it. This means that, at least, MPICH is incorrect here. I’d expect most MPICH derivatives to be incorrect for this as well. >>>>>>>> >>>>>>>> However, I’m trying to understand if that was the intention (and just forgotten to describe in the standard). We never discussed this in the MPI-3 working group. >>>>>>>> >>>>>>>> — Pavan >>>>>>>> >>>>>>>> On Jun 20, 2014, at 12:46 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: >>>>>>>> >>>>>>>>> On Jun 20, 2014, at 11:16 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Is the following code correct? Two threads of a process both do the following: >>>>>>>>>> >>>>>>>>>> MPI_PUT >>>>>>>>>> MPI_PUT >>>>>>>>>> MPI_WIN_FENCE >>>>>>>>>> >>>>>>>>>> Collectives are not allowed on the same communicator simultaneously. But we don’t say anything similar for windows. >>>>>>>>> >>>>>>>>> As I read it, it should be fine according to the standard as long as you haven't specified any assertions that would cause you to be lying to the implementation. >>>>>>>>> >>>>>>>>> When we don't specifically disallow concurrent multithreaded operations, then the general threading text seems to allow this. See MPI-3, p.483, l.1-11: >>>>>>>>> >>>>>>>>> ----8<---- >>>>>>>>> The two main requirements for a thread-compliant implementation are listed below. >>>>>>>>> >>>>>>>>> 1. All MPI calls are thread-safe, i.e., two concurrently running threads may make MPI calls and the outcome will be as if the calls executed in some order, even if their execution is interleaved. >>>>>>>>> >>>>>>>>> 2. Blocking MPI calls will block the calling thread only, allowing another thread to execute, if available. The calling thread will be blocked until the event on which it is waiting occurs. Once the blocked communication is enabled and can proceed, then the call will complete and the thread will be marked runnable, within a finite time. A blocked thread will not prevent progress of other runnable threads on the same process, and will not prevent them from executing MPI calls. >>>>>>>>> ----8<---- >>>>>>>>> >>>>>>>>> So, absent some external inter-thread synchronization, it will be undefined exactly which epoch the MPI_PUTs end up in. >>>>>>>>> >>>>>>>>> -Dave >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> mpiwg-rma mailing list >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Fence + threads
by Thakur, Rajeev 05 Jul '14

05 Jul '14

BTW, which working group does this fall under? It needs to make sure that MPI_Win_fence, MPI_File_sync, and anything else for which this has not been specified are covered. Rajeev On Jul 2, 2014, at 3:12 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > I agree; I think this was just an oversight. > > > On Sat, Jun 21, 2014 at 11:24 AM, William Gropp <wgropp(a)illinois.edu> wrote: > I agree with Rajeev, and I suspect the text that only mentions communicators was an oversight - it probably came from the language in MPI-1.1 where only communicators existed. I do think that this needs to be clarified. After all, it is (as it should be) incorrect for two threads in the same process to call MPI_Barrier on the same communicator, even though we could define the semantics in this specific situation. > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > University of Illinois Urbana-Champaign > > > > > > On Jun 20, 2014, at 1:36 PM, Rajeev Thakur wrote: > >> I always assumed that concurrent calls from multiple threads to collective functions on the same comm, win, or fh are not allowed. For example, there is also a collective MPI_File_sync(fh). >> >> Rajeev >> >> >> On Jun 20, 2014, at 1:32 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >> >>> >>> Right. We don’t say something similar for windows. Was it intentional or just an accidental ommission? >>> >>> — Pavan >>> >>> On Jun 20, 2014, at 1:29 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>> >>>> FWIW, the collective communication chapter says this on 217:24-27. >>>> >>>> "Finally, in multithreaded implementations, one can have more than one, concurrently >>>> executing, collective communication call at a process. In these situations, it is the user's responsibility >>>> to ensure that the same communicator is not used concurrently by two di erent >>>> collective communication calls at the same process." >>>> >>>> >>>> >>>> On Jun 20, 2014, at 1:18 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>> >>>>> >>>>> That doesn’t disallow it. It just says the ordering is arbitrary, which is OK in my example. >>>>> >>>>> — Pavan >>>>> >>>>> On Jun 20, 2014, at 1:16 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>>>> >>>>>> There is this text on pg 484, ln 18-22 >>>>>> >>>>>> "Collective calls: Matching of collective calls on a communicator, window, or file handle is done according to the order in which the calls are issued at each process. If concurrent threads issue such calls on the same communicator, window or file handle, it is up to the user to make sure the calls are correctly ordered, using interthread synchronization." >>>>>> >>>>>> Rajeev >>>>>> >>>>>> On Jun 20, 2014, at 1:11 PM, "Balaji, Pavan" <balaji(a)anl.gov> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> The standard itself doesn’t seem to disallow it. This means that, at least, MPICH is incorrect here. I’d expect most MPICH derivatives to be incorrect for this as well. >>>>>>> >>>>>>> However, I’m trying to understand if that was the intention (and just forgotten to describe in the standard). We never discussed this in the MPI-3 working group. >>>>>>> >>>>>>> — Pavan >>>>>>> >>>>>>> On Jun 20, 2014, at 12:46 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: >>>>>>> >>>>>>>> On Jun 20, 2014, at 11:16 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Is the following code correct? Two threads of a process both do the following: >>>>>>>>> >>>>>>>>> MPI_PUT >>>>>>>>> MPI_PUT >>>>>>>>> MPI_WIN_FENCE >>>>>>>>> >>>>>>>>> Collectives are not allowed on the same communicator simultaneously. But we don’t say anything similar for windows. >>>>>>>> >>>>>>>> As I read it, it should be fine according to the standard as long as you haven't specified any assertions that would cause you to be lying to the implementation. >>>>>>>> >>>>>>>> When we don't specifically disallow concurrent multithreaded operations, then the general threading text seems to allow this. See MPI-3, p.483, l.1-11: >>>>>>>> >>>>>>>> ----8<---- >>>>>>>> The two main requirements for a thread-compliant implementation are listed below. >>>>>>>> >>>>>>>> 1. All MPI calls are thread-safe, i.e., two concurrently running threads may make MPI calls and the outcome will be as if the calls executed in some order, even if their execution is interleaved. >>>>>>>> >>>>>>>> 2. Blocking MPI calls will block the calling thread only, allowing another thread to execute, if available. The calling thread will be blocked until the event on which it is waiting occurs. Once the blocked communication is enabled and can proceed, then the call will complete and the thread will be marked runnable, within a finite time. A blocked thread will not prevent progress of other runnable threads on the same process, and will not prevent them from executing MPI calls. >>>>>>>> ----8<---- >>>>>>>> >>>>>>>> So, absent some external inter-thread synchronization, it will be undefined exactly which epoch the MPI_PUTs end up in. >>>>>>>> >>>>>>>> -Dave >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions
by Rolf Rabenseifner 04 Jul '14

04 Jul '14

Pavan, > Ticket 435 should be cleaned up ... I would recommend to substitute in ticket #435 the text Ticket #434 proposes to require some sort of synchronization by adding the following additional rule after the 6 rules on page 454: 7.An RMA operation issued at the origin after MPI_WIN_START or MPI_WIN_FENCE to a specific target, accesses the public window copy at the target that is available after the matching MPI_WIN_POST or MPI_WIN_FENCE at the target. This, however, only impacts RMA operations, but not load/store accesses on shared memory windows. by the following new text: MPI-3.0 p441:34-35 defines "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process." If a remote load/store on shared memory is not treated as an RMA operation, the fence will not synchronize a sender process issuing local store before fence and a receiver process issuing a remote load to the same memory location after the fence. MPI3.0 p442:28-33 defines MPI_Win_start: "Starts an RMA access epoch for win. RMA calls issued on win during this epoch must access only windows at processes in group. Each process in group must issue a matching call to MPI_WIN_POST. RMA accesses to each target window will be delayed, if necessary, until the target process executed the matching call to MPI_WIN_POST." If a remote load/store on shared memory is not treated as an RMA operation, then remote load/store are not valid between MPI_Win_start and MPI_Win_complete, and the post-start syncronization will not synchronize a sender process issuing a local store before post and receiver process issuing a remote load to the same memory Location after the start operation. MPI-3.0 p453:44 - p454.3 rules 2 and 3: "2. If an RMA operation is completed at the origin by a call to MPI_WIN_FENCE then the operation is completed at the target by the matching call to MPI_WIN_FENCE by the target process. 3. If an RMA operation is completed at the origin by a call to MPI_WIN_COMPLETE then the operation is completed at the target by the matching call to MPI_WIN_WAIT by the target process." If a remote load/store on shared memory is not treated as an RMA operation, then a remote store before fence or complete at a sender process will not be synchronized with a local load after the matching fence or wait at the receiver process. Such synchronizing behavior of remote and local load/stores on shared Memory windows was expected in the paper published at EuroMPI 2012 by several members of the RMA WG: Torsten Hoefler, James Dinan, Darius Buntinas, Pavan Balaji, Brian Barrett, Ron Brightwell, William Gropp, Vivek Kale, Rajeev Thakur: "MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared Memory". There are two options to fix this problem: A) To define that remote load and store on a shared memory window is treated as an RMA operation. This would imply that all one-sided sync primitives must explicit synchronize. B) To define that a remote and a local store or load are treated not as RMA Operation and to explicitely define additional process-synchronization behavior of some one-sided sync routines. The proposal below is based on B) and currently restricted to fence and post-start-complete-wait doing such process-to-process synchronization on shared memory windows. Additionally we need to modify the text to move it into the function definitions of MPI_Win_start, MPI_Win_wait, and MPI_Fence. But before this, the RMA Group should discuss A or B. Best regards Rolf ----- Original Message ----- > From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Torsten Hoefler" <htor(a)inf.ethz.ch>, "Bill Gropp" <wgropp(a)uiuc.edu> > Sent: Wednesday, July 2, 2014 9:30:23 PM > Subject: Re: [mpiwg-rma] Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions > > Ticket 435 should be cleaned up to reflect the corrections pointed > out in 434, so that we can focus specifically on the problem of > direct load/stores to shared memory. > > Rajeev > > On Jul 2, 2014, at 9:38 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> > wrote: > > > Bill, Rajeev, and all other RMA WG members, > > > > Hubert and Torsten discussed already in 2012 the meaning of > > the MPI one-sided synchronization routines for MPI-3.0 shared > > memory. > > > > This question is still unresolved in the MPI-3.0 + errata. > > > > Does the term "RMA Operation" include "a remote load/store > > from an origin process to the window Memory on a target"? > > > > Or not? > > > > The ticket #435 expects "not". > > > > In this case, MPI_Win_fence and post-start-complete-wait > > cannot be used for synchronizing the sending process of data > > with the receiving process of data that use only > > local and remote load/stores on shared memory windows. > > > > Ticket 435 extends the meaning of > > MPI_Win_fence and post-start-complete-wait that they > > provide sender-Receiver synchronization between processes > > that use local and remote load/stores > > on shared memory windows. > > > > I hope that all RMA working group members, agree that > > - currently the behavior of these sync-routines for > > shared memory remote load/stores is undefined due to > > the undefined definition of "RMA Operation" > > - and that we need an errata that resolves this problem. > > > > What is your opinion about the solution provided in > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/435 ? > > > > Best regards > > Rolf > > > > PS: Ticket 435 is the result of a discussion of Pavan, Hubert and I > > at ISC2014. > > > > ----- Original Message ----- > >> From: Hubert Ritzdorf > >> Sent: Tuesday, September 11, 2012 7:26 PM > >> To: mpi3-rma(a)lists.mpi-forum.org > >> Subject: MPI_Win_allocate_shared and synchronization functions > >> > >> Hi, > >> > >> it's quite unclear what Page 410, Lines 17-19 > >> > >> A consistent view can be created in the uni?fied > >> memory model (see Section 11.4) by utilizing the window > >> synchronization functions (see > >> Section 11.5) > >> > >> really means. Section 11.5 doesn't mention any (load/store) access > >> to > >> shared memory. > >> Thus, must > >> > >> (*) RMA communication calls and RMA operations > >> be interpreted as RMA communication calls (MPI_GET, MPI_PUT, > >> ...) and > >> ANY load/store access to shared > >> window > >> (*) put call as put call and any store to shared > >> memory > >> (*) get call as get call and any load from shared > >> memory > >> (*) accumulate call as accumulate call and any load or store > >> access > >> to shared window ? > >> > >> Example: Assertion MPI_MODE_NOPRECEDE > >> > >> Does > >> > >> the fence does not complete any sequence of locally issued RMA > >> calls > >> > >> mean for windows created by MPI_Win_Allocate_shared () > >> > >> the fence does not complete any sequence of locally issued RMA > >> calls > >> or > >> any load/store access to the window memory ? > >> > >> It's not clear to me. I will be probably not clear for the > >> standard > >> MPI user. > >> RMA operations are defined only MPI functions for window objects > >> (as far as I can see). > >> But possibly I'm totally wrong and the synchronization functions > >> synchronize > >> only the RMA communication calls (MPI_GET, MPI_PUT, ...). > >> > >> Hubert > >> > >> ----------------------------------------------------------------------------- > >> > >> Wednesday, September 12, 2012 11:37 AM > >> > >> Hubert, > >> > >> This is what I was referring to. I'm in favor of this proposal. > >> > >> Torsten > >> > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Fence + threads
by Jim Dinan 02 Jul '14

02 Jul '14

I agree; I think this was just an oversight. On Sat, Jun 21, 2014 at 11:24 AM, William Gropp <wgropp(a)illinois.edu> wrote: > I agree with Rajeev, and I suspect the text that only mentions > communicators was an oversight - it probably came from the language in > MPI-1.1 where only communicators existed. I do think that this needs to be > clarified. After all, it is (as it should be) incorrect for two threads in > the same process to call MPI_Barrier on the same communicator, even though > we could define the semantics in this specific situation. > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > University of Illinois Urbana-Champaign > > > > > > On Jun 20, 2014, at 1:36 PM, Rajeev Thakur wrote: > > I always assumed that concurrent calls from multiple threads to collective > functions on the same comm, win, or fh are not allowed. For example, there > is also a collective MPI_File_sync(fh). > > Rajeev > > > On Jun 20, 2014, at 1:32 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > > Right. We don’t say something similar for windows. Was it intentional or > just an accidental ommission? > > > — Pavan > > > On Jun 20, 2014, at 1:29 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > > > FWIW, the collective communication chapter says this on 217:24-27. > > > "Finally, in multithreaded implementations, one can have more than one, > concurrently > > executing, collective communication call at a process. In these > situations, it is the user's responsibility > > to ensure that the same communicator is not used concurrently by two di > erent > > collective communication calls at the same process." > > > > > On Jun 20, 2014, at 1:18 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > > > That doesn’t disallow it. It just says the ordering is arbitrary, which > is OK in my example. > > > — Pavan > > > On Jun 20, 2014, at 1:16 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > > > There is this text on pg 484, ln 18-22 > > > "Collective calls: Matching of collective calls on a communicator, window, > or file handle is done according to the order in which the calls are issued > at each process. If concurrent threads issue such calls on the same > communicator, window or file handle, it is up to the user to make sure the > calls are correctly ordered, using interthread synchronization." > > > Rajeev > > > On Jun 20, 2014, at 1:11 PM, "Balaji, Pavan" <balaji(a)anl.gov> > > wrote: > > > > The standard itself doesn’t seem to disallow it. This means that, at > least, MPICH is incorrect here. I’d expect most MPICH derivatives to be > incorrect for this as well. > > > However, I’m trying to understand if that was the intention (and just > forgotten to describe in the standard). We never discussed this in the > MPI-3 working group. > > > — Pavan > > > On Jun 20, 2014, at 12:46 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> > wrote: > > > On Jun 20, 2014, at 11:16 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > > > Is the following code correct? Two threads of a process both do the > following: > > > MPI_PUT > > MPI_PUT > > MPI_WIN_FENCE > > > Collectives are not allowed on the same communicator simultaneously. But > we don’t say anything similar for windows. > > > As I read it, it should be fine according to the standard as long as you > haven't specified any assertions that would cause you to be lying to the > implementation. > > > When we don't specifically disallow concurrent multithreaded operations, > then the general threading text seems to allow this. See MPI-3, p.483, > l.1-11: > > > ----8<---- > > The two main requirements for a thread-compliant implementation are listed > below. > > > 1. All MPI calls are thread-safe, i.e., two concurrently running threads > may make MPI calls and the outcome will be as if the calls executed in some > order, even if their execution is interleaved. > > > 2. Blocking MPI calls will block the calling thread only, allowing another > thread to execute, if available. The calling thread will be blocked until > the event on which it is waiting occurs. Once the blocked communication is > enabled and can proceed, then the call will complete and the thread will be > marked runnable, within a finite time. A blocked thread will not prevent > progress of other runnable threads on the same process, and will not > prevent them from executing MPI calls. > > ----8<---- > > > So, absent some external inter-thread synchronization, it will be > undefined exactly which epoch the MPI_PUTs end up in. > > > -Dave > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions
by Thakur, Rajeev 02 Jul '14

02 Jul '14

Ticket 435 should be cleaned up to reflect the corrections pointed out in 434, so that we can focus specifically on the problem of direct load/stores to shared memory. Rajeev On Jul 2, 2014, at 9:38 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Bill, Rajeev, and all other RMA WG members, > > Hubert and Torsten discussed already in 2012 the meaning of > the MPI one-sided synchronization routines for MPI-3.0 shared memory. > > This question is still unresolved in the MPI-3.0 + errata. > > Does the term "RMA Operation" include "a remote load/store > from an origin process to the window Memory on a target"? > > Or not? > > The ticket #435 expects "not". > > In this case, MPI_Win_fence and post-start-complete-wait > cannot be used for synchronizing the sending process of data > with the receiving process of data that use only > local and remote load/stores on shared memory windows. > > Ticket 435 extends the meaning of > MPI_Win_fence and post-start-complete-wait that they > provide sender-Receiver synchronization between processes > that use local and remote load/stores > on shared memory windows. > > I hope that all RMA working group members, agree that > - currently the behavior of these sync-routines for > shared memory remote load/stores is undefined due to > the undefined definition of "RMA Operation" > - and that we need an errata that resolves this problem. > > What is your opinion about the solution provided in > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/435 ? > > Best regards > Rolf > > PS: Ticket 435 is the result of a discussion of Pavan, Hubert and I at ISC2014. > > ----- Original Message ----- >> From: Hubert Ritzdorf >> Sent: Tuesday, September 11, 2012 7:26 PM >> To: mpi3-rma(a)lists.mpi-forum.org >> Subject: MPI_Win_allocate_shared and synchronization functions >> >> Hi, >> >> it's quite unclear what Page 410, Lines 17-19 >> >> A consistent view can be created in the uni?fied >> memory model (see Section 11.4) by utilizing the window >> synchronization functions (see >> Section 11.5) >> >> really means. Section 11.5 doesn't mention any (load/store) access to >> shared memory. >> Thus, must >> >> (*) RMA communication calls and RMA operations >> be interpreted as RMA communication calls (MPI_GET, MPI_PUT, >> ...) and >> ANY load/store access to shared >> window >> (*) put call as put call and any store to shared memory >> (*) get call as get call and any load from shared memory >> (*) accumulate call as accumulate call and any load or store access >> to shared window ? >> >> Example: Assertion MPI_MODE_NOPRECEDE >> >> Does >> >> the fence does not complete any sequence of locally issued RMA calls >> >> mean for windows created by MPI_Win_Allocate_shared () >> >> the fence does not complete any sequence of locally issued RMA calls >> or >> any load/store access to the window memory ? >> >> It's not clear to me. I will be probably not clear for the standard >> MPI user. >> RMA operations are defined only MPI functions for window objects >> (as far as I can see). >> But possibly I'm totally wrong and the synchronization functions >> synchronize >> only the RMA communication calls (MPI_GET, MPI_PUT, ...). >> >> Hubert >> >> ----------------------------------------------------------------------------- >> >> Wednesday, September 12, 2012 11:37 AM >> >> Hubert, >> >> This is what I was referring to. I'm in favor of this proposal. >> >> Torsten >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions
by Thakur, Rajeev 02 Jul '14

02 Jul '14

Ticket 429 (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/429) is also connected with whether remote loads/stores are to be considered as local loads/stores or RMA operations. In the discussion on the mailing list about that ticket, some said they should not be considered as local loads/stores. That discussion is now linked from the ticket. Rajeev On Jul 2, 2014, at 10:31 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: >> In general I am opposed to decreasing the performance of applications >> by mandating strong synchronization semantics. >> >> I will have to study this particular instance carefully. > I agree. > > If the user of a shared memory window only want to issue > a memory fence, he/she can use the non-collective MPI_Win_sync. > > I would expect that MPI_Win_fence and > post-start-complete-wait should be used on a shared memory window > only if the user wants process-to-process synchronization. > > In all other cases, he/she would use lock/unlock or pure win_sync. > The proposal does not touch lock/unlock and win_sync. > > A possible alternative would define remote load/store as RMA > and would terribly touch all synchronization routines. > We all would be against this, although this was the first idea > in 2012 with probably the historical remark > "This is what I was referring to. I'm in favor of this proposal." > > Rolf > > > ----- Original Message ----- >> From: "William Gropp" <wgropp(a)illinois.edu> >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >> Cc: "Bill Gropp" <wgropp(a)uiuc.edu>, "Rajeev Thakur" <thakur(a)anl.gov>, "Hubert Ritzdorf" >> <Hubert.Ritzdorf(a)EMEA.NEC.COM>, "Pavan Balaji" <balaji(a)anl.gov>, "Torsten Hoefler" <htor(a)inf.ethz.ch>, "MPI WG >> Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Wednesday, July 2, 2014 4:46:40 PM >> Subject: Re: Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions >> >> In general I am opposed to decreasing the performance of applications >> by mandating strong synchronization semantics. >> >> >> I will have to study this particular instance carefully. >> >> >> Bill >> >> >> >> >> >> >> William Gropp >> Director, Parallel Computing Institute Thomas M. Siebel Chair in >> Computer Science >> >> >> University of Illinois Urbana-Champaign >> >> >> >> >> >> >> >> On Jul 2, 2014, at 9:38 AM, Rolf Rabenseifner wrote: >> >> >> >> Bill, Rajeev, and all other RMA WG members, >> >> Hubert and Torsten discussed already in 2012 the meaning of >> the MPI one-sided synchronization routines for MPI-3.0 shared memory. >> >> This question is still unresolved in the MPI-3.0 + errata. >> >> Does the term "RMA Operation" include "a remote load/store >> from an origin process to the window Memory on a target"? >> >> Or not? >> >> The ticket #435 expects "not". >> >> In this case, MPI_Win_fence and post-start-complete-wait >> cannot be used for synchronizing the sending process of data >> with the receiving process of data that use only >> local and remote load/stores on shared memory windows. >> >> Ticket 435 extends the meaning of >> MPI_Win_fence and post-start-complete-wait that they >> provide sender-Receiver synchronization between processes >> that use local and remote load/stores >> on shared memory windows. >> >> I hope that all RMA working group members, agree that >> - currently the behavior of these sync-routines for >> shared memory remote load/stores is undefined due to >> the undefined definition of "RMA Operation" >> - and that we need an errata that resolves this problem. >> >> What is your opinion about the solution provided in >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/435 ? >> >> Best regards >> Rolf >> >> PS: Ticket 435 is the result of a discussion of Pavan, Hubert and I >> at ISC2014. >> >> ----- Original Message ----- >> >> >> From: Hubert Ritzdorf >> >> >> Sent: Tuesday, September 11, 2012 7:26 PM >> >> >> To: mpi3-rma(a)lists.mpi-forum.org >> >> >> Subject: MPI_Win_allocate_shared and synchronization functions >> >> >> >> >> >> Hi, >> >> >> >> >> >> it's quite unclear what Page 410, Lines 17-19 >> >> >> >> >> >> A consistent view can be created in the uni?fied >> >> >> memory model (see Section 11.4) by utilizing the window >> >> >> synchronization functions (see >> >> >> Section 11.5) >> >> >> >> >> >> really means. Section 11.5 doesn't mention any (load/store) access to >> >> >> shared memory. >> >> >> Thus, must >> >> >> >> >> >> (*) RMA communication calls and RMA operations >> >> >> be interpreted as RMA communication calls (MPI_GET, MPI_PUT, >> >> >> ...) and >> >> >> ANY load/store access to shared >> >> >> window >> >> >> (*) put call as put call and any store to shared memory >> >> >> (*) get call as get call and any load from shared memory >> >> >> (*) accumulate call as accumulate call and any load or store access >> >> >> to shared window ? >> >> >> >> >> >> Example: Assertion MPI_MODE_NOPRECEDE >> >> >> >> >> >> Does >> >> >> >> >> >> the fence does not complete any sequence of locally issued RMA calls >> >> >> >> >> >> mean for windows created by MPI_Win_Allocate_shared () >> >> >> >> >> >> the fence does not complete any sequence of locally issued RMA calls >> >> >> or >> >> >> any load/store access to the window memory ? >> >> >> >> >> >> It's not clear to me. I will be probably not clear for the standard >> >> >> MPI user. >> >> >> RMA operations are defined only MPI functions for window objects >> >> >> (as far as I can see). >> >> >> But possibly I'm totally wrong and the synchronization functions >> >> >> synchronize >> >> >> only the RMA communication calls (MPI_GET, MPI_PUT, ...). >> >> >> >> >> >> Hubert >> >> >> >> >> >> ----------------------------------------------------------------------------- >> >> >> >> >> >> Wednesday, September 12, 2012 11:37 AM >> >> >> >> >> >> Hubert, >> >> >> >> >> >> This is what I was referring to. I'm in favor of this proposal. >> >> >> >> >> >> Torsten >> >> >> >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >> >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions
by Hubert Ritzdorf 02 Jul '14

02 Jul '14

Hi, I don't know whether this is significant (important). There is an example in the paper "MPI+MPI: A New, Hybrid Approach to Parallel Programming with MPI Plus Shared Memory" by Torsten Hoefler, James Dinan, ?Darius Buntinas, and others on this mailing list (EuroMPI 2012) MPI_Comm_split_type(comm, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &shmcomm); MPI_Win_allocate_shared(size*sizeof(double), info, shmcomm, &mem, &win); MPI_Win_shared_query(win, north, &sz, &northptr); MPI_Win_shared_query(win, south, &sz, &southptr); MPI_Win_shared_query(win, east, &sz, &eastptr); MPI_Win_shared_query(win, west, &sz, &westptr); for(iter=0; iter<niters; ++iter) { MPI_Win_fence(0, win); // start new access and exposure epoch if(north != MPI_PROC_NULL) // the north "communication" for(int i=0; i<bx; ++i) a2[ind(i+1,0)] = northptr[ind(i+1,by)]; if(south != MPI_PROC_NULL) // the south "communication" for(int i=0; i<bx; ++i) a2[ind(i+1,by+1)] = southptr[ind(i+1,1)]; if(east != MPI_PROC_NULL) // the east "communication" for(int i=0; i<by; ++i) a2[ind(bx+1,i+1)] = eastptr[ind(1,i+1)]; if(west != MPI_PROC_NULL) // the west "communication" for(int i=0; i<by; ++i) a2[ind(0,i+1)] = westptr[ind(bx,i+1)]; update_grid(&a1, &a2); // apply operator and swap arrays } When I understand it correctly, the MPI_Win_fence() synchronizes processes in this example. Otherwise, the example would not work correctly. Hubert ________________________________________ From: Rolf Rabenseifner [rabenseifner(a)hlrs.de] Sent: Wednesday, July 02, 2014 5:31 PM To: William Gropp Cc: Bill Gropp; Rajeev Thakur; Hubert Ritzdorf; Pavan Balaji; Torsten Hoefler; MPI WG Remote Memory Access working group Subject: Re: Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions > In general I am opposed to decreasing the performance of applications > by mandating strong synchronization semantics. > > I will have to study this particular instance carefully. I agree. If the user of a shared memory window only want to issue a memory fence, he/she can use the non-collective MPI_Win_sync. I would expect that MPI_Win_fence and post-start-complete-wait should be used on a shared memory window only if the user wants process-to-process synchronization. In all other cases, he/she would use lock/unlock or pure win_sync. The proposal does not touch lock/unlock and win_sync. A possible alternative would define remote load/store as RMA and would terribly touch all synchronization routines. We all would be against this, although this was the first idea in 2012 with probably the historical remark "This is what I was referring to. I'm in favor of this proposal." Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "Bill Gropp" <wgropp(a)uiuc.edu>, "Rajeev Thakur" <thakur(a)anl.gov>, "Hubert Ritzdorf" > <Hubert.Ritzdorf(a)EMEA.NEC.COM>, "Pavan Balaji" <balaji(a)anl.gov>, "Torsten Hoefler" <htor(a)inf.ethz.ch>, "MPI WG > Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Wednesday, July 2, 2014 4:46:40 PM > Subject: Re: Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions > > In general I am opposed to decreasing the performance of applications > by mandating strong synchronization semantics. > > > I will have to study this particular instance carefully. > > > Bill > > > > > > > William Gropp > Director, Parallel Computing Institute Thomas M. Siebel Chair in > Computer Science > > > University of Illinois Urbana-Champaign > > > > > > > > On Jul 2, 2014, at 9:38 AM, Rolf Rabenseifner wrote: > > > > Bill, Rajeev, and all other RMA WG members, > > Hubert and Torsten discussed already in 2012 the meaning of > the MPI one-sided synchronization routines for MPI-3.0 shared memory. > > This question is still unresolved in the MPI-3.0 + errata. > > Does the term "RMA Operation" include "a remote load/store > from an origin process to the window Memory on a target"? > > Or not? > > The ticket #435 expects "not". > > In this case, MPI_Win_fence and post-start-complete-wait > cannot be used for synchronizing the sending process of data > with the receiving process of data that use only > local and remote load/stores on shared memory windows. > > Ticket 435 extends the meaning of > MPI_Win_fence and post-start-complete-wait that they > provide sender-Receiver synchronization between processes > that use local and remote load/stores > on shared memory windows. > > I hope that all RMA working group members, agree that > - currently the behavior of these sync-routines for > shared memory remote load/stores is undefined due to > the undefined definition of "RMA Operation" > - and that we need an errata that resolves this problem. > > What is your opinion about the solution provided in > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/435 ? > > Best regards > Rolf > > PS: Ticket 435 is the result of a discussion of Pavan, Hubert and I > at ISC2014. > > ----- Original Message ----- > > > From: Hubert Ritzdorf > > > Sent: Tuesday, September 11, 2012 7:26 PM > > > To: mpi3-rma(a)lists.mpi-forum.org > > > Subject: MPI_Win_allocate_shared and synchronization functions > > > > > > Hi, > > > > > > it's quite unclear what Page 410, Lines 17-19 > > > > > > A consistent view can be created in the uni?fied > > > memory model (see Section 11.4) by utilizing the window > > > synchronization functions (see > > > Section 11.5) > > > > > > really means. Section 11.5 doesn't mention any (load/store) access to > > > shared memory. > > > Thus, must > > > > > > (*) RMA communication calls and RMA operations > > > be interpreted as RMA communication calls (MPI_GET, MPI_PUT, > > > ...) and > > > ANY load/store access to shared > > > window > > > (*) put call as put call and any store to shared memory > > > (*) get call as get call and any load from shared memory > > > (*) accumulate call as accumulate call and any load or store access > > > to shared window ? > > > > > > Example: Assertion MPI_MODE_NOPRECEDE > > > > > > Does > > > > > > the fence does not complete any sequence of locally issued RMA calls > > > > > > mean for windows created by MPI_Win_Allocate_shared () > > > > > > the fence does not complete any sequence of locally issued RMA calls > > > or > > > any load/store access to the window memory ? > > > > > > It's not clear to me. I will be probably not clear for the standard > > > MPI user. > > > RMA operations are defined only MPI functions for window objects > > > (as far as I can see). > > > But possibly I'm totally wrong and the synchronization functions > > > synchronize > > > only the RMA communication calls (MPI_GET, MPI_PUT, ...). > > > > > > Hubert > > > > > > ----------------------------------------------------------------------------- > > > > > > Wednesday, September 12, 2012 11:37 AM > > > > > > Hubert, > > > > > > This is what I was referring to. I'm in favor of this proposal. > > > > > > Torsten > > > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) Click https://www.mailcontrol.com/sr/QhgXRFXs23fGX2PQPOmvUgMAe56rOQQtRy!ZSQ6Yj1df… to report this email as spam.

1 0

Re: [mpiwg-rma] Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions
by Rolf Rabenseifner 02 Jul '14

02 Jul '14

> In general I am opposed to decreasing the performance of applications > by mandating strong synchronization semantics. > > I will have to study this particular instance carefully. I agree. If the user of a shared memory window only want to issue a memory fence, he/she can use the non-collective MPI_Win_sync. I would expect that MPI_Win_fence and post-start-complete-wait should be used on a shared memory window only if the user wants process-to-process synchronization. In all other cases, he/she would use lock/unlock or pure win_sync. The proposal does not touch lock/unlock and win_sync. A possible alternative would define remote load/store as RMA and would terribly touch all synchronization routines. We all would be against this, although this was the first idea in 2012 with probably the historical remark "This is what I was referring to. I'm in favor of this proposal." Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "Bill Gropp" <wgropp(a)uiuc.edu>, "Rajeev Thakur" <thakur(a)anl.gov>, "Hubert Ritzdorf" > <Hubert.Ritzdorf(a)EMEA.NEC.COM>, "Pavan Balaji" <balaji(a)anl.gov>, "Torsten Hoefler" <htor(a)inf.ethz.ch>, "MPI WG > Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Wednesday, July 2, 2014 4:46:40 PM > Subject: Re: Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions > > In general I am opposed to decreasing the performance of applications > by mandating strong synchronization semantics. > > > I will have to study this particular instance carefully. > > > Bill > > > > > > > William Gropp > Director, Parallel Computing Institute Thomas M. Siebel Chair in > Computer Science > > > University of Illinois Urbana-Champaign > > > > > > > > On Jul 2, 2014, at 9:38 AM, Rolf Rabenseifner wrote: > > > > Bill, Rajeev, and all other RMA WG members, > > Hubert and Torsten discussed already in 2012 the meaning of > the MPI one-sided synchronization routines for MPI-3.0 shared memory. > > This question is still unresolved in the MPI-3.0 + errata. > > Does the term "RMA Operation" include "a remote load/store > from an origin process to the window Memory on a target"? > > Or not? > > The ticket #435 expects "not". > > In this case, MPI_Win_fence and post-start-complete-wait > cannot be used for synchronizing the sending process of data > with the receiving process of data that use only > local and remote load/stores on shared memory windows. > > Ticket 435 extends the meaning of > MPI_Win_fence and post-start-complete-wait that they > provide sender-Receiver synchronization between processes > that use local and remote load/stores > on shared memory windows. > > I hope that all RMA working group members, agree that > - currently the behavior of these sync-routines for > shared memory remote load/stores is undefined due to > the undefined definition of "RMA Operation" > - and that we need an errata that resolves this problem. > > What is your opinion about the solution provided in > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/435 ? > > Best regards > Rolf > > PS: Ticket 435 is the result of a discussion of Pavan, Hubert and I > at ISC2014. > > ----- Original Message ----- > > > From: Hubert Ritzdorf > > > Sent: Tuesday, September 11, 2012 7:26 PM > > > To: mpi3-rma(a)lists.mpi-forum.org > > > Subject: MPI_Win_allocate_shared and synchronization functions > > > > > > Hi, > > > > > > it's quite unclear what Page 410, Lines 17-19 > > > > > > A consistent view can be created in the uni?fied > > > memory model (see Section 11.4) by utilizing the window > > > synchronization functions (see > > > Section 11.5) > > > > > > really means. Section 11.5 doesn't mention any (load/store) access to > > > shared memory. > > > Thus, must > > > > > > (*) RMA communication calls and RMA operations > > > be interpreted as RMA communication calls (MPI_GET, MPI_PUT, > > > ...) and > > > ANY load/store access to shared > > > window > > > (*) put call as put call and any store to shared memory > > > (*) get call as get call and any load from shared memory > > > (*) accumulate call as accumulate call and any load or store access > > > to shared window ? > > > > > > Example: Assertion MPI_MODE_NOPRECEDE > > > > > > Does > > > > > > the fence does not complete any sequence of locally issued RMA calls > > > > > > mean for windows created by MPI_Win_Allocate_shared () > > > > > > the fence does not complete any sequence of locally issued RMA calls > > > or > > > any load/store access to the window memory ? > > > > > > It's not clear to me. I will be probably not clear for the standard > > > MPI user. > > > RMA operations are defined only MPI functions for window objects > > > (as far as I can see). > > > But possibly I'm totally wrong and the synchronization functions > > > synchronize > > > only the RMA communication calls (MPI_GET, MPI_PUT, ...). > > > > > > Hubert > > > > > > ----------------------------------------------------------------------------- > > > > > > Wednesday, September 12, 2012 11:37 AM > > > > > > Hubert, > > > > > > This is what I was referring to. I'm in favor of this proposal. > > > > > > Torsten > > > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Ticket 435 and Re: MPI_Win_allocate_shared and synchronization functions
by William Gropp 02 Jul '14

02 Jul '14

In general I am opposed to decreasing the performance of applications by mandating strong synchronization semantics. I will have to study this particular instance carefully. Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Jul 2, 2014, at 9:38 AM, Rolf Rabenseifner wrote: > Bill, Rajeev, and all other RMA WG members, > > Hubert and Torsten discussed already in 2012 the meaning of > the MPI one-sided synchronization routines for MPI-3.0 shared memory. > > This question is still unresolved in the MPI-3.0 + errata. > > Does the term "RMA Operation" include "a remote load/store > from an origin process to the window Memory on a target"? > > Or not? > > The ticket #435 expects "not". > > In this case, MPI_Win_fence and post-start-complete-wait > cannot be used for synchronizing the sending process of data > with the receiving process of data that use only > local and remote load/stores on shared memory windows. > > Ticket 435 extends the meaning of > MPI_Win_fence and post-start-complete-wait that they > provide sender-Receiver synchronization between processes > that use local and remote load/stores > on shared memory windows. > > I hope that all RMA working group members, agree that > - currently the behavior of these sync-routines for > shared memory remote load/stores is undefined due to > the undefined definition of "RMA Operation" > - and that we need an errata that resolves this problem. > > What is your opinion about the solution provided in > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/435 ? > > Best regards > Rolf > > PS: Ticket 435 is the result of a discussion of Pavan, Hubert and I at ISC2014. > > ----- Original Message ----- >> From: Hubert Ritzdorf >> Sent: Tuesday, September 11, 2012 7:26 PM >> To: mpi3-rma(a)lists.mpi-forum.org >> Subject: MPI_Win_allocate_shared and synchronization functions >> >> Hi, >> >> it's quite unclear what Page 410, Lines 17-19 >> >> A consistent view can be created in the uni?fied >> memory model (see Section 11.4) by utilizing the window >> synchronization functions (see >> Section 11.5) >> >> really means. Section 11.5 doesn't mention any (load/store) access to >> shared memory. >> Thus, must >> >> (*) RMA communication calls and RMA operations >> be interpreted as RMA communication calls (MPI_GET, MPI_PUT, >> ...) and >> ANY load/store access to shared >> window >> (*) put call as put call and any store to shared memory >> (*) get call as get call and any load from shared memory >> (*) accumulate call as accumulate call and any load or store access >> to shared window ? >> >> Example: Assertion MPI_MODE_NOPRECEDE >> >> Does >> >> the fence does not complete any sequence of locally issued RMA calls >> >> mean for windows created by MPI_Win_Allocate_shared () >> >> the fence does not complete any sequence of locally issued RMA calls >> or >> any load/store access to the window memory ? >> >> It's not clear to me. I will be probably not clear for the standard >> MPI user. >> RMA operations are defined only MPI functions for window objects >> (as far as I can see). >> But possibly I'm totally wrong and the synchronization functions >> synchronize >> only the RMA communication calls (MPI_GET, MPI_PUT, ...). >> >> Hubert >> >> ----------------------------------------------------------------------------- >> >> Wednesday, September 12, 2012 11:37 AM >> >> Hubert, >> >> This is what I was referring to. I'm in favor of this proposal. >> >> Torsten >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0