June 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Added 4 extra tickets to the RMA wiki
by Jeff Hammond 25 Jun '14

25 Jun '14

This is my guess: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi3-rma-agenda?action=di… But I agree some more detail from Pavan would help. And I want to add at least one more to the docket. Jeff Sent from my iPhone > On Jun 25, 2014, at 6:38 AM, William Gropp <wgropp(a)illinois.edu> wrote: > > What do you want to discuss? This was the agenda to the March meeting; are you asking about the September meeting? > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > University of Illinois Urbana-Champaign > > > > > >> On Jun 25, 2014, at 4:22 AM, Balaji, Pavan wrote: >> >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi3-rma-agenda >> >> RMA WG chairs: Can we schedule some telecon time to discuss this? >> >> — Pavan >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Added 4 extra tickets to the RMA wiki
by William Gropp 25 Jun '14

25 Jun '14

What do you want to discuss? This was the agenda to the March meeting; are you asking about the September meeting? Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Jun 25, 2014, at 4:22 AM, Balaji, Pavan wrote: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi3-rma-agenda > > RMA WG chairs: Can we schedule some telecon time to discuss this? > > — Pavan > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Added 4 extra tickets to the RMA wiki
by Rajeev Thakur 25 Jun '14

25 Jun '14

Are they listed under "The following (tickets) have been passed…"? Rajeev On Jun 25, 2014, at 4:22 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi3-rma-agenda > > RMA WG chairs: Can we schedule some telecon time to discuss this? > > — Pavan > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Added 4 extra tickets to the RMA wiki
by Balaji, Pavan 25 Jun '14

25 Jun '14

https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi3-rma-agenda RMA WG chairs: Can we schedule some telecon time to discuss this? — Pavan

1 0

Re: [mpiwg-rma] Fence + threads
by maik peterson 24 Jun '14

24 Jun '14

pavan, what a disaster for such a simple operation. go ahead mpi rma... 2014-06-20 18:16 GMT+02:00 Balaji, Pavan <balaji(a)anl.gov>: > > Is the following code correct? Two threads of a process both do the > following: > > MPI_PUT > MPI_PUT > MPI_WIN_FENCE > > Collectives are not allowed on the same communicator simultaneously. But > we don’t say anything similar for windows. > > — Pavan > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] Fence + threads
by Rolf Rabenseifner 22 Jun '14

22 Jun '14

Are you sure that you want to use MPI_Fence? MPI_Fence is collective because a Group of processes (here threads in different proceeses) have done some RMA and want to tell all others in the group that they did. If they do not know to whom they want to tell some information, and only these receivers want to be blocked, then non-collective methods based on MPI_Lock/Unlock may be more appropriate? If I understand right, then you have the following issues: - the MPI_PUT is locally finished - the data stored with MPI_PUT is remotely accessible - some other thread are informed about that this data is now available. - some restrictions on who knows about who needs the information, i.e., a fence over all threads of all processes is the wrong way. Rolf ----- Original Message ----- > From: "Jed Brown" <jedbrown(a)mcs.anl.gov> > To: "William Gropp" <wgropp(a)illinois.edu>, "MPI WG Remote Memory Access working group" > <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Saturday, June 21, 2014 11:12:49 PM > Subject: Re: [mpiwg-rma] Fence + threads > > William Gropp <wgropp(a)illinois.edu> writes: > > > There are lots of choices. Passive target can be used. Or a > > single > > thread can call MPI_Win_fence - all threads can call the RMA > > communication routines. > > The point of my example was that the threads are doing divergent > tasks, > so wrapping a user mutex around MPI_Win_fence makes the algorithm > more > synchronous and may result in idle/spinning threads. > > The argument that passive target can be used isn't compelling to me > since it also applies in other cases that we use to justify active > target. > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Fence + threads
by Jed Brown 21 Jun '14

21 Jun '14

William Gropp <wgropp(a)illinois.edu> writes: > There are lots of choices. Passive target can be used. Or a single > thread can call MPI_Win_fence - all threads can call the RMA > communication routines. The point of my example was that the threads are doing divergent tasks, so wrapping a user mutex around MPI_Win_fence makes the algorithm more synchronous and may result in idle/spinning threads. The argument that passive target can be used isn't compelling to me since it also applies in other cases that we use to justify active target.

1 0

Re: [mpiwg-rma] Fence + threads
by William Gropp 21 Jun '14

21 Jun '14

There are lots of choices. Passive target can be used. Or a single thread can call MPI_Win_fence - all threads can call the RMA communication routines. Its only having multiple threads making a collective call to MPI_Win_fence on the same window that is an issue (and not consistent with how MPI handles other collective operations). Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Jun 21, 2014, at 10:24 AM, William Gropp wrote: > I agree with Rajeev, and I suspect the text that only mentions communicators was an oversight - it probably came from the language in MPI-1.1 where only communicators existed. I do think that this needs to be clarified. After all, it is (as it should be) incorrect for two threads in the same process to call MPI_Barrier on the same communicator, even though we could define the semantics in this specific situation. > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > University of Illinois Urbana-Champaign > > > > > > On Jun 20, 2014, at 1:36 PM, Rajeev Thakur wrote: > >> I always assumed that concurrent calls from multiple threads to collective functions on the same comm, win, or fh are not allowed. For example, there is also a collective MPI_File_sync(fh). >> >> Rajeev >> >> >> On Jun 20, 2014, at 1:32 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >> >>> >>> Right. We don’t say something similar for windows. Was it intentional or just an accidental ommission? >>> >>> — Pavan >>> >>> On Jun 20, 2014, at 1:29 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>> >>>> FWIW, the collective communication chapter says this on 217:24-27. >>>> >>>> "Finally, in multithreaded implementations, one can have more than one, concurrently >>>> executing, collective communication call at a process. In these situations, it is the user's responsibility >>>> to ensure that the same communicator is not used concurrently by two dierent >>>> collective communication calls at the same process." >>>> >>>> >>>> >>>> On Jun 20, 2014, at 1:18 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>> >>>>> >>>>> That doesn’t disallow it. It just says the ordering is arbitrary, which is OK in my example. >>>>> >>>>> — Pavan >>>>> >>>>> On Jun 20, 2014, at 1:16 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>>>> >>>>>> There is this text on pg 484, ln 18-22 >>>>>> >>>>>> "Collective calls: Matching of collective calls on a communicator, window, or file handle is done according to the order in which the calls are issued at each process. If concurrent threads issue such calls on the same communicator, window or file handle, it is up to the user to make sure the calls are correctly ordered, using interthread synchronization." >>>>>> >>>>>> Rajeev >>>>>> >>>>>> On Jun 20, 2014, at 1:11 PM, "Balaji, Pavan" <balaji(a)anl.gov> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> The standard itself doesn’t seem to disallow it. This means that, at least, MPICH is incorrect here. I’d expect most MPICH derivatives to be incorrect for this as well. >>>>>>> >>>>>>> However, I’m trying to understand if that was the intention (and just forgotten to describe in the standard). We never discussed this in the MPI-3 working group. >>>>>>> >>>>>>> — Pavan >>>>>>> >>>>>>> On Jun 20, 2014, at 12:46 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: >>>>>>> >>>>>>>> On Jun 20, 2014, at 11:16 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Is the following code correct? Two threads of a process both do the following: >>>>>>>>> >>>>>>>>> MPI_PUT >>>>>>>>> MPI_PUT >>>>>>>>> MPI_WIN_FENCE >>>>>>>>> >>>>>>>>> Collectives are not allowed on the same communicator simultaneously. But we don’t say anything similar for windows. >>>>>>>> >>>>>>>> As I read it, it should be fine according to the standard as long as you haven't specified any assertions that would cause you to be lying to the implementation. >>>>>>>> >>>>>>>> When we don't specifically disallow concurrent multithreaded operations, then the general threading text seems to allow this. See MPI-3, p.483, l.1-11: >>>>>>>> >>>>>>>> ----8<---- >>>>>>>> The two main requirements for a thread-compliant implementation are listed below. >>>>>>>> >>>>>>>> 1. All MPI calls are thread-safe, i.e., two concurrently running threads may make MPI calls and the outcome will be as if the calls executed in some order, even if their execution is interleaved. >>>>>>>> >>>>>>>> 2. Blocking MPI calls will block the calling thread only, allowing another thread to execute, if available. The calling thread will be blocked until the event on which it is waiting occurs. Once the blocked communication is enabled and can proceed, then the call will complete and the thread will be marked runnable, within a finite time. A blocked thread will not prevent progress of other runnable threads on the same process, and will not prevent them from executing MPI calls. >>>>>>>> ----8<---- >>>>>>>> >>>>>>>> So, absent some external inter-thread synchronization, it will be undefined exactly which epoch the MPI_PUTs end up in. >>>>>>>> >>>>>>>> -Dave >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Fence + threads
by Jed Brown 21 Jun '14

21 Jun '14

William Gropp <wgropp(a)illinois.edu> writes: > After all, it is (as it should be) incorrect for two threads in the > same process to call MPI_Barrier on the same communicator, even though > we could define the semantics in this specific situation. I don't see a serious use case for that, and it's natural to lump in with the other collectives. Consider this use case for MPI_Win_fence: In a multi-layer geophysical flow, there exists stiff coupling in the vertical with sufficiently large data sizes to justify a horizontal-only decomposition over MPI ranks, but some other processes are contained within layers (e.g., gravity waves in the ocean) which are subcycled, justifying a vertical decomposition across threads. Different layers may subcycle a different number of times. Bill has a cartoon slide advocating decompositions of this sort for future systems. A similar layout could arise in non-square dense linear algebra where instead of creating 1D subcomms for row and column blocks, one of the decomposition directions only uses threads. I would expect the computation to be more regular for DLA and thus less compelling than the stratified flow example, but irregularity would be present for adaptive H-matrix methods, for example. If someone wants to use windows for the MPI communication in this case and if MPI_Win_fence cannot be overlapped from threads, what should the user do? Create different windows for each layer? What if their scheme depends on adjacent layers (so that there is overlap)?

1 0

Re: [mpiwg-rma] Fence + threads
by William Gropp 21 Jun '14

21 Jun '14

I agree with Rajeev, and I suspect the text that only mentions communicators was an oversight - it probably came from the language in MPI-1.1 where only communicators existed. I do think that this needs to be clarified. After all, it is (as it should be) incorrect for two threads in the same process to call MPI_Barrier on the same communicator, even though we could define the semantics in this specific situation. Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Jun 20, 2014, at 1:36 PM, Rajeev Thakur wrote: > I always assumed that concurrent calls from multiple threads to collective functions on the same comm, win, or fh are not allowed. For example, there is also a collective MPI_File_sync(fh). > > Rajeev > > > On Jun 20, 2014, at 1:32 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > >> >> Right. We don’t say something similar for windows. Was it intentional or just an accidental ommission? >> >> — Pavan >> >> On Jun 20, 2014, at 1:29 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> >>> FWIW, the collective communication chapter says this on 217:24-27. >>> >>> "Finally, in multithreaded implementations, one can have more than one, concurrently >>> executing, collective communication call at a process. In these situations, it is the user's responsibility >>> to ensure that the same communicator is not used concurrently by two dierent >>> collective communication calls at the same process." >>> >>> >>> >>> On Jun 20, 2014, at 1:18 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>> >>>> >>>> That doesn’t disallow it. It just says the ordering is arbitrary, which is OK in my example. >>>> >>>> — Pavan >>>> >>>> On Jun 20, 2014, at 1:16 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>>> >>>>> There is this text on pg 484, ln 18-22 >>>>> >>>>> "Collective calls: Matching of collective calls on a communicator, window, or file handle is done according to the order in which the calls are issued at each process. If concurrent threads issue such calls on the same communicator, window or file handle, it is up to the user to make sure the calls are correctly ordered, using interthread synchronization." >>>>> >>>>> Rajeev >>>>> >>>>> On Jun 20, 2014, at 1:11 PM, "Balaji, Pavan" <balaji(a)anl.gov> >>>>> wrote: >>>>> >>>>>> >>>>>> The standard itself doesn’t seem to disallow it. This means that, at least, MPICH is incorrect here. I’d expect most MPICH derivatives to be incorrect for this as well. >>>>>> >>>>>> However, I’m trying to understand if that was the intention (and just forgotten to describe in the standard). We never discussed this in the MPI-3 working group. >>>>>> >>>>>> — Pavan >>>>>> >>>>>> On Jun 20, 2014, at 12:46 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: >>>>>> >>>>>>> On Jun 20, 2014, at 11:16 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>>>>> >>>>>>>> >>>>>>>> Is the following code correct? Two threads of a process both do the following: >>>>>>>> >>>>>>>> MPI_PUT >>>>>>>> MPI_PUT >>>>>>>> MPI_WIN_FENCE >>>>>>>> >>>>>>>> Collectives are not allowed on the same communicator simultaneously. But we don’t say anything similar for windows. >>>>>>> >>>>>>> As I read it, it should be fine according to the standard as long as you haven't specified any assertions that would cause you to be lying to the implementation. >>>>>>> >>>>>>> When we don't specifically disallow concurrent multithreaded operations, then the general threading text seems to allow this. See MPI-3, p.483, l.1-11: >>>>>>> >>>>>>> ----8<---- >>>>>>> The two main requirements for a thread-compliant implementation are listed below. >>>>>>> >>>>>>> 1. All MPI calls are thread-safe, i.e., two concurrently running threads may make MPI calls and the outcome will be as if the calls executed in some order, even if their execution is interleaved. >>>>>>> >>>>>>> 2. Blocking MPI calls will block the calling thread only, allowing another thread to execute, if available. The calling thread will be blocked until the event on which it is waiting occurs. Once the blocked communication is enabled and can proceed, then the call will complete and the thread will be marked runnable, within a finite time. A blocked thread will not prevent progress of other runnable threads on the same process, and will not prevent them from executing MPI calls. >>>>>>> ----8<---- >>>>>>> >>>>>>> So, absent some external inter-thread synchronization, it will be undefined exactly which epoch the MPI_PUTs end up in. >>>>>>> >>>>>>> -Dave >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0