March 2011 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 16 Mar '11

16 Mar '11

Right. Some things that need to be touched up are that flush can only be called within lock-unlock currently. Also flush takes a target rank, which is the rank passed to the put/get. Flush_all has the notion of all ranks used in preceding puts/gets. Such things would need to be addressed. Rajeev On Mar 15, 2011, at 11:22 PM, Torsten Hoefler wrote: > On Tue, Mar 15, 2011 at 09:39:24PM -0500, Rajeev Thakur wrote: >> That's what I meant in my first mail. We need to say whether this example will work or not. >> >> A=10 >> barrier barrier >> x = A >> >> If not, which synchronization functions need to be used and how? The >> current text does not automatically cover this example. > Right, I never claimed that the proposal is complete :-). It's basically > a quick poll to see if there are major flaws and I should retract or if > it is worth discussing at the Forum (to flesh out such things like you > mentioned). > > To answer your question: I think this should either be: > > A=10 > MPI_Win_flush > barrier barrier > x=A > > This would basically insert a membar in MPI_Win_flush which makes > sure that A is committed before the barrier completes (a barrier flag is > set and read). > > This fits our current semantics where we suggest that win_flush can be > used for ordering. Keep in mind that it's still the unified model, i.e., > the value would commit eventually, the flush is just needed to enforce > ordering with regards to the barrier. > > Thanks & Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > "A couple of months in the laboratory can frequently save a couple of > hours in the library." - Westheimer (contemporary) > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by James Dinan 15 Mar '11

15 Mar '11

Hi Torsten, This sounds promising, I think we should pursue it. I initially had a question about how the allocation would be freed, but then remembered that we've covered this already for MPI_Win_allocate (Section 11.2.4 of the RMA proposal): "If the window was created with MPI_WIN_ALLOCATE, MPI_WIN_FREE will free the window memory that was allocated in MPI_WIN_ALLOCATE." Best, ~Jim. On 3/15/11 11:22 PM, Torsten Hoefler wrote: > On Tue, Mar 15, 2011 at 09:39:24PM -0500, Rajeev Thakur wrote: >> That's what I meant in my first mail. We need to say whether this example will work or not. >> >> A=10 >> barrier barrier >> x = A >> >> If not, which synchronization functions need to be used and how? The >> current text does not automatically cover this example. > Right, I never claimed that the proposal is complete :-). It's basically > a quick poll to see if there are major flaws and I should retract or if > it is worth discussing at the Forum (to flesh out such things like you > mentioned). > > To answer your question: I think this should either be: > > A=10 > MPI_Win_flush > barrier barrier > x=A > > This would basically insert a membar in MPI_Win_flush which makes > sure that A is committed before the barrier completes (a barrier flag is > set and read). > > This fits our current semantics where we suggest that win_flush can be > used for ordering. Keep in mind that it's still the unified model, i.e., > the value would commit eventually, the flush is just needed to enforce > ordering with regards to the barrier. > > Thanks& Best, > Torsten >

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 09:39:24PM -0500, Rajeev Thakur wrote: > That's what I meant in my first mail. We need to say whether this example will work or not. > > A=10 > barrier barrier > x = A > > If not, which synchronization functions need to be used and how? The > current text does not automatically cover this example. Right, I never claimed that the proposal is complete :-). It's basically a quick poll to see if there are major flaws and I should retract or if it is worth discussing at the Forum (to flesh out such things like you mentioned). To answer your question: I think this should either be: A=10 MPI_Win_flush barrier barrier x=A This would basically insert a membar in MPI_Win_flush which makes sure that A is committed before the barrier completes (a barrier flag is set and read). This fits our current semantics where we suggest that win_flush can be used for ordering. Keep in mind that it's still the unified model, i.e., the value would commit eventually, the flush is just needed to enforce ordering with regards to the barrier. Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- "A couple of months in the laboratory can frequently save a couple of hours in the library." - Westheimer (contemporary)

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 09:48:40PM -0500, Rajeev Thakur wrote: > Also, at the last meeting we had trouble defining the MPI_SHM_Fence > function. Dave said we won't be able to define it portably for all > processor architectures wrt memory orderings etc. Someone else pointed > out that, in Fortran, code can be moved around it. Those issues won't > go away by effectively renaming MPI_SHM_Fence to MPI_Win_flush. (These > issues are wrt direct loads/stores to shared memory.) I know, those issues are the same for a fast implementation on a shared memory system (one that doesn't do a membar in every put/get/...). So I think this issue is orthogonal and independent of the interface. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 15 Mar '11

15 Mar '11

Also, at the last meeting we had trouble defining the MPI_SHM_Fence function. Dave said we won't be able to define it portably for all processor architectures wrt memory orderings etc. Someone else pointed out that, in Fortran, code can be moved around it. Those issues won't go away by effectively renaming MPI_SHM_Fence to MPI_Win_flush. (These issues are wrt direct loads/stores to shared memory.) Rajeev On Mar 15, 2011, at 9:39 PM, Rajeev Thakur wrote: > That's what I meant in my first mail. We need to say whether this example will work or not. > > A=10 > barrier barrier > x = A > > If not, which synchronization functions need to be used and how? The current text does not automatically cover this example. > > Rajeev > > > On Mar 15, 2011, at 9:28 PM, Torsten Hoefler wrote: > >> On Tue, Mar 15, 2011 at 09:22:49PM -0500, Rajeev Thakur wrote: >>> All can be used, but none are needed, right? >> We might need to talk about synch or flush. Some shared memory systems >> are loosely coupled and need membars to guarantee correct ordering. >> >>> Currently, in the unified model you can do: >>> >>> lock >>> put(A) >>> unlock >>> barrier barrier >>> x = A >>> >>> With shared memory, you could do >>> >>> A=10 >>> barrier barrier >>> x = A >>> >>> Right? >> Not really. The barrier and the x=A might be re-ordered. This is also >> the biggest issue with our definition of the unified model, some shared >> memory systems might actually not be able to support it :-(. We had a >> long discussion in Stuttgart about this. >> >>> If the above is true, it might be a bit confusing to users that in the >>> RMA chapter you use put, get, and synchronization functions for some >>> things and can directly read/write memory for some other things. >> Well, the standard is not a user documentation :-). But yes, having this >> special kind of window requires some effort to understand it. >> >> Best, >> Torsten >> >> -- >> bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- >> Torsten Hoefler | Performance Modeling and Simulation Lead >> Blue Waters Directorate | University of Illinois (UIUC) >> 1205 W Clark Street | Urbana, IL, 61801 >> NCSA Building | +01 (217) 244-7736 >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 15 Mar '11

15 Mar '11

That's what I meant in my first mail. We need to say whether this example will work or not. A=10 barrier barrier x = A If not, which synchronization functions need to be used and how? The current text does not automatically cover this example. Rajeev On Mar 15, 2011, at 9:28 PM, Torsten Hoefler wrote: > On Tue, Mar 15, 2011 at 09:22:49PM -0500, Rajeev Thakur wrote: >> All can be used, but none are needed, right? > We might need to talk about synch or flush. Some shared memory systems > are loosely coupled and need membars to guarantee correct ordering. > >> Currently, in the unified model you can do: >> >> lock >> put(A) >> unlock >> barrier barrier >> x = A >> >> With shared memory, you could do >> >> A=10 >> barrier barrier >> x = A >> >> Right? > Not really. The barrier and the x=A might be re-ordered. This is also > the biggest issue with our definition of the unified model, some shared > memory systems might actually not be able to support it :-(. We had a > long discussion in Stuttgart about this. > >> If the above is true, it might be a bit confusing to users that in the >> RMA chapter you use put, get, and synchronization functions for some >> things and can directly read/write memory for some other things. > Well, the standard is not a user documentation :-). But yes, having this > special kind of window requires some effort to understand it. > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 09:22:49PM -0500, Rajeev Thakur wrote: > All can be used, but none are needed, right? We might need to talk about synch or flush. Some shared memory systems are loosely coupled and need membars to guarantee correct ordering. > Currently, in the unified model you can do: > > lock > put(A) > unlock > barrier barrier > x = A > > With shared memory, you could do > > A=10 > barrier barrier > x = A > > Right? Not really. The barrier and the x=A might be re-ordered. This is also the biggest issue with our definition of the unified model, some shared memory systems might actually not be able to support it :-(. We had a long discussion in Stuttgart about this. > If the above is true, it might be a bit confusing to users that in the > RMA chapter you use put, get, and synchronization functions for some > things and can directly read/write memory for some other things. Well, the standard is not a user documentation :-). But yes, having this special kind of window requires some effort to understand it. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 15 Mar '11

15 Mar '11

All can be used, but none are needed, right? Currently, in the unified model you can do: lock put(A) unlock barrier barrier x = A With shared memory, you could do A=10 barrier barrier x = A Right? If the above is true, it might be a bit confusing to users that in the RMA chapter you use put, get, and synchronization functions for some things and can directly read/write memory for some other things. Rajeev On Mar 15, 2011, at 9:17 PM, Torsten Hoefler wrote: > On Tue, Mar 15, 2011 at 09:03:40PM -0500, Rajeev Thakur wrote: >> Since processes sharing memory can/will access it directly using >> memory references (not put/get), and this proposal is for the unified >> memory model, we need to say whether any of the synchronization >> functions in the RMA chapter are needed at all for such usage. > Well, put/get to local windows are also allowed, so this would not be an > exception. It's even nicely defined because the chapter defines the > behavior for overlapping windows, and a shared window could be treated > semantically like multiple overlapping windows. I am not sure if all are > needed but I would say that all can be used. > > Yes, load/store accesses would probably be the most common use (plus > maybe the new (portable) atomics!). > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 06:44:49PM -0700, Bronis R. de Supinski wrote: > > Torsten: > > I suggested a joint meeting because that will make it > clear on the schedule. I was not thinking about adding > administrative overhead. I would like to be present. > I will likely miss the Tuesday morning sessions due to > a long telecon that I need to attend. Ok. This decision is up to the WG chairs. > I think some sort of "best effort" wording will be needed. > It is possible that processes are on the same node but > that no shared memory is available. In that case distinct > physical memory regions could be returned. Yes. > I am not quite following the group comments. Are you saying > that the different windows are returned to different shared > memory domains? Yes > I thought the window would be available to all processes but > physically shared memory is used when possible. Well, this would be an option too -- but I think it's not catching the gist of the idea. > How is the group returned? How do you query it? MPI_Win_get_group :-). Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 09:03:40PM -0500, Rajeev Thakur wrote: > Since processes sharing memory can/will access it directly using > memory references (not put/get), and this proposal is for the unified > memory model, we need to say whether any of the synchronization > functions in the RMA chapter are needed at all for such usage. Well, put/get to local windows are also allowed, so this would not be an exception. It's even nicely defined because the chapter defines the behavior for overlapping windows, and a shared window could be treated semantically like multiple overlapping windows. I am not sure if all are needed but I would say that all can be used. Yes, load/store accesses would probably be the most common use (plus maybe the new (portable) atomics!). Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0