March 2011 - mpiwg-hybridpm - mailman3.cels.anl.gov

[Mpi3-hybridpm] Fwd: MPI shared memory allocation issues
by Dave Goodell 28 Mar '11

28 Mar '11

Re-forwarding to mpi3-hybridpm@ now that I have subscribed so that the previous forward is not dependent on whoever the list admin actually is. If you have followup correspondence on this thread, please CC mpi3-rma(a)lists.mpi-forum.org as well. -Dave Begin forwarded message: > From: Dave Goodell <goodell(a)mcs.anl.gov> > Date: March 28, 2011 1:28:48 PM CDT > To: mpi3-hybridpm(a)lists.mpi-forum.org, mpi3-rma(a)lists.mpi-forum.org > Subject: Fwd: MPI shared memory allocation issues > > Forwarding this mail to a broader audience, based on our discussion here at the March 2011 MPI Forum meeting. There was additional correspondence on this thread that I can forward as needed, but this forwarded mail contains my core argument against the original (allocate+free+fence) proposal. > > -Dave > > Begin forwarded message: > >> From: Dave Goodell <goodell(a)mcs.anl.gov> >> Date: February 24, 2011 11:18:20 AM CST >> To: Ron Brightwell <rbbrigh(a)sandia.gov>, Douglas Miller <dougmill(a)us.ibm.com>, "Bronis R. de Supinski" <bronis(a)llnl.gov>, Jim Dinan <dinan(a)mcs.anl.gov>, Pavan Balaji <balaji(a)mcs.anl.gov>, Marc Snir <snir(a)illinois.edu> >> Subject: MPI shared memory allocation issues >> >> I voiced concerns at the last MPI forum meeting about the proposed MPI extensions for allocating shared memory. In particular I was concerned about "MPI_Shm_fence". Pavan asked me to write up a quick email to this group in order to help clarify my view in the discussion; this is that email. Please widen the distribution list as appropriate, I just mailed the addresses that Pavan indicated to me. FYI, I am not currently subscribed to the mpi3-hybrid list. >> >> First, I view multithreaded programming within an OS process and multiprocess programming using a shared memory region to be essentially the same problem. There are probably alternative interpretations of the words "process" and "thread" that could muddy the picture here, but for the sake of clarity, let's use the conventional meanings for the moment. Also, I am not interested in discussing distributed shared memory (DSM) here, I think that bringing it up just confuses the discussion further. My primary objections to the proposal are valid entirely within a discussion of conventional shared memory, processes, and threads. >> >> Given that preface, I believe that many, if not all, of the issues raised by Boehm's paper, "Threads Cannot Be Implemented As a Library" [1], apply here. In particular, some variation on the example from section 4.3 is probably an issue, but the others seem to apply as well. The performance example is also relevant here, but in an even more dramatic fashion given the dearth of synchronization primitives offered by the proposal. >> >> I do not believe that we can specify a way to program the provided shared memory in any way that is robust and useful to the user, because C and Fortran do not give us enough of a specification in order to do so. Without getting into the business of compiler writing, MPI has no way to give the user any meaningful guarantees. Just as Boehm noted about pthreads, we can probably come up with an approach that will work most of the time. But that's a pretty flimsy guarantee for a standard like MPI. >> >> If you ignore the difficulty in specifying an interface that can actually be used correctly, then another issue arises. The only proposed synchronization mechanism virtually guarantees that the user can at best utilize the allocated shared memory region to share data that is written once and otherwise read-only. Any other shared memory programming techniques are either going to be non-portable (e.g., using pthread mutexes or calls/macros from some atomic operations library), or they will be limited to potentially slow dark-ages techniques such as Dekker's Algorithm with excessive MPI_Shm_fence-ing. So does this proposal really empower the user in any meaningful way? >> >> I don't see a compelling advantage to putting this into MPI as opposed to providing this as some third-party library on top of MPI. Sure, it's easy to implement the allocate/free calls inside of MPI because the machinery is typically there. But a third-party library would be able to escape some of the extremely generic portability constraints of the MPI standard and would therefore be able to provide a more robust interface to the user. A discussion of DSM might make putting it into MPI more compelling because access to the network hardware might be involved, but I'm not particularly interested in having that discussion right now. I think that MPI-3 RMA would probably be more suitable for that use case. >> >> -Dave >> >> [1] http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf >> >

1 0

[Mpi3-hybridpm] Fwd: MPI shared memory allocation issues
by Dave Goodell 28 Mar '11

28 Mar '11

Forwarding this mail to a broader audience, based on our discussion here at the March 2011 MPI Forum meeting. There was additional correspondence on this thread that I can forward as needed, but this forwarded mail contains my core argument against the original (allocate+free+fence) proposal. -Dave Begin forwarded message: > From: Dave Goodell <goodell(a)mcs.anl.gov> > Date: February 24, 2011 11:18:20 AM CST > To: Ron Brightwell <rbbrigh(a)sandia.gov>, Douglas Miller <dougmill(a)us.ibm.com>, "Bronis R. de Supinski" <bronis(a)llnl.gov>, Jim Dinan <dinan(a)mcs.anl.gov>, Pavan Balaji <balaji(a)mcs.anl.gov>, Marc Snir <snir(a)illinois.edu> > Subject: MPI shared memory allocation issues > > I voiced concerns at the last MPI forum meeting about the proposed MPI extensions for allocating shared memory. In particular I was concerned about "MPI_Shm_fence". Pavan asked me to write up a quick email to this group in order to help clarify my view in the discussion; this is that email. Please widen the distribution list as appropriate, I just mailed the addresses that Pavan indicated to me. FYI, I am not currently subscribed to the mpi3-hybrid list. > > First, I view multithreaded programming within an OS process and multiprocess programming using a shared memory region to be essentially the same problem. There are probably alternative interpretations of the words "process" and "thread" that could muddy the picture here, but for the sake of clarity, let's use the conventional meanings for the moment. Also, I am not interested in discussing distributed shared memory (DSM) here, I think that bringing it up just confuses the discussion further. My primary objections to the proposal are valid entirely within a discussion of conventional shared memory, processes, and threads. > > Given that preface, I believe that many, if not all, of the issues raised by Boehm's paper, "Threads Cannot Be Implemented As a Library" [1], apply here. In particular, some variation on the example from section 4.3 is probably an issue, but the others seem to apply as well. The performance example is also relevant here, but in an even more dramatic fashion given the dearth of synchronization primitives offered by the proposal. > > I do not believe that we can specify a way to program the provided shared memory in any way that is robust and useful to the user, because C and Fortran do not give us enough of a specification in order to do so. Without getting into the business of compiler writing, MPI has no way to give the user any meaningful guarantees. Just as Boehm noted about pthreads, we can probably come up with an approach that will work most of the time. But that's a pretty flimsy guarantee for a standard like MPI. > > If you ignore the difficulty in specifying an interface that can actually be used correctly, then another issue arises. The only proposed synchronization mechanism virtually guarantees that the user can at best utilize the allocated shared memory region to share data that is written once and otherwise read-only. Any other shared memory programming techniques are either going to be non-portable (e.g., using pthread mutexes or calls/macros from some atomic operations library), or they will be limited to potentially slow dark-ages techniques such as Dekker's Algorithm with excessive MPI_Shm_fence-ing. So does this proposal really empower the user in any meaningful way? > > I don't see a compelling advantage to putting this into MPI as opposed to providing this as some third-party library on top of MPI. Sure, it's easy to implement the allocate/free calls inside of MPI because the machinery is typically there. But a third-party library would be able to escape some of the extremely generic portability constraints of the MPI standard and would therefore be able to provide a more robust interface to the user. A discussion of DSM might make putting it into MPI more compelling because access to the network hardware might be involved, but I'm not particularly interested in having that discussion right now. I think that MPI-3 RMA would probably be more suitable for that use case. > > -Dave > > [1] http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf >

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Pavan Balaji 27 Mar '11

27 Mar '11

I'm not entirely sure this is addressing the complains raised by Dave about the shared memory functionality. Basically, the concern (as I understood it) was that shared memory functionality cannot be correctly implemented as a library alone without compiler support. We are still in the same state here as with Ron's proposal. -- Pavan On 03/16/2011 08:51 AM, Torsten Hoefler wrote: > On Wed, Mar 16, 2011 at 08:44:14AM -0500, Rajeev Thakur wrote: >> You could, but it starts looking more and more like RMA. The purpose >> of SHM_allocate was to provide an easy way to allocate shared memory >> and access it directly with memory references, instead of the user >> having to do that himself using POSIX or System V shared memory >> allocation functions. Now if he has to read the whole RMA chapter for >> that... :-) > I thought the motivation was to have a portable interface? The old > functionality could be emulated by a library that calls allocate and > starts a lock-all epoch. Not sure how this is conceptually more complex. > > But we shall wait until we get to the consistency and ordering details > ;-). > > Torsten > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 16 Mar '11

16 Mar '11

On Wed, Mar 16, 2011 at 08:44:14AM -0500, Rajeev Thakur wrote: > You could, but it starts looking more and more like RMA. The purpose > of SHM_allocate was to provide an easy way to allocate shared memory > and access it directly with memory references, instead of the user > having to do that himself using POSIX or System V shared memory > allocation functions. Now if he has to read the whole RMA chapter for > that... :-) I thought the motivation was to have a portable interface? The old functionality could be emulated by a library that calls allocate and starts a lock-all epoch. Not sure how this is conceptually more complex. But we shall wait until we get to the consistency and ordering details ;-). Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 16 Mar '11

16 Mar '11

You could, but it starts looking more and more like RMA. The purpose of SHM_allocate was to provide an easy way to allocate shared memory and access it directly with memory references, instead of the user having to do that himself using POSIX or System V shared memory allocation functions. Now if he has to read the whole RMA chapter for that... :-) Rajeev On Mar 16, 2011, at 8:22 AM, James Dinan wrote: > Hi Rajeev, > > Why not use lock/unlock? Lock_all seems like a good fit here. > > ~Jim. > > On 3/16/11 8:09 AM, Rajeev Thakur wrote: >> Right. Some things that need to be touched up are that flush can only be called within lock-unlock currently. Also flush takes a target rank, which is the rank passed to the put/get. Flush_all has the notion of all ranks used in preceding puts/gets. Such things would need to be addressed. >> >> Rajeev >> >> >> On Mar 15, 2011, at 11:22 PM, Torsten Hoefler wrote: >> >>> On Tue, Mar 15, 2011 at 09:39:24PM -0500, Rajeev Thakur wrote: >>>> That's what I meant in my first mail. We need to say whether this example will work or not. >>>> >>>> A=10 >>>> barrier barrier >>>> x = A >>>> >>>> If not, which synchronization functions need to be used and how? The >>>> current text does not automatically cover this example. >>> Right, I never claimed that the proposal is complete :-). It's basically >>> a quick poll to see if there are major flaws and I should retract or if >>> it is worth discussing at the Forum (to flesh out such things like you >>> mentioned). >>> >>> To answer your question: I think this should either be: >>> >>> A=10 >>> MPI_Win_flush >>> barrier barrier >>> x=A >>> >>> This would basically insert a membar in MPI_Win_flush which makes >>> sure that A is committed before the barrier completes (a barrier flag is >>> set and read). >>> >>> This fits our current semantics where we suggest that win_flush can be >>> used for ordering. Keep in mind that it's still the unified model, i.e., >>> the value would commit eventually, the flush is just needed to enforce >>> ordering with regards to the barrier. >>> >>> Thanks& Best, >>> Torsten >>> >>> -- >>> bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- >>> "A couple of months in the laboratory can frequently save a couple of >>> hours in the library." - Westheimer (contemporary) >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 16 Mar '11

16 Mar '11

Right. Some things that need to be touched up are that flush can only be called within lock-unlock currently. Also flush takes a target rank, which is the rank passed to the put/get. Flush_all has the notion of all ranks used in preceding puts/gets. Such things would need to be addressed. Rajeev On Mar 15, 2011, at 11:22 PM, Torsten Hoefler wrote: > On Tue, Mar 15, 2011 at 09:39:24PM -0500, Rajeev Thakur wrote: >> That's what I meant in my first mail. We need to say whether this example will work or not. >> >> A=10 >> barrier barrier >> x = A >> >> If not, which synchronization functions need to be used and how? The >> current text does not automatically cover this example. > Right, I never claimed that the proposal is complete :-). It's basically > a quick poll to see if there are major flaws and I should retract or if > it is worth discussing at the Forum (to flesh out such things like you > mentioned). > > To answer your question: I think this should either be: > > A=10 > MPI_Win_flush > barrier barrier > x=A > > This would basically insert a membar in MPI_Win_flush which makes > sure that A is committed before the barrier completes (a barrier flag is > set and read). > > This fits our current semantics where we suggest that win_flush can be > used for ordering. Keep in mind that it's still the unified model, i.e., > the value would commit eventually, the flush is just needed to enforce > ordering with regards to the barrier. > > Thanks & Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > "A couple of months in the laboratory can frequently save a couple of > hours in the library." - Westheimer (contemporary) > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 09:39:24PM -0500, Rajeev Thakur wrote: > That's what I meant in my first mail. We need to say whether this example will work or not. > > A=10 > barrier barrier > x = A > > If not, which synchronization functions need to be used and how? The > current text does not automatically cover this example. Right, I never claimed that the proposal is complete :-). It's basically a quick poll to see if there are major flaws and I should retract or if it is worth discussing at the Forum (to flesh out such things like you mentioned). To answer your question: I think this should either be: A=10 MPI_Win_flush barrier barrier x=A This would basically insert a membar in MPI_Win_flush which makes sure that A is committed before the barrier completes (a barrier flag is set and read). This fits our current semantics where we suggest that win_flush can be used for ordering. Keep in mind that it's still the unified model, i.e., the value would commit eventually, the flush is just needed to enforce ordering with regards to the barrier. Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- "A couple of months in the laboratory can frequently save a couple of hours in the library." - Westheimer (contemporary)

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 09:48:40PM -0500, Rajeev Thakur wrote: > Also, at the last meeting we had trouble defining the MPI_SHM_Fence > function. Dave said we won't be able to define it portably for all > processor architectures wrt memory orderings etc. Someone else pointed > out that, in Fortran, code can be moved around it. Those issues won't > go away by effectively renaming MPI_SHM_Fence to MPI_Win_flush. (These > issues are wrt direct loads/stores to shared memory.) I know, those issues are the same for a fast implementation on a shared memory system (one that doesn't do a membar in every put/get/...). So I think this issue is orthogonal and independent of the interface. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 15 Mar '11

15 Mar '11

Also, at the last meeting we had trouble defining the MPI_SHM_Fence function. Dave said we won't be able to define it portably for all processor architectures wrt memory orderings etc. Someone else pointed out that, in Fortran, code can be moved around it. Those issues won't go away by effectively renaming MPI_SHM_Fence to MPI_Win_flush. (These issues are wrt direct loads/stores to shared memory.) Rajeev On Mar 15, 2011, at 9:39 PM, Rajeev Thakur wrote: > That's what I meant in my first mail. We need to say whether this example will work or not. > > A=10 > barrier barrier > x = A > > If not, which synchronization functions need to be used and how? The current text does not automatically cover this example. > > Rajeev > > > On Mar 15, 2011, at 9:28 PM, Torsten Hoefler wrote: > >> On Tue, Mar 15, 2011 at 09:22:49PM -0500, Rajeev Thakur wrote: >>> All can be used, but none are needed, right? >> We might need to talk about synch or flush. Some shared memory systems >> are loosely coupled and need membars to guarantee correct ordering. >> >>> Currently, in the unified model you can do: >>> >>> lock >>> put(A) >>> unlock >>> barrier barrier >>> x = A >>> >>> With shared memory, you could do >>> >>> A=10 >>> barrier barrier >>> x = A >>> >>> Right? >> Not really. The barrier and the x=A might be re-ordered. This is also >> the biggest issue with our definition of the unified model, some shared >> memory systems might actually not be able to support it :-(. We had a >> long discussion in Stuttgart about this. >> >>> If the above is true, it might be a bit confusing to users that in the >>> RMA chapter you use put, get, and synchronization functions for some >>> things and can directly read/write memory for some other things. >> Well, the standard is not a user documentation :-). But yes, having this >> special kind of window requires some effort to understand it. >> >> Best, >> Torsten >> >> -- >> bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- >> Torsten Hoefler | Performance Modeling and Simulation Lead >> Blue Waters Directorate | University of Illinois (UIUC) >> 1205 W Clark Street | Urbana, IL, 61801 >> NCSA Building | +01 (217) 244-7736 >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 15 Mar '11

15 Mar '11

That's what I meant in my first mail. We need to say whether this example will work or not. A=10 barrier barrier x = A If not, which synchronization functions need to be used and how? The current text does not automatically cover this example. Rajeev On Mar 15, 2011, at 9:28 PM, Torsten Hoefler wrote: > On Tue, Mar 15, 2011 at 09:22:49PM -0500, Rajeev Thakur wrote: >> All can be used, but none are needed, right? > We might need to talk about synch or flush. Some shared memory systems > are loosely coupled and need membars to guarantee correct ordering. > >> Currently, in the unified model you can do: >> >> lock >> put(A) >> unlock >> barrier barrier >> x = A >> >> With shared memory, you could do >> >> A=10 >> barrier barrier >> x = A >> >> Right? > Not really. The barrier and the x=A might be re-ordered. This is also > the biggest issue with our definition of the unified model, some shared > memory systems might actually not be able to support it :-(. We had a > long discussion in Stuttgart about this. > >> If the above is true, it might be a bit confusing to users that in the >> RMA chapter you use put, get, and synchronization functions for some >> things and can directly read/write memory for some other things. > Well, the standard is not a user documentation :-). But yes, having this > special kind of window requires some effort to understand it. > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0