February 2012 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] Next hybrid telecon
by Jim Dinan 24 Feb '12

24 Feb '12

Hi All, Thanks for responding so quickly. We'll connect next Thursday, March 1 at 11am CT. Call-in details are on the wiki. ~Jim. On 2/24/12 4:06 PM, Jim Dinan wrote: > Hi All, > > Please input your availability to connect next week: > > http://www.doodle.com/8m58yk9uyvdm7ab7 > > The agenda for this call will be to wrap up discussion on the helper > threads and endpoints proposals. > > ~Jim. > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

[Mpi3-hybridpm] Next hybrid telecon
by Jim Dinan 24 Feb '12

24 Feb '12

Hi All, Please input your availability to connect next week: http://www.doodle.com/8m58yk9uyvdm7ab7 The agenda for this call will be to wrap up discussion on the helper threads and endpoints proposals. ~Jim.

1 0

[Mpi3-hybridpm] Fwd: tickets 310 313 317
by Marc Snir 24 Feb '12

24 Feb '12

Begin forwarded message: > From: Douglas Miller <dougmill(a)us.ibm.com> > Subject: Re: [Mpi3-hybridpm] tickets 310 313 317 > Date: February 24, 2012 4:17:46 PM EST > To: mpi3-hybridpm(a)lists.mpi-forum.org > Reply-To: mpi3-hybridpm(a)lists.mpi-forum.org > > Last time we discussed these, I thought there were going to be some minor changes made. Unfortunately, these revised texts seem to have significant changes, and are farther from what I was comfortable with. We don't seem to be converging on a target. Here's what I've been thinking: > > * One MPI_INIT (or MPI_INIT_THREAD) and one MPI_FINALIZE per OS process. After MPI_INIT*, all MPI Processes are initialized. > > You have this in my proposal. However, I don't prohibit one MPI_Init call per MPI process. > * Every thread is associated with one MPI Process by default. Which MPI Process is implementation defined. A call to MPI_COMM_RANK on MPI_COMM_WORLD will tell which. (sub-communicators might return MPI_PROC_NULL) > You have this in my proposal if you use MPI_THREAD_MULTIPLE > > * changing associated is permitted with MPI_THREAD_ATTACH (not sure I'm comfortable with the abstract "index" version of this, I like (MPI_COMM, rank) more) > > If we are sure we can get in the standard the ability to migrate a thread from one MPI process to another, then your approach is fine. But, if we are not sure that this will pass, then I prefer to push for a new level of thread support. Whether threads are attached by default in this case or not is something I can go either way. With the interface I propose, threads need not be attached to any MPI process, which I think is an advantage. In an environment, such as OpenMP, with dynamic thread support, where threads come and go, I doubt that it makes much sense to attach threads to MPI processes by default; the user will want to control which thread is attached to which MPI process. > * No new thread levels. Specify that thread level is relative to the MPI Process. I'm OK with requiring that multiple MPI processes requires thread level > MPI_THREAD_SINGLE, although I'm not sure it is necessary. > Thread level is relative to address space -- otherwise how can you have only one MPI_INIT_THREAD call per address space? > > MPI_COMM_SPLIT_TYPE is fine as-is, I think. > > > > > _______________________________________________ > Douglas Miller BlueGene Messaging Development > IBM Corp., Rochester, MN USA Bldg 030-2 A401 > dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > > Marc Snir ---02/22/2012 09:12:08 AM---Marc Snir <snir(a)mcs.anl.gov> > > Marc Snir <snir(a)mcs.anl.gov> > Sent by: mpi3-hybridpm-bounces(a)lists.mpi-forum.org > 02/22/2012 09:07 AM > Please respond to > mpi3-hybridpm(a)lists.mpi-forum.org > > To > > mpi3-hybridpm(a)lists.mpi-forum.org, > > cc > > > Subject > > [Mpi3-hybridpm] tickets 310 313 317 > > I posted text for each of them. Text for ticket 311 is now part of text for ticket 310. Please read and comment. > > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] tickets 310 313 317
by Douglas Miller 24 Feb '12

24 Feb '12

Last time we discussed these, I thought there were going to be some minor changes made. Unfortunately, these revised texts seem to have significant changes, and are farther from what I was comfortable with. We don't seem to be converging on a target. Here's what I've been thinking: * One MPI_INIT (or MPI_INIT_THREAD) and one MPI_FINALIZE per OS process. After MPI_INIT*, all MPI Processes are initialized. * Every thread is associated with one MPI Process by default. Which MPI Process is implementation defined. A call to MPI_COMM_RANK on MPI_COMM_WORLD will tell which. (sub-communicators might return MPI_PROC_NULL) * changing associated is permitted with MPI_THREAD_ATTACH (not sure I'm comfortable with the abstract "index" version of this, I like (MPI_COMM, rank) more) * No new thread levels. Specify that thread level is relative to the MPI Process. I'm OK with requiring that multiple MPI processes requires thread level > MPI_THREAD_SINGLE, although I'm not sure it is necessary. MPI_COMM_SPLIT_TYPE is fine as-is, I think. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A401 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Marc Snir <snir(a)mcs.anl.gov > To Sent by: mpi3-hybridpm(a)lists.mpi-forum.org, mpi3-hybridpm-bou cc nces(a)lists.mpi-fo rum.org Subject [Mpi3-hybridpm] tickets 310 313 317 02/22/2012 09:07 AM Please respond to mpi3-hybridpm@lis ts.mpi-forum.org I posted text for each of them. Text for ticket 311 is now part of text for ticket 310. Please read and comment. _______________________________________________ Mpi3-hybridpm mailing list Mpi3-hybridpm(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

[Mpi3-hybridpm] tickets 310 313 317
by Marc Snir 22 Feb '12

22 Feb '12

I posted text for each of them. Text for ticket 311 is now part of text for ticket 310. Please read and comment.

1 0

Re: [Mpi3-hybridpm] Next hybrid telecon
by Jim Dinan 13 Feb '12

13 Feb '12

Hi All, Notes from today's meeting: 1. Helper threads - did a reading of the new proposal draft - Rhe API has been reduced to just join/leave functions - Helping epochs have been defined as a means for clearly specifying the semantics - Feedback to these changes was positive 2. Continued discussion on initialization / finalization in the endpoints proposal - Discussion centered around several cases and how they impact initialization: 1. Number of threads is fixed 2. Number of threads varies with user control 3. Number of threads varies without user control ~Jim. On 2/7/12 3:23 PM, Pavan Balaji wrote: > > On Monday (02/13) 11am central time. > > Agenda items: > > 1. Epochs in the helper threads proposal. > > 2. Continue discussion on the endpoints proposal initialization. > > -- Pavan >

1 0

[Mpi3-hybridpm] endpoints -- initialize & finalize
by Marc Snir 13 Feb '12

13 Feb '12

I did not get to right a details proposal -- so I send a few points for discussion today Let's assume first that the use decides what thread is attached to what endpoint. I propose that we leave INIT and FINALIZE as is, and add an MPI_ATTACH(int) that can be called before MPI is initialized, or after, for a thread to join an endpoint (an MPI_PROCESS). We can decide later if this is a one time deal, or whether a thread can move form an endpoint to another. We have three cases: 1. Set of threads used by a program in an address space is fixed and set before the user code starts executing. This is the case for UPC, or PGAS, in general, and for OpenMP, when dyn_var=false. A programmer can enforce that OpenMP use a fixed number of threads by calling mop_set_dynamic(0). This is an easy case -- no matter how we define the behavior of MPI_INIT and MPI_FINALIZE: We use MPI_ATTACH to attach each thread to an endpoint, and than have one thread per endpoint call INIT. Life would be easier if replicated initializations were allowed, but that's second order. Finalization replicates the initialization process. 2. Set of threads used by a program changes, under user control. This will be the case, for programs written using the pthread library. Typically, such a program will start single-threaded. The user will need to attach newly created threads to the proper endpoint. Allowing only one initialization per address space -- and also one finalization -- makes life easier. But the difference is not great, since each thread needs to be explicitly attached to an endpoint. Either the user "knows" whether the endpoint was already initialized and writes code accordingly; or a thread firs attaches to an endpoint, then invokes MPI_INITIALIZED to decide whether the endpoint needs to be initialized. 3. Set of threads used by a program changes, and the changes are not under user control. This is the case for OpenMP, with dyn_var=true. I see no convenient way of handling this case: At any point where the set of threads may change (in OpenMP, at any point where a parallel construct starts or ends), one will need (a) to make sure that all MPI calls are complete; (b) reattach threads to endpoints. I suggest not to handle this case. The alternative is that what thread is attached to what endpoint is not under user control -- but is done automatically by the system. This is the current situation when we have only one endpoint, and we should preserve this option, for compatibility: No call to MPI_ATTACH needed in such a case. In such a case, it would be convenient to allow only one call to MPI_INIT and one call to MPI_FINALIZE per (address space) program. But I am not sure I want to spend time arguing about this model, where the runtime dynamically attaches threads to endpoints.

1 0

Re: [Mpi3-hybridpm] Fwd: MPI shared memory allocation issues
by Dave Goodell 13 Feb '12

13 Feb '12

I'm not sure what happened here. I didn't re-send this. So a mail server or maybe the listserv must have re-sent this for some bizarre reason. -Dave On Mar 28, 2011, at 1:28 PM CDT, Dave Goodell wrote: > Forwarding this mail to a broader audience, based on our discussion here at the March 2011 MPI Forum meeting. There was additional correspondence on this thread that I can forward as needed, but this forwarded mail contains my core argument against the original (allocate+free+fence) proposal. > > -Dave > > Begin forwarded message: > >> From: Dave Goodell <goodell(a)mcs.anl.gov> >> Date: February 24, 2011 11:18:20 AM CST >> To: Ron Brightwell <rbbrigh(a)sandia.gov>, Douglas Miller <dougmill(a)us.ibm.com>, "Bronis R. de Supinski" <bronis(a)llnl.gov>, Jim Dinan <dinan(a)mcs.anl.gov>, Pavan Balaji <balaji(a)mcs.anl.gov>, Marc Snir <snir(a)illinois.edu> >> Subject: MPI shared memory allocation issues >> >> I voiced concerns at the last MPI forum meeting about the proposed MPI extensions for allocating shared memory. In particular I was concerned about "MPI_Shm_fence". Pavan asked me to write up a quick email to this group in order to help clarify my view in the discussion; this is that email. Please widen the distribution list as appropriate, I just mailed the addresses that Pavan indicated to me. FYI, I am not currently subscribed to the mpi3-hybrid list. >> >> First, I view multithreaded programming within an OS process and multiprocess programming using a shared memory region to be essentially the same problem. There are probably alternative interpretations of the words "process" and "thread" that could muddy the picture here, but for the sake of clarity, let's use the conventional meanings for the moment. Also, I am not interested in discussing distributed shared memory (DSM) here, I think that bringing it up just confuses the discussion further. My primary objections to the proposal are valid entirely within a discussion of conventional shared memory, processes, and threads. >> >> Given that preface, I believe that many, if not all, of the issues raised by Boehm's paper, "Threads Cannot Be Implemented As a Library" [1], apply here. In particular, some variation on the example from section 4.3 is probably an issue, but the others seem to apply as well. The performance example is also relevant here, but in an even more dramatic fashion given the dearth of synchronization primitives offered by the proposal. >> >> I do not believe that we can specify a way to program the provided shared memory in any way that is robust and useful to the user, because C and Fortran do not give us enough of a specification in order to do so. Without getting into the business of compiler writing, MPI has no way to give the user any meaningful guarantees. Just as Boehm noted about pthreads, we can probably come up with an approach that will work most of the time. But that's a pretty flimsy guarantee for a standard like MPI. >> >> If you ignore the difficulty in specifying an interface that can actually be used correctly, then another issue arises. The only proposed synchronization mechanism virtually guarantees that the user can at best utilize the allocated shared memory region to share data that is written once and otherwise read-only. Any other shared memory programming techniques are either going to be non-portable (e.g., using pthread mutexes or calls/macros from some atomic operations library), or they will be limited to potentially slow dark-ages techniques such as Dekker's Algorithm with excessive MPI_Shm_fence-ing. So does this proposal really empower the user in any meaningful way? >> >> I don't see a compelling advantage to putting this into MPI as opposed to providing this as some third-party library on top of MPI. Sure, it's easy to implement the allocate/free calls inside of MPI because the machinery is typically there. But a third-party library would be able to escape some of the extremely generic portability constraints of the MPI standard and would therefore be able to provide a more robust interface to the user. A discussion of DSM might make putting it into MPI more compelling because access to the network hardware might be involved, but I'm not particularly interested in having that discussion right now. I think that MPI-3 RMA would probably be more suitable for that use case. >> >> -Dave >> >> [1] http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf >> > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] Fwd: MPI shared memory allocation issues
by Torsten Hoefler 12 Feb '12

12 Feb '12

Dave, On Mon, Mar 28, 2011 at 01:28:48PM -0500, Dave Goodell wrote: > Forwarding this mail to a broader audience, based on our discussion > here at the March 2011 MPI Forum meeting. There was additional > correspondence on this thread that I can forward as needed, but this > forwarded mail contains my core argument against the original > (allocate+free+fence) proposal. Thanks! This is an important discussion. Let me recap below what we discussed in the RMA group when moving towards the newer integrated allocate_shared. > Begin forwarded message: > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > > Date: February 24, 2011 11:18:20 AM CST > > To: Ron Brightwell <rbbrigh(a)sandia.gov>, Douglas Miller <dougmill(a)us.ibm.com>, "Bronis R. de Supinski" <bronis(a)llnl.gov>, Jim Dinan <dinan(a)mcs.anl.gov>, Pavan Balaji <balaji(a)mcs.anl.gov>, Marc Snir <snir(a)illinois.edu> > > Subject: MPI shared memory allocation issues > > > > I voiced concerns at the last MPI forum meeting about the proposed > > MPI extensions for allocating shared memory. In particular I was > > concerned about "MPI_Shm_fence". Pavan asked me to write up a quick > > email to this group in order to help clarify my view in the > > discussion; this is that email. Please widen the distribution list > > as appropriate, I just mailed the addresses that Pavan indicated to > > me. FYI, I am not currently subscribed to the mpi3-hybrid list. > > > > First, I view multithreaded programming within an OS process and > > multiprocess programming using a shared memory region to be > > essentially the same problem. There are probably alternative > > interpretations of the words "process" and "thread" that could muddy > > the picture here, but for the sake of clarity, let's use the > > conventional meanings for the moment. Also, I am not interested in > > discussing distributed shared memory (DSM) here, I think that > > bringing it up just confuses the discussion further. My primary > > objections to the proposal are valid entirely within a discussion of > > conventional shared memory, processes, and threads. > > > > Given that preface, I believe that many, if not all, of the issues > > raised by Boehm's paper, "Threads Cannot Be Implemented As a > > Library" [1], apply here. In particular, some variation on the > > example from section 4.3 is probably an issue, but the others seem > > to apply as well. The performance example is also relevant here, > > but in an even more dramatic fashion given the dearth of > > synchronization primitives offered by the proposal. > > > > I do not believe that we can specify a way to program the provided > > shared memory in any way that is robust and useful to the user, > > because C and Fortran do not give us enough of a specification in > > order to do so. Without getting into the business of compiler > > writing, MPI has no way to give the user any meaningful guarantees. > > Just as Boehm noted about pthreads, we can probably come up with an > > approach that will work most of the time. But that's a pretty > > flimsy guarantee for a standard like MPI. Yes, as Boehm points out, serial compiler optimizations can have very bad effects on concurrently running code accessing close-by memory. However, as you point out, the situation is equivalent to what we have today in pthreads and the proposal does not claim any more, it says "The consistency of load/store accesses from/to the shared memory as observed by the user program depends on the architecture.". We can extend it to include the compiler and maybe reference Boehm's paper (I would see this as a ticket 0 change). I agree to the general sentiment that it is impossible to implement shared memory semantics in a language that doesn't even have a real memory model. However, at the same time, I want to remind us that the behavior of Fortran was *never* 100% correct in MPI <= 2.2 (and we rely in this TR for MPI-3.0). At the same time, Fortran/MPI programs are ubiquitous :-). But let's discuss Boehm's identified correctness issues here: * 4.1 Concurrent modification This is only an issue if users rely on the consistency of the underlying hardware, if they use Win_flush and friends (as advised), such a reordering would be illegal. One downside is that this will only work in C and Fortran will probably may have all kinds of wacky problems with code movement as usual (however, they should be in a position to fix this with the new bindings and the Fortran TR). * 4.2 Rewriting of Adjacent Data This applies to the unified window as well where we simply specify the byte granularity of updates (an architecture could work with larger chunks (e.g., words) and cause the same trouble). So this issue is not limited to the shared memory window, especially when fast remote memory access hardware is used. Here we face the general trade-off between fast hardware access and safety (securing it through a software layer). We decided that byte-consistency is something we can expect from vendors. Also, the vendor library is always free to return MPI_ERR_RMA_SHARED (like he can always choose to not offer the unified memory model). * 4.3 Register Promotion While this is certainly a problem with threads, it would not be one with MPI windows because they have to dereference the address of the accessed memory and would thus prevent register promotion. Copying the data into a faster memory region would also not harm because the remote side has to query the addresses anyway. Again, restrictions may apply for Fortran codes. > > If you ignore the difficulty in specifying an interface that can > > actually be used correctly, then another issue arises. The only > > proposed synchronization mechanism virtually guarantees that the > > user can at best utilize the allocated shared memory region to share > > data that is written once and otherwise read-only. Any other shared > > memory programming techniques are either going to be non-portable > > (e.g., using pthread mutexes or calls/macros from some atomic > > operations library), or they will be limited to potentially slow > > dark-ages techniques such as Dekker's Algorithm with excessive > > MPI_Shm_fence-ing. So does this proposal really empower the user in > > any meaningful way? I agree. This should be addressed by the merge into the RMA context which offers all required functionality (we avoided memory locks on purpose because they are evil). > > I don't see a compelling advantage to putting this into MPI as > > opposed to providing this as some third-party library on top of MPI. > > Sure, it's easy to implement the allocate/free calls inside of MPI > > because the machinery is typically there. But a third-party library > > would be able to escape some of the extremely generic portability > > constraints of the MPI standard and would therefore be able to > > provide a more robust interface to the user. A discussion of DSM > > might make putting it into MPI more compelling because access to the > > network hardware might be involved, but I'm not particularly > > interested in having that discussion right now. I think that MPI-3 > > RMA would probably be more suitable for that use case. First, I am against DSM. Second, I believe that it may be very valuable to have this kind of functionality in MPI because virtually all large-scale codes have to become hybrid. The main issue are the associated memory saving (on-node communication with MPI is often sufficiently fast). I believe the current practice of mixing OpenMP and MPI to achieve this simple goal may be suboptimal (OpenMP supports only the "shared everything" (threaded) model and enables thus a whole new class of bugs and races). All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

[Mpi3-hybridpm] Next hybrid telecon
by Pavan Balaji 07 Feb '12

07 Feb '12

On Monday (02/13) 11am central time. Agenda items: 1. Epochs in the helper threads proposal. 2. Continue discussion on the endpoints proposal initialization. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0