December 2010 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Pavan Balaji 07 Dec '10

07 Dec '10

On 12/07/2010 03:02 PM, Douglas Miller wrote: > There is already precedence, isn't there? At the very least > MPI_IS_THREAD_MAIN can be called by any thread in those modes. Also > MPI_QUERY_THREAD? and MPI_INITIALIZED? MPI_INITIALIZED is the only exception. Everything else requires the right thread-level. However, helper thread functionality is a lot more complex in that it does actual communication or more internal MPI processing than MPI_INITIALIZED. The complexity to benefit ratio doesn't seem to be low enough for this. The only benefit is the difference between single-thread performance in THREAD_MULTIPLE vs. THREAD_FUNNELED modes, which is fairly minimal. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Douglas Miller 07 Dec '10

07 Dec '10

There is already precedence, isn't there? At the very least MPI_IS_THREAD_MAIN can be called by any thread in those modes. Also MPI_QUERY_THREAD? and MPI_INITIALIZED? _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Pavan Balaji <balaji(a)mcs.anl.g ov> To mpi3-hybridpm(a)lists.mpi-forum.org 12/07/2010 10:12 cc AM Douglas Miller/Rochester/IBM@IBMUS Subject Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum On 12/07/2010 09:47 AM, Douglas Miller wrote: > While the implementation I'd be making would probably not distinguish > between MULTIPLE, SERIAL, and FUNNEL, I still believe that there are > implementations that do benefit from using a lesser threading model and > would still be able to use helper threads. I think it is short-sighted > to rule out those modes for helper threads. The question is not about the utility, but rather of the introduction of a new concept -- that is, MPI functions that can be called by multiple threads even in SERIAL or FUNNELED mode. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Ron Brightwell 07 Dec '10

07 Dec '10

On 12/07/2010 08:55 AM, James Dinan wrote: > If it's implemented using BSD mmap then all nodes will need to give the > same filename to map. My feeling is that if MPI can reliably generate > the key/filename/whatever for us then it's unnecessary complexity to put > this in the user interface. > > ~Jim. > > On 12/7/10 7:58 AM, Douglas Miller wrote: >> In order to ensure all nodes get the same memory segment, they all have >> to use the same "key". At least if this is implemented on Unix SysV or >> POSIX shmem. Then there's the possibility of some nodes allocing on a >> sub-comm first, etc. I think its just avoiding a lot of potential >> complications and don't see what the downside is. >> I agree with Jim. Communicators are supposed to hide this kind of complexity from the user. We could also limit the number of shared memory regions to one per communicator, essentially forcing the user to dup a communicator if it wants more than one. -Ron

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Pavan Balaji 07 Dec '10

07 Dec '10

On 12/07/2010 09:47 AM, Douglas Miller wrote: > While the implementation I'd be making would probably not distinguish > between MULTIPLE, SERIAL, and FUNNEL, I still believe that there are > implementations that do benefit from using a lesser threading model and > would still be able to use helper threads. I think it is short-sighted > to rule out those modes for helper threads. The question is not about the utility, but rather of the introduction of a new concept -- that is, MPI functions that can be called by multiple threads even in SERIAL or FUNNELED mode. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by James Dinan 07 Dec '10

07 Dec '10

If it's implemented using BSD mmap then all nodes will need to give the same filename to map. My feeling is that if MPI can reliably generate the key/filename/whatever for us then it's unnecessary complexity to put this in the user interface. ~Jim. On 12/7/10 7:58 AM, Douglas Miller wrote: > In order to ensure all nodes get the same memory segment, they all have > to use the same "key". At least if this is implemented on Unix SysV or > POSIX shmem. Then there's the possibility of some nodes allocing on a > sub-comm first, etc. I think its just avoiding a lot of potential > complications and don't see what the downside is. > > > _______________________________________________ > Douglas Miller BlueGene Messaging Development > IBM Corp., Rochester, MN USA Bldg 030-2 A410 > dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > > Inactive hide details for "Ron Brightwell" ---12/07/2010 12:14:38 > AM---On 12/06/2010 05:59 PM, James Dinan wrote:"Ron Brightwell" > ---12/07/2010 12:14:38 AM---On 12/06/2010 05:59 PM, James Dinan wrote: > > *"Ron Brightwell" <rbbrigh(a)sandia.gov>* > Sent by: mpi3-hybridpm-bounces(a)lists.mpi-forum.org > > 12/07/2010 12:11 AM > Please respond to > mpi3-hybridpm(a)lists.mpi-forum.org > > > > To > > > "mpi3-hybridpm(a)lists.mpi-forum.org" > <mpi3-hybridpm(a)lists.mpi-forum.org> > > > cc > > > Subject > > > Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum > > > > > On 12/06/2010 05:59 PM, James Dinan wrote: > > On 12/6/10 5:49 PM, Pavan Balaji wrote: > >> Hi Doug, > >> > >> On 12/06/2010 11:42 AM, Douglas Miller wrote: > >>> Re: SHMEM ALLOC: I thought we were considering adding a "const char > >>> *key" param so that shared memory allocations better match OS (e.g. > >>> POSIX) shmem allocations and also to allow the call to be > >>> non-synchronizing. > >> > >> As I recalled, this was still being discussed. Did we decide on adding > >> the key? Ron? > >> > >> I'll add it in as a discussion item. > > > > I think that the key is primarily helpful as a bootstrap mechanism for > > processes to ensure they connect to the right shared segment. I think > > the intent was to have shmem alloc be collective so this should be > > extraneous. > > Right. I think you lose several important optimizations for shared > memory if the call is not collective. It doesn't have to be synchronous, > but it does have to be collective. > > -Ron > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Douglas Miller 07 Dec '10

07 Dec '10

While the implementation I'd be making would probably not distinguish between MULTIPLE, SERIAL, and FUNNEL, I still believe that there are implementations that do benefit from using a lesser threading model and would still be able to use helper threads. I think it is short-sighted to rule out those modes for helper threads. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Pavan Balaji <balaji(a)mcs.anl.g ov> To Douglas Miller/Rochester/IBM@IBMUS 12/07/2010 08:08 cc AM mpi3-hybridpm(a)lists.mpi-forum.org Subject Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum On 12/07/2010 08:03 AM, Douglas Miller wrote: > Well, I was thinking we did not need to restrict MPI_HELPER_* to only > MPI_THREAD_MULTIPLE, as I think these routines are easily made thread > safe independently of the rest of MPI. I would like to see these be > more-widely usable. Convincing the Forum about that is an uphill battle, AFAICT (not sure which way I'll vote on that, either). I'd like to not add that in the initial proposal. If you like, we can create a second proposal chapter (like what we are doing in the RMA WG) with these "additional changes" which will be good to have, but not critical. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Douglas Miller 07 Dec '10

07 Dec '10

It's just that providing a "more meaningful" key can help when debugging or managing through the OS. You still have to generate some sort of "sequence" number per-communicator, as successive allocations on the same communicator must return different memory. So, communicator alone is not enough to uniquely identify/select an allocation. Implementations will have to generate a "key" to pass to the OS, and so all ranks must be able to generate identical keys for corresponding calls. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Pavan Balaji <balaji(a)mcs.anl.g ov> To mpi3-hybridpm(a)lists.mpi-forum.org 12/07/2010 08:08 cc AM Douglas Miller/Rochester/IBM@IBMUS Subject Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum On 12/07/2010 07:58 AM, Douglas Miller wrote: > In order to ensure all nodes get the same memory segment, they all have > to use the same "key". At least if this is implemented on Unix SysV or > POSIX shmem. Then there's the possibility of some nodes allocing on a > sub-comm first, etc. I think its just avoiding a lot of potential > complications and don't see what the downside is. In that sense, isn't the communicator a "key"? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Pavan Balaji 07 Dec '10

07 Dec '10

On 12/07/2010 08:03 AM, Douglas Miller wrote: > Well, I was thinking we did not need to restrict MPI_HELPER_* to only > MPI_THREAD_MULTIPLE, as I think these routines are easily made thread > safe independently of the rest of MPI. I would like to see these be > more-widely usable. Convincing the Forum about that is an uphill battle, AFAICT (not sure which way I'll vote on that, either). I'd like to not add that in the initial proposal. If you like, we can create a second proposal chapter (like what we are doing in the RMA WG) with these "additional changes" which will be good to have, but not critical. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Pavan Balaji 07 Dec '10

07 Dec '10

On 12/07/2010 07:58 AM, Douglas Miller wrote: > In order to ensure all nodes get the same memory segment, they all have > to use the same "key". At least if this is implemented on Unix SysV or > POSIX shmem. Then there's the possibility of some nodes allocing on a > sub-comm first, etc. I think its just avoiding a lot of potential > complications and don't see what the downside is. In that sense, isn't the communicator a "key"? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum
by Douglas Miller 07 Dec '10

07 Dec '10

Well, I was thinking we did not need to restrict MPI_HELPER_* to only MPI_THREAD_MULTIPLE, as I think these routines are easily made thread safe independently of the rest of MPI. I would like to see these be more-widely usable. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Pavan Balaji <balaji(a)mcs.anl.g ov> To mpi3-hybridpm(a)lists.mpi-forum.org 12/06/2010 05:49 cc PM Douglas Miller/Rochester/IBM@IBMUS Subject Re: [Mpi3-hybridpm] Hybrid WG meeting at the Forum Hi Doug, [...snip...] > A) I don't see mention of the exception to MPI threading rules for the > MPI_HELPER_* calls. Was that intentionally removed? Did I forget a > discussion of that? I was thinking we need to state to implementers that > MPI_HELPER_* must be thread safe (in all modes except MPI_THREAD_SINGLE) > and to users that these calls may be used by any/all threads in any mode > (except MPI_THREAD_SINGLE). Do we need to discuss this more? We decided to drop at the Forum (3-4 Forums back, or whenever we had the plenary session). I had accidentally added it in the first draft, but removed it since. Do you mean to say that you do want the modified threading rules back? [...snip...] -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0