April 2013 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Jeff Squyres (jsquyres) 24 Apr '13

24 Apr '13

On Apr 24, 2013, at 10:46 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> a) it could solve a lot of Jeff Hammond's originally-cited problems > > I think I said already that MPI_INIT* cannot be a solution to my > problem because it is collective and I might not want to use the > theoretical threaded library described in my ticket on every MPI > process. Rain on my parade. Matches the weather here, today, too. But I guess this brings up another subtle point (if no one brought it up already): since INIT is collective, if we have multiple threads invoking INIT simultaneously in each MPI process, we effectively have a matching problem -- how do you match each INIT (or even just the first/actually-initializing INIT) from each process? The obvious answer is to add a tag to INIT. But I will shoot anyone who suggests such a monstrosity (that doesn't solve all matching issues, anyway). And then I will deny that I ever mentioned it. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Jeff Hammond 24 Apr '13

24 Apr '13

> My overall point: if we start exploring the path of re-initialization > > a) it could solve a lot of Jeff Hammond's originally-cited problems I think I said already that MPI_INIT* cannot be a solution to my problem because it is collective and I might not want to use the theoretical threaded library described in my ticket on every MPI process. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Jeff Squyres (jsquyres) 24 Apr '13

24 Apr '13

On Apr 22, 2013, at 6:17 PM, "Schulz, Martin" <schulzm(a)llnl.gov> wrote: >> Some MPI implementations might need an explicit *real* finalize at some >> point, if they need to clean up OS resources. For most MPI > > I was wondering about that - what platforms would be affected by this? OS resources shouldn't be an issue, since the OS should take care of this at process termination. Network resources may be a different issue, though. I find it ironic that a tools guy would ask this question. :-) Don't you want memory/resource-checking tools to be able to tell you if you've leaked MPI resources when MPI is actually finalized? (regardless of whether you're going to initialize it again) This means that when MPI_Finalize causes a ref count to go to zero, it would be good to actually finalize. That being said, such behavior doesn't have to be mandated -- it could be a quality of implementation issue. Additionally, you might want this kind of behavior: ----- MPI_Init_thread(NULL, NULL, MPI_THREAD_SINGLE, &provided); assert(provided == MPI_THREAD_SINGLE); MPI_Finalize(); MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, &provided); assert(provided == MPI_THREAD_MULTIPLE); MPI_Finalize(); MPI_Init_thread(NULL, NULL, MPI_THREAD_SINGLE, &provided); assert(provided == MPI_THREAD_SINGLE); MPI_Finalize(); ----- I.e., the implementation actually changes the thread level each time it is initialized after being finalized -- implying that the implementation actually finalizes and then actually re-initializes, assumedly from scratch. But again, this could be a quality of implementation issue -- MPI_Init_thread() makes no guarantees about what is returned in provided. If an implementation stays initialized under the covers, it could return THREAD_SINGLE in all 3 cases, above. One last thing: Brian raised a good, subtle point, too: once MPI gets finalized, then all MPI handles become stale. This might be a little tricky to do if MPI stays fully (or mostly) initialized under the covers. But this is *also* a quality of implementation issue. If you do something like this: ----- MPI_Init(...); MPI_Comm_dup(MPI_COMM_WORLD, &comm); MPI_Finalize(); MPI_Init(...); MPI_Send(..., comm); MPI_Finalize(); ----- Then you deserve whatever you get, such as: 1. MPI_Send will work (if you get [un]lucky -- this certainly won't be portable!) 2. MPI_Send segv's (because the handle now points off into la-la land) 3. MPI_Send returns some kind of "stale handle" error (high quality implementation) My overall point: if we start exploring the path of re-initialization a) it could solve a lot of Jeff Hammond's originally-cited problems b) and a lot of the issues that stemmed from a) c) we should make it allowable to stay initialized under the covers (i.e., a quality of implementation issue). -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Pavan Balaji 23 Apr '13

23 Apr '13

On 04/22/2013 05:17 PM US Central Time, Schulz, Martin wrote: >> Some MPI implementations might need an explicit *real* finalize at some >> point, if they need to clean up OS resources. For most MPI > > I was wondering about that - what platforms would be affected by > this? OS resources shouldn't be an issue, since the OS should take > care of this at process termination. Network resources may be a > different issue, though. Named pipes are an example of this. IBM's POE is notorious for such restrictions, though I don't know the technical reasons for why it is so, and whether it can be fixed. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Schulz, Martin 22 Apr '13

22 Apr '13

Hi Pavan, On Apr 22, 2013, at 3:13 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > I need to read the rest of the emails, so my email might not capture the > entire context -- > > Some MPI implementations might need an explicit *real* finalize at some > point, if they need to clean up OS resources. For most MPI I was wondering about that - what platforms would be affected by this? OS resources shouldn't be an issue, since the OS should take care of this at process termination. Network resources may be a different issue, though. > implementations this might not be a problem. For some MPI > implementations + compiler combinations, we can ask the compiler to > throw an "at_exit()" callback to do the cleanup. But this is something In most OSes, this should be doable at library end as well. > we need to explicitly consider before making a decision. Agreed. Martin > > -- Pavan > > On 04/22/2013 05:09 PM US Central Time, Schulz, Martin wrote: >> Would we actually specify what "not being initialized" anymore means, except that users are no longer allowed to call MPI routines before they call MPI_Init again? If so, the impact on the implementation should be minimal, since it could just keep MPI "alive" under the hood. >> >> Martin >> >> >> On Apr 22, 2013, at 12:08 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >> >>> I think the new semantics for MPI_Init() would be backward compatible, but we could also add a new routine: MPI_Init_awesome(). :) >>> >>> On 4/22/13 11:42 AM, Jeff Squyres (jsquyres) wrote: >>>> Sayantan and I talked about this on the phone today. It seems like re-initialization might be a good idea, but a good first step might be asking all the hardware/software vendors if there's a technical reason they don't allow re-initialzation today (other than "because it's not required by MPI"). I.e., I think that some API's/networks (like PSM) don't allow re-initialization -- is there a technical reason for that, or is it just an overcome-able software limitation? >>>> >>>> I'm not a big fan of re-defining the default behavior of MPI_INIT, however -- I think there might be a big impact on legacy applications. >>>> >>>> >>>> >>>> On Apr 19, 2013, at 3:06 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >>>> >>>>> Another issue that ought to consider is that MPI currently can only be initialized/finalized once. This requirement breaks the "MPI is a library" semantic and leads to some of the nastiness Jeff S. mentioned below. I think we should re-evaluate if this restriction is really required, or if it's just convenient for implementers. >>>>> >>>>> Another suggestion on this front -- Why not modify the semantics of MPI_Init to match what we want? >>>>> >>>>> MPI_Init: >>>>> >>>>> - Always THREAD_MULTIPLE >>>>> - Always a thread safe call >>>>> - Ref-counted >>>>> - Can be used to initialize/finalize MPI multiple times >>>>> - Cannot be combined with MPI_Init_thread >>>>> >>>>> If apps really care about getting rid of threading overhead, then they should use MPI_Init_thread() and use the thread level argument to give a performance hint. >>>>> >>>>> ~Jim. >>>>> >>>>> On 4/19/13 1:11 PM, Jeff Squyres (jsquyres) wrote: >>>>>> Points to think about for the Monday teleconf... >>>>>> >>>>>> With regards to ref-counted MPI_INIT / MPI_FINALIZE (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302) >>>>>> >>>>>> PROBLEMS IT SOLVES: >>>>>> - multiple, separate libraries in a single process needing access to MPI >>>>>> ==> works best when all entities call MPI_INIT* at the beginning of time, work for a "long" period of time, and then call MPI_FINALIZE at the end of time (i.e., there's no race condition -- see below) >>>>>> >>>>>> PROBLEMS IT DOES NOT SOLVE: >>>>>> - Implementation not providing MPI_THREAD_MULTIPLE support (e.g., if the separate libraries are in different threads and someone already initialized MPI with THREAD_SINGLE, other threads can't know if it's safe to call MPI_INIT* or not) >>>>>> - The "finalize" problem (i.e., can't guarantee to know if MPI has been finalized or not -- there's a race between calling MPI_FINALIZED, seeing that MPI is not finalized, and then calling MPI_INIT) >>>>>> >>>>>> PROBLEMS IT CREATES: >>>>>> - Will need to change the definition of "main thread" >>>>>> - Possibly also need to change the definitions of MPI_THREAD_SERIALIZED and MPI_THREAD_FUNNELED >>>>>> >>>>>> OPEN QUESTIONS: >>>>>> - Do we still need to keep the restriction that the thread that initializes MPI is the same thread that finalizes MPI? >>>>>> - Should we allow re-initialization? This effectively solves some (but not all) of the problems that have been discussed, but probably opens a new can of worms... >>>>>> >>>>>> >>>>>> >>>>>> On Apr 12, 2013, at 8:27 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>>>> >>>>>>> All, >>>>>>> >>>>>>> Jim and I will be late for the April 22nd meeting. So we decided to >>>>>>> move the endpoints discussion to the telecon after this one. >>>>>>> >>>>>>> I chatted with Jeff Squyres yesterday. He'll be driving the April 22nd >>>>>>> telecon to discuss more details on the ref-counted init/finalize issue. >>>>>>> He'll be sending out some notes before the call for discussion. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> -- Pavan >>>>>>> >>>>>>> On 04/08/2013 11:47 AM US Central Time, Pavan Balaji wrote: >>>>>>>> >>>>>>>> The next call will be on April 22nd, 11am central. Same telecon number. >>>>>>>> >>>>>>>> -- Pavan >>>>>>>> >>>>>>>> On 04/08/2013 11:42 AM US Central Time, Jim Dinan wrote: >>>>>>>>> Meeting notes are on the wiki: >>>>>>>>> >>>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2013-04-… >>>>>>>>> >>>>>>>>> >>>>>>>>> ~Jim. >>>>>>>>> >>>>>>>>> On 4/7/13 12:45 PM, Pavan Balaji wrote: >>>>>>>>>> All, >>>>>>>>>> >>>>>>>>>> This is a reminder that we'll have our hybrid telecon tomorrow at 11am. >>>>>>>>>> Here's the telecon information: >>>>>>>>>> >>>>>>>>>> International dial-in number: 1-719-234-7800 >>>>>>>>>> Domestic dial-in number: 1-888-850-4523 >>>>>>>>>> Participant Passcode: 314159 >>>>>>>>>> >>>>>>>>>> The main item we'll be discussing is Jeff Squyres' ref-count proposal. >>>>>>>>>> >>>>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302 >>>>>>>>>> >>>>>>>>>> -- Pavan >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Mpi3-hybridpm mailing list >>>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pavan Balaji >>>>>>> http://www.mcs.anl.gov/~balaji >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Mpi3-hybridpm mailing list >>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>> >>>> >>> _______________________________________________ >>> Mpi3-hybridpm mailing list >>> Mpi3-hybridpm(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> >> ________________________________________________________________________ >> Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm >> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >> >> >> >> >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Pavan Balaji 22 Apr '13

22 Apr '13

I need to read the rest of the emails, so my email might not capture the entire context -- Some MPI implementations might need an explicit *real* finalize at some point, if they need to clean up OS resources. For most MPI implementations this might not be a problem. For some MPI implementations + compiler combinations, we can ask the compiler to throw an "at_exit()" callback to do the cleanup. But this is something we need to explicitly consider before making a decision. -- Pavan On 04/22/2013 05:09 PM US Central Time, Schulz, Martin wrote: > Would we actually specify what "not being initialized" anymore means, except that users are no longer allowed to call MPI routines before they call MPI_Init again? If so, the impact on the implementation should be minimal, since it could just keep MPI "alive" under the hood. > > Martin > > > On Apr 22, 2013, at 12:08 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > >> I think the new semantics for MPI_Init() would be backward compatible, but we could also add a new routine: MPI_Init_awesome(). :) >> >> On 4/22/13 11:42 AM, Jeff Squyres (jsquyres) wrote: >>> Sayantan and I talked about this on the phone today. It seems like re-initialization might be a good idea, but a good first step might be asking all the hardware/software vendors if there's a technical reason they don't allow re-initialzation today (other than "because it's not required by MPI"). I.e., I think that some API's/networks (like PSM) don't allow re-initialization -- is there a technical reason for that, or is it just an overcome-able software limitation? >>> >>> I'm not a big fan of re-defining the default behavior of MPI_INIT, however -- I think there might be a big impact on legacy applications. >>> >>> >>> >>> On Apr 19, 2013, at 3:06 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >>> >>>> Another issue that ought to consider is that MPI currently can only be initialized/finalized once. This requirement breaks the "MPI is a library" semantic and leads to some of the nastiness Jeff S. mentioned below. I think we should re-evaluate if this restriction is really required, or if it's just convenient for implementers. >>>> >>>> Another suggestion on this front -- Why not modify the semantics of MPI_Init to match what we want? >>>> >>>> MPI_Init: >>>> >>>> - Always THREAD_MULTIPLE >>>> - Always a thread safe call >>>> - Ref-counted >>>> - Can be used to initialize/finalize MPI multiple times >>>> - Cannot be combined with MPI_Init_thread >>>> >>>> If apps really care about getting rid of threading overhead, then they should use MPI_Init_thread() and use the thread level argument to give a performance hint. >>>> >>>> ~Jim. >>>> >>>> On 4/19/13 1:11 PM, Jeff Squyres (jsquyres) wrote: >>>>> Points to think about for the Monday teleconf... >>>>> >>>>> With regards to ref-counted MPI_INIT / MPI_FINALIZE (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302) >>>>> >>>>> PROBLEMS IT SOLVES: >>>>> - multiple, separate libraries in a single process needing access to MPI >>>>> ==> works best when all entities call MPI_INIT* at the beginning of time, work for a "long" period of time, and then call MPI_FINALIZE at the end of time (i.e., there's no race condition -- see below) >>>>> >>>>> PROBLEMS IT DOES NOT SOLVE: >>>>> - Implementation not providing MPI_THREAD_MULTIPLE support (e.g., if the separate libraries are in different threads and someone already initialized MPI with THREAD_SINGLE, other threads can't know if it's safe to call MPI_INIT* or not) >>>>> - The "finalize" problem (i.e., can't guarantee to know if MPI has been finalized or not -- there's a race between calling MPI_FINALIZED, seeing that MPI is not finalized, and then calling MPI_INIT) >>>>> >>>>> PROBLEMS IT CREATES: >>>>> - Will need to change the definition of "main thread" >>>>> - Possibly also need to change the definitions of MPI_THREAD_SERIALIZED and MPI_THREAD_FUNNELED >>>>> >>>>> OPEN QUESTIONS: >>>>> - Do we still need to keep the restriction that the thread that initializes MPI is the same thread that finalizes MPI? >>>>> - Should we allow re-initialization? This effectively solves some (but not all) of the problems that have been discussed, but probably opens a new can of worms... >>>>> >>>>> >>>>> >>>>> On Apr 12, 2013, at 8:27 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>>> >>>>>> All, >>>>>> >>>>>> Jim and I will be late for the April 22nd meeting. So we decided to >>>>>> move the endpoints discussion to the telecon after this one. >>>>>> >>>>>> I chatted with Jeff Squyres yesterday. He'll be driving the April 22nd >>>>>> telecon to discuss more details on the ref-counted init/finalize issue. >>>>>> He'll be sending out some notes before the call for discussion. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -- Pavan >>>>>> >>>>>> On 04/08/2013 11:47 AM US Central Time, Pavan Balaji wrote: >>>>>>> >>>>>>> The next call will be on April 22nd, 11am central. Same telecon number. >>>>>>> >>>>>>> -- Pavan >>>>>>> >>>>>>> On 04/08/2013 11:42 AM US Central Time, Jim Dinan wrote: >>>>>>>> Meeting notes are on the wiki: >>>>>>>> >>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2013-04-… >>>>>>>> >>>>>>>> >>>>>>>> ~Jim. >>>>>>>> >>>>>>>> On 4/7/13 12:45 PM, Pavan Balaji wrote: >>>>>>>>> All, >>>>>>>>> >>>>>>>>> This is a reminder that we'll have our hybrid telecon tomorrow at 11am. >>>>>>>>> Here's the telecon information: >>>>>>>>> >>>>>>>>> International dial-in number: 1-719-234-7800 >>>>>>>>> Domestic dial-in number: 1-888-850-4523 >>>>>>>>> Participant Passcode: 314159 >>>>>>>>> >>>>>>>>> The main item we'll be discussing is Jeff Squyres' ref-count proposal. >>>>>>>>> >>>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302 >>>>>>>>> >>>>>>>>> -- Pavan >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mpi3-hybridpm mailing list >>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>> >>>>>> >>>>>> -- >>>>>> Pavan Balaji >>>>>> http://www.mcs.anl.gov/~balaji >>>>> >>>>> >>>> _______________________________________________ >>>> Mpi3-hybridpm mailing list >>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>> >>> >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Schulz, Martin 22 Apr '13

22 Apr '13

Would we actually specify what "not being initialized" anymore means, except that users are no longer allowed to call MPI routines before they call MPI_Init again? If so, the impact on the implementation should be minimal, since it could just keep MPI "alive" under the hood. Martin On Apr 22, 2013, at 12:08 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > I think the new semantics for MPI_Init() would be backward compatible, but we could also add a new routine: MPI_Init_awesome(). :) > > On 4/22/13 11:42 AM, Jeff Squyres (jsquyres) wrote: >> Sayantan and I talked about this on the phone today. It seems like re-initialization might be a good idea, but a good first step might be asking all the hardware/software vendors if there's a technical reason they don't allow re-initialzation today (other than "because it's not required by MPI"). I.e., I think that some API's/networks (like PSM) don't allow re-initialization -- is there a technical reason for that, or is it just an overcome-able software limitation? >> >> I'm not a big fan of re-defining the default behavior of MPI_INIT, however -- I think there might be a big impact on legacy applications. >> >> >> >> On Apr 19, 2013, at 3:06 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >> >>> Another issue that ought to consider is that MPI currently can only be initialized/finalized once. This requirement breaks the "MPI is a library" semantic and leads to some of the nastiness Jeff S. mentioned below. I think we should re-evaluate if this restriction is really required, or if it's just convenient for implementers. >>> >>> Another suggestion on this front -- Why not modify the semantics of MPI_Init to match what we want? >>> >>> MPI_Init: >>> >>> - Always THREAD_MULTIPLE >>> - Always a thread safe call >>> - Ref-counted >>> - Can be used to initialize/finalize MPI multiple times >>> - Cannot be combined with MPI_Init_thread >>> >>> If apps really care about getting rid of threading overhead, then they should use MPI_Init_thread() and use the thread level argument to give a performance hint. >>> >>> ~Jim. >>> >>> On 4/19/13 1:11 PM, Jeff Squyres (jsquyres) wrote: >>>> Points to think about for the Monday teleconf... >>>> >>>> With regards to ref-counted MPI_INIT / MPI_FINALIZE (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302) >>>> >>>> PROBLEMS IT SOLVES: >>>> - multiple, separate libraries in a single process needing access to MPI >>>> ==> works best when all entities call MPI_INIT* at the beginning of time, work for a "long" period of time, and then call MPI_FINALIZE at the end of time (i.e., there's no race condition -- see below) >>>> >>>> PROBLEMS IT DOES NOT SOLVE: >>>> - Implementation not providing MPI_THREAD_MULTIPLE support (e.g., if the separate libraries are in different threads and someone already initialized MPI with THREAD_SINGLE, other threads can't know if it's safe to call MPI_INIT* or not) >>>> - The "finalize" problem (i.e., can't guarantee to know if MPI has been finalized or not -- there's a race between calling MPI_FINALIZED, seeing that MPI is not finalized, and then calling MPI_INIT) >>>> >>>> PROBLEMS IT CREATES: >>>> - Will need to change the definition of "main thread" >>>> - Possibly also need to change the definitions of MPI_THREAD_SERIALIZED and MPI_THREAD_FUNNELED >>>> >>>> OPEN QUESTIONS: >>>> - Do we still need to keep the restriction that the thread that initializes MPI is the same thread that finalizes MPI? >>>> - Should we allow re-initialization? This effectively solves some (but not all) of the problems that have been discussed, but probably opens a new can of worms... >>>> >>>> >>>> >>>> On Apr 12, 2013, at 8:27 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>> >>>>> All, >>>>> >>>>> Jim and I will be late for the April 22nd meeting. So we decided to >>>>> move the endpoints discussion to the telecon after this one. >>>>> >>>>> I chatted with Jeff Squyres yesterday. He'll be driving the April 22nd >>>>> telecon to discuss more details on the ref-counted init/finalize issue. >>>>> He'll be sending out some notes before the call for discussion. >>>>> >>>>> Thanks, >>>>> >>>>> -- Pavan >>>>> >>>>> On 04/08/2013 11:47 AM US Central Time, Pavan Balaji wrote: >>>>>> >>>>>> The next call will be on April 22nd, 11am central. Same telecon number. >>>>>> >>>>>> -- Pavan >>>>>> >>>>>> On 04/08/2013 11:42 AM US Central Time, Jim Dinan wrote: >>>>>>> Meeting notes are on the wiki: >>>>>>> >>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2013-04-… >>>>>>> >>>>>>> >>>>>>> ~Jim. >>>>>>> >>>>>>> On 4/7/13 12:45 PM, Pavan Balaji wrote: >>>>>>>> All, >>>>>>>> >>>>>>>> This is a reminder that we'll have our hybrid telecon tomorrow at 11am. >>>>>>>> Here's the telecon information: >>>>>>>> >>>>>>>> International dial-in number: 1-719-234-7800 >>>>>>>> Domestic dial-in number: 1-888-850-4523 >>>>>>>> Participant Passcode: 314159 >>>>>>>> >>>>>>>> The main item we'll be discussing is Jeff Squyres' ref-count proposal. >>>>>>>> >>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302 >>>>>>>> >>>>>>>> -- Pavan >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Mpi3-hybridpm mailing list >>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>> >>>>> >>>>> -- >>>>> Pavan Balaji >>>>> http://www.mcs.anl.gov/~balaji >>>> >>>> >>> _______________________________________________ >>> Mpi3-hybridpm mailing list >>> Mpi3-hybridpm(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> >> > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Jim Dinan 22 Apr '13

22 Apr '13

I think the new semantics for MPI_Init() would be backward compatible, but we could also add a new routine: MPI_Init_awesome(). :) On 4/22/13 11:42 AM, Jeff Squyres (jsquyres) wrote: > Sayantan and I talked about this on the phone today. It seems like re-initialization might be a good idea, but a good first step might be asking all the hardware/software vendors if there's a technical reason they don't allow re-initialzation today (other than "because it's not required by MPI"). I.e., I think that some API's/networks (like PSM) don't allow re-initialization -- is there a technical reason for that, or is it just an overcome-able software limitation? > > I'm not a big fan of re-defining the default behavior of MPI_INIT, however -- I think there might be a big impact on legacy applications. > > > > On Apr 19, 2013, at 3:06 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > >> Another issue that ought to consider is that MPI currently can only be initialized/finalized once. This requirement breaks the "MPI is a library" semantic and leads to some of the nastiness Jeff S. mentioned below. I think we should re-evaluate if this restriction is really required, or if it's just convenient for implementers. >> >> Another suggestion on this front -- Why not modify the semantics of MPI_Init to match what we want? >> >> MPI_Init: >> >> - Always THREAD_MULTIPLE >> - Always a thread safe call >> - Ref-counted >> - Can be used to initialize/finalize MPI multiple times >> - Cannot be combined with MPI_Init_thread >> >> If apps really care about getting rid of threading overhead, then they should use MPI_Init_thread() and use the thread level argument to give a performance hint. >> >> ~Jim. >> >> On 4/19/13 1:11 PM, Jeff Squyres (jsquyres) wrote: >>> Points to think about for the Monday teleconf... >>> >>> With regards to ref-counted MPI_INIT / MPI_FINALIZE (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302) >>> >>> PROBLEMS IT SOLVES: >>> - multiple, separate libraries in a single process needing access to MPI >>> ==> works best when all entities call MPI_INIT* at the beginning of time, work for a "long" period of time, and then call MPI_FINALIZE at the end of time (i.e., there's no race condition -- see below) >>> >>> PROBLEMS IT DOES NOT SOLVE: >>> - Implementation not providing MPI_THREAD_MULTIPLE support (e.g., if the separate libraries are in different threads and someone already initialized MPI with THREAD_SINGLE, other threads can't know if it's safe to call MPI_INIT* or not) >>> - The "finalize" problem (i.e., can't guarantee to know if MPI has been finalized or not -- there's a race between calling MPI_FINALIZED, seeing that MPI is not finalized, and then calling MPI_INIT) >>> >>> PROBLEMS IT CREATES: >>> - Will need to change the definition of "main thread" >>> - Possibly also need to change the definitions of MPI_THREAD_SERIALIZED and MPI_THREAD_FUNNELED >>> >>> OPEN QUESTIONS: >>> - Do we still need to keep the restriction that the thread that initializes MPI is the same thread that finalizes MPI? >>> - Should we allow re-initialization? This effectively solves some (but not all) of the problems that have been discussed, but probably opens a new can of worms... >>> >>> >>> >>> On Apr 12, 2013, at 8:27 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>> >>>> All, >>>> >>>> Jim and I will be late for the April 22nd meeting. So we decided to >>>> move the endpoints discussion to the telecon after this one. >>>> >>>> I chatted with Jeff Squyres yesterday. He'll be driving the April 22nd >>>> telecon to discuss more details on the ref-counted init/finalize issue. >>>> He'll be sending out some notes before the call for discussion. >>>> >>>> Thanks, >>>> >>>> -- Pavan >>>> >>>> On 04/08/2013 11:47 AM US Central Time, Pavan Balaji wrote: >>>>> >>>>> The next call will be on April 22nd, 11am central. Same telecon number. >>>>> >>>>> -- Pavan >>>>> >>>>> On 04/08/2013 11:42 AM US Central Time, Jim Dinan wrote: >>>>>> Meeting notes are on the wiki: >>>>>> >>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2013-04-… >>>>>> >>>>>> >>>>>> ~Jim. >>>>>> >>>>>> On 4/7/13 12:45 PM, Pavan Balaji wrote: >>>>>>> All, >>>>>>> >>>>>>> This is a reminder that we'll have our hybrid telecon tomorrow at 11am. >>>>>>> Here's the telecon information: >>>>>>> >>>>>>> International dial-in number: 1-719-234-7800 >>>>>>> Domestic dial-in number: 1-888-850-4523 >>>>>>> Participant Passcode: 314159 >>>>>>> >>>>>>> The main item we'll be discussing is Jeff Squyres' ref-count proposal. >>>>>>> >>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302 >>>>>>> >>>>>>> -- Pavan >>>>>>> >>>>>> _______________________________________________ >>>>>> Mpi3-hybridpm mailing list >>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>> >>>> >>>> -- >>>> Pavan Balaji >>>> http://www.mcs.anl.gov/~balaji >>> >>> >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > >

1 0

Re: [Mpi3-hybridpm] [EXTERNAL] Re: Reminder for the hybrid telecon tomorrow
by Barrett, Brian W 22 Apr '13

22 Apr '13

Portals has traditionally disallowed reinitialization, although I'm not sure there's a good reason other than it's a bunch of internal bookkeeping (ie, we need to make sure user handles are somehow cleaned up so they can't be reused). We could probably make it work, but not with the guarantee that the endpoint address wouldn't change, so you'd have to go through the entire initialization process. Given the cost of reinitialization, I think it's a design point we need to push users to avoid. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories ________________________________________ From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org [mpi3-hybridpm-bounces(a)lists.mpi-forum.org] on behalf of Jeff Squyres (jsquyres) [jsquyres(a)cisco.com] Sent: Monday, April 22, 2013 10:42 AM To: <mpi3-hybridpm(a)lists.mpi-forum.org> Subject: [EXTERNAL] Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow Sayantan and I talked about this on the phone today. It seems like re-initialization might be a good idea, but a good first step might be asking all the hardware/software vendors if there's a technical reason they don't allow re-initialzation today (other than "because it's not required by MPI"). I.e., I think that some API's/networks (like PSM) don't allow re-initialization -- is there a technical reason for that, or is it just an overcome-able software limitation? I'm not a big fan of re-defining the default behavior of MPI_INIT, however -- I think there might be a big impact on legacy applications. On Apr 19, 2013, at 3:06 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > Another issue that ought to consider is that MPI currently can only be initialized/finalized once. This requirement breaks the "MPI is a library" semantic and leads to some of the nastiness Jeff S. mentioned below. I think we should re-evaluate if this restriction is really required, or if it's just convenient for implementers. > > Another suggestion on this front -- Why not modify the semantics of MPI_Init to match what we want? > > MPI_Init: > > - Always THREAD_MULTIPLE > - Always a thread safe call > - Ref-counted > - Can be used to initialize/finalize MPI multiple times > - Cannot be combined with MPI_Init_thread > > If apps really care about getting rid of threading overhead, then they should use MPI_Init_thread() and use the thread level argument to give a performance hint. > > ~Jim. > > On 4/19/13 1:11 PM, Jeff Squyres (jsquyres) wrote: >> Points to think about for the Monday teleconf... >> >> With regards to ref-counted MPI_INIT / MPI_FINALIZE (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302) >> >> PROBLEMS IT SOLVES: >> - multiple, separate libraries in a single process needing access to MPI >> ==> works best when all entities call MPI_INIT* at the beginning of time, work for a "long" period of time, and then call MPI_FINALIZE at the end of time (i.e., there's no race condition -- see below) >> >> PROBLEMS IT DOES NOT SOLVE: >> - Implementation not providing MPI_THREAD_MULTIPLE support (e.g., if the separate libraries are in different threads and someone already initialized MPI with THREAD_SINGLE, other threads can't know if it's safe to call MPI_INIT* or not) >> - The "finalize" problem (i.e., can't guarantee to know if MPI has been finalized or not -- there's a race between calling MPI_FINALIZED, seeing that MPI is not finalized, and then calling MPI_INIT) >> >> PROBLEMS IT CREATES: >> - Will need to change the definition of "main thread" >> - Possibly also need to change the definitions of MPI_THREAD_SERIALIZED and MPI_THREAD_FUNNELED >> >> OPEN QUESTIONS: >> - Do we still need to keep the restriction that the thread that initializes MPI is the same thread that finalizes MPI? >> - Should we allow re-initialization? This effectively solves some (but not all) of the problems that have been discussed, but probably opens a new can of worms... >> >> >> >> On Apr 12, 2013, at 8:27 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >>> All, >>> >>> Jim and I will be late for the April 22nd meeting. So we decided to >>> move the endpoints discussion to the telecon after this one. >>> >>> I chatted with Jeff Squyres yesterday. He'll be driving the April 22nd >>> telecon to discuss more details on the ref-counted init/finalize issue. >>> He'll be sending out some notes before the call for discussion. >>> >>> Thanks, >>> >>> -- Pavan >>> >>> On 04/08/2013 11:47 AM US Central Time, Pavan Balaji wrote: >>>> >>>> The next call will be on April 22nd, 11am central. Same telecon number. >>>> >>>> -- Pavan >>>> >>>> On 04/08/2013 11:42 AM US Central Time, Jim Dinan wrote: >>>>> Meeting notes are on the wiki: >>>>> >>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2013-04-… >>>>> >>>>> >>>>> ~Jim. >>>>> >>>>> On 4/7/13 12:45 PM, Pavan Balaji wrote: >>>>>> All, >>>>>> >>>>>> This is a reminder that we'll have our hybrid telecon tomorrow at 11am. >>>>>> Here's the telecon information: >>>>>> >>>>>> International dial-in number: 1-719-234-7800 >>>>>> Domestic dial-in number: 1-888-850-4523 >>>>>> Participant Passcode: 314159 >>>>>> >>>>>> The main item we'll be discussing is Jeff Squyres' ref-count proposal. >>>>>> >>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302 >>>>>> >>>>>> -- Pavan >>>>>> >>>>> _______________________________________________ >>>>> Mpi3-hybridpm mailing list >>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>> >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >> >> > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ _______________________________________________ Mpi3-hybridpm mailing list Mpi3-hybridpm(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow
by Sur, Sayantan 22 Apr '13

22 Apr '13

I do not have wiki modification privileges so treat this (+Jeff's email about re-init) as the minutes. Another item we talked about in addition to re-initialization was a side library that helps coordination between MPI and other threading models. This "side library" which is an impl of a side document is not a part of the spec, but approved by the Forum anyways. The API for that library could expose the number of threads that want to call into MPI, Infos, Asserts (although currently MPI doesn't init with asserts and infos, but there has been talk about it in the future), and Helper threads like feature. The API could provide simple features such as ref-counted Init and error-detection functionality such as Thread X's assert conflicts Thread Y's assert. Etc. Of course, if we could fix some of the issues within MPI spec, it would be preferable, but if the issues prove to be intractable, a standalone library could provide practical useful functionality. Sayantan > -----Original Message----- > From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org [mailto:mpi3-hybridpm- > bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Squyres (jsquyres) > Sent: Monday, April 22, 2013 9:42 AM > To: <mpi3-hybridpm(a)lists.mpi-forum.org> > Subject: Re: [Mpi3-hybridpm] Reminder for the hybrid telecon tomorrow > > Sayantan and I talked about this on the phone today. It seems like re- > initialization might be a good idea, but a good first step might be asking all the > hardware/software vendors if there's a technical reason they don't allow re- > initialzation today (other than "because it's not required by MPI"). I.e., I > think that some API's/networks (like PSM) don't allow re-initialization -- is > there a technical reason for that, or is it just an overcome-able software > limitation? > > I'm not a big fan of re-defining the default behavior of MPI_INIT, however -- > I think there might be a big impact on legacy applications. > > > > On Apr 19, 2013, at 3:06 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > > > Another issue that ought to consider is that MPI currently can only be > initialized/finalized once. This requirement breaks the "MPI is a library" > semantic and leads to some of the nastiness Jeff S. mentioned below. I think > we should re-evaluate if this restriction is really required, or if it's just > convenient for implementers. > > > > Another suggestion on this front -- Why not modify the semantics of > MPI_Init to match what we want? > > > > MPI_Init: > > > > - Always THREAD_MULTIPLE > > - Always a thread safe call > > - Ref-counted > > - Can be used to initialize/finalize MPI multiple times > > - Cannot be combined with MPI_Init_thread > > > > If apps really care about getting rid of threading overhead, then they > should use MPI_Init_thread() and use the thread level argument to give a > performance hint. > > > > ~Jim. > > > > On 4/19/13 1:11 PM, Jeff Squyres (jsquyres) wrote: > >> Points to think about for the Monday teleconf... > >> > >> With regards to ref-counted MPI_INIT / MPI_FINALIZE (https://svn.mpi- > forum.org/trac/mpi-forum-web/ticket/302) > >> > >> PROBLEMS IT SOLVES: > >> - multiple, separate libraries in a single process needing access to MPI > >> ==> works best when all entities call MPI_INIT* at the beginning of > >> time, work for a "long" period of time, and then call MPI_FINALIZE at > >> the end of time (i.e., there's no race condition -- see below) > >> > >> PROBLEMS IT DOES NOT SOLVE: > >> - Implementation not providing MPI_THREAD_MULTIPLE support (e.g., if > >> the separate libraries are in different threads and someone already > >> initialized MPI with THREAD_SINGLE, other threads can't know if it's > >> safe to call MPI_INIT* or not) > >> - The "finalize" problem (i.e., can't guarantee to know if MPI has > >> been finalized or not -- there's a race between calling > >> MPI_FINALIZED, seeing that MPI is not finalized, and then calling > >> MPI_INIT) > >> > >> PROBLEMS IT CREATES: > >> - Will need to change the definition of "main thread" > >> - Possibly also need to change the definitions of > >> MPI_THREAD_SERIALIZED and MPI_THREAD_FUNNELED > >> > >> OPEN QUESTIONS: > >> - Do we still need to keep the restriction that the thread that initializes > MPI is the same thread that finalizes MPI? > >> - Should we allow re-initialization? This effectively solves some (but not > all) of the problems that have been discussed, but probably opens a new can > of worms... > >> > >> > >> > >> On Apr 12, 2013, at 8:27 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> > >>> All, > >>> > >>> Jim and I will be late for the April 22nd meeting. So we decided to > >>> move the endpoints discussion to the telecon after this one. > >>> > >>> I chatted with Jeff Squyres yesterday. He'll be driving the April > >>> 22nd telecon to discuss more details on the ref-counted init/finalize > issue. > >>> He'll be sending out some notes before the call for discussion. > >>> > >>> Thanks, > >>> > >>> -- Pavan > >>> > >>> On 04/08/2013 11:47 AM US Central Time, Pavan Balaji wrote: > >>>> > >>>> The next call will be on April 22nd, 11am central. Same telecon number. > >>>> > >>>> -- Pavan > >>>> > >>>> On 04/08/2013 11:42 AM US Central Time, Jim Dinan wrote: > >>>>> Meeting notes are on the wiki: > >>>>> > >>>>> https://svn.mpi-forum.org/trac/mpi-forum- > web/wiki/MPI3Hybrid/notes > >>>>> -2013-04-08 > >>>>> > >>>>> > >>>>> ~Jim. > >>>>> > >>>>> On 4/7/13 12:45 PM, Pavan Balaji wrote: > >>>>>> All, > >>>>>> > >>>>>> This is a reminder that we'll have our hybrid telecon tomorrow at > 11am. > >>>>>> Here's the telecon information: > >>>>>> > >>>>>> International dial-in number: 1-719-234-7800 Domestic dial-in > >>>>>> number: 1-888-850-4523 Participant Passcode: 314159 > >>>>>> > >>>>>> The main item we'll be discussing is Jeff Squyres' ref-count proposal. > >>>>>> > >>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302 > >>>>>> > >>>>>> -- Pavan > >>>>>> > >>>>> _______________________________________________ > >>>>> Mpi3-hybridpm mailing list > >>>>> Mpi3-hybridpm(a)lists.mpi-forum.org > >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > >>>> > >>> > >>> -- > >>> Pavan Balaji > >>> http://www.mcs.anl.gov/~balaji > >> > >> > > _______________________________________________ > > Mpi3-hybridpm mailing list > > Mpi3-hybridpm(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0