September 2013 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [mpiwg-hybridpm] Reminder: Hybrid WG call tomorrow
by Pavan Balaji 23 Sep '13

23 Sep '13

Here are the minutes for the call: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2013-09-… The minutes Jeff took down from the Forum meeting are also uploaded to the wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2013-09-… -- Pavan On Sep 22, 2013, at 11:46 PM, Pavan Balaji wrote: > Folks, > > Reminder for the hybrid WG call tomorrow at 11am central. > > Note that we moved to webex since the last call. > > https://cisco.webex.com/ciscosales/j.php?ED=236535652&UID=0&PW=NOGE0NDk5MmV… > > If prompted, the password is "hybrid". > > The main agenda item for the call is the INIT/FINALIZE proposal that was discussed at the Forum meeting. Time permitting, we'll discuss the endpoints proposal as well. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Please re-join the webex
by Jeff Squyres (jsquyres) 23 Sep '13

23 Sep '13

Sorry about that folks; please re-join the webex... My fault... (I hit the wrong button to end the meeting) -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-hybridpm] Reminder: Hybrid WG call tomorrow
by Pavan Balaji 23 Sep '13

23 Sep '13

What happened? I got kicked out and it's not allowing me to join back. -- Pavan On Sep 22, 2013, at 11:46 PM, Pavan Balaji wrote: > Folks, > > Reminder for the hybrid WG call tomorrow at 11am central. > > Note that we moved to webex since the last call. > > https://cisco.webex.com/ciscosales/j.php?ED=236535652&UID=0&PW=NOGE0NDk5MmV… > > If prompted, the password is "hybrid". > > The main agenda item for the call is the INIT/FINALIZE proposal that was discussed at the Forum meeting. Time permitting, we'll discuss the endpoints proposal as well. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Reminder: Hybrid WG call tomorrow
by Pavan Balaji 22 Sep '13

22 Sep '13

Folks, Reminder for the hybrid WG call tomorrow at 11am central. Note that we moved to webex since the last call. https://cisco.webex.com/ciscosales/j.php?ED=236535652&UID=0&PW=NOGE0NDk5MmV… If prompted, the password is "hybrid". The main agenda item for the call is the INIT/FINALIZE proposal that was discussed at the Forum meeting. Time permitting, we'll discuss the endpoints proposal as well. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Comments from the INIT/FINALIZE presentation today
by Jeff Squyres (jsquyres) 13 Sep '13

13 Sep '13

Short version: the proposal went down in flames. More detail: Here's some of the specific points that caused a lot of contention: 1. Although it took me a while to understand his comments, Torsten more-or-less immediately identified a major flaw in the proposal (and Bill Gropp agreed): the idea of INIT/FINALIZE only being collective when opening/closing an epoch leads to real problems when combined with the idea of multiple threads (assume THREAD_MULTIPLE for the purpose of this discussion point). The real problem is that individual threads have the power to alter global process state. One example: it becomes racy if multiple threads in multiple MPI processes repeatedly invoke the following: // In each thread for (i = 0; i < LARGE_NUMBER; ++i) { MPI_Init(NULL, NULL); MPI_Finalize(); } You can't know how many epochs there will be in a given process because it's racy. And therefore you can't guarantee correctness, because you can/will have a different number of epochs in each process. Making INIT/FINALIZE always collective solves this issue, but I recall we had other issues with that (I'm too jet lagged/tired to remember what they were offhand). Regardless, this is a major issue that we'll need to figure out if this proposal is going to go any further. 2. One possible solution to this entire mess was proposed by Alexander Supalov: - the first call to MPI_INIT initializes MPI - subsequent calls to MPI_INIT are effectively no-ops - make MPI_Finalize a no-op I.e., you initialize once and never finalize. There's no issue with threads, no issues with collective/local-only INIT/FINALIZE behavior, ...etc. It's a simple solution. 3. Another possible solution was proposed: create MPI_INIT2 / MPI_FINALIZE2 (pick better names, I know), something like this: MPI_Init2(&my_mpi_comm_world, &my_mpi_comm_self); MPI_Send(..., my_mpi_comm_world); MPI_Finalize2(&my_mpi_comm_world, &my_mpi_comm_self); I.e., you basically get a handle that identifies *your* "session" of MPI usage. INIT2/FINALIZE2 are always collective, they return a "personalized"/unique COMM_WORLD/COMM_SELF that basically identify the caller's MPI "session", they are thread safe (e.g., if I ATTR_PUT on my_mpi_comm_world, that attr is not available on your_mpi_comm_world), etc. This avoids the problem of threads calling INIT and effectively altering global state (especially when it's racy). Granted, there's lot of other MPI global symbols (e.g., datatypes, etc.). But many, many things come from MPI_COMM_WORLD (e.g., all non-dynamic communicators, files, and windows -- need to think about dynamics, though...). So perhaps having *your* copy of MPI_COMM_WORLD/SELF would be sufficient...? It's an interesting idea; needs some thought (e.g., what to do with the existing MPI_COMM_WORLD and MPI_COMM_SELF symbols -- I'm sure there's lots of other details to think through). 4. Torsten was also concerned that there was a lack of a solid use case for stacked libraries that call INIT/FINALIZE themselves (because today's libraries currently rely on the application to call INIT/FINALIZE), and also a lack of a solid use case for re-initialization of MPI. To me, this is a chicken/egg issue (i.e., no one does it because you can't), but it's hard to argue against it. Perhaps those of you who are closer to app developers can cite specific needs/use cases...? 5. Bill/Torsten pointed out that in slide 6, INITIALIZED/FINALIZED are *not* inherently racy per MPI-2+ definitions. Jeff Hammond's original question was how to make multiple threads in an application (presumably from multiple different libraries) safe to initialize MPI, perhaps where they all do something like this: // Disregarding issues about <THREAD_MULTIPLE for this discussion point MPI_Initialized(&flag); if (0 == flag) MPI_Init_thread(...MULTIPLE...); Bill pointed out that MPI (2 and 3) says that for THREAD_MULTIPLE, the program must be correct for any interleaving of MPI calls from multiple threads. By this definition, what JeffH wants to do is erroneous because multiple threads would call MPI_INIT, and is therefore a new use case. While this is jaw-dropping to me, technically it is correct. The standard answer to do this is that the different threads must use some mechanism outside of MPI for synchronization/state/whatever. (I realize this doesn't help JeffH's use case at all, because different libraries have no knowledge of each other and can't coordinate who will call MPI_INIT. This is just more background info for those of you who weren't there, because we've been actively working on extending the standard to allow JeffH's use case -- I'm just reporting on what was discussed) ----- Martin took notes for me while I was presenting. His notes are below, interspersed with my comments and further explanations on his notes. Begin forwarded message: > From: Martin Schulz <schulz6(a)llnl.gov> > Subject: Hybrid comments > Date: September 13, 2013 3:12:07 PM GMT+02:00 > To: Jeff Squyres <jsquyres(a)cisco.com> > > Hybrid Init/Finalize Notes > > Distinction of undefined vs. prohibited things after epoch > - handle don't have to go stale > - behavior is undefined > - goal should be to keep the spirit of the current Finalize --> Torsten was picking on slide 21, where we said a "low quality implementation must behave as if finalized when refcount decrements to 0". His point is that if someone does something erroneous, the behavior is actually *undefined* -- we can't say the implementation *must behave* a certain way if a user does something erroneous. We're certainly not going for that; it was just a poorly worded slide (although it took me quite a while to understand his objection!). > > What does "collective" mean after close of an epoch > (if multiple processes are left after a finalize and we initialize again) --> The question was raised in slide 23: what happens if all 3 MPI processes call MPI_INIT again? Dunno -- we didn't talk about that... > Launch a legacy library inside a new MPI code on a subset of nodes > problem: you can't restrict MPI_COMM_WORLD --> This comment was basically the result of a lengthy discussion about the "INIT/FINALIZE not collective unless opening/closing an epoch" issue. If you allow such behavior, then you could allow > > Big issue: you get a potentially different MCW if you do local inits (not sure who else does this) --> This issue is also related to the "INIT/FINALIZE not collective unless opening/closing an epoch..." discussion. > > Corner case 6: > thread ordering can result in more or less epochs --> See my point #1 up above. > > Initialized/Finalized is not racy because MPI 2 requires that all interleavings have to be valid - otherwise the code is erroneous --> See my point #5 up above. > > create bubbles with their own MCW > have init return separate MCW > enables separation --> See my point #3 up above -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-hybridpm] Does anyone anticipate joining the webex tomorrow?
by Alice Koniges 12 Sep '13

12 Sep '13

I will try to join if I can from Beijing China. Not sure how well it will work. I think that is 8:30PM china time, but if someone thinks otherwise please let me know. Alice On Sep 13, 2013, at 1:13 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > Sounds good. > > On Sep 12, 2013, at 6:10 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> >> I was hoping to, but it's 6.30am here, so I can't say for sure. >> >> Can I ping you over email if I plan to join, so you don't have to set it up if you don't hear from me? >> >> -- Pavan >> >> On Sep 12, 2013, at 10:59 AM, Jeff Squyres (jsquyres) wrote: >> >>> Does anyone anticipate joining the Hybrid WG plenary webex where I present the INIT/FINALIZE slides tomorrow? >>> >>> It's at 1:30pm Madrid time. >>> >>> I ask because we're wondering if it's worthwhile to setup the speakerphone in the room, etc. >>> >>> -- >>> Jeff Squyres >>> jsquyres(a)cisco.com >>> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >>> >>> _______________________________________________ >>> mpiwg-hybridpm mailing list >>> mpiwg-hybridpm(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> mpiwg-hybridpm mailing list >> mpiwg-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm

1 0

Re: [mpiwg-hybridpm] Does anyone anticipate joining the webex tomorrow?
by Jim Dinan 12 Sep '13

12 Sep '13

Eek, that's 5:30am EST. I think you've got it under control without me. :) On Thu, Sep 12, 2013 at 12:13 PM, Jeff Squyres (jsquyres) < jsquyres(a)cisco.com> wrote: > Sounds good. > > On Sep 12, 2013, at 6:10 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > > > I was hoping to, but it's 6.30am here, so I can't say for sure. > > > > Can I ping you over email if I plan to join, so you don't have to set it > up if you don't hear from me? > > > > -- Pavan > > > > On Sep 12, 2013, at 10:59 AM, Jeff Squyres (jsquyres) wrote: > > > >> Does anyone anticipate joining the Hybrid WG plenary webex where I > present the INIT/FINALIZE slides tomorrow? > >> > >> It's at 1:30pm Madrid time. > >> > >> I ask because we're wondering if it's worthwhile to setup the > speakerphone in the room, etc. > >> > >> -- > >> Jeff Squyres > >> jsquyres(a)cisco.com > >> For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > >> _______________________________________________ > >> mpiwg-hybridpm mailing list > >> mpiwg-hybridpm(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > > > _______________________________________________ > > mpiwg-hybridpm mailing list > > mpiwg-hybridpm(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm >

1 0

Re: [mpiwg-hybridpm] Does anyone anticipate joining the webex tomorrow?
by Jeff Squyres (jsquyres) 12 Sep '13

12 Sep '13

Sounds good. On Sep 12, 2013, at 6:10 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > I was hoping to, but it's 6.30am here, so I can't say for sure. > > Can I ping you over email if I plan to join, so you don't have to set it up if you don't hear from me? > > -- Pavan > > On Sep 12, 2013, at 10:59 AM, Jeff Squyres (jsquyres) wrote: > >> Does anyone anticipate joining the Hybrid WG plenary webex where I present the INIT/FINALIZE slides tomorrow? >> >> It's at 1:30pm Madrid time. >> >> I ask because we're wondering if it's worthwhile to setup the speakerphone in the room, etc. >> >> -- >> Jeff Squyres >> jsquyres(a)cisco.com >> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> _______________________________________________ >> mpiwg-hybridpm mailing list >> mpiwg-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-hybridpm] Does anyone anticipate joining the webex tomorrow?
by Pavan Balaji 12 Sep '13

12 Sep '13

I was hoping to, but it's 6.30am here, so I can't say for sure. Can I ping you over email if I plan to join, so you don't have to set it up if you don't hear from me? -- Pavan On Sep 12, 2013, at 10:59 AM, Jeff Squyres (jsquyres) wrote: > Does anyone anticipate joining the Hybrid WG plenary webex where I present the INIT/FINALIZE slides tomorrow? > > It's at 1:30pm Madrid time. > > I ask because we're wondering if it's worthwhile to setup the speakerphone in the room, etc. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Does anyone anticipate joining the webex tomorrow?
by Jeff Squyres (jsquyres) 12 Sep '13

12 Sep '13

Does anyone anticipate joining the Hybrid WG plenary webex where I present the INIT/FINALIZE slides tomorrow? It's at 1:30pm Madrid time. I ask because we're wondering if it's worthwhile to setup the speakerphone in the room, etc. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0