June 2013 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] First cut at slides
by Jeff Squyres (jsquyres) 29 Jun '13

29 Jun '13

On Jun 28, 2013, at 10:05 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> Put differently: A's FINALIZE is collective with B/first FINALIZE (by >> definition), but in practice it may need to synchronize with B/second >> FINALIZE. > > Good point. Would requiring MPI to not synchronize when it's not finalizing fix this? There be dragons thar... ___====-_ _-====___ __--^^^ // \\ ^^^--_ _-^ // ( ) \\ ^-_ - // |\^^/| \\ - _/ // (0::0) \\ \_ / (( \\// )) \ - MPI \\ (oo) // FINALIZE - - \\ / \/ \ // - - \\/ \// - / /| /\ ( ) /\ |\ \ |/ | /\_/\_/\_/ \_/\ ( /\ ) /\_/ \_/\_/\_/\ | \| ` |/ V V ` V \_(| | | |)_/ V ' V V \| ' ` ` ` ` / | | | | \ ' ' ' ' <( | | | | )> <__\_| | | |_\__> ^^^^ ^^^ ^^^ ^^^^^ -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-hybridpm] First cut at slides
by Pavan Balaji 28 Jun '13

28 Jun '13

On 06/24/2013 10:01 AM, Jeff Squyres (jsquyres) wrote: > Put differently: A's FINALIZE is collective with B/first FINALIZE (by > definition), but in practice it may need to synchronize with B/second > FINALIZE. Good point. Would requiring MPI to not synchronize when it's not finalizing fix this? -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Reminder for call on Monday
by Jim Dinan 28 Jun '13

28 Jun '13

Hi Pavan, Sorry, I won't be able to attend the next meeting because I'm moving. The following Monday will be no good either, since I won't have time to prepare anything between now and then. Could we discuss this at the 4/15 meeting? Cheers, ~Jim. On Thu, Jun 27, 2013 at 10:09 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > Reminder that our next Hybrid call is on Monday at 11am central: > > Toll-free dial-in number: 1-866-398-2885 > Toll dial-in number: 1-719-457-6209 > Participant Passcode: 3141592 > > We'll discuss the slides JeffS put together. > > I'm not sure Jim is ready with the Endpoints proposal yet. Jim, if you > are, can you send it around before the call so we have some time to look > through it? > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > ______________________________**_________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.**org <Mpi3-hybridpm(a)lists.mpi-forum.org> > http://lists.mpi-forum.org/**mailman/listinfo.cgi/mpi3-**hybridpm<http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm> >

1 0

[Mpi3-hybridpm] Reminder for call on Monday
by Pavan Balaji 27 Jun '13

27 Jun '13

Reminder that our next Hybrid call is on Monday at 11am central: Toll-free dial-in number: 1-866-398-2885 Toll dial-in number: 1-719-457-6209 Participant Passcode: 3141592 We'll discuss the slides JeffS put together. I'm not sure Jim is ready with the Endpoints proposal yet. Jim, if you are, can you send it around before the call so we have some time to look through it? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] First cut at slides
by Jeff Squyres (jsquyres) 24 Jun '13

24 Jun '13

On Jun 22, 2013, at 9:30 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> I guess my point is that process A must wait for the *actual* >> finalization in process B (which is the 2nd one), and if Finalize >> synchronizes (which most.. but not all.. MPI implementations do), >> then A will block until B calls the 2nd Finalize. > > Eh? My reading was that they *might* synchronize on every FINALIZE call, but will *finalize* only when ref-count is zero. Yes, that is definitely possible (since "collective" *might* mean synchronize). But I gave this some more thought over the weekend, and I'm now even more uncomfortable with it: in my example, process A calls FINALIZE once, and that's when it actually finalizes process A. It's collective with the *first* call to FINALIZE on proc B, which *doesn't* finalize process B. So yes, A/FINALIZE *might* synch with B/first FINALIZE -- but that's not really the point. A also has the added requirement that it needs to *actually finalize* A. And it therefore needs to synchronize with B/second FINALIZE, for which it was *not* explicitly paired via a collective operation. Put differently: A's FINALIZE is collective with B/first FINALIZE (by definition), but in practice it may need to synchronize with B/second FINALIZE. That's just weird. >> But I think I'm coming to the conclusion that if you want to not be >> affected by Finalize-possibly-blocking semantics, then you should >> just call MPI_COMM_DISCONNECT first. > > I don't think so. What do you mean? -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-hybridpm] First cut at slides
by Martin Schulz 23 Jun '13

23 Jun '13

Hi Jeff, all, Just catching up with this thread now - looking over the slides, I had a few questions/comments: Slide 3: why does this eliminates the issues of calling before Init? Wouldn't the user still have to call Init and an MPI call at a point when the init refcount is 0 would call an error? Slide 6: what do you mean with actually finalize? Will a user see a difference (unless he/she keeps track of how many times this was called, i.e., has a separate ref count)? Slide 7: I am not sure if the term epoch really helps (we had the same situation in MPI_T and we just talked about MPI_T being initialized by saying more init calls than finalize calls). The reason why I am saying this is a user cannot and should not think of epochs. After a part of a program that does its own initialization (e.g., a library) calls finalize, this component should not ever make any MPI calls until it calls init again, independent of whether we are in a different epoch or not. Epoch should only make sense as implementation details in some implementations. Slide 8: why not deprecate and later remove these functions? Nobody should have a need to check - if a component wants to call MPI, it can initialize it. We had the same discussion in MPI_T and never found a good use for these functions (that's why they are not there). I know the semantics are slightly different there, but I just not sure we should expose the epoch concept or the ability to test for it to users. Having said that, though, it may worthwhile to expose the ref count through a mandatory control variable in MPI_T for tools. Slide 9: I don't think this restrictions makes any sense anymore - a component would have to check whether it is the main thread. If yes, its own called must be delayed until the no-op finalizes are called (which is impossible since the component will have shutdown at that point) or if not, it has to check whether it is the one that drops the ref count to zero. If so, it is not allowed to call finalize, which again is problematic because it can't call it later. Do we still need the notion of a main thread (I realize it may/will impact implementations, but from the perspective of creating a clean and consistent standard)? Slide 14: Is there any guarantee that the size of COMM_WORLD stays the same? If processes drop out between epochs, COMM_WORLD would be smaller, right? Also, if you spawn processes, finalize, and reinitialize on all processes, do the new ones become part of COMM_WORLD? Or should all intracommunicators be recreated? Martin On Jun 18, 2013, at 11:00 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > On Jun 18, 2013, at 11:25 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > >>> But maybe that isn't disastrous. If you don't want that (potential) >>> behavior, you should COMM_DISCONNECT before finalize. >> >> Does FINALIZE imply DISCONNECT? > > Technically speaking, no. FINALIZE is collective over all connected processes (where "connected" includes those who have not yet DISCONNECTed). > > In one sense, everyone disconnects because FINALIZE completes. But technically, none of them explicitly called DISCONNECT. > >> Could we add that behavior without >> breaking existing semantics? > > Getting more specific on FINALIZE semantics is a deep, dark, rat hole with dragons and other terrible monsters in it... > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-hybridpm] First cut at slides
by Pavan Balaji 22 Jun '13

22 Jun '13

One more issue -- Consider a library libfoo.la, which provides: foo_init() { MPI_INIT_THREAD(THREAD_MULTIPLE); } foo_finalize() { MPI_FINALIZE(); } int main() { MPI_INIT_THREAD(FUNNELED); foo_init(); level = QUERY_THREAD_LEVEL(); foo_finalize(); if (level == MULTIPLE) { /* do some THREAD_MULTIPLE stuff */ } return 0; } I guess my questions are: 1. can any library function call increase my thread level? I guess yes. 2. can any library function call decrease my thread level? My guess is yes, but then things get messy. -- Pavan On 06/22/2013 08:03 AM, Pavan Balaji wrote: > Hi Jeff, > > Thanks. The slides seem to be confusing collective vs. synchronizing. I > think you mean to say that INIT/FINALIZE are always collective, and they > *might* be synchronizing (though we expect most implementations to not > be synchronizing when they are not actually initializing or finalizing). > > Also, doesn't IS_THREAD_MAIN need to be thread safe as well, like > QUERY_THREAD_LEVEL? > > #pragma omp parallel for > for (i = 0 to gazillion) { > if (IS_THREAD_MAIN()) { > MPI_SUPER_CALL(...); > } > } > > I agree with JeffH that CONNECT/DISCONNECT is not a problem. MPI cannot > actually finalize in the first call to MPI_FINALIZE in your example, > since the ref-count didn't reach zero. > > -- Pavan > > On 06/18/2013 07:47 AM, Jeff Squyres (jsquyres) wrote: >> For comment... >> >> Here's a first cut at slides that we discussed yesterday. It's meant >> to be a rollup of all the issues surrounding: >> >> - thread safe INIT / FINALIZE >> - reference-counting INIT / FINALIZE >> - nesting of INIT / FINALIZE >> >> I discovered a new issue when making up these slides: with >> ref-counting INIT/FINALIZE, since INIT/FINALIZE are collective, we >> have the same ref counts in all procs in MPI_COMM_WORLD. But the ref >> counts may be different in non-COMM_WORLD connected processes. What >> happens when they try to finalize over all connected processes? See >> slide 12 for an example. I'm not sure what the right answer is yet. >> >> >> >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] First cut at slides
by Pavan Balaji 22 Jun '13

22 Jun '13

On 06/22/2013 08:27 AM, Jeff Squyres (jsquyres) wrote: > On Jun 22, 2013, at 9:03 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> Thanks. The slides seem to be confusing collective vs. synchronizing. I think you mean to say that INIT/FINALIZE are always collective, and they *might* be synchronizing (though we expect most implementations to not be synchronizing when they are not actually initializing or finalizing). > > How so? I thought slide 13 was fairly explicit about that: > > - INIT and FINALIZE still collective > - Continue to not specify if they synchronize or not Ok, thanks. I might have missed that. >> I agree with JeffH that CONNECT/DISCONNECT is not a problem. MPI >> cannot actually finalize in the first call to MPI_FINALIZE in your >> example, since the ref-count didn't reach zero. > > I guess my point is that process A must wait for the *actual* > finalization in process B (which is the 2nd one), and if Finalize > synchronizes (which most.. but not all.. MPI implementations do), > then A will block until B calls the 2nd Finalize. Eh? My reading was that they *might* synchronize on every FINALIZE call, but will *finalize* only when ref-count is zero. > But I think I'm coming to the conclusion that if you want to not be > affected by Finalize-possibly-blocking semantics, then you should > just call MPI_COMM_DISCONNECT first. I don't think so. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] First cut at slides
by Jeff Squyres (jsquyres) 22 Jun '13

22 Jun '13

On Jun 22, 2013, at 9:03 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Thanks. The slides seem to be confusing collective vs. synchronizing. I think you mean to say that INIT/FINALIZE are always collective, and they *might* be synchronizing (though we expect most implementations to not be synchronizing when they are not actually initializing or finalizing). How so? I thought slide 13 was fairly explicit about that: - INIT and FINALIZE still collective - Continue to not specify if they synchronize or not (although to be fair, I'm not sure if that is a local edit or not -- I can't remember offhand what was in the last version of the PPTX I sent out, so I've attached a current copy of the slides; see slide 13) > Also, doesn't IS_THREAD_MAIN need to be thread safe as well, like QUERY_THREAD_LEVEL? Yes, I missed that -- fixed. > I agree with JeffH that CONNECT/DISCONNECT is not a problem. MPI cannot actually finalize in the first call to MPI_FINALIZE in your example, since the ref-count didn't reach zero. I guess my point is that process A must wait for the *actual* finalization in process B (which is the 2nd one), and if Finalize synchronizes (which most.. but not all.. MPI implementations do), then A will block until B calls the 2nd Finalize. But I think I'm coming to the conclusion that if you want to not be affected by Finalize-possibly-blocking semantics, then you should just call MPI_COMM_DISCONNECT first. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-hybridpm] First cut at slides
by Pavan Balaji 22 Jun '13

22 Jun '13

Hi Jeff, Thanks. The slides seem to be confusing collective vs. synchronizing. I think you mean to say that INIT/FINALIZE are always collective, and they *might* be synchronizing (though we expect most implementations to not be synchronizing when they are not actually initializing or finalizing). Also, doesn't IS_THREAD_MAIN need to be thread safe as well, like QUERY_THREAD_LEVEL? #pragma omp parallel for for (i = 0 to gazillion) { if (IS_THREAD_MAIN()) { MPI_SUPER_CALL(...); } } I agree with JeffH that CONNECT/DISCONNECT is not a problem. MPI cannot actually finalize in the first call to MPI_FINALIZE in your example, since the ref-count didn't reach zero. -- Pavan On 06/18/2013 07:47 AM, Jeff Squyres (jsquyres) wrote: > For comment... > > Here's a first cut at slides that we discussed yesterday. It's meant to be a rollup of all the issues surrounding: > > - thread safe INIT / FINALIZE > - reference-counting INIT / FINALIZE > - nesting of INIT / FINALIZE > > I discovered a new issue when making up these slides: with ref-counting INIT/FINALIZE, since INIT/FINALIZE are collective, we have the same ref counts in all procs in MPI_COMM_WORLD. But the ref counts may be different in non-COMM_WORLD connected processes. What happens when they try to finalize over all connected processes? See slide 12 for an example. I'm not sure what the right answer is yet. > > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0