February 2021 - mpiwg-sessions - mailman3.cels.anl.gov

Re: [mpiwg-sessions] more excitement - more nuanced response to issue 435
by Daniel Holmes 19 Feb '21

19 Feb '21

Hi Martin, Comm size example: local procedure reads local information, result is 2, no problems. Pending window sync: erroneous, "user must call all procedures necessary to complete involvement in communication", no problems. One-sided passive target put from left to right: seriously bad idea, might work if target hardware capable, might fail if target needs software agent, although such agent should still be active because window has not been freed, discovered by win unlock, if it must fail then it should probably be prohibited in mpi-4.1 somehow - wording is complex. What about process ending normally/properly immediately after session finalize? Note local is weak local - permitted to depend on remote progress - making sure stuff gets onto/off of the wire before returning is permitted. Intrigue galore! Cheers, Dan. 19 Feb 2021 19:48:43 Martin Schulz <schulzm(a)in.tum.de>: > Hi Dan, all, > > To the first part: I would also prefer Rolf’s text with the unspecified collectives (we probably need to add files and windows, but that is easy) – it leave quite a bit of wiggle room. With Ialltoall, I think we would be overspecifying, as this would declare session_finalize not only collective, but also synchronizing. > > However, also here we don’t say what really happens with the MPI objects and what can and cannot be done afterwards – I think it would be legal (as of now) to create a persistent communication, call finalize (no pending communication) and then call start. We obviously don’t want that. > > > As for the second part: > > Branch at Q1, freeing all – at the end, I am not sure that this against the rule of least astonishment. It is different than Finalize, but that should not be astonishing. I would compare this to memory allocation: if I write a program that allocates a lot of memory, I normally don’t bother freeing it before exit, I simply don’t care. If I write a library with an “end” call, though, I better clean everything up or I wrote a bad library. I think this would be second nature to programmers. > > If we go on the premise, though, that this is too much for MPI programmers, I principally like the idea of keeping session_finalize local, as this is the only option to keep init and finalize parallel, i.e., have matching semantics. However, I am not sure what this would mean in some areas: > > Session Init A Session Init A > C1=from_g(from_s,A) C1=from_g(from_s,A) > Session Finalize > Barrier on COMM_WORLD Barrier on COMM_WORLD > Comm_get_size(C1) > > What is the value here? > > If we create a window with memory from 3 processes and then one finalizes the session, is the remaining memory still available? What does this mean for pending synchronization operations and open communication windows between the other 2? > > We can probably construct more corner cases. > > In general, I am not trying to shoot down the idea (I find it intriguing, as it preserves the symmetry if init and finalize), but I think it could have quite some consequences. Forcing a free or describing the interleaved collective nature would be a small step (text and correctness wise) and would allow most (all?) other solutions to be added in 4.1 with some time to discuss it. > > Martin > > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de > > > > From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of Dan Holmes via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> > Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> > Date: Friday, 19. February 2021 at 20:16 > To: mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> > Cc: Dan Holmes <danholmes(a)chi.scot> > Subject: Re: [mpiwg-sessions] more excitement - more nuanced response to issue 435 > > Hi Howard, > > My initial impression (from reading your email but not looking at the PDF yet), is: > * I much prefer Rolf’s suggested reference to generic/unspecified “collective operations” rather than nailing it down to MPI_Ialltoall. > * I don’t like the restriction that the user must finalise sessions in a particular order to match the internal implementation of a single session finalise at some remote process (e.g. the scenario of Rolf’s case A on issue 435). > > More fundamental: we need a decision tree to tease apart the design decisions we are making at pace and with no reference implementation. > > First choice: does MPI_SESSION_FINALISE do anything non-local? If so, what? > If no, then next choice is: > > Root Q1: Do we wish to mandate that the user must do clean up prior to MPI_SESSION_FINALISE? If so, then eek! Breach of rule of least astonishment. > Branch Q2: If no, then do we wish to mandate that MPI_SESSION_FINALISE does whatever clean up has not been done by the user? If so, eek! Significant change to accepted text. > Branch Q3: If no, then does MPI_SESSION_FINALISE do anything non-local? If so, eek! What does it do? Panic. > Branch Q4: If no, then does MPI_SESSION_FINALISE need to be defined as collective? If so, eek! Why? Why does it need that semantic? Panic. > Branch Q5: If no, then does MPI_SESSION_FINALISE need to be defined as non-local? If so, eek! Why? Why does it need that semantic? Panic. > Branch Q6: If no, then we should define MPI_SESSION_FINALISE as local (meaning weak-local, of course)? If so, strike all text about collective operation(s) of any kind and strike any restriction on ordering of calls and strike any restriction on the permitted associations/derivations of communicators from sessions. > > > This is a linear decision tree that leads to: > > > " > MPI_SESSION_FINALIZE is a local procedure; it does not free MPI objects derived from the session. It is erroneous to use MPI objects derived from a session after calling MPI_SESSION_FINALIZE for that session. > > > If the user wishes to recover resources from MPI objects derived from a session, then appropriate calls to MPI procedures must be made by the user prior to calling MPI_SESSION_FINALIZE, such as MPI_COMM_DISCONNECT (from communicators), MPI_WIN_FREE (for windows), and MPI_FILE_CLOSE (for files). > “ > > > Discuss. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Executive Director > Chief Technology Officer > CHI Ltd > danholmes(a)chi.scot > > > > >> On 19 Feb 2021, at 18:13, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: >> >> HI All, >> >> Ah this is exciting. So I spent some time on baking verbiage to add about MPI_Session_finalize non-local behavior. >> >> See the attached cutout from the results pdf. >> >> I’ve added verbiage describing the semantics (copying some wording from MPI_Sendrecv, or at least the flavor) of session finalize in the event that the user has not cleaned up MPI objects associated with the session(s). >> It’s a simple easy to understand (I think) model. Basically session finalize has the semantics of a MPI_Ialltoall for each communicators still associated with the session at finalize, followed by a waitall. As long as all other processes finalizing their sessions generate in aggregate, a message pattern which matches, no deadlock. If not, potential deadlock. One takeaway from this is that we can’t support arbitrary associations of communicators to sessions in each MPI process when the app doesn’t do its own cleanup so as to make MPI_Session_finalize a local op. >> >> I’ve added some examples and we can add more as we think needed. May have to change the presentation mechanism however. >> >> I didn’t want to open this as a PR at this point, hence this notification mechanism. >> >> Howard >> >> -- >> >> Howard Pritchard >> HPC-ENV >> Los Alamos National Laboratory >> >> <temp.pdf>_______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions >

1 0

Re: [mpiwg-sessions] more excitement - more nuanced response to issue 435
by Martin Schulz 19 Feb '21

19 Feb '21

Hi Dan, all, To the first part: I would also prefer Rolf’s text with the unspecified collectives (we probably need to add files and windows, but that is easy) – it leave quite a bit of wiggle room. With Ialltoall, I think we would be overspecifying, as this would declare session_finalize not only collective, but also synchronizing. However, also here we don’t say what really happens with the MPI objects and what can and cannot be done afterwards – I think it would be legal (as of now) to create a persistent communication, call finalize (no pending communication) and then call start. We obviously don’t want that. As for the second part: Branch at Q1, freeing all – at the end, I am not sure that this against the rule of least astonishment. It is different than Finalize, but that should not be astonishing. I would compare this to memory allocation: if I write a program that allocates a lot of memory, I normally don’t bother freeing it before exit, I simply don’t care. If I write a library with an “end” call, though, I better clean everything up or I wrote a bad library. I think this would be second nature to programmers. If we go on the premise, though, that this is too much for MPI programmers, I principally like the idea of keeping session_finalize local, as this is the only option to keep init and finalize parallel, i.e., have matching semantics. However, I am not sure what this would mean in some areas: Session Init A Session Init A C1=from_g(from_s,A) C1=from_g(from_s,A) Session Finalize Barrier on COMM_WORLD Barrier on COMM_WORLD Comm_get_size(C1) What is the value here? If we create a window with memory from 3 processes and then one finalizes the session, is the remaining memory still available? What does this mean for pending synchronization operations and open communication windows between the other 2? We can probably construct more corner cases. In general, I am not trying to shoot down the idea (I find it intriguing, as it preserves the symmetry if init and finalize), but I think it could have quite some consequences. Forcing a free or describing the interleaved collective nature would be a small step (text and correctness wise) and would allow most (all?) other solutions to be added in 4.1 with some time to discuss it. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of Dan Holmes via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Friday, 19. February 2021 at 20:16 To: mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Dan Holmes <danholmes(a)chi.scot> Subject: Re: [mpiwg-sessions] more excitement - more nuanced response to issue 435 Hi Howard, My initial impression (from reading your email but not looking at the PDF yet), is: * I much prefer Rolf’s suggested reference to generic/unspecified “collective operations” rather than nailing it down to MPI_Ialltoall. * I don’t like the restriction that the user must finalise sessions in a particular order to match the internal implementation of a single session finalise at some remote process (e.g. the scenario of Rolf’s case A on issue 435). More fundamental: we need a decision tree to tease apart the design decisions we are making at pace and with no reference implementation. First choice: does MPI_SESSION_FINALISE do anything non-local? If so, what? If no, then next choice is: Root Q1: Do we wish to mandate that the user must do clean up prior to MPI_SESSION_FINALISE? If so, then eek! Breach of rule of least astonishment. Branch Q2: If no, then do we wish to mandate that MPI_SESSION_FINALISE does whatever clean up has not been done by the user? If so, eek! Significant change to accepted text. Branch Q3: If no, then does MPI_SESSION_FINALISE do anything non-local? If so, eek! What does it do? Panic. Branch Q4: If no, then does MPI_SESSION_FINALISE need to be defined as collective? If so, eek! Why? Why does it need that semantic? Panic. Branch Q5: If no, then does MPI_SESSION_FINALISE need to be defined as non-local? If so, eek! Why? Why does it need that semantic? Panic. Branch Q6: If no, then we should define MPI_SESSION_FINALISE as local (meaning weak-local, of course)? If so, strike all text about collective operation(s) of any kind and strike any restriction on ordering of calls and strike any restriction on the permitted associations/derivations of communicators from sessions. This is a linear decision tree that leads to: " MPI_SESSION_FINALIZE is a local procedure; it does not free MPI objects derived from the session. It is erroneous to use MPI objects derived from a session after calling MPI_SESSION_FINALIZE for that session. If the user wishes to recover resources from MPI objects derived from a session, then appropriate calls to MPI procedures must be made by the user prior to calling MPI_SESSION_FINALIZE, such as MPI_COMM_DISCONNECT (from communicators), MPI_WIN_FREE (for windows), and MPI_FILE_CLOSE (for files). “ Discuss. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot On 19 Feb 2021, at 18:13, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: HI All, Ah this is exciting. So I spent some time on baking verbiage to add about MPI_Session_finalize non-local behavior. See the attached cutout from the results pdf. I’ve added verbiage describing the semantics (copying some wording from MPI_Sendrecv, or at least the flavor) of session finalize in the event that the user has not cleaned up MPI objects associated with the session(s). It’s a simple easy to understand (I think) model. Basically session finalize has the semantics of a MPI_Ialltoall for each communicators still associated with the session at finalize, followed by a waitall. As long as all other processes finalizing their sessions generate in aggregate, a message pattern which matches, no deadlock. If not, potential deadlock. One takeaway from this is that we can’t support arbitrary associations of communicators to sessions in each MPI process when the app doesn’t do its own cleanup so as to make MPI_Session_finalize a local op. I’ve added some examples and we can add more as we think needed. May have to change the presentation mechanism however. I didn’t want to open this as a PR at this point, hence this notification mechanism. Howard -- Howard Pritchard HPC-ENV Los Alamos National Laboratory <temp.pdf>_______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] [EXTERNAL] Re: more excitement - more nuanced response to issue 435
by Pritchard Jr., Howard 19 Feb '21

19 Feb '21

HI Dan, Oh I like this decision tree – responses interleaved below From: Dan Holmes <danholmes(a)chi.scot> Date: Friday, February 19, 2021 at 12:16 PM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: [EXTERNAL] Re: [mpiwg-sessions] more excitement - more nuanced response to issue 435 Hi Howard, My initial impression (from reading your email but not looking at the PDF yet), is: * I much prefer Rolf’s suggested reference to generic/unspecified “collective operations” rather than nailing it down to MPI_Ialltoall. * I don’t like the restriction that the user must finalise sessions in a particular order to match the internal implementation of a single session finalise at some remote process (e.g. the scenario of Rolf’s case A on issue 435). More fundamental: we need a decision tree to tease apart the design decisions we are making at pace and with no reference implementation. First choice: does MPI_SESSION_FINALISE do anything non-local? If so, what? If we think mpi_session_finalize should clean up what the user didn’t do, then for most interconnects we’d need some mechanism for insuring pending sends, etc. got off on the wire. Otherwise I can’t think of anything from an implementation point of view. If no, then next choice is: Root Q1: Do we wish to mandate that the user must do clean up prior to MPI_SESSION_FINALISE? If so, then eek! Breach of rule of least astonishment. No but cleaning up should be recommended best practice Branch Q2: If no, then do we wish to mandate that MPI_SESSION_FINALISE does whatever clean up has not been done by the user? If so, eek! Significant change to accepted text. We do have this sentence starting out the function description currently - This routine cleans up all MPI state associated with the supplied session. Up for interpretation what that means I guess. This alone doesn’t imply anything about non-local. Branch Q3: If no, then does MPI_SESSION_FINALISE do anything non-local? If so, eek! What does it do? Panic. If no then there ought not to be anything non-lcocal Branch Q4: If no, then does MPI_SESSION_FINALISE need to be defined as collective? If so, eek! Why? Why does it need that semantic? Panic. No in this event Branch Q5: If no, then does MPI_SESSION_FINALISE need to be defined as non-local? If so, eek! Why? Why does it need that semantic? Panic. Again no if we take Q4 answer to be negative. Branch Q6: If no, then we should define MPI_SESSION_FINALISE as local (meaning weak-local, of course)? If so, strike all text about collective operation(s) of any kind and strike any restriction on ordering of calls and strike any restriction on the permitted associations/derivations of communicators from sessions. agreed This is a linear decision tree that leads to: " MPI_SESSION_FINALIZE is a local procedure; it does not free MPI objects derived from the session. It is erroneous to use MPI objects derived from a session after calling MPI_SESSION_FINALIZE for that session. If the user wishes to recover resources from MPI objects derived from a session, then appropriate calls to MPI procedures must be made by the user prior to calling MPI_SESSION_FINALIZE, such as MPI_COMM_DISCONNECT (from communicators), MPI_WIN_FREE (for windows), and MPI_FILE_CLOSE (for files). “ Discuss. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot<mailto:[email protected]> On 19 Feb 2021, at 18:13, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: HI All, Ah this is exciting. So I spent some time on baking verbiage to add about MPI_Session_finalize non-local behavior. See the attached cutout from the results pdf. I’ve added verbiage describing the semantics (copying some wording from MPI_Sendrecv, or at least the flavor) of session finalize in the event that the user has not cleaned up MPI objects associated with the session(s). It’s a simple easy to understand (I think) model. Basically session finalize has the semantics of a MPI_Ialltoall for each communicators still associated with the session at finalize, followed by a waitall. As long as all other processes finalizing their sessions generate in aggregate, a message pattern which matches, no deadlock. If not, potential deadlock. One takeaway from this is that we can’t support arbitrary associations of communicators to sessions in each MPI process when the app doesn’t do its own cleanup so as to make MPI_Session_finalize a local op. I’ve added some examples and we can add more as we think needed. May have to change the presentation mechanism however. I didn’t want to open this as a PR at this point, hence this notification mechanism. Howard -- Howard Pritchard HPC-ENV Los Alamos National Laboratory <temp.pdf>_______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] more excitement - more nuanced response to issue 435
by Dan Holmes 19 Feb '21

19 Feb '21

Hi Howard, My initial impression (from reading your email but not looking at the PDF yet), is: * I much prefer Rolf’s suggested reference to generic/unspecified “collective operations” rather than nailing it down to MPI_Ialltoall. * I don’t like the restriction that the user must finalise sessions in a particular order to match the internal implementation of a single session finalise at some remote process (e.g. the scenario of Rolf’s case A on issue 435). More fundamental: we need a decision tree to tease apart the design decisions we are making at pace and with no reference implementation. First choice: does MPI_SESSION_FINALISE do anything non-local? If so, what? If no, then next choice is: Root Q1: Do we wish to mandate that the user must do clean up prior to MPI_SESSION_FINALISE? If so, then eek! Breach of rule of least astonishment. Branch Q2: If no, then do we wish to mandate that MPI_SESSION_FINALISE does whatever clean up has not been done by the user? If so, eek! Significant change to accepted text. Branch Q3: If no, then does MPI_SESSION_FINALISE do anything non-local? If so, eek! What does it do? Panic. Branch Q4: If no, then does MPI_SESSION_FINALISE need to be defined as collective? If so, eek! Why? Why does it need that semantic? Panic. Branch Q5: If no, then does MPI_SESSION_FINALISE need to be defined as non-local? If so, eek! Why? Why does it need that semantic? Panic. Branch Q6: If no, then we should define MPI_SESSION_FINALISE as local (meaning weak-local, of course)? If so, strike all text about collective operation(s) of any kind and strike any restriction on ordering of calls and strike any restriction on the permitted associations/derivations of communicators from sessions. This is a linear decision tree that leads to: " MPI_SESSION_FINALIZE is a local procedure; it does not free MPI objects derived from the session. It is erroneous to use MPI objects derived from a session after calling MPI_SESSION_FINALIZE for that session. If the user wishes to recover resources from MPI objects derived from a session, then appropriate calls to MPI procedures must be made by the user prior to calling MPI_SESSION_FINALIZE, such as MPI_COMM_DISCONNECT (from communicators), MPI_WIN_FREE (for windows), and MPI_FILE_CLOSE (for files). “ Discuss. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot > On 19 Feb 2021, at 18:13, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > HI All, > > Ah this is exciting. So I spent some time on baking verbiage to add about MPI_Session_finalize non-local behavior. > > See the attached cutout from the results pdf. > > I’ve added verbiage describing the semantics (copying some wording from MPI_Sendrecv, or at least the flavor) of session finalize in the event that the user has not cleaned up MPI objects associated with the session(s). > It’s a simple easy to understand (I think) model. Basically session finalize has the semantics of a MPI_Ialltoall for each communicators still associated with the session at finalize, followed by a waitall. As long as all other processes finalizing their sessions generate in aggregate, a message pattern which matches, no deadlock. If not, potential deadlock. One takeaway from this is that we can’t support arbitrary associations of communicators to sessions in each MPI process when the app doesn’t do its own cleanup so as to make MPI_Session_finalize a local op. > > I’ve added some examples and we can add more as we think needed. May have to change the presentation mechanism however. > > I didn’t want to open this as a PR at this point, hence this notification mechanism. > > Howard > > -- > > Howard Pritchard > HPC-ENV > Los Alamos National Laboratory > > <temp.pdf>_______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions <https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions>

1 0

more excitement - more nuanced response to issue 435
by Pritchard Jr., Howard 19 Feb '21

19 Feb '21

HI All, Ah this is exciting. So I spent some time on baking verbiage to add about MPI_Session_finalize non-local behavior. See the attached cutout from the results pdf. I’ve added verbiage describing the semantics (copying some wording from MPI_Sendrecv, or at least the flavor) of session finalize in the event that the user has not cleaned up MPI objects associated with the session(s). It’s a simple easy to understand (I think) model. Basically session finalize has the semantics of a MPI_Ialltoall for each communicators still associated with the session at finalize, followed by a waitall. As long as all other processes finalizing their sessions generate in aggregate, a message pattern which matches, no deadlock. If not, potential deadlock. One takeaway from this is that we can’t support arbitrary associations of communicators to sessions in each MPI process when the app doesn’t do its own cleanup so as to make MPI_Session_finalize a local op. I’ve added some examples and we can add more as we think needed. May have to change the presentation mechanism however. I didn’t want to open this as a PR at this point, hence this notification mechanism. Howard -- Howard Pritchard HPC-ENV Los Alamos National Laboratory

1 0

Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED
by Dan Holmes 19 Feb '21

19 Feb '21

Hi Howard, The third option given by Martin, isolation, is (and has been for years) the intent of sessions. The meaning of isolation w.r.t. thread support levels is that the thread support level is scoped within each session. Calling MPI_SESSION_INIT and requesting MPI_THREAD_FUNNELED is a promise by the user to only ever access that session (and everything derived from it) using the exact same thread that called MPI_SESSION_INIT. Calling MPI_SESSION_INIT and requesting MPI_THREAD_SERIALIZED is a promise by the user to only ever access that session (and everything derived from it) using the exactly one thread at a time, with no restriction on which thread is in MPI accessing that session. MPI can (trivially) provide those levels of thread support by initialising itself using the traditional MPI_THREAD_MULTIPLE level. In this minimal implementation, the provided thread support level could be returned as anything between the requested level and MPI_THREAD_MULTIPLE inclusive. If the MPI library wishes to take advantage of the user promise, then it can relax some protections that it would otherwise be required to enforce. In this high quality implementation, the provided thread support level should be equal to the request level and the library will assume the user complies with the promise and assume single-thread-at-a-time or always-the-same-thread accesses to the session and derived resources, permitting some/most/all locks and atomics to be removed. Your implementation that has shared state applying to all sessions breaches that high-quality isolation, because it requires a cross-session lock/atomic. If the common block is read-only data, then shared/multi-threaded access is probably not a big problem [ED: please check, especially when one thread finalises a session]. If the common block is read-write but there is no producer-consumer interaction, then this could be resolved by replication, one distinct copy per session. If there is thread-to-thread/session-to-session interaction via the common block, then the implementation is probably incorrect because there is no need for that because sessions are isolated from each other. To make this a little more concrete, for discussion purposes, consider an application that uses 3 libraries, one that is semi-independent from the rest of the application and is executed on a dedicated thread that makes calls into MPI, another that needs to be called every now and again but does not care which single thread accesses it although it will use that thread to call into MPI, and a third that performs multi-threaded operations, including concurrent calls from multiple threads into MPI. This application could use all three thread support levels at the same time. The first library would create a session, requesting MPI_THREAD_FUNNELED, and be provided MPI_THREAD_FUNNELED (for a high-quality MPI library or greater if not). The second library creates its own session, requesting and being provided MPI_THREAD_SERIALIZED (for a high-quality MPI library or *greater/lower* if not). The third library creates a third session, requesting and being provided MPI_THREAD_MULTIPLE (for a high-quality MPI library or *lower* if not). Depending on the execution pattern permitted by the application (in compliance with the requirements of all three libraries), all three libraries might be calling into MPI at the same time, with different threads. From a traditional pre-MPI-4.0 point-of-view, this must be MPI_THREAD_MULTIPLE internally to the MPI library, even if it returns a lower level for one or more of the sessions. From MPI-4.0 onwards, there is another implementation path: the resources used exclusively by each session can be accessed using the thread support level provided to that session. Shared resources, those used by multiple sessions, must be protected as if accessed in compliance with the MPI_THREAD_MULTIPLE thread support level. Thus, the ompi_comm_t struct and ompi_request_t struct data derived from the sessions created by library 1 and library 2 above can be accessed without thread-safety protections, but the ompi_comm_t struct and ompi_request_t struct data derived from the session created by library 3 must be protected against multiple concurrent accesses. On a system with a single network card, where the network driver requires a particular thread to make all calls into the driver (e.g. because it uses per-thread storage), then all/most network calls inside MPI must be marshalled onto that single special thread, which requires a cross-thread whole-MPI-library protection mechanism, i.e. traditional MPI_THREAD_MULTIPLE. In conclusion, the prior effort to deprecate MPI_THREAD_FUNNELED and MPI_THREAD_SERIALIZED (was it Jim, I thought it was Nathan?) fell for “good” reasons, i.e. reasons that made a lot of sense in common important past architectures, still make some small amount of sense for current architectures, and may make much more sense in arbitrary unknown future architectures. An effort to restrict sessions to the MPI_THREAD_MULTIPLE thread support level should probably fail for the exact same “good“ reasons. However, isolation AKA scoping the thread support to a per-session context is a step forward, IMHO. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot > On 19 Feb 2021, at 15:17, Pritchard Jr., Howard <howardp(a)lanl.gov> wrote: > > HI Martin, > > Open MPI internally behaves as if there are only either MPI_THREAD_SERIALIZED or MPI_THREAD_MULTIPLE. > Except for odds and ends like supporting the MPI_Is_thread_main, it doesn’t behave differently whether the application > requests for MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, or MPI_THREAD_SERIALIZED. > > Now as for the prototype, there is a block of common resources used by all MPI “instances” – what the developer of the prototype was calling “sessions”, and it is initialized either for thread safety or not. So its whoever makes the first call to MPI_Session_init of MPI_Init/MPI_Init_thread which determines the thread support provided for subsequent calls to either MPI_Session_init or MPI_Init_thread, etc. > > I suspect other MPI implementations would have a similar set of common resources that would have to be shared by all MPI sessions. > > In some ways from the implementor’s point of view this is not that important as the only sane thing from a software engineering point of view is if you know your code may be operating in a multi-threaded environment, you should code it that way from the start. In the Open MPI code there are many places where, for example, counters are incremented/decremented using atomic ops irrespective of what the application asked for in terms of thread support. Indeed, we several major releases ago removed the option to build Open MPI for thread serial only, as much of the plumbing code now assumes it can be invoked concurrently from multiple threads. > > Here's my thoughts on the suggestions: > > · We actually deprecate funneled and serialized - this would be my preferred option but I thought Jim Dinan already fought that battle and lost for MPI 4 – but maybe I’m not remembering right > · We don’t deprecate, but state that they should no longer be returned in the session model – that would be easy enough for 4.1 > · We isolate sessions against each other – not sure I understand this one > > > Howard > > From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> > Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> > Date: Thursday, February 18, 2021 at 3:43 PM > To: Daniel Holmes <danholmes(a)chi.scot> > Cc: Martin Schulz <schulzm(a)in.tum.de>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> > Subject: Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED > > Agreed – not sure this comes across in the text, though. We may want to think about clarifying this in 4.1. > > Howard, is this how the Open MPI works already or what impact do you expect? > > Thanks, > > Martin > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de > > > > From: Daniel Holmes <danholmes(a)chi.scot> > Date: Thursday, 18. February 2021 at 23:13 > To: Martin Schulz <schulzm(a)in.tum.de> > Cc: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> > Subject: Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED > > Hi Martin, > > Your third option, isolation, is (and has been for years) the intent of sessions. > > Cheers, > Dan. > > 18 Feb 2021 22:04:51 Martin Schulz <schulzm(a)in.tum.de>: > >> Hi Dan, >> >> >> I agree – single has a good reason and I think would be fine – in this case the clause that the first init anywhere may dictate thread mode anymore else would indicate to other session inits later on that only single is now supported. >> >> >> I am wondering about serialized and funneled, though: for funneled, it kind of leads to a requirement that session_init can only be called on the same thread again. If not, it should actually fail, as nothing else makes sense (if two session inits on different threads return funneled). For serialized, it requires threads know of each other so they can synchronize, which is against what we wanted to achieve. >> >> >> I actually only see three options (perhaps there are more): >> · We actually deprecate funneled and serialized >> · We don’t deprecate, but state that they should no longer be returned in the session model >> · We isolate sessions against each other >> >> >> In the latter case, returning thread funneled means that MPI calls associated with objects derived from that session have to be called from that thread. For serialized it would disallow two MPI calls at the same time from two threads, if they both use an object derived from the same session. Across sessions there would be no restrictions. >> >> >> I kind of like the latter one, as it goes with the spirit of session, but I wonder what implementers say to that. >> >> >> Martin >> >> >> >> >> >> >> >> >> -- >> Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems >> Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching >> Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) >> Email: schulzm(a)in.tum.de >> >> >> >> >> >> >> From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of Dan Holmes via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> >> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> >> Date: Thursday, 18. February 2021 at 21:07 >> To: mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> >> Cc: Dan Holmes <danholmes(a)chi.scot> >> Subject: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED >> >> >> Hi Martin, >> >> >> This is an interesting argument for disallowing use of MPI_THREAD_SINGLE and MPI_THREAD_FUNNELED with the sessions model (and with the world model in combination with the sessions model). >> >> >> Modern MPI libraries effectively only have MPI_THREAD_SERIALIZED and MPI_THREAD_MULTIPLE anyway. >> >> >> However, a single thread could create a bunch of sessions and initialise the world model and comply with MPI_THREAD_SINGLE and/or MPI_THREAD_FUNNELED. >> >> >> Two libraries might be called by a single thread and thereby be thread compliant. >> >> >> So, prohibiting these thread support levels seems too severe. >> >> >> Library writers need to document the restrictions they place on the main application and all the other libraries that it uses. If a library tells MPI it will only use a particular thread then the caller into that library must comply, etc. >> >> >> An MPI library is free to upgrade later (if it is capable of that) without telling already-initialised sessions (because superset) - except for MPI_THREAD_SINGLE (there can be only one). It is also free to return a lower provided than the requested. >> >> >> The only problem occurs if two distinct threads ask for MPI_THREAD_FUNNELED concurrently and MPI provides MPI_THREAD_FUNNELED to one of them but is not able to upgrade internally - it is then stuck for when the second thread makes its request. >> >> >> One of the main points of the isolation between threads was that no cross-session protection was needed. Thus, session S1 funnelled onto thread T1 and S2 funnelled onto T2 should be easy to implement, even though that scenario is not currently possible with MPI-3.1 and no-one has actually coded it yet. >> >> Cheers, >> Dan. >> — >> Dr Daniel Holmes PhD >> Executive Director >> Chief Technology Officer >> CHI Ltd >> danholmes(a)chi.scot <mailto:[email protected]> >> >> >>> Hi Howard, >>> >>> >>> >> I recall we decided to leave the behavior of thread support level when using multiple sessions/world model up to the implementation. However, in the case of MPI_THREAD_FUNNELED we should probably add a blurb about not doing this in the advice to users in the Session Creation and Destruction Methods. >>> >>> >>> I think the problem also appears if the original MPI_Init_thread uses FUNNELED. We probably need to add something there as well. >>> >>> >>> >> My first guess based on the openmpi prototype is that if one session is initialized with funneled, then all subsequent created sessions or mpi init thread will be supported at funnel level. >>> >>> >>> That kind of makes sense - but what happens if the second Session_Init is then on another thread? The funneled refers to a different thread. >>> >>> >>> >> For your thread serialized question I would say the answer is no, unless under the covers the implementation is always working in effectively thread multiple mode. >>> >>> >>> Hmm, I understand the motivation and implementation implications, but this would kill the isolation then, which is also not good. Also, if two libraries have separate sessions, it is often not even possible to avoid this. >>> >>> >>> Almost seems like we should deprecate funneled and serialized. >>> >>> >>> Another item to cover next Monday __. >>> >>> >>> Martin

1 0

Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED
by Pritchard Jr., Howard 19 Feb '21

19 Feb '21

HI Martin, Open MPI internally behaves as if there are only either MPI_THREAD_SERIALIZED or MPI_THREAD_MULTIPLE. Except for odds and ends like supporting the MPI_Is_thread_main, it doesn’t behave differently whether the application requests for MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, or MPI_THREAD_SERIALIZED. Now as for the prototype, there is a block of common resources used by all MPI “instances” – what the developer of the prototype was calling “sessions”, and it is initialized either for thread safety or not. So its whoever makes the first call to MPI_Session_init of MPI_Init/MPI_Init_thread which determines the thread support provided for subsequent calls to either MPI_Session_init or MPI_Init_thread, etc. I suspect other MPI implementations would have a similar set of common resources that would have to be shared by all MPI sessions. In some ways from the implementor’s point of view this is not that important as the only sane thing from a software engineering point of view is if you know your code may be operating in a multi-threaded environment, you should code it that way from the start. In the Open MPI code there are many places where, for example, counters are incremented/decremented using atomic ops irrespective of what the application asked for in terms of thread support. Indeed, we several major releases ago removed the option to build Open MPI for thread serial only, as much of the plumbing code now assumes it can be invoked concurrently from multiple threads. Here's my thoughts on the suggestions: · We actually deprecate funneled and serialized - this would be my preferred option but I thought Jim Dinan already fought that battle and lost for MPI 4 – but maybe I’m not remembering right · We don’t deprecate, but state that they should no longer be returned in the session model – that would be easy enough for 4.1 · We isolate sessions against each other – not sure I understand this one Howard From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Thursday, February 18, 2021 at 3:43 PM To: Daniel Holmes <danholmes(a)chi.scot> Cc: Martin Schulz <schulzm(a)in.tum.de>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Subject: Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED Agreed – not sure this comes across in the text, though. We may want to think about clarifying this in 4.1. Howard, is this how the Open MPI works already or what impact do you expect? Thanks, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: Daniel Holmes <danholmes(a)chi.scot> Date: Thursday, 18. February 2021 at 23:13 To: Martin Schulz <schulzm(a)in.tum.de> Cc: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Subject: Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED Hi Martin, Your third option, isolation, is (and has been for years) the intent of sessions. Cheers, Dan. 18 Feb 2021 22:04:51 Martin Schulz <schulzm(a)in.tum.de>: Hi Dan, I agree – single has a good reason and I think would be fine – in this case the clause that the first init anywhere may dictate thread mode anymore else would indicate to other session inits later on that only single is now supported. I am wondering about serialized and funneled, though: for funneled, it kind of leads to a requirement that session_init can only be called on the same thread again. If not, it should actually fail, as nothing else makes sense (if two session inits on different threads return funneled). For serialized, it requires threads know of each other so they can synchronize, which is against what we wanted to achieve. I actually only see three options (perhaps there are more): · We actually deprecate funneled and serialized · We don’t deprecate, but state that they should no longer be returned in the session model · We isolate sessions against each other In the latter case, returning thread funneled means that MPI calls associated with objects derived from that session have to be called from that thread. For serialized it would disallow two MPI calls at the same time from two threads, if they both use an object derived from the same session. Across sessions there would be no restrictions. I kind of like the latter one, as it goes with the spirit of session, but I wonder what implementers say to that. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of Dan Holmes via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Thursday, 18. February 2021 at 21:07 To: mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Dan Holmes <danholmes(a)chi.scot> Subject: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED Hi Martin, This is an interesting argument for disallowing use of MPI_THREAD_SINGLE and MPI_THREAD_FUNNELED with the sessions model (and with the world model in combination with the sessions model). Modern MPI libraries effectively only have MPI_THREAD_SERIALIZED and MPI_THREAD_MULTIPLE anyway. However, a single thread could create a bunch of sessions and initialise the world model and comply with MPI_THREAD_SINGLE and/or MPI_THREAD_FUNNELED. Two libraries might be called by a single thread and thereby be thread compliant. So, prohibiting these thread support levels seems too severe. Library writers need to document the restrictions they place on the main application and all the other libraries that it uses. If a library tells MPI it will only use a particular thread then the caller into that library must comply, etc. An MPI library is free to upgrade later (if it is capable of that) without telling already-initialised sessions (because superset) - except for MPI_THREAD_SINGLE (there can be only one). It is also free to return a lower provided than the requested. The only problem occurs if two distinct threads ask for MPI_THREAD_FUNNELED concurrently and MPI provides MPI_THREAD_FUNNELED to one of them but is not able to upgrade internally - it is then stuck for when the second thread makes its request. One of the main points of the isolation between threads was that no cross-session protection was needed. Thus, session S1 funnelled onto thread T1 and S2 funnelled onto T2 should be easy to implement, even though that scenario is not currently possible with MPI-3.1 and no-one has actually coded it yet. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot<mailto:[email protected]> Hi Howard, >> I recall we decided to leave the behavior of thread support level when using multiple sessions/world model up to the implementation. However, in the case of MPI_THREAD_FUNNELED we should probably add a blurb about not doing this in the advice to users in the Session Creation and Destruction Methods. I think the problem also appears if the original MPI_Init_thread uses FUNNELED. We probably need to add something there as well. >> My first guess based on the openmpi prototype is that if one session is initialized with funneled, then all subsequent created sessions or mpi init thread will be supported at funnel level. That kind of makes sense - but what happens if the second Session_Init is then on another thread? The funneled refers to a different thread. >> For your thread serialized question I would say the answer is no, unless under the covers the implementation is always working in effectively thread multiple mode. Hmm, I understand the motivation and implementation implications, but this would kill the isolation then, which is also not good. Also, if two libraries have separate sessions, it is often not even possible to avoid this. Almost seems like we should deprecate funneled and serialized. Another item to cover next Monday __. Martin

1 0

Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED
by Martin Schulz 18 Feb '21

18 Feb '21

Agreed – not sure this comes across in the text, though. We may want to think about clarifying this in 4.1. Howard, is this how the Open MPI works already or what impact do you expect? Thanks, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: Daniel Holmes <danholmes(a)chi.scot> Date: Thursday, 18. February 2021 at 23:13 To: Martin Schulz <schulzm(a)in.tum.de> Cc: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Subject: Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED Hi Martin, Your third option, isolation, is (and has been for years) the intent of sessions. Cheers, Dan. 18 Feb 2021 22:04:51 Martin Schulz <schulzm(a)in.tum.de>: Hi Dan, I agree – single has a good reason and I think would be fine – in this case the clause that the first init anywhere may dictate thread mode anymore else would indicate to other session inits later on that only single is now supported. I am wondering about serialized and funneled, though: for funneled, it kind of leads to a requirement that session_init can only be called on the same thread again. If not, it should actually fail, as nothing else makes sense (if two session inits on different threads return funneled). For serialized, it requires threads know of each other so they can synchronize, which is against what we wanted to achieve. I actually only see three options (perhaps there are more): We actually deprecate funneled and serialized We don’t deprecate, but state that they should no longer be returned in the session model We isolate sessions against each other In the latter case, returning thread funneled means that MPI calls associated with objects derived from that session have to be called from that thread. For serialized it would disallow two MPI calls at the same time from two threads, if they both use an object derived from the same session. Across sessions there would be no restrictions. I kind of like the latter one, as it goes with the spirit of session, but I wonder what implementers say to that. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of Dan Holmes via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Thursday, 18. February 2021 at 21:07 To: mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Dan Holmes <danholmes(a)chi.scot> Subject: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED Hi Martin, This is an interesting argument for disallowing use of MPI_THREAD_SINGLE and MPI_THREAD_FUNNELED with the sessions model (and with the world model in combination with the sessions model). Modern MPI libraries effectively only have MPI_THREAD_SERIALIZED and MPI_THREAD_MULTIPLE anyway. However, a single thread could create a bunch of sessions and initialise the world model and comply with MPI_THREAD_SINGLE and/or MPI_THREAD_FUNNELED. Two libraries might be called by a single thread and thereby be thread compliant. So, prohibiting these thread support levels seems too severe. Library writers need to document the restrictions they place on the main application and all the other libraries that it uses. If a library tells MPI it will only use a particular thread then the caller into that library must comply, etc. An MPI library is free to upgrade later (if it is capable of that) without telling already-initialised sessions (because superset) - except for MPI_THREAD_SINGLE (there can be only one). It is also free to return a lower provided than the requested. The only problem occurs if two distinct threads ask for MPI_THREAD_FUNNELED concurrently and MPI provides MPI_THREAD_FUNNELED to one of them but is not able to upgrade internally - it is then stuck for when the second thread makes its request. One of the main points of the isolation between threads was that no cross-session protection was needed. Thus, session S1 funnelled onto thread T1 and S2 funnelled onto T2 should be easy to implement, even though that scenario is not currently possible with MPI-3.1 and no-one has actually coded it yet. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot Hi Howard, >> I recall we decided to leave the behavior of thread support level when using multiple sessions/world model up to the implementation. However, in the case of MPI_THREAD_FUNNELED we should probably add a blurb about not doing this in the advice to users in the Session Creation and Destruction Methods. I think the problem also appears if the original MPI_Init_thread uses FUNNELED. We probably need to add something there as well. >> My first guess based on the openmpi prototype is that if one session is initialized with funneled, then all subsequent created sessions or mpi init thread will be supported at funnel level. That kind of makes sense - but what happens if the second Session_Init is then on another thread? The funneled refers to a different thread. >> For your thread serialized question I would say the answer is no, unless under the covers the implementation is always working in effectively thread multiple mode. Hmm, I understand the motivation and implementation implications, but this would kill the isolation then, which is also not good. Also, if two libraries have separate sessions, it is often not even possible to avoid this. Almost seems like we should deprecate funneled and serialized. Another item to cover next Monday __. Martin

1 0

Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED
by Daniel Holmes 18 Feb '21

18 Feb '21

Hi Martin, Your third option, isolation, is (and has been for years) the intent of sessions. Cheers, Dan. 18 Feb 2021 22:04:51 Martin Schulz <schulzm(a)in.tum.de>: > Hi Dan, > > I agree – single has a good reason and I think would be fine – in this case the clause that the first init anywhere may dictate thread mode anymore else would indicate to other session inits later on that only single is now supported. > > I am wondering about serialized and funneled, though: for funneled, it kind of leads to a requirement that session_init can only be called on the same thread again. If not, it should actually fail, as nothing else makes sense (if two session inits on different threads return funneled). For serialized, it requires threads know of each other so they can synchronize, which is against what we wanted to achieve. > > I actually only see three options (perhaps there are more): * > We actually deprecate funneled and serialized * > We don’t deprecate, but state that they should no longer be returned in the session model * > We isolate sessions against each other > > In the latter case, returning thread funneled means that MPI calls associated with objects derived from that session have to be called from that thread. For serialized it would disallow two MPI calls at the same time from two threads, if they both use an object derived from the same session. Across sessions there would be no restrictions. > > I kind of like the latter one, as it goes with the spirit of session, but I wonder what implementers say to that. > > Martin > > > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de > > > > From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of Dan Holmes via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> > Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> > Date: Thursday, 18. February 2021 at 21:07 > To: mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> > Cc: Dan Holmes <danholmes(a)chi.scot> > Subject: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED > > Hi Martin, > > This is an interesting argument for disallowing use of MPI_THREAD_SINGLE and MPI_THREAD_FUNNELED with the sessions model (and with the world model in combination with the sessions model). > > Modern MPI libraries effectively only have MPI_THREAD_SERIALIZED and MPI_THREAD_MULTIPLE anyway. > > However, a single thread could create a bunch of sessions and initialise the world model and comply with MPI_THREAD_SINGLE and/or MPI_THREAD_FUNNELED. > > Two libraries might be called by a single thread and thereby be thread compliant. > > So, prohibiting these thread support levels seems too severe. > > > Library writers need to document the restrictions they place on the main application and all the other libraries that it uses. If a library tells MPI it will only use a particular thread then the caller into that library must comply, etc. > > > An MPI library is free to upgrade later (if it is capable of that) without telling already-initialised sessions (because superset) - except for MPI_THREAD_SINGLE (there can be only one). It is also free to return a lower provided than the requested. > > > The only problem occurs if two distinct threads ask for MPI_THREAD_FUNNELED concurrently and MPI provides MPI_THREAD_FUNNELED to one of them but is not able to upgrade internally - it is then stuck for when the second thread makes its request. > > > One of the main points of the isolation between threads was that no cross-session protection was needed. Thus, session S1 funnelled onto thread T1 and S2 funnelled onto T2 should be easy to implement, even though that scenario is not currently possible with MPI-3.1 and no-one has actually coded it yet. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Executive Director > Chief Technology Officer > CHI Ltd > danholmes(a)chi.scot > >> Hi Howard, >> >>>> I recall we decided to leave the behavior of thread support level when using multiple sessions/world model up to the implementation. However, in the case of MPI_THREAD_FUNNELED we should probably add a blurb about not doing this in the advice to users in the Session Creation and Destruction Methods. >> >> I think the problem also appears if the original MPI_Init_thread uses FUNNELED. We probably need to add something there as well. >> >>>> My first guess based on the openmpi prototype is that if one session is initialized with funneled, then all subsequent created sessions or mpi init thread will be supported at funnel level. >> >> That kind of makes sense - but what happens if the second Session_Init is then on another thread? The funneled refers to a different thread. >> >>>> For your thread serialized question I would say the answer is no, unless under the covers the implementation is always working in effectively thread multiple mode. >> >> Hmm, I understand the motivation and implementation implications, but this would kill the isolation then, which is also not good. Also, if two libraries have separate sessions, it is often not even possible to avoid this. >> >> Almost seems like we should deprecate funneled and serialized. >> >> Another item to cover next Monday __. >> >> Martin >

1 0

Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED
by Martin Schulz 18 Feb '21

18 Feb '21

Hi Dan, I agree – single has a good reason and I think would be fine – in this case the clause that the first init anywhere may dictate thread mode anymore else would indicate to other session inits later on that only single is now supported. I am wondering about serialized and funneled, though: for funneled, it kind of leads to a requirement that session_init can only be called on the same thread again. If not, it should actually fail, as nothing else makes sense (if two session inits on different threads return funneled). For serialized, it requires threads know of each other so they can synchronize, which is against what we wanted to achieve. I actually only see three options (perhaps there are more): We actually deprecate funneled and serialized We don’t deprecate, but state that they should no longer be returned in the session model We isolate sessions against each other In the latter case, returning thread funneled means that MPI calls associated with objects derived from that session have to be called from that thread. For serialized it would disallow two MPI calls at the same time from two threads, if they both use an object derived from the same session. Across sessions there would be no restrictions. I kind of like the latter one, as it goes with the spirit of session, but I wonder what implementers say to that. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of Dan Holmes via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Thursday, 18. February 2021 at 21:07 To: mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Dan Holmes <danholmes(a)chi.scot> Subject: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED Hi Martin, This is an interesting argument for disallowing use of MPI_THREAD_SINGLE and MPI_THREAD_FUNNELED with the sessions model (and with the world model in combination with the sessions model). Modern MPI libraries effectively only have MPI_THREAD_SERIALIZED and MPI_THREAD_MULTIPLE anyway. However, a single thread could create a bunch of sessions and initialise the world model and comply with MPI_THREAD_SINGLE and/or MPI_THREAD_FUNNELED. Two libraries might be called by a single thread and thereby be thread compliant. So, prohibiting these thread support levels seems too severe. Library writers need to document the restrictions they place on the main application and all the other libraries that it uses. If a library tells MPI it will only use a particular thread then the caller into that library must comply, etc. An MPI library is free to upgrade later (if it is capable of that) without telling already-initialised sessions (because superset) - except for MPI_THREAD_SINGLE (there can be only one). It is also free to return a lower provided than the requested. The only problem occurs if two distinct threads ask for MPI_THREAD_FUNNELED concurrently and MPI provides MPI_THREAD_FUNNELED to one of them but is not able to upgrade internally - it is then stuck for when the second thread makes its request. One of the main points of the isolation between threads was that no cross-session protection was needed. Thus, session S1 funnelled onto thread T1 and S2 funnelled onto T2 should be easy to implement, even though that scenario is not currently possible with MPI-3.1 and no-one has actually coded it yet. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot Hi Howard, >> I recall we decided to leave the behavior of thread support level when using multiple sessions/world model up to the implementation. However, in the case of MPI_THREAD_FUNNELED we should probably add a blurb about not doing this in the advice to users in the Session Creation and Destruction Methods. I think the problem also appears if the original MPI_Init_thread uses FUNNELED. We probably need to add something there as well. >> My first guess based on the openmpi prototype is that if one session is initialized with funneled, then all subsequent created sessions or mpi init thread will be supported at funnel level. That kind of makes sense - but what happens if the second Session_Init is then on another thread? The funneled refers to a different thread. >> For your thread serialized question I would say the answer is no, unless under the covers the implementation is always working in effectively thread multiple mode. Hmm, I understand the motivation and implementation implications, but this would kill the isolation then, which is also not good. Also, if two libraries have separate sessions, it is often not even possible to avoid this. Almost seems like we should deprecate funneled and serialized. Another item to cover next Monday __. Martin

1 0