February 2021 - mpiwg-sessions - mailman3.cels.anl.gov

[EXTERNAL] Question on MPI_THREAD_FUNNELED
by Dan Holmes 18 Feb '21

18 Feb '21

Hi Martin, This is an interesting argument for disallowing use of MPI_THREAD_SINGLE and MPI_THREAD_FUNNELED with the sessions model (and with the world model in combination with the sessions model). Modern MPI libraries effectively only have MPI_THREAD_SERIALIZED and MPI_THREAD_MULTIPLE anyway. However, a single thread could create a bunch of sessions and initialise the world model and comply with MPI_THREAD_SINGLE and/or MPI_THREAD_FUNNELED. Two libraries might be called by a single thread and thereby be thread compliant. So, prohibiting these thread support levels seems too severe. Library writers need to document the restrictions they place on the main application and all the other libraries that it uses. If a library tells MPI it will only use a particular thread then the caller into that library must comply, etc. An MPI library is free to upgrade later (if it is capable of that) without telling already-initialised sessions (because superset) - except for MPI_THREAD_SINGLE (there can be only one). It is also free to return a lower provided than the requested. The only problem occurs if two distinct threads ask for MPI_THREAD_FUNNELED concurrently and MPI provides MPI_THREAD_FUNNELED to one of them but is not able to upgrade internally - it is then stuck for when the second thread makes its request. One of the main points of the isolation between threads was that no cross-session protection was needed. Thus, session S1 funnelled onto thread T1 and S2 funnelled onto T2 should be easy to implement, even though that scenario is not currently possible with MPI-3.1 and no-one has actually coded it yet. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot > Hi Howard, > > >> I recall we decided to leave the behavior of thread support level when using multiple sessions/world model up to the implementation. However, in the case of MPI_THREAD_FUNNELED we should probably add a blurb about not doing this in the advice to users in the Session Creation and Destruction Methods. > > I think the problem also appears if the original MPI_Init_thread uses FUNNELED. We probably need to add something there as well. > > >> My first guess based on the openmpi prototype is that if one session is initialized with funneled, then all subsequent created sessions or mpi init thread will be supported at funnel level. > > That kind of makes sense - but what happens if the second Session_Init is then on another thread? The funneled refers to a different thread. > > >> For your thread serialized question I would say the answer is no, unless under the covers the implementation is always working in effectively thread multiple mode. > > Hmm, I understand the motivation and implementation implications, but this would kill the isolation then, which is also not good. Also, if two libraries have separate sessions, it is often not even possible to avoid this. > > Almost seems like we should deprecate funneled and serialized. > > Another item to cover next Monday __. > > Martin

1 0

followup to discussions at yesterday's MPI virtual forum
by Pritchard Jr., Howard 18 Feb '21

18 Feb '21

HI All, Following up on the Session’s related discussions we had at yesterday’s virtual forum. To review, we discussed three open issues: https://github.com/mpi-forum/mpi-issues/issues/424 - Sessions: need to update sections of standard describing initial error handler https://github.com/mpi-forum/mpi-issues/issues/434 - Sessions: need to clarify limitations when mixing the sessions model with dynamic process model https://github.com/mpi-forum/mpi-issues/issues/435 - Is definition of local Session_init + collective Session_finalize pair broken? 424 is easy to address and I’ll rebase PR 519 on top of the rc branch and we can maybe get it added to the agenda for next week. 435 led to lots of lively conversation, but at the end of the day, my takeaway was that this is addressable via careful rewording and additional text plus some examples similar to the ones in 11.2.2 concerning MPI_Finalize. I have a problem though with some of the discussion around 434. I think it’s a big mistake to break with what had been our philosophy to date with sessions – that is don’t potentially break existing programs and library stacks. The text near the beginning of chapter 11 currently has a blurb about interoperability: An application can employ both of these Process Models concurrently. In multi-component MPI applications, for example, a component such as a library can make use of the Sessions Model to instantiate MPI resources without impacting the rest of the application. And this Both of these models also support the Dynamic Process Model (see Section 11.7), which provides for the creation and management of additional processes after an MPI application has been started. If we were going to say that a communicator derived via the sessions model can’t be used with MPI_Comm_spawn or MPI_Comm_spawn_multiple then potentially an existing library which under the covers uses these functions in its implementation, would stop working if the consumer of this library’s functionality (which probably involves some kind of input MPI communicator as an initialization argument) was converted to using sessions and supplied a communicator derived via the sessions mechanism. I suspect if we opened a PR to declare that such communicators can’t be used with these spawn functions, then someone will want us to add a function to test whether a communicator was associated with a session. Then there would be questions about MPI_Comm_accept/connect and so on. From an implementor’s point of view, I don’t see any major challenges supporting the dynamic process model with sessions – within the known limitations of having to use the world process model in the spawnee processes. I’m most reluctant to start putting in disclaimers and caveats about using communicators derived via the sessions mechanism as it seems to open us up to problems like these. Note I think some clarifying text about limitations when using the dynamic process model in the context of sessions is definitely warranted, and will work on a PR to add such text to the chapter. Howard -- Howard Pritchard HPC-ENV Los Alamos National Laboratory

1 0

Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED
by Martin Schulz 15 Feb '21

15 Feb '21

Hi Howard, >> I recall we decided to leave the behavior of thread support level when using multiple sessions/world model up to the implementation. However, in the case of MPI_THREAD_FUNNELED we should probably add a blurb about not doing this in the advice to users in the Session Creation and Destruction Methods. I think the problem also appears if the original MPI_Init_thread uses FUNNELED. We probably need to add something there as well. >> My first guess based on the openmpi prototype is that if one session is initialized with funneled, then all subsequent created sessions or mpi init thread will be supported at funnel level. That kind of makes sense - but what happens if the second Session_Init is then on another thread? The funneled refers to a different thread. >> For your thread serialized question I would say the answer is no, unless under the covers the implementation is always working in effectively thread multiple mode. Hmm, I understand the motivation and implementation implications, but this would kill the isolation then, which is also not good. Also, if two libraries have separate sessions, it is often not even possible to avoid this. Almost seems like we should deprecate funneled and serialized. Another item to cover next Monday __. Martin Howard On 2/13/21, 2:53 PM, "mpiwg-sessions on behalf of Martin Schulz via mpiwg-sessions" <mpiwg-sessions-bounces(a)lists.mpi-forum.org on behalf of mpiwg-sessions(a)lists.mpi-forum.org> wrote: Hi all, Can MPI_Session_Init be called with requesting MPI_THREAD_FUNNELED from two different threads? If so, does MPI return MPI_THREAD_FUNNELED or MPI_THREAD_SERIALIZED? Can I match this with MPI_Init_thread using MPI_THREAD_FUNNELED from a different thread? If we assume full isolation, that should work, right? Also, can I make two concurrent MPI calls from two different threads at the same time, assuming the MPI calls use MPI objects derived from two different sessions, both initialized with MPI_THREAD_SERIALIZED? Thanks, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] [EXTERNAL] Question on MPI_THREAD_FUNNELED
by Pritchard Jr., Howard 15 Feb '21

15 Feb '21

Hi Martin, I recall we decided to leave the behavior of thread support level when using multiple sessions/world model up to the implementation. However, in the case of MPI_THREAD_FUNNELED we should probably add a blurb about not doing this in the advice to users in the Session Creation and Destruction Methods. My first guess based on the openmpi prototype is that if one session is initialized with funneled, then all subsequent created sessions or mpi init thread will be supported at funnel level. For your thread serialized question I would say the answer is no, unless under the covers the implementation is always working in effectively thread multiple mode. Howard On 2/13/21, 2:53 PM, "mpiwg-sessions on behalf of Martin Schulz via mpiwg-sessions" <mpiwg-sessions-bounces(a)lists.mpi-forum.org on behalf of mpiwg-sessions(a)lists.mpi-forum.org> wrote: Hi all, Can MPI_Session_Init be called with requesting MPI_THREAD_FUNNELED from two different threads? If so, does MPI return MPI_THREAD_FUNNELED or MPI_THREAD_SERIALIZED? Can I match this with MPI_Init_thread using MPI_THREAD_FUNNELED from a different thread? If we assume full isolation, that should work, right? Also, can I make two concurrent MPI calls from two different threads at the same time, assuming the MPI calls use MPI objects derived from two different sessions, both initialized with MPI_THREAD_SERIALIZED? Thanks, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Question on MPI_THREAD_FUNNELED
by Martin Schulz 13 Feb '21

13 Feb '21

Hi all, Can MPI_Session_Init be called with requesting MPI_THREAD_FUNNELED from two different threads? If so, does MPI return MPI_THREAD_FUNNELED or MPI_THREAD_SERIALIZED? Can I match this with MPI_Init_thread using MPI_THREAD_FUNNELED from a different thread? If we assume full isolation, that should work, right? Also, can I make two concurrent MPI calls from two different threads at the same time, assuming the MPI calls use MPI objects derived from two different sessions, both initialized with MPI_THREAD_SERIALIZED? Thanks, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de

1 0

change in plan - Sessions WG will meet on 2/8
by Pritchard Jr., Howard 07 Feb '21

07 Feb '21

HI All, It looks like we need to discuss an issue open by Rolf last week: https://github.com/mpi-forum/mpi-issues/issues/435 He does have a point that as the Sessions Model related text currently stands, there is no explicit prohibition of applications using Sessions asymmetrically. By asymmetric I mean that different MPI processes derive MPI communicators, files, or windows from sessions handles differently. The current text implies that this is not allowed, or at least leads to logical inconsistencies, owing to the description of the collective behavior of Session finalize in the event that there are communicators, windows, or files still associated with a session handle when Session finalize is invoked. I propose that for 4.1, we add an additional paragraph to 11.3 that states something to the effect that this asymmetry is not allowed, or at the minimum that if an application does so, the behavior of Session finalize may be undefined. Other ideas on how to address 435 would be welcome! Thanks, Howard [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory lanl.gov [signature_493573494]<https://www.instagram.com/losalamosnatlab/>[signature_1756087032]<https://twitter.com/LosAlamosNatLab>[signature_1451995439]<https://www.linkedin.com/company/los-alamos-national-laboratory/>[signature_1759973918]<https://www.facebook.com/LosAlamosNationalLab/>

1 0

Re: [mpiwg-sessions] [EXTERNAL] Sessions/Spawn
by Pritchard Jr., Howard 03 Feb '21

03 Feb '21

Issue 434 - https://github.com/mpi-forum/mpi-issues/issues/434 On 2/3/21, 8:58 AM, "mpiwg-sessions on behalf of Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions-bounces(a)lists.mpi-forum.org on behalf of mpiwg-sessions(a)lists.mpi-forum.org> wrote: Hi Martin, Your proposal sounds good to me. Howard On 2/3/21, 8:16 AM, "Martin Schulz" <schulzm(a)in.tum.de> wrote: OK, that makes sense. Since we pre-announced, yet, perhaps we should just briefly state the topic, but don't go into solutions and then point folks to the WG if they want to weigh in. What do you think? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de On 03.02.21, 16:11, "Pritchard Jr., Howard" <howardp(a)lanl.gov> wrote: Hi Martin, I mulled this over some more and think we need to discuss the implications here further within the WG before presenting something to the forum. So let's hold off bringing this topic up at today's MPI forum. Howard On 2/2/21, 8:28 AM, "Schulz, Martin" <martin.w.j.schulz(a)tum.de> wrote: Hi Howard, Based on the discussion yesterday in the WG, should we mention this at the whole forum tomorrow and possible suggest the one sentence errata that Spawn is only of for the WPM, or is this too early? Thanks, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] [EXTERNAL] Sessions/Spawn
by Pritchard Jr., Howard 03 Feb '21

03 Feb '21

Hi Martin, Your proposal sounds good to me. Howard On 2/3/21, 8:16 AM, "Martin Schulz" <schulzm(a)in.tum.de> wrote: OK, that makes sense. Since we pre-announced, yet, perhaps we should just briefly state the topic, but don't go into solutions and then point folks to the WG if they want to weigh in. What do you think? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de On 03.02.21, 16:11, "Pritchard Jr., Howard" <howardp(a)lanl.gov> wrote: Hi Martin, I mulled this over some more and think we need to discuss the implications here further within the WG before presenting something to the forum. So let's hold off bringing this topic up at today's MPI forum. Howard On 2/2/21, 8:28 AM, "Schulz, Martin" <martin.w.j.schulz(a)tum.de> wrote: Hi Howard, Based on the discussion yesterday in the WG, should we mention this at the whole forum tomorrow and possible suggest the one sentence errata that Spawn is only of for the WPM, or is this too early? Thanks, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de

1 0

Re: [mpiwg-sessions] [EXTERNAL] Sessions/Spawn
by Pritchard Jr., Howard 03 Feb '21

03 Feb '21

HI Wesley, We don't think this is a show stopper for the MPI 4.0. For 4.1 we need to add some clarifying text to indicate that there are certain limitations to using the dynamic process model and sessions. I'll open an issue to track this for 4.1. Howard On 2/3/21, 8:32 AM, "Martin Schulz" <schulzm(a)in.tum.de> wrote: Sure, that is an option, of course, but I don't think it is that critical and I think Howard sees it the same way (please speak up if you disagree). However, if MPI should slip for other reasons, I think we should add a fix. Just my 2c, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de On 03.02.21, 16:21, "Wesley Bland" <work(a)wesbland.com> wrote: If that’s the solution, that’s fine. The other option is to hold MPI 4.0 if this is a showstopper. I’m not that familiar with the problem so I was trying to figure out which side this is on. Thanks, Wes > On Feb 3, 2021, at 9:18 AM, Martin Schulz <schulzm(a)in.tum.de> wrote: > > Hi Wesley, > > Sorry, we overlapped - based on the discussion we had on Monday, there is no way we can get this into MPI 4.0 the "right" way. So, the idea to put a stop-gap measure in that for now disallows certain parts, and that could be done as an errata. Hence, it would be nice to get this cleaned and ready, but I don't think we have a huge rush here. > > Martin > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de > > > > On 03.02.21, 16:16, "Bland, Wesley" <wesley.bland(a)intel.com> wrote: > > Is this something that needs to be in MPI 4.0? If so, we need to at least decide what we want to do along that path. If this can be fixed by an errata later, that’s fine. > > Thanks, > Wes > >> On Feb 3, 2021, at 9:11 AM, Pritchard Jr., Howard <howardp(a)lanl.gov> wrote: >> >> Hi Martin, >> >> I mulled this over some more and think we need to discuss the implications here further within the WG before presenting something to the forum. >> So let's hold off bringing this topic up at today's MPI forum. >> >> Howard >> >> >> On 2/2/21, 8:28 AM, "Schulz, Martin" <martin.w.j.schulz(a)tum.de> wrote: >> >> Hi Howard, >> >> Based on the discussion yesterday in the WG, should we mention this at the whole forum tomorrow and possible suggest the one sentence errata that Spawn is only of for the WPM, or is this too early? >> >> Thanks, >> >> Martin >> >> >> -- >> Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems >> Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching >> Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) >> Email: schulzm(a)in.tum.de >> >> >> >> >> > > >

1 0

Re: [mpiwg-sessions] [EXTERNAL] Sessions/Spawn
by Martin Schulz 03 Feb '21

03 Feb '21

Sure, that is an option, of course, but I don't think it is that critical and I think Howard sees it the same way (please speak up if you disagree). However, if MPI should slip for other reasons, I think we should add a fix. Just my 2c, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de On 03.02.21, 16:21, "Wesley Bland" <work(a)wesbland.com> wrote: If that’s the solution, that’s fine. The other option is to hold MPI 4.0 if this is a showstopper. I’m not that familiar with the problem so I was trying to figure out which side this is on. Thanks, Wes > On Feb 3, 2021, at 9:18 AM, Martin Schulz <schulzm(a)in.tum.de> wrote: > > Hi Wesley, > > Sorry, we overlapped - based on the discussion we had on Monday, there is no way we can get this into MPI 4.0 the "right" way. So, the idea to put a stop-gap measure in that for now disallows certain parts, and that could be done as an errata. Hence, it would be nice to get this cleaned and ready, but I don't think we have a huge rush here. > > Martin > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de > > > > On 03.02.21, 16:16, "Bland, Wesley" <wesley.bland(a)intel.com> wrote: > > Is this something that needs to be in MPI 4.0? If so, we need to at least decide what we want to do along that path. If this can be fixed by an errata later, that’s fine. > > Thanks, > Wes > >> On Feb 3, 2021, at 9:11 AM, Pritchard Jr., Howard <howardp(a)lanl.gov> wrote: >> >> Hi Martin, >> >> I mulled this over some more and think we need to discuss the implications here further within the WG before presenting something to the forum. >> So let's hold off bringing this topic up at today's MPI forum. >> >> Howard >> >> >> On 2/2/21, 8:28 AM, "Schulz, Martin" <martin.w.j.schulz(a)tum.de> wrote: >> >> Hi Howard, >> >> Based on the discussion yesterday in the WG, should we mention this at the whole forum tomorrow and possible suggest the one sentence errata that Spawn is only of for the WPM, or is this too early? >> >> Thanks, >> >> Martin >> >> >> -- >> Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems >> Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching >> Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) >> Email: schulzm(a)in.tum.de >> >> >> >> >> > > >

1 0