Hi Dan, I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. An example could be newly spawned processes  and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: [email protected] From: "Holmes, Daniel John" <[email protected]> Date: Wednesday, 4. January 2023 at 10:17 To: Martin Schulz <[email protected]>, MPI Sessions working group <[email protected]> Cc: "Pritchard Jr., Howard" <[email protected]> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. Best wishes, Dan. From: Martin Schulz <[email protected]> Sent: 04 January 2023 19:43 To: MPI Sessions working group <[email protected]>; Holmes, Daniel John <[email protected]> Cc: Pritchard Jr., Howard <[email protected]> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi all, I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: [email protected] From: mpiwg-sessions <[email protected]> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <[email protected]> Reply to: MPI Sessions working group <[email protected]> Date: Wednesday, 4. January 2023 at 09:30 To: "Holmes, Daniel John" <[email protected]>, MPI Sessions working group <[email protected]> Cc: "Pritchard Jr., Howard" <[email protected]> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll HI Dan, Yes that was my interpretation as well. We can discuss at our next meeting 1/9/23 if there’s time. Howard From: "Holmes, Daniel John" <[email protected]> Date: Wednesday, January 4, 2023 at 12:05 PM To: MPI Sessions working group <[email protected]> Cc: "Pritchard Jr., Howard" <[email protected]> Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Howard, It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: MPI_Session_init // local – PMIx fence prohibited MPI_Group_from_pset // local – PMIx fence prohibited MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). Best wishes, Dan. From: mpiwg-sessions <[email protected]> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions Sent: 04 January 2023 18:32 To: MPI Sessions working group <[email protected]> Cc: Pritchard Jr., Howard <[email protected]> Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll Hi All, First, Happy New Year! I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166 . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. Thanks for any ideas, Howard — Howard PritchardResearch ScientistHPC-ENV Los Alamos National [email protected]