January 2023 - mpiwg-sessions - mailman3.cels.anl.gov

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Ralph Castain 10 Jan '23

10 Jan '23

I'm not sure how anyone can meet some of those requirements as they lead into the infinite exchange of "yes I am at that point" handshakes. I currently let you know that the process set has been defined - i.e., all of the MPI procs involved have called 'construct'. I cannot let you know that all of those procs have actually exited that function call as I have no idea how they are progressing such things. So if you take some action based on the process set membership, it is entirely possible that some remote member is still in the 'construct' function when they receive that action. Adjustments to membership face the same issue. I can only notify you that the membership has been adjusted (meaning everyone is being notified of the updated membership), but there is no way to tell you when every member has adjusted their internal tracking of that membership. Thus, if you allow dynamic membership, then query of membership should always be regarded as a point-in-time response as by the time you use it, the membership might be in flux. I believe this is why communicators are immutable, yes? Ralph > On Jan 10, 2023, at 5:35 AM, Holmes, Daniel John via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > Hi Martin, > > So, in summary: > > An immediate procedure is a local procedure that does not require progress for its own semantic action but is still permitted to do some unrelated progress – in practice, it is very hard to distinguish between immediate and local – pragmatically, there might be no reason to attempt to distinguish immediate from local. Implications: > The MPI Standard should not specify “does not involve communication” for persistent point-to-point because it is unenforceable nonsense that attempts to distinguish immediate from local. > The MPI Standard should acknowledge that “basic runtime routines” such as printf are local in the MPI sense of the term, even if they do not directly contain any MPI-related code. > Upgrading the implementation of an MPI procedure from immediate to local in a PMPI wrapper should be permitted in all cases because tools. > MPI_Session_init is local semantically, whether the implementation complies with immediate or with local. > Any chunk of code that is not nonlocal is local semantically, whether it complies with immediate or with local. > Discovery of “new” process set names involves a consensus issue – either there is consensus before discovery or there is potential for race-conditions. Concerns: > There are no standardised mechanisms for adding new process sets, so everything is speculation here. > If future runtimes are permitted to adjust process set names unilaterally (without request/demand/control of any MPI procedures/operations), will they be required to ensure consensus before discovery or will they be permitted to introduce race-conditions into user code that uses MPI to query for process set names? Hint: sane runtimes only please. > If the future mechanism for adjusting process sets will be an MPI operation, will it have passive target one-sided semantics or collective semantics or something else non-yet-MPI? Hint: sane MPI operations only please. > Is there anything down this road that requires MPI_Session_init to be nonlocal? Hint: no. > Is there anything down this road that requires “query process set names” to be nonlocal? Hint: no. > > Best wishes, > Dan. > > From: Martin Schulz <schulzm(a)in.tum.de> > Sent: 10 January 2023 06:36 > To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; Martin Schulz <schulzm(a)in.tum.de>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> > Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Dan, all, > > More comments inline. > > Martin > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de <mailto:[email protected]> > > > From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>> > Date: Monday, 9. January 2023 at 16:36 > To: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Martin, > > Responses inline. > > Best wishes. > Dan. > > From: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>> > Sent: 06 January 2023 02:14 > To: Holmes, Daniel John <daniel.john.holmes(a)intel.com <mailto:[email protected]>>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: Pritchard Jr., Howard <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Dan, > > A few comments (kind of many intertwined good discussions 😊 ): > > Have we ever been able to define the difference between immediate and local? I know there is a perception difference, but what is difference to the user? What is too long for an immediate call? Are we disallowing progress? If so, what happens if the OS switches threads (this “blocks” the current one) and then the other thread makes progress? I don’t we can define this – I think for the user these two things will always be the same (from how they have to program) and an implementation will always have the freedom to do what it wants – there could at most be a performance expectation or advice to implementors. > > [DAN] This has always been the argument against attempting to differentiate between immediate and local; it has always been a successful argument. It means text stating that persistent point-to-point initialization procedures “involve no communication” (MPI-4.0 section 3.9 page 94 line 37) is an erratum we should fix – how can we say that when the OS might context switch to another thread that does legitimately does communication? Also, we do define the semantic of immediate without naming it – when we describe the independence of basic runtime routines (MPI-4.0 section 2.9.1, page 28, lines 24-27) – all “library routines that are part of the basic language environment” have this unnamed immediate semantic, even though the OS could preempt a call to “printf”, context-switch to a different MPI process, and do MPI things. > > [MARTIN] I think we agree here then, we need to treat immediate and local as the same from a user’s point of view – is this what you mean? > > Why should an MPI implementation be prevented from calling the progress engine in a Comm_rank call? Why would it not be able to communicate, if it wants to? E.g., is a PMPI wrapper implementation that sends an acknowledgement that this call has been made to a central location and waits for it being received an incorrect wrapper? > > [DAN] Permitting poking the (local) progress engine during the call and requiring some (remote) progress to happen before returning from the call are observably different – the first cannot delay until remote action, the second can be arbitrarily delayed by refusing to enter MPI at another MPI process. OTOH, the example you give (reliably log procedure call) is not an MPI progress action, so it is irrelevant to the discussion of immediate vs. local. If the PMPI wrapper used MPI to send the log entry and MPI to ensure it has been received, then the wrapper would be nonlocal, so you cannot mean that. You must be intending usage of some non-MPI communication mechanism, which is a bad idea in the presence of pending MPI communication (MPI-4.0 section 11.10.5 page 549 line 21 says it’s UB), which makes calling your intercepted MPI_COMM_RANK risky. > > [MARTIN] No, I actually meant using MPI send/recv to send the logs – If I intercept an MPI API with PMPI, I (as the tool writer) need to make sure that the (modified) API is consistent with the MPI standard, i.e., I offer a local semantics in case of calls with Comm_rank. However, how I do this, should be up to me as the tool writer. If I reserve an MPI process for tools work, create a new “fake” COMM_WORLD with that process and then have the application work on that, plus I use that extra process in a way that guarantees that any send to that process from the wrappers is always enabled (in the sense we are trying to define it), I am still honoring that guarantee and keeping Comm_rank local for the application running on top of my PMPI layer. However, if we start distinguishing local and immediate and we disallow such weak local behavior to occur in some calls, then all the tools that use this become incorrect, which would be bad. > > As for the Sessions API – I agree with most of what you say on local vs. non-local, but this is the user’s perspective. I was more asking about whether there is ever a need for an implementation to wait for something to complete. If we can really exclude that, then we should be good – however, I doubt we can do that for sure going forward. > > [DAN] We can always exclude that wait because we can always postpone that wait. > > [MARTIN] That would be good – not 100% convinced, yet, if we can have the needed information handy at this point – but I am happy to be convinced otherwise. > > As for the waiting for changes – shouldn’t this be part of the query of the process list? Why a separate call? > > [DAN] No. Because we already have the separate calls and all of them have the necessary semantic. The only process sets that could be *discovered* by a subsequent query call are ones where the calling MPI process is not a member. > > [MARTIN] The list of process sets can grow and hence it should be able to discover new process sets, even with the local process being a member – we just cannot remove processes sets. > > [DAN] The existing immediate/local query will check for such changes but not delay until such a change happens. > > [MARTIN] I did not mean to wait until a change happens, but we probably want some consistency or mechanisms to ensure this. Consider the following code > > SESSION_INIT > QUERY PROCESS SETS > … > BARRIER > … > QUERY PROCESS SETS > > If the runtime adds a process set between the first query operation and the barrier that includes more than one MPI process in the communicator that is used in the barrier, would you expect for this process set to show up in the second query calls in all of the MPI processes? If so, there may be the need for some synchronization in the processes set engine. You could do that in each and every barrier, but this could be costly. It is likely very advantageous to have the ability in the query routine to synchronize. If the process sets can be different, what mechanism can the user us to ensure he/she can “trust” the process set information to be accurate, i.e., when can they call a communicator creation safely? > > [DAN] If you want “I know a change is supposed to happen eventually, delay here until it does and tell me about it” then you need a new probably-nonlocal procedure (it’s return depends on remote action, but not yet an MPI procedure because we have no MPI way to create new process sets). > > [MARTIN] From an MPI perspective, this would still be local, as no semantically related MPI procedure call is required. > > [DAN] This is all very speculative until we define sessions 2.0 with mechanisms to create new process sets. Achieving consensus on the new process set (content and naming) is on our list of semantic concerns in the Sessions WG for such functionality. > > [MARTIN] Agreed, actually reacting or explicitly waiting for process set changes is something for MPI 5.0. > > Why do you want to see session_init as an immediate call? Shouldn’t this be able to setup things, which could include allocations possible triggering longer resource negotiations in the OS? That call should not be in the critical path? > > [DAN] There is no requirement for MPI_Session_init to do anything complex, why do you want to add complexity where there is none and there is no need for it? > > [MARTIN] See above – we cannot find a clear separation of local and immediate and once this should be a moot point. As for a use case, sending a log message for a tool would be important. > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de <mailto:[email protected]> > > > From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>> > Date: Thursday, 5. January 2023 at 02:50 > To: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Martin, > > This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. > > immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call > local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call > nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call > > > Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. > > I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. > > The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. > > Best wishes, > Dan. > > From: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>> > Sent: 04 January 2023 22:43 > To: Holmes, Daniel John <daniel.john.holmes(a)intel.com <mailto:[email protected]>>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: Pritchard Jr., Howard <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Dan, > > I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). > > However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. > > An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. > > This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. > > Martin > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de <mailto:[email protected]> > > > From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>> > Date: Wednesday, 4. January 2023 at 10:17 > To: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Martin, > > MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. > > For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. > > Best wishes, > Dan. > > From: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>> > Sent: 04 January 2023 19:43 > To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>>; Holmes, Daniel John <daniel.john.holmes(a)intel.com <mailto:[email protected]>> > Cc: Pritchard Jr., Howard <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi all, > > I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. > > The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? > > Martin > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de <mailto:[email protected]> > > > From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Date: Wednesday, 4. January 2023 at 09:30 > To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > HI Dan, > > Yes that was my interpretation as well. > > We can discuss at our next meeting 1/9/23 if there’s time. > > Howard > > > From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>> > Date: Wednesday, January 4, 2023 at 12:05 PM > To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Howard, > > It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. > > Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: > > MPI_Session_init // local – PMIx fence prohibited > MPI_Group_from_pset // local – PMIx fence prohibited > MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed > > The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). > > Best wishes, > Dan. > > From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions > Sent: 04 January 2023 18:32 > To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: Pritchard Jr., Howard <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll > > Hi All, > > First, Happy New Year! > > I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. > We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. > > I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166 <https://urldefense.com/v3/__https:/github.com/open-mpi/ompi/issues/11166__;…> . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. > > I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. > > Thanks for any ideas, > > Howard > > > — > > <image001.png> > Howard Pritchard > Research Scientist > HPC-ENV > > Los Alamos National Laboratory > howardp(a)lanl.gov <mailto:[email protected]> > > <image002.png> <https://urldefense.com/v3/__https:/www.instagram.com/losalamosnatlab/__;!!B…><image003.png> <https://urldefense.com/v3/__https:/twitter.com/LosAlamosNatLab__;!!Bt8fGhp8…><image004.png> <https://urldefense.com/v3/__https:/www.linkedin.com/company/los-alamos-nati…><image005.png> <https://urldefense.com/v3/__https:/www.facebook.com/LosAlamosNationalLab/__…> > > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Holmes, Daniel John 10 Jan '23

10 Jan '23

Hi Martin, So, in summary: 1. An immediate procedure is a local procedure that does not require progress for its own semantic action but is still permitted to do some unrelated progress – in practice, it is very hard to distinguish between immediate and local – pragmatically, there might be no reason to attempt to distinguish immediate from local. Implications: * The MPI Standard should not specify “does not involve communication” for persistent point-to-point because it is unenforceable nonsense that attempts to distinguish immediate from local. * The MPI Standard should acknowledge that “basic runtime routines” such as printf are local in the MPI sense of the term, even if they do not directly contain any MPI-related code. * Upgrading the implementation of an MPI procedure from immediate to local in a PMPI wrapper should be permitted in all cases because tools. * MPI_Session_init is local semantically, whether the implementation complies with immediate or with local. * Any chunk of code that is not nonlocal is local semantically, whether it complies with immediate or with local. 2. Discovery of “new” process set names involves a consensus issue – either there is consensus before discovery or there is potential for race-conditions. Concerns: * There are no standardised mechanisms for adding new process sets, so everything is speculation here. * If future runtimes are permitted to adjust process set names unilaterally (without request/demand/control of any MPI procedures/operations), will they be required to ensure consensus before discovery or will they be permitted to introduce race-conditions into user code that uses MPI to query for process set names? Hint: sane runtimes only please. * If the future mechanism for adjusting process sets will be an MPI operation, will it have passive target one-sided semantics or collective semantics or something else non-yet-MPI? Hint: sane MPI operations only please. * Is there anything down this road that requires MPI_Session_init to be nonlocal? Hint: no. * Is there anything down this road that requires “query process set names” to be nonlocal? Hint: no. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 10 January 2023 06:36 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; Martin Schulz <schulzm(a)in.tum.de>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, all, More comments inline. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Monday, 9. January 2023 at 16:36 To: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, Responses inline. Best wishes. Dan. From: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>> Sent: 06 January 2023 02:14 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com<mailto:[email protected]>>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, A few comments (kind of many intertwined good discussions 😊 ): * Have we ever been able to define the difference between immediate and local? I know there is a perception difference, but what is difference to the user? What is too long for an immediate call? Are we disallowing progress? If so, what happens if the OS switches threads (this “blocks” the current one) and then the other thread makes progress? I don’t we can define this – I think for the user these two things will always be the same (from how they have to program) and an implementation will always have the freedom to do what it wants – there could at most be a performance expectation or advice to implementors. [DAN] This has always been the argument against attempting to differentiate between immediate and local; it has always been a successful argument. It means text stating that persistent point-to-point initialization procedures “involve no communication” (MPI-4.0 section 3.9 page 94 line 37) is an erratum we should fix – how can we say that when the OS might context switch to another thread that does legitimately does communication? Also, we do define the semantic of immediate without naming it – when we describe the independence of basic runtime routines (MPI-4.0 section 2.9.1, page 28, lines 24-27) – all “library routines that are part of the basic language environment” have this unnamed immediate semantic, even though the OS could preempt a call to “printf”, context-switch to a different MPI process, and do MPI things. [MARTIN] I think we agree here then, we need to treat immediate and local as the same from a user’s point of view – is this what you mean? * Why should an MPI implementation be prevented from calling the progress engine in a Comm_rank call? Why would it not be able to communicate, if it wants to? E.g., is a PMPI wrapper implementation that sends an acknowledgement that this call has been made to a central location and waits for it being received an incorrect wrapper? [DAN] Permitting poking the (local) progress engine during the call and requiring some (remote) progress to happen before returning from the call are observably different – the first cannot delay until remote action, the second can be arbitrarily delayed by refusing to enter MPI at another MPI process. OTOH, the example you give (reliably log procedure call) is not an MPI progress action, so it is irrelevant to the discussion of immediate vs. local. If the PMPI wrapper used MPI to send the log entry and MPI to ensure it has been received, then the wrapper would be nonlocal, so you cannot mean that. You must be intending usage of some non-MPI communication mechanism, which is a bad idea in the presence of pending MPI communication (MPI-4.0 section 11.10.5 page 549 line 21 says it’s UB), which makes calling your intercepted MPI_COMM_RANK risky. [MARTIN] No, I actually meant using MPI send/recv to send the logs – If I intercept an MPI API with PMPI, I (as the tool writer) need to make sure that the (modified) API is consistent with the MPI standard, i.e., I offer a local semantics in case of calls with Comm_rank. However, how I do this, should be up to me as the tool writer. If I reserve an MPI process for tools work, create a new “fake” COMM_WORLD with that process and then have the application work on that, plus I use that extra process in a way that guarantees that any send to that process from the wrappers is always enabled (in the sense we are trying to define it), I am still honoring that guarantee and keeping Comm_rank local for the application running on top of my PMPI layer. However, if we start distinguishing local and immediate and we disallow such weak local behavior to occur in some calls, then all the tools that use this become incorrect, which would be bad. * As for the Sessions API – I agree with most of what you say on local vs. non-local, but this is the user’s perspective. I was more asking about whether there is ever a need for an implementation to wait for something to complete. If we can really exclude that, then we should be good – however, I doubt we can do that for sure going forward. [DAN] We can always exclude that wait because we can always postpone that wait. [MARTIN] That would be good – not 100% convinced, yet, if we can have the needed information handy at this point – but I am happy to be convinced otherwise. * As for the waiting for changes – shouldn’t this be part of the query of the process list? Why a separate call? [DAN] No. Because we already have the separate calls and all of them have the necessary semantic. The only process sets that could be *discovered* by a subsequent query call are ones where the calling MPI process is not a member. [MARTIN] The list of process sets can grow and hence it should be able to discover new process sets, even with the local process being a member – we just cannot remove processes sets. [DAN] The existing immediate/local query will check for such changes but not delay until such a change happens. [MARTIN] I did not mean to wait until a change happens, but we probably want some consistency or mechanisms to ensure this. Consider the following code SESSION_INIT QUERY PROCESS SETS … BARRIER … QUERY PROCESS SETS If the runtime adds a process set between the first query operation and the barrier that includes more than one MPI process in the communicator that is used in the barrier, would you expect for this process set to show up in the second query calls in all of the MPI processes? If so, there may be the need for some synchronization in the processes set engine. You could do that in each and every barrier, but this could be costly. It is likely very advantageous to have the ability in the query routine to synchronize. If the process sets can be different, what mechanism can the user us to ensure he/she can “trust” the process set information to be accurate, i.e., when can they call a communicator creation safely? [DAN] If you want “I know a change is supposed to happen eventually, delay here until it does and tell me about it” then you need a new probably-nonlocal procedure (it’s return depends on remote action, but not yet an MPI procedure because we have no MPI way to create new process sets). [MARTIN] From an MPI perspective, this would still be local, as no semantically related MPI procedure call is required. [DAN] This is all very speculative until we define sessions 2.0 with mechanisms to create new process sets. Achieving consensus on the new process set (content and naming) is on our list of semantic concerns in the Sessions WG for such functionality. [MARTIN] Agreed, actually reacting or explicitly waiting for process set changes is something for MPI 5.0. * Why do you want to see session_init as an immediate call? Shouldn’t this be able to setup things, which could include allocations possible triggering longer resource negotiations in the OS? That call should not be in the critical path? [DAN] There is no requirement for MPI_Session_init to do anything complex, why do you want to add complexity where there is none and there is no need for it? [MARTIN] See above – we cannot find a clear separation of local and immediate and once this should be a moot point. As for a use case, sending a log message for a tool would be important. -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Thursday, 5. January 2023 at 02:50 To: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>> Sent: 04 January 2023 22:43 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com<mailto:[email protected]>>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Wednesday, 4. January 2023 at 10:17 To: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>> Sent: 04 January 2023 19:43 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>>; Holmes, Daniel John <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi all, I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Date: Wednesday, 4. January 2023 at 09:30 To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll HI Dan, Yes that was my interpretation as well. We can discuss at our next meeting 1/9/23 if there’s time. Howard From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Wednesday, January 4, 2023 at 12:05 PM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Howard, It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: MPI_Session_init // local – PMIx fence prohibited MPI_Group_from_pset // local – PMIx fence prohibited MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). Best wishes, Dan. From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions Sent: 04 January 2023 18:32 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll Hi All, First, Happy New Year! I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166<https://urldefense.com/v3/__https:/github.com/open-mpi/ompi/issues/11166__;…> . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. Thanks for any ideas, Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_1293224934]<https://urldefense.com/v3/__https:/www.instagram.com/losalamosnatlab/__;!!B…>[signature_2498822630]<https://urldefense.com/v3/__https:/twitter.com/LosAlamosNatLab__;!!Bt8fGhp8…>[signature_1283032776]<https://urldefense.com/v3/__https:/www.linkedin.com/company/los-alamos-nati…>[signature_3959178607]<https://urldefense.com/v3/__https:/www.facebook.com/LosAlamosNationalLab/__…>

1 0

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Martin Schulz 10 Jan '23

10 Jan '23

Hi Dan, all, More comments inline. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Monday, 9. January 2023 at 16:36 To: Martin Schulz <schulzm(a)in.tum.de>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, Responses inline. Best wishes. Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 06 January 2023 02:14 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, A few comments (kind of many intertwined good discussions 😊 ): Have we ever been able to define the difference between immediate and local? I know there is a perception difference, but what is difference to the user? What is too long for an immediate call? Are we disallowing progress? If so, what happens if the OS switches threads (this “blocks” the current one) and then the other thread makes progress? I don’t we can define this – I think for the user these two things will always be the same (from how they have to program) and an implementation will always have the freedom to do what it wants – there could at most be a performance expectation or advice to implementors. [DAN] This has always been the argument against attempting to differentiate between immediate and local; it has always been a successful argument. It means text stating that persistent point-to-point initialization procedures “involve no communication” (MPI-4.0 section 3.9 page 94 line 37) is an erratum we should fix – how can we say that when the OS might context switch to another thread that does legitimately does communication? Also, we do define the semantic of immediate without naming it – when we describe the independence of basic runtime routines (MPI-4.0 section 2.9.1, page 28, lines 24-27) – all “library routines that are part of the basic language environment” have this unnamed immediate semantic, even though the OS could preempt a call to “printf”, context-switch to a different MPI process, and do MPI things. [MARTIN] I think we agree here then, we need to treat immediate and local as the same from a user’s point of view – is this what you mean? Why should an MPI implementation be prevented from calling the progress engine in a Comm_rank call? Why would it not be able to communicate, if it wants to? E.g., is a PMPI wrapper implementation that sends an acknowledgement that this call has been made to a central location and waits for it being received an incorrect wrapper? [DAN] Permitting poking the (local) progress engine during the call and requiring some (remote) progress to happen before returning from the call are observably different – the first cannot delay until remote action, the second can be arbitrarily delayed by refusing to enter MPI at another MPI process. OTOH, the example you give (reliably log procedure call) is not an MPI progress action, so it is irrelevant to the discussion of immediate vs. local. If the PMPI wrapper used MPI to send the log entry and MPI to ensure it has been received, then the wrapper would be nonlocal, so you cannot mean that. You must be intending usage of some non-MPI communication mechanism, which is a bad idea in the presence of pending MPI communication (MPI-4.0 section 11.10.5 page 549 line 21 says it’s UB), which makes calling your intercepted MPI_COMM_RANK risky. [MARTIN] No, I actually meant using MPI send/recv to send the logs – If I intercept an MPI API with PMPI, I (as the tool writer) need to make sure that the (modified) API is consistent with the MPI standard, i.e., I offer a local semantics in case of calls with Comm_rank. However, how I do this, should be up to me as the tool writer. If I reserve an MPI process for tools work, create a new “fake” COMM_WORLD with that process and then have the application work on that, plus I use that extra process in a way that guarantees that any send to that process from the wrappers is always enabled (in the sense we are trying to define it), I am still honoring that guarantee and keeping Comm_rank local for the application running on top of my PMPI layer. However, if we start distinguishing local and immediate and we disallow such weak local behavior to occur in some calls, then all the tools that use this become incorrect, which would be bad. As for the Sessions API – I agree with most of what you say on local vs. non-local, but this is the user’s perspective. I was more asking about whether there is ever a need for an implementation to wait for something to complete. If we can really exclude that, then we should be good – however, I doubt we can do that for sure going forward. [DAN] We can always exclude that wait because we can always postpone that wait. [MARTIN] That would be good – not 100% convinced, yet, if we can have the needed information handy at this point – but I am happy to be convinced otherwise. As for the waiting for changes – shouldn’t this be part of the query of the process list? Why a separate call? [DAN] No. Because we already have the separate calls and all of them have the necessary semantic. The only process sets that could be *discovered* by a subsequent query call are ones where the calling MPI process is not a member. [MARTIN] The list of process sets can grow and hence it should be able to discover new process sets, even with the local process being a member – we just cannot remove processes sets. [DAN] The existing immediate/local query will check for such changes but not delay until such a change happens. [MARTIN] I did not mean to wait until a change happens, but we probably want some consistency or mechanisms to ensure this. Consider the following code SESSION_INIT QUERY PROCESS SETS … BARRIER … QUERY PROCESS SETS If the runtime adds a process set between the first query operation and the barrier that includes more than one MPI process in the communicator that is used in the barrier, would you expect for this process set to show up in the second query calls in all of the MPI processes? If so, there may be the need for some synchronization in the processes set engine. You could do that in each and every barrier, but this could be costly. It is likely very advantageous to have the ability in the query routine to synchronize. If the process sets can be different, what mechanism can the user us to ensure he/she can “trust” the process set information to be accurate, i.e., when can they call a communicator creation safely? [DAN] If you want “I know a change is supposed to happen eventually, delay here until it does and tell me about it” then you need a new probably-nonlocal procedure (it’s return depends on remote action, but not yet an MPI procedure because we have no MPI way to create new process sets). [MARTIN] From an MPI perspective, this would still be local, as no semantically related MPI procedure call is required. [DAN] This is all very speculative until we define sessions 2.0 with mechanisms to create new process sets. Achieving consensus on the new process set (content and naming) is on our list of semantic concerns in the Sessions WG for such functionality. [MARTIN] Agreed, actually reacting or explicitly waiting for process set changes is something for MPI 5.0. Why do you want to see session_init as an immediate call? Shouldn’t this be able to setup things, which could include allocations possible triggering longer resource negotiations in the OS? That call should not be in the critical path? [DAN] There is no requirement for MPI_Session_init to do anything complex, why do you want to add complexity where there is none and there is no need for it? [MARTIN] See above – we cannot find a clear separation of local and immediate and once this should be a moot point. As for a use case, sending a log message for a tool would be important. -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Thursday, 5. January 2023 at 02:50 To: Martin Schulz <schulzm(a)in.tum.de>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 04 January 2023 22:43 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Wednesday, 4. January 2023 at 10:17 To: Martin Schulz <schulzm(a)in.tum.de>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 04 January 2023 19:43 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org>; Holmes, Daniel John <daniel.john.holmes(a)intel.com> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi all, I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org> Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Wednesday, 4. January 2023 at 09:30 To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll HI Dan, Yes that was my interpretation as well. We can discuss at our next meeting 1/9/23 if there’s time. Howard From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Wednesday, January 4, 2023 at 12:05 PM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Howard, It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: MPI_Session_init // local – PMIx fence prohibited MPI_Group_from_pset // local – PMIx fence prohibited MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). Best wishes, Dan. From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions Sent: 04 January 2023 18:32 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll Hi All, First, Happy New Year! I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166 . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. Thanks for any ideas, Howard — Howard PritchardResearch ScientistHPC-ENV Los Alamos National Laboratoryhowardp(a)lanl.gov

1 0

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Holmes, Daniel John 09 Jan '23

09 Jan '23

Hi Martin, Responses inline. Best wishes. Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 06 January 2023 02:14 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, A few comments (kind of many intertwined good discussions 😊 ): * Have we ever been able to define the difference between immediate and local? I know there is a perception difference, but what is difference to the user? What is too long for an immediate call? Are we disallowing progress? If so, what happens if the OS switches threads (this “blocks” the current one) and then the other thread makes progress? I don’t we can define this – I think for the user these two things will always be the same (from how they have to program) and an implementation will always have the freedom to do what it wants – there could at most be a performance expectation or advice to implementors. [DAN] This has always been the argument against attempting to differentiate between immediate and local; it has always been a successful argument. It means text stating that persistent point-to-point initialization procedures “involve no communication” (MPI-4.0 section 3.9 page 94 line 37) is an erratum we should fix – how can we say that when the OS might context switch to another thread that does legitimately does communication? Also, we do define the semantic of immediate without naming it – when we describe the independence of basic runtime routines (MPI-4.0 section 2.9.1, page 28, lines 24-27) – all “library routines that are part of the basic language environment” have this unnamed immediate semantic, even though the OS could preempt a call to “printf”, context-switch to a different MPI process, and do MPI things. * Why should an MPI implementation be prevented from calling the progress engine in a Comm_rank call? Why would it not be able to communicate, if it wants to? E.g., is a PMPI wrapper implementation that sends an acknowledgement that this call has been made to a central location and waits for it being received an incorrect wrapper? [DAN] Permitting poking the (local) progress engine during the call and requiring some (remote) progress to happen before returning from the call are observably different – the first cannot delay until remote action, the second can be arbitrarily delayed by refusing to enter MPI at another MPI process. OTOH, the example you give (reliably log procedure call) is not an MPI progress action, so it is irrelevant to the discussion of immediate vs. local. If the PMPI wrapper used MPI to send the log entry and MPI to ensure it has been received, then the wrapper would be nonlocal, so you cannot mean that. You must be intending usage of some non-MPI communication mechanism, which is a bad idea in the presence of pending MPI communication (MPI-4.0 section 11.10.5 page 549 line 21 says it’s UB), which makes calling your intercepted MPI_COMM_RANK risky. * As for the Sessions API – I agree with most of what you say on local vs. non-local, but this is the user’s perspective. I was more asking about whether there is ever a need for an implementation to wait for something to complete. If we can really exclude that, then we should be good – however, I doubt we can do that for sure going forward. [DAN] We can always exclude that wait because we can always postpone that wait. * As for the waiting for changes – shouldn’t this be part of the query of the process list? Why a separate call? [DAN] No. Because we already have the separate calls and all of them have the necessary semantic. The only process sets that could be *discovered* by a subsequent query call are ones where the calling MPI process is not a member. The existing immediate/local query will check for such changes but not delay until such a change happens. If you want “I know a change is supposed to happen eventually, delay here until it does and tell me about it” then you need a new probably-nonlocal procedure (it’s return depends on remote action, but not yet an MPI procedure because we have no MPI way to create new process sets). This is all very speculative until we define sessions 2.0 with mechanisms to create new process sets. Achieving consensus on the new process set (content and naming) is on our list of semantic concerns in the Sessions WG for such functionality. * Why do you want to see session_init as an immediate call? Shouldn’t this be able to setup things, which could include allocations possible triggering longer resource negotiations in the OS? That call should not be in the critical path? [DAN] There is no requirement for MPI_Session_init to do anything complex, why do you want to add complexity where there is none and there is no need for it? Cheers, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Thursday, 5. January 2023 at 02:50 To: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>> Sent: 04 January 2023 22:43 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com<mailto:[email protected]>>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Wednesday, 4. January 2023 at 10:17 To: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>> Sent: 04 January 2023 19:43 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>>; Holmes, Daniel John <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi all, I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Date: Wednesday, 4. January 2023 at 09:30 To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll HI Dan, Yes that was my interpretation as well. We can discuss at our next meeting 1/9/23 if there’s time. Howard From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Wednesday, January 4, 2023 at 12:05 PM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Howard, It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: MPI_Session_init // local – PMIx fence prohibited MPI_Group_from_pset // local – PMIx fence prohibited MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). Best wishes, Dan. From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions Sent: 04 January 2023 18:32 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll Hi All, First, Happy New Year! I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166<https://urldefense.com/v3/__https:/github.com/open-mpi/ompi/issues/11166__;…> . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. Thanks for any ideas, Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_1293224934]<https://urldefense.com/v3/__https:/www.instagram.com/losalamosnatlab/__;!!B…>[signature_2498822630]<https://urldefense.com/v3/__https:/twitter.com/LosAlamosNatLab__;!!Bt8fGhp8…>[signature_1283032776]<https://urldefense.com/v3/__https:/www.linkedin.com/company/los-alamos-nati…>[signature_3959178607]<https://urldefense.com/v3/__https:/www.facebook.com/LosAlamosNationalLab/__…>

1 0

Sessions WG - meet today
by Pritchard Jr., Howard 09 Jan '23

09 Jan '23

Hi All, Happy New Year! We’ll start back today on Sessions WG meetings. Topics for today: * Sessions and parallel debuggers * MPI_Session_init – do we need something more precise than “lightweight” * Proposal from Tools working group (contingent on Marc-Andre Hermmans availability) * Future meeting agenda items As a reminder, info for connecting to the zoom meeting can be found at https://github.com/mpiwg-sessions/mpi-standard/wiki Connection info has not changed from last year – or at least I don’t think so. Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov [signature_3888582037]<https://www.instagram.com/losalamosnatlab/>[signature_2370638264]<https://twitter.com/LosAlamosNatLab>[signature_3255071010]<https://www.linkedin.com/company/los-alamos-national-laboratory/>[signature_872298555]<https://www.facebook.com/LosAlamosNationalLab/>

1 0

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Ralph Castain 06 Jan '23

06 Jan '23

Just so we are on the same page: a process set is some collection of processes that have agreed to call themselves "foo" - yes? Kind of like arbitrary named groupings. In PMIx-land, we call those "groups" - which is immaterial, but I wanted to ensure anyone looking thru the standard knew the right place to find it. I'll follow your 'set" terminology here. Definition of a set can be initiated from two sources: * external to the application - e.g., someone using a tool to establish the definition * from the application itself. PMIx supports two methods here: one that is driven by a single process (that invites others to join), and the other being a collective. Looking solely at the application-initiated side, we have the possibility that a proc may augment the set membership when it joins. Essentially, the situation is where a process knows of some other procs that need to be members of the set, but aren't necessarily known to all set members just yet. When this proc joins, it has to add those procs to the set (via an attribute). The bottom line is that application procs don't actually know the final set membership until they are handed it at completion of the construct operation. Thus, the operation is always a global one across the nodes where member procs are executing. So the event notifying that a process set has been newly defined isn't generated until the operation completes (since the event includes the membership, which we don't know until it does complete). It is guaranteed visible to all processes within its scope (see the Standard for scope rules), not just the ones involved in the set as they really don't need it (the operation return tells them the set construct is complete). Receipt of the event is ansync - no guarantee of ordering. If the app is doing the single-process driven method for constructing the set, then there are additional events involved (notifying that you have been invited, notifying the leader that you accepted/rejected the invite, etc). No real way for another process to interject itself into that handshake exchange. I am well aware of Howard's scenario as I raised the symptoms of it to him over in OMPI-land a little whiule ago. I'm not sure, though, if what I'm answering really pertains to what he's trying to solve. I'll try toop around with him to see if I'm just dragging you all down a whirlpool - could well be that he hit something I don't know about :-) Ralph > On Jan 5, 2023, at 6:33 PM, Martin Schulz <schulzm(a)in.tum.de> wrote: > > Hi Ralph, > > What does the definition of a new process set mean? Is this just local or then guaranteed visible across all involved processes? When is that coordination and/or synchronization happen and is that visible at any time to the user? > > I guess this goes back to Howard’s original question on when an MPI implementations is required to add PMIx fences (not necessarily for a particular implementation for a portable usage of the API across any implementation). > > Thanks, > > Martin > > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de <mailto:[email protected]> > > > From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> on behalf of Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Date: Thursday, 5. January 2023 at 07:09 > To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: Ralph Castain <rhc(a)pmix.org <mailto:[email protected]>> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Just FWIW: PMIx provides an event you can register to receive whenever a new process set has been defined. We could, I imagine, add one for process set deletion if that is useful. > > And you can always query for the complete list of current process set names (as well as memberships, if you want it). > Ralph > > > >> On Jan 5, 2023, at 4:50 AM, Holmes, Daniel John via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: >> >> Hi Martin, >> >> This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. >> >> immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call >> local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call >> nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call >> >> >> Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. >> >> I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. >> >> The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. >> >> Best wishes, >> Dan. >> >> From: Martin Schulz <schulzm(a)in.tum.de> >> Sent: 04 January 2023 22:43 >> To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> >> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> >> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll >> >> Hi Dan, >> >> I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). >> >> However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. >> >> An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. >> >> This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. >> >> Martin >> >> >> -- >> Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems >> Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching >> Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) >> Email: schulzm(a)in.tum.de <mailto:[email protected]> >> >> >> From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>> >> Date: Wednesday, 4. January 2023 at 10:17 >> To: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> >> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> >> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll >> >> Hi Martin, >> >> MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. >> >> For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. >> >> Best wishes, >> Dan. >> >> From: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>> >> Sent: 04 January 2023 19:43 >> To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>>; Holmes, Daniel John <daniel.john.holmes(a)intel.com <mailto:[email protected]>> >> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov <mailto:[email protected]>> >> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll >> >> Hi all, >> >> I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. >> >> The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? >> >> Martin >> >> >> -- >> Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems >> Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching >> Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) >> Email: schulzm(a)in.tum.de <mailto:[email protected]> >> >> >> From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> >> Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> >> Date: Wednesday, 4. January 2023 at 09:30 >> To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> >> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> >> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll >> >> HI Dan, >> >> Yes that was my interpretation as well. >> >> We can discuss at our next meeting 1/9/23 if there’s time. >> >> Howard >> >> >> From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>> >> Date: Wednesday, January 4, 2023 at 12:05 PM >> To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> >> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> >> Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll >> >> Hi Howard, >> >> It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. >> >> Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: >> >> MPI_Session_init // local – PMIx fence prohibited >> MPI_Group_from_pset // local – PMIx fence prohibited >> MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed >> >> The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). >> >> Best wishes, >> Dan. >> >> From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions >> Sent: 04 January 2023 18:32 >> To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> >> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov <mailto:[email protected]>> >> Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll >> >> Hi All, >> >> First, Happy New Year! >> >> I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. >> We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. >> >> I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166 <https://urldefense.com/v3/__https:/github.com/open-mpi/ompi/issues/11166__;…> . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. >> >> I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. >> >> Thanks for any ideas, >> >> Howard >> >> >> — >> >> <image001.png> >> Howard Pritchard >> Research Scientist >> HPC-ENV >> >> Los Alamos National Laboratory >> howardp(a)lanl.gov <mailto:[email protected]> >> >> <image002.png> <https://urldefense.com/v3/__https:/www.instagram.com/losalamosnatlab/__;!!B…><image003.png> <https://urldefense.com/v3/__https:/twitter.com/LosAlamosNatLab__;!!Bt8fGhp8…><image004.png> <https://urldefense.com/v3/__https:/www.linkedin.com/company/los-alamos-nati…><image005.png> <https://urldefense.com/v3/__https:/www.facebook.com/LosAlamosNationalLab/__…> >> >> >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Martin Schulz 05 Jan '23

05 Jan '23

Hi Ralph, What does the definition of a new process set mean? Is this just local or then guaranteed visible across all involved processes? When is that coordination and/or synchronization happen and is that visible at any time to the user? I guess this goes back to Howard’s original question on when an MPI implementations is required to add PMIx fences (not necessarily for a particular implementation for a portable usage of the API across any implementation). Thanks, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Thursday, 5. January 2023 at 07:09 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Ralph Castain <rhc(a)pmix.org> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Just FWIW: PMIx provides an event you can register to receive whenever a new process set has been defined. We could, I imagine, add one for process set deletion if that is useful. And you can always query for the complete list of current process set names (as well as memberships, if you want it). Ralph On Jan 5, 2023, at 4:50 AM, Holmes, Daniel John via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: Hi Martin, This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 04 January 2023 22:43 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Wednesday, 4. January 2023 at 10:17 To: Martin Schulz <schulzm(a)in.tum.de>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 04 January 2023 19:43 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org>; Holmes, Daniel John <daniel.john.holmes(a)intel.com> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi all, I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org> Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Wednesday, 4. January 2023 at 09:30 To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll HI Dan, Yes that was my interpretation as well. We can discuss at our next meeting 1/9/23 if there’s time. Howard From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Wednesday, January 4, 2023 at 12:05 PM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Howard, It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: MPI_Session_init // local – PMIx fence prohibited MPI_Group_from_pset // local – PMIx fence prohibited MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). Best wishes, Dan. From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions Sent: 04 January 2023 18:32 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll Hi All, First, Happy New Year! I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166 . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. Thanks for any ideas, Howard — <image001.png> Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov <image002.png><image003.png><image004.png><image005.png> _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Martin Schulz 05 Jan '23

05 Jan '23

Hi Dan, A few comments (kind of many intertwined good discussions 😊 ): Have we ever been able to define the difference between immediate and local? I know there is a perception difference, but what is difference to the user? What is too long for an immediate call? Are we disallowing progress? If so, what happens if the OS switches threads (this “blocks” the current one) and then the other thread makes progress? I don’t we can define this – I think for the user these two things will always be the same (from how they have to program) and an implementation will always have the freedom to do what it wants – there could at most be a performance expectation or advice to implementors. Why should an MPI implementation be prevented from calling the progress engine in a Comm_rank call? Why would it not be able to communicate, if it wants to? E.g., is a PMPI wrapper implementation that sends an acknowledgement that this call has been made to a central location and waits for it being received an incorrect wrapper? As for the Sessions API – I agree with most of what you say on local vs. non-local, but this is the user’s perspective. I was more asking about whether there is ever a need for an implementation to wait for something to complete. If we can really exclude that, then we should be good – however, I doubt we can do that for sure going forward. As for the waiting for changes – shouldn’t this be part of the query of the process list? Why a separate call? Why do you want to see session_init as an immediate call? Shouldn’t this be able to setup things, which could include allocations possible triggering longer resource negotiations in the OS? That call should not be in the critical path? Cheers, Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Thursday, 5. January 2023 at 02:50 To: Martin Schulz <schulzm(a)in.tum.de>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 04 January 2023 22:43 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Wednesday, 4. January 2023 at 10:17 To: Martin Schulz <schulzm(a)in.tum.de>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 04 January 2023 19:43 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org>; Holmes, Daniel John <daniel.john.holmes(a)intel.com> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi all, I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org> Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Wednesday, 4. January 2023 at 09:30 To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll HI Dan, Yes that was my interpretation as well. We can discuss at our next meeting 1/9/23 if there’s time. Howard From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com> Date: Wednesday, January 4, 2023 at 12:05 PM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov> Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Howard, It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: MPI_Session_init // local – PMIx fence prohibited MPI_Group_from_pset // local – PMIx fence prohibited MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). Best wishes, Dan. From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions Sent: 04 January 2023 18:32 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll Hi All, First, Happy New Year! I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166 . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. Thanks for any ideas, Howard — Howard PritchardResearch ScientistHPC-ENV Los Alamos National Laboratoryhowardp(a)lanl.gov

1 0

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Ralph Castain 05 Jan '23

05 Jan '23

Just FWIW: PMIx provides an event you can register to receive whenever a new process set has been defined. We could, I imagine, add one for process set deletion if that is useful. And you can always query for the complete list of current process set names (as well as memberships, if you want it). Ralph > On Jan 5, 2023, at 4:50 AM, Holmes, Daniel John via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > Hi Martin, > > This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. > > immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call > local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call > nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call > > > Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. > > I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. > > The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. > > Best wishes, > Dan. > > From: Martin Schulz <schulzm(a)in.tum.de> > Sent: 04 January 2023 22:43 > To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> > Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Dan, > > I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). > > However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. > > An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. > > This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. > > Martin > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de <mailto:[email protected]> > > > From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>> > Date: Wednesday, 4. January 2023 at 10:17 > To: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Martin, > > MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. > > For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. > > Best wishes, > Dan. > > From: Martin Schulz <schulzm(a)in.tum.de <mailto:[email protected]>> > Sent: 04 January 2023 19:43 > To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>>; Holmes, Daniel John <daniel.john.holmes(a)intel.com <mailto:[email protected]>> > Cc: Pritchard Jr., Howard <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi all, > > I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. > > The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? > > Martin > > > -- > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) > Email: schulzm(a)in.tum.de <mailto:[email protected]> > > > From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Date: Wednesday, 4. January 2023 at 09:30 > To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll > > HI Dan, > > Yes that was my interpretation as well. > > We can discuss at our next meeting 1/9/23 if there’s time. > > Howard > > > From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com <mailto:[email protected]>> > Date: Wednesday, January 4, 2023 at 12:05 PM > To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll > > Hi Howard, > > It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. > > Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: > > MPI_Session_init // local – PMIx fence prohibited > MPI_Group_from_pset // local – PMIx fence prohibited > MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed > > The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). > > Best wishes, > Dan. > > From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions > Sent: 04 January 2023 18:32 > To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> > Cc: Pritchard Jr., Howard <howardp(a)lanl.gov <mailto:[email protected]>> > Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll > > Hi All, > > First, Happy New Year! > > I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. > We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. > > I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166 <https://urldefense.com/v3/__https:/github.com/open-mpi/ompi/issues/11166__;…> . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. > > I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. > > Thanks for any ideas, > > Howard > > > — > > <image001.png> > Howard Pritchard > Research Scientist > HPC-ENV > > Los Alamos National Laboratory > howardp(a)lanl.gov <mailto:[email protected]> > > <image002.png> <https://urldefense.com/v3/__https:/www.instagram.com/losalamosnatlab/__;!!B…><image003.png> <https://urldefense.com/v3/__https:/twitter.com/LosAlamosNatLab__;!!Bt8fGhp8…><image004.png> <https://urldefense.com/v3/__https:/www.linkedin.com/company/los-alamos-nati…><image005.png> <https://urldefense.com/v3/__https:/www.facebook.com/LosAlamosNationalLab/__…> > > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll
by Holmes, Daniel John 05 Jan '23

05 Jan '23

Hi Martin, This seems like the best reason so far for differentiating “immediate” as the term that refers to the stronger semantic. immediate = prohibited to delay its return – neither until (remote) progress nor until (remote) specific semantically-related MPI procedure call local = permitted to delay its return until (remote) progress but not until (remote) specific semantically-related MPI procedure call nonlocal = permitted to delay its return (remote) progress and/or until (remote) specific semantically-related MPI procedure call Permitting progress in an API that was intended to be “lightweight” is bad. Progress could mean that MPI_Session_init delays its return while it pushes a few GB of buffered-mode send message data into a network – correctness is not affected, but performance expectations are. The canonical example is whether MPI_Comm_rank can delay its return while it does (remote) progress – currently “yes because local, but why would anyone implement it that way? It should be immediate.” The recent debate about whether MPI_Put is nonlocal is relevant here – correctness allows us to say “it doesn’t matter if MPI_Put is nonlocal because the user cannot create a deadlock” but the performance expectations of separating synchronisation from data movement instructions suggest that MPI_Put should be immediate, not even local, but definitely not nonlocal. I would be much happier with some kind of explicit “update the list of process set names” API that has whatever semantic is needed, rather than allowing (remote) progress in any of the existing immediate APIs: “initialise a session”, “get the number of process set names”, “get the nth process set name”, and “make a group from this process set name”. The trouble is: what is the appropriate semantic for the “update the list of process set names” API? We already have MPI_Session_init that will give an updated list of process set names if it is already different to previous lists given by previous sessions. We only need a “delay until something changes” semantic. What needs to change to satisfy this semantic? Is the first difference sufficient? Can/should the user specify what they are looking for? Is any difference actually necessary (should it have a timeout, in case nothing changes)? Is it collective (over which group)? Those questions are currently answered by the mechanism(s) that could possibly cause change to the list of process sets – spawn is an MPI operation, connect/accept are MPI operations, and so on. Initialising another session after one of these operations has completed will already get the up-to-date list of process sets (whether it changed because of the dynamic process model operation or not). Any future proposal for an API that extends/modifies/prunes the list of process set names needs to have a semantic that permits the appropriate consensus in its implementation. The consensus is needed by operations like “give me more resources”, “take these resources back”, etc – not by query functions. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de> Sent: 04 January 2023 22:43 To: Holmes, Daniel John <daniel.john.holmes(a)intel.com>; MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Dan, I fully agree with you on the difference between MPI and PVM and the need to keep things constant from the user’s perspective. They should not be impacted by this and hence also the agreement that none of this can be non-local (as this will impact the user). However, at some point the underlying runtime will have to come to a consensus and that may come after a user would expect it relative to a local procedure call. This is not likely or probably not even possible with the current static scheme and implementation, but it will likely happen when we add more dynamic behavior. An example could be newly spawned processes and when they are ready and have established all their individual process set memberships. This could be, of course, pushed off to the user who would need to ensure the right timing, but it may also be better to give the runtime some leeway – again, not in the sense of non-local, but in the sense of weak progress. This was actually my understanding of weak progress anyway – that any MPI routine could delay return for weak progress, but he hardened that only recently to operations only. For the current set of MPI routines that actually does not make a difference in execution, but here we could – by accident – add a limiter for future implementations and this is what I would like to avoid by opening up the chance for these routines to participate in progress. Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Wednesday, 4. January 2023 at 10:17 To: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: RE: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Martin, MPI is not PVM. We do not wait to see which/how many processes start and join the group/process set before deciding on the membership of the group/process set. The names and the membership of all (built-in/predefined) process sets are known a priori without coordination during the initialisation procedure call(s). Deviation from that membership (e.g. a process fails to start or fails to join up with the other processes) is a fault, which will cause a failure (e.g. a collective operation cannot complete), which will manifest as an error. The process set still exists and a group can still be formed from it; the communicator creation procedure that uses that group will raise an error. For scenarios/implementations where additional process sets “appear” during the execution, those new process sets might not appear until all involved processes will see the same new set name (depending on what the implementation can support); that might mean every involved process will have to have done some progress after the process set was created internally before any process will expose it to the user via MPI calls. That delay must never happen for the built-in/predefined process sets, so we have no conflict or difficulty. Best wishes, Dan. From: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>> Sent: 04 January 2023 19:43 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>>; Holmes, Daniel John <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi all, I agree with this interpretation – I always thought that was the original intent; non-local work should be able to be push off to the first communicator creation. The question about it being an operation and/or a local call is interesting, though – I tend to also see it the same as Dan, but is there a scenario in implementations that may require some kind of progress in other MPI processes (e.g., to internally synchronize on process sets)? If so, would we have to classify at least some calls (perhaps only the query of the process sets) as (local) operations so we can mandate progress? Or maybe “have to” is to harsh, but it would implementations to be more efficient? Martin -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de<mailto:[email protected]> From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of "Pritchard Jr., Howard via mpiwg-sessions" <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Reply to: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Date: Wednesday, 4. January 2023 at 09:30 To: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: Re: [mpiwg-sessions] [EXTERNAL] RE: MPI_Session_init semantics question/poll HI Dan, Yes that was my interpretation as well. We can discuss at our next meeting 1/9/23 if there’s time. Howard From: "Holmes, Daniel John" <daniel.john.holmes(a)intel.com<mailto:[email protected]>> Date: Wednesday, January 4, 2023 at 12:05 PM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: "Pritchard Jr., Howard" <howardp(a)lanl.gov<mailto:[email protected]>> Subject: [EXTERNAL] RE: MPI_Session_init semantics question/poll Hi Howard, It was always intended that MPI_Session_init was a local procedure. In fact, “initialise a session” is not even an MPI operation, so it doesn’t make sense for it to be expressed via a nonlocal procedure. Further, it was intended that the nonlocal portion of the work done by MPI_Init that is eventually needed in the pure sessions pattern would be done during the first nonlocal procedure call in that pattern, as follows: MPI_Session_init // local – PMIx fence prohibited MPI_Group_from_pset // local – PMIx fence prohibited MPI_Comm_create_from_group // nonlocal – PMIx fence permitted, if needed The nonlocal work should be unnecessary until the first nonlocal procedure call, so this should all work out fine (modulo some refactoring/debugging). Best wishes, Dan. From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Pritchard Jr., Howard via mpiwg-sessions Sent: 04 January 2023 18:32 To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov<mailto:[email protected]>> Subject: [mpiwg-sessions] MPI_Session_init semantics question/poll Hi All, First, Happy New Year! I’ve got a question about the semantics of MPI_Session_init. In particular, I’d be interested in knowing people’s opinion on whether this function is nonlocal or local. We don’t have any text in the current version of the standard that states whether or not MPI_Session_init is a nonlocal operation. I’m considering options for handling this issue: https://github.com/open-mpi/ompi/issues/11166<https://urldefense.com/v3/__https:/github.com/open-mpi/ompi/issues/11166__;…> . It turns out that the way to properly resolve this issue depends on whether or not MPI_Session_init has local or nonlocal semantics. I had been working under the assumption that we had intended session initialization to be a local function, but considering how to resolve issue 11166 made me begin to question this assumption. Thanks for any ideas, Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_1293224934]<https://urldefense.com/v3/__https:/www.instagram.com/losalamosnatlab/__;!!B…>[signature_2498822630]<https://urldefense.com/v3/__https:/twitter.com/LosAlamosNatLab__;!!Bt8fGhp8…>[signature_1283032776]<https://urldefense.com/v3/__https:/www.linkedin.com/company/los-alamos-nati…>[signature_3959178607]<https://urldefense.com/v3/__https:/www.facebook.com/LosAlamosNationalLab/__…>

1 0