- mpiwg-sessions - mailman3.cels.anl.gov

Slides from previous meeting
by Martin SCHREIBER 11 Jan '22

11 Jan '22

Bonjour, Dominik asked me to forward his slides to this mailing list. Here they are: www.martin-schreiber.info/pub/tmp/2022_01_11_dominik_huber_dynamic_sessions… He also extended them with more information. All the best, Martin -- Prof. Dr. Martin Schreiber Applied Mathematics / Scientific and High-Performance Computing Université Grenoble Alpes / Laboratoire Jean Kuntzmann, France Time-X Euro-HPC project: Informatics / Computer Architecture and Parallel Systems Technical University of Munich, Germany

1 0

Sessions WG - meet today
by Pritchard Jr., Howard 10 Jan '22

10 Jan '22

Hello All, We will meet today. Dominik Huber will give a presentation summarizing elements of his thesis – Towards Dynamic Resource Management with MPI Sessions and PMIx<https://www.martin-schreiber.info/data/student_projects/2021_MA_dominik_hub…>. Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov [signature_1079070393]<https://www.instagram.com/losalamosnatlab/>[signature_574451709]<https://twitter.com/LosAlamosNatLab>[signature_720027535]<https://www.linkedin.com/company/los-alamos-national-laboratory/>[signature_1078058180]<https://www.facebook.com/LosAlamosNationalLab/>

1 0

Re: [mpiwg-sessions] [EXTERNAL] We missed a bit -- §19.3.3
by Pritchard Jr., Howard 06 Jan '22

06 Jan '22

HI Dan, Thanks for the catch. Are you going to open an issue? Otherwise I will. Howard From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Thursday, January 6, 2022 at 7:51 AM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Dan Holmes <danholmes(a)chi.scot> Subject: [EXTERNAL] [mpiwg-sessions] We missed a bit -- §19.3.3 Hi all, I found section 19.3.3 by accident while looking for something else. It talks about initialising MPI for multiple language support but only mentions the world model procedures (without even using the phrase “world model”). We should have inserted the sessions model way of initialising MPI, IMHO. This should be tidied up for MPI-4.1, if others agree. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot<mailto:[email protected]>

1 0

We missed a bit -- §19.3.3
by Dan Holmes 06 Jan '22

06 Jan '22

Hi all, I found section 19.3.3 by accident while looking for something else. It talks about initialising MPI for multiple language support but only mentions the world model procedures (without even using the phrase “world model”). We should have inserted the sessions model way of initialising MPI, IMHO. This should be tidied up for MPI-4.1, if others agree. Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot

1 0

Re: [mpiwg-sessions] Meet today
by Ralph Castain 03 Jan '22

03 Jan '22

Hello folks I had a chance over the holidays to catch up on your docs regarding dynamic sessions - quite interesting. I believe there is support in PMIx for pretty much everything I saw being discussed. We have APIs by which an application can directly request allocation changes from the RM, and events by which the RM can notify (and subsequently negotiate) an application regarding changes to its allocation. So each side has the ability to initiate the process, and then both sides negotiate to a common conclusion. We also have an API by which an application can "register" its willingness to accept RM-initiated "preemption" requests so the RM can incorporate that willingness in its pricing and planning procedures. Unfortunately, while we have that infrastructure defined in the PMIx Standard and implemented in OpenPMIx, we have not yet seen the required backend support implemented in an RM. I have started working with some folks on integrating support into Slurm, but I do not know the timetable for public release of that work. SchedMD has been ambivalent towards accepting pull requests that extend its PMIx support, so this may well have to be released as a side-project. I have previously approached Altair about adding support to PBS - nothing has happened yet. I suspect they are waiting for customer demand. I have no knowledge of any other RMs looking into it. As a gap-filling measure, I am adding simulated support in PRRTE so that anyone wanting to develop dynamic resource code can at least have a place where they can develop it and do a little testing. PRRTE doesn't include a scheduler, but I can simulate it by retaining some of the RM-allocated resources as part of a PRRTE-managed "pool". Meantime, I have started a little personal project to add PMIx support to Kubernetes, hopefully giving it more capability to support HPC applications. The Kubeflow community has a degree of PMIx support, but I want to directly integrate it to Kubernetes itself, including the dynamic resource elements described above. I have no timetable for completing that work - as many of you may know, I am retired and so this is something to do in my spare time. If anyone is interested on tracking progress on this, please let me know. Thus, I would encourage you to start prodding your favorite RM vendors as this may prove the critical timeline in making dynamic sessions a reality! Also, if you identify any "gaps" in the PMIx support, please do let me know - I'd be happy to work with you to fill them. The current definitions were developed primarily to support workflow operations and the needs of the dynamic programming model communities (e.g., TensorFlow and Data Analytics). I think those are very similar to what you are identifying, but may perhaps need some tweaking. Ralph On Jan 3, 2022, at 9:28 AM, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > wrote: Hello All, Happy New Year! Let’s try to meet today. Items on the agenda: * PR #629 (issue #511) - https://github.com/mpi-forum/mpi-issues/issues/511 <https://github.com/mpi-forum/mpi-issues/issues/511> * Pick up where we were on discussion of dynamic sessions requirements, see: * https://miro.com/app/board/o9J_l_Rxe9Q=/ <https://miro.com/app/board/o9J_l_Rxe9Q=/> * https://docs.google.com/document/d/1l7LQ8eeVOUW69TDVG9LjKJUuerfE3S3teaMFG5D… <https://docs.google.com/document/d/1l7LQ8eeVOUW69TDVG9LjKJUuerfE3S3teaMFG5D…> * * If my calendar calculation is right, we will be meeting with the FT WG today Thanks, Howard — <image001.png> Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov <mailto:[email protected]> <image002.png> <https://www.instagram.com/losalamosnatlab/> <image003.png> <https://twitter.com/LosAlamosNatLab> <image004.png> <https://www.linkedin.com/company/los-alamos-national-laboratory/> <image005.png> <https://www.facebook.com/LosAlamosNationalLab/> _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions <https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions>

1 0

Meet today
by Pritchard Jr., Howard 03 Jan '22

03 Jan '22

Hello All, Happy New Year! Let’s try to meet today. Items on the agenda: * PR #629 (issue #511) - https://github.com/mpi-forum/mpi-issues/issues/511 * Pick up where we were on discussion of dynamic sessions requirements, see: * https://miro.com/app/board/o9J_l_Rxe9Q=/ * https://docs.google.com/document/d/1l7LQ8eeVOUW69TDVG9LjKJUuerfE3S3teaMFG5D… * * If my calendar calculation is right, we will be meeting with the FT WG today Thanks, Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov [signature_690465573]<https://www.instagram.com/losalamosnatlab/>[signature_457751801]<https://twitter.com/LosAlamosNatLab>[signature_205049194]<https://www.linkedin.com/company/los-alamos-national-laboratory/>[signature_39351274]<https://www.facebook.com/LosAlamosNationalLab/>

1 0

Sessions WG - no more meetings this year
by Pritchard Jr., Howard 13 Dec '21

13 Dec '21

Hi All, We’ll skip holding Sessions WG meetings for the remainder of this month. We will resume on 1/3/2022. Have an enjoyable holiday season. Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov [signature_87017459]<https://www.instagram.com/losalamosnatlab/>[signature_371222934]<https://twitter.com/LosAlamosNatLab>[signature_763336158]<https://www.linkedin.com/company/los-alamos-national-laboratory/>[signature_1978777853]<https://www.facebook.com/LosAlamosNationalLab/>

1 0

Re: [mpiwg-sessions] sessions WG meeting today?
by Dan Holmes 13 Dec '21

13 Dec '21

HI Howard, I was kinda assuming it would be cancelled. I’m not averse to relaying the discussion/outcome of the reading for PR #629 (issue #511). Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot > On 13 Dec 2021, at 16:22, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > HI All, > > Does anyone have agenda items for the Sessions WG meeting today? > > Howard > > — > > <image001.png> > Howard Pritchard > Research Scientist > HPC-ENV > > Los Alamos National Laboratory > howardp(a)lanl.gov <mailto:[email protected]> > > <image002.png> <https://www.instagram.com/losalamosnatlab/><image003.png> <https://twitter.com/LosAlamosNatLab><image004.png> <https://www.linkedin.com/company/los-alamos-national-laboratory/><image005.png> <https://www.facebook.com/LosAlamosNationalLab/> > > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions <https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions>

1 0

sessions WG meeting today?
by Pritchard Jr., Howard 13 Dec '21

13 Dec '21

HI All, Does anyone have agenda items for the Sessions WG meeting today? Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov [signature_476022104]<https://www.instagram.com/losalamosnatlab/>[signature_1182217576]<https://twitter.com/LosAlamosNatLab>[signature_1172155898]<https://www.linkedin.com/company/los-alamos-national-laboratory/>[signature_2057245918]<https://www.facebook.com/LosAlamosNationalLab/>

1 0

Re: [mpiwg-sessions] WPM Session Handle
by Dan Holmes 01 Dec '21

01 Dec '21

Hi Joseph, Thanks for bringing this back to mind. Apologies for drop ping the ball :) To clarify and to refresh my memory of previous discussions: 1) The new request type, called a continuation, is only “floating” until it is bound to an MPI operation, right? AFAIK, every MPI operation (*not* every MPI procedure) is “in” a single/particular session, because all MPI operations require one of the four central objects that provide nonlocal functionality. Thus, for a nontrivial continuation, i.e. for a continuation that actually continues an MPI operation, the association with a particular session is clearly defined already. Does the brief period of time during which it is, technically, “floating” matter? If so, how/why? (I can’t remember the discussion we had on this point.) 2) Can a single continuation request be bound to more than one MPI operation at the same time? I can’t remember which way the continuations proposal lands on this design choice. Whichever choice is made, all MPI operations bound to a single continuation would have to be “in” the same session as each other, otherwise the isolation rule for sessions is breached. Specifying that session up front at creation-time for the continuation permits checking of the first MPI operation bound to it as well as the rest, whereas not specifying it explicitly means that the first MPI operation bound is always okay, but subsequent ones might be incompatible because of a session mismatch. Thus, isolation of sessions can be specified and enforced with implicit or with explicit selection of the session. 3) The idea of having a WPM session handle is almost entirely independent of the continuations proposal, although the latter might be able to make use of the former if it existed, right? I would adjust the WPM session handle proposal to avoid a global symbol by providing instead a procedure that gave (a copy of) the WPM session handle. Global symbols created problems, c.f. the symbol MPI_COMM_WORLD w.r.t. FT. The Sessions WG did not propose any mechanism to get a session handle for the world model because it would have added complexity to the MPI-4.0 discussions. We did, repeatedly, intimate that the world model is effectively like a session to which the user never gets a session handle. We even went as far as showing pseudo-code that implements MPI_INIT using MPI_SESSION_INIT so illustrate this point. The motivation there was to make the proposal less scary, rather than to offer this as new functionality for users of MPI. However, I don’t see any technical reasons why any MPI library could not provide a world model session handle. The only mandated pset names are “mpi://world <mpi://world>” and “mpi://self <mpi://self>” which have obvious meanings in the world model, so the minimal compliant outcomes of MPI_SESSION_GET_NUM_PSETS and MPI_SESSION_GET_NTH_PSET seem straightforward. We could make the actions of MPI_SESSION_[G|S]ET_ERRHANDLER (when called for the WPM session handle) be identical to MPI_COMM_[G|S]ET_ERRHANDLER (when called for the MPI_COMM_SELF communicator handle). Setting with one affects the result of getting with the other. One point of possible contention would be: is calling MPI_SESSION_FINALIZE using the WPM session handle permitted and, if so, is it identical to (and a replacement for) calling MPI_FINALIZE? Specifically, if the WPM session is finalised, does the return value of MPI_FINALIZED change and is a subsequent call to MPI_FINALIZE erroneous? Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot > On 1 Dec 2021, at 15:26, Joseph Schuchart via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > Ping. Any opinions/inputs on this? > > Thanks > Joseph > > > -------- Forwarded Message -------- > Subject: WPM Session Handle > Date: Sun, 31 Oct 2021 11:08:06 -0400 > From: Joseph Schuchart <schuchart(a)icl.utk.edu> > To: mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> > > > > Hi all, > > The Hybrid-WG is currently discussing the Continuations proposal. Part of that proposal is a new continuation request type that is not bound to any communication operation, and thus not connected to any communicator, window, or file. That floating state is somewhat disatisfying. We are thus considering connecting it to a session, i.e., provide a session handle at creation. This would potentially allow us to isolate the execution of continuations from different sessions. Unfortunately, this is not possible at the moment because there is no default session handle for the WPM model. > > From my (arguably uninformed outside) point of view, the WPM is a special type of session worthy of its own handle (MPI_SESSION_WPM, for example), which can be used with all functions expecting session handles as input. As an example, `MPI_Session_get_info(MPI_SESSION_WPM)` would return an info object containing the "thread_level" info key and the value passed to MPI_Init_thread (or "single" for MPI_Init). I'm sure there are other more hairy procedures, but you get the point. > > Has this been discussed before? Was there a good reason not to have a WPM session handle? If not, can we add something like MPI_SESSION_WPM to allow future API extensions to use session handles? > > Thanks > Joseph > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0