- mpiwg-sessions - mailman3.cels.anl.gov

Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion
by Dominik Huber 20 Aug '25

20 Aug '25

Hi Ralph and Michael, Thank you for the feedback. I added a draft of PMIx attributes to support the MPI API on slides 20 + 21. The current prototype uses PMIx_Allocation_request and PMIx_Notify_event for this purpose. Best regards, Dominik ________________________________ Von: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> im Auftrag von Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Gesendet: Dienstag, 19. August 2025 15:47:23 An: Michael Raymond Cc: Ralph Castain; MPI Sessions working group Betreff: Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix. sec. ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix.sec.ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use // in environments where multiple authentication mechanisms // may be available. When returned in a callback function, a // string identifier of the credential type #define PMIX_CRYPTO_KEY “pmix.sec.key” // (pmix_byte_object_t) blob containing crypto key You can include those in the “spawn” request as part of the job-level information to be passed to all the new processes. Providing the child procs with access to the parent’s credentials can be done by including those credentials in the job-level info to spawn, or using PMIx_Connect or PMIx_Group_construct after spawn as they will distribute the parent’s job-level info. Note that OpenMPI has chosen the latter approach as it proved beneficial for the child procs to have a full picture of their parent job. The info is retrieved by individual procs with a simple PMIx_Get, passing the ID of the proc whose credential you are seeking (rank=wildcard since we assume it’s the same for all procs in the namespace) and the PMIX_CRYPTO_KEY key (if you need the type, you can ask for that too with PMIX_CRED_TYPE). I suppose it is possible (perhaps in the future) for a process/job to be assigned multiple security keys - e.g., if the system supported multiple parallel fabrics, each with its own security key. If that happens, we’ll have two choices: (a) define a new attribute for passing an array of structs containing keys and associated fabric ID so you know which key goes with which fabric; and/or (b) packing that info into the PMIX_CRYPTO_KEY byte object in a specified manner so it can be unpacked at the proc. We also have the PMIx_Get_credential API you can use to request and obtain a new credential. Of course, that requires that the host support it on the backend! As you probably recall, we had talked during the Aurora days about having PMIx assist with obtaining the credential from the system (or perhaps generating it algorithmically like Slurm does for VNI), but never quite got to that point. At this time, we would rely on the PMIx server’s upcall to the host to obtain the credential. In terms of injecting the key into the appropriate privileged service, there are a couple of options. One could use PMIx to connect to the privileged service and pass the key that way. I suspect system security folks would be leery of that no matter how carefully we tried to construct the connection validation handshake. Another option might be to create a setuid script that operates at the service-privileged level and communicates the key to the service. Might be more palatable to the security folks as the script could be tightly constrained as to its abilities and kept very simple. Or maybe have the vendors provide an appropriate library that PMIx can call to set the credential (thereby allowing the vendor to control the communication handshake). Or we could assign the responsibility for setting the network key to the host. If we pass the key in the spawn command, then the host could search for it and inject it into the services where required. Vendor environments would probably be able to do that as they usually have an accepted method for talking to a privileged service. Probably won’t work for mpirun, though, as that normally doesn’t run at privilege and therefore couldn’t do the injection. So if we want mpirun to support network credentials for spawn, then we probably need to either find an acceptable way to connect to the service, or get acceptance for a setuid script or library to span the gap. Happy to discuss this more over the phone or at a meeting, if people would like. Ralph On Aug 19, 2025, at 5:29 AM, Raymond, Michael <mraymond(a)hpe.com> wrote: On slide 11, “divers” works, but you may mean “diverse”. With the PMIx APIs, is there a way to get back and/or specify which network security key should be used with the new Pset? With HPE’s Slingshot and the upcoming UEC, different applications are run with different network security keys (VNIs and JobIDs respectively). Two separately-launched applications will likely have different keys and be unable to intercommunicate. MPI is going to need PMI to tell it which key to use to talk to the processes that were launched separately. MPI will also need a privileged service to add the key to its network context. With Slingshot, applications launched under the same job share a JobVNI in addition to their individual job-step VNIs. Applications from separate jobs and users can use DRC2 to agree upon a VNI. We’ll need a generic attribute to pass this information in PMIx. Michael Raymond HPE HPC Development Environment 1 (612) 851-0500 On Aug 18, 2025, at 20:37, Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd #pfptBannervq9swly { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important; max-width: none !important; max-height: none !important } .pfptPrimaryButtonvq9swly:hover, .pfptPrimaryButtonvq9swly:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonvq9swly:active { background-color: #90a4ae !important; } Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a means of making it easier to extend in the future. I suspect the community at-large will discover additional use-cases that require new inputs, and it would be unfortunate if we had to create new APIs every time that happened. BTW: I have the resource usage code working in both PMIx and PRRTE now - getting ready to commit those changes. Just need to clean up a couple of things first. Ralph On Aug 18, 2025, at 12:12 PM, Dominik Huber via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: This Message Is From an External Sender This message came from outside your organization. Hi all, As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi… <https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi…> I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. Best regards, Dominik _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi… <https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi…> _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]> https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp…

1 0

Re: [mpiwg-sessions] Cancel 25th August sessions WG meeting?
by Ralph Castain 19 Aug '25

19 Aug '25

Same here > On Aug 19, 2025, at 9:29 AM, Raymond, Michael via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > I’m fine either way. > > Michael Raymond > HPE HPC Development Environment > 1 (612) 851-0500 > > > On Aug 19, 2025, at 10:27, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Hi All, > > I think when we were discussing the WG’s August schedule we forgot about Europar. > > So kind of a poll – who would be interested in meeting this coming Monday? > > Howard > > ------- > > <image001.png>Howard Pritchard > > Research Scientist > > HPC-ENV > > Los Alamos National Laboratory > > howardp(a)lanl.gov <mailto:[email protected]> > > <image002.png><image003.png><image004.png><image005.png> _______________________________________________ > > mpiwg-sessions mailing list > > mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > > https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp… > > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi…

1 0

Re: [mpiwg-sessions] Cancel 25th August sessions WG meeting?
by Raymond, Michael 19 Aug '25

19 Aug '25

1 0

Cancel 25th August sessions WG meeting?
by Pritchard Jr., Howard 19 Aug '25

19 Aug '25

Hi All, I think when we were discussing the WG’s August schedule we forgot about Europar. So kind of a poll – who would be interested in meeting this coming Monday? Howard ------- [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_3189307141]<https://urldefense.us/v3/__https://www.instagram.com/losalamosnatlab/__;!!G… >[signature_3469584182]<https://urldefense.us/v3/__https://twitter.com/LosAlamosNatLab__;!!G_uCfscf… >[signature_4021162776]<https://urldefense.us/v3/__https://www.linkedin.com/company/los-alamos-nati… >[signature_393059152]<https://urldefense.us/v3/__https://www.facebook.com/LosAlamosNationalLab/__… >

1 0

Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion
by Ralph Castain 19 Aug '25

19 Aug '25

Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix.sec.ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use // in environments where multiple authentication mechanisms // may be available. When returned in a callback function, a // string identifier of the credential type #define PMIX_CRYPTO_KEY “pmix.sec.key” // (pmix_byte_object_t) blob containing crypto key You can include those in the “spawn” request as part of the job-level information to be passed to all the new processes. Providing the child procs with access to the parent’s credentials can be done by including those credentials in the job-level info to spawn, or using PMIx_Connect or PMIx_Group_construct after spawn as they will distribute the parent’s job-level info. Note that OpenMPI has chosen the latter approach as it proved beneficial for the child procs to have a full picture of their parent job. The info is retrieved by individual procs with a simple PMIx_Get, passing the ID of the proc whose credential you are seeking (rank=wildcard since we assume it’s the same for all procs in the namespace) and the PMIX_CRYPTO_KEY key (if you need the type, you can ask for that too with PMIX_CRED_TYPE). I suppose it is possible (perhaps in the future) for a process/job to be assigned multiple security keys - e.g., if the system supported multiple parallel fabrics, each with its own security key. If that happens, we’ll have two choices: (a) define a new attribute for passing an array of structs containing keys and associated fabric ID so you know which key goes with which fabric; and/or (b) packing that info into the PMIX_CRYPTO_KEY byte object in a specified manner so it can be unpacked at the proc. We also have the PMIx_Get_credential API you can use to request and obtain a new credential. Of course, that requires that the host support it on the backend! As you probably recall, we had talked during the Aurora days about having PMIx assist with obtaining the credential from the system (or perhaps generating it algorithmically like Slurm does for VNI), but never quite got to that point. At this time, we would rely on the PMIx server’s upcall to the host to obtain the credential. In terms of injecting the key into the appropriate privileged service, there are a couple of options. One could use PMIx to connect to the privileged service and pass the key that way. I suspect system security folks would be leery of that no matter how carefully we tried to construct the connection validation handshake. Another option might be to create a setuid script that operates at the service-privileged level and communicates the key to the service. Might be more palatable to the security folks as the script could be tightly constrained as to its abilities and kept very simple. Or maybe have the vendors provide an appropriate library that PMIx can call to set the credential (thereby allowing the vendor to control the communication handshake). Or we could assign the responsibility for setting the network key to the host. If we pass the key in the spawn command, then the host could search for it and inject it into the services where required. Vendor environments would probably be able to do that as they usually have an accepted method for talking to a privileged service. Probably won’t work for mpirun, though, as that normally doesn’t run at privilege and therefore couldn’t do the injection. So if we want mpirun to support network credentials for spawn, then we probably need to either find an acceptable way to connect to the service, or get acceptance for a setuid script or library to span the gap. Happy to discuss this more over the phone or at a meeting, if people would like. Ralph > On Aug 19, 2025, at 5:29 AM, Raymond, Michael <mraymond(a)hpe.com> wrote: > > On slide 11, “divers” works, but you may mean “diverse”. > > With the PMIx APIs, is there a way to get back and/or specify which network security key should be used with the new Pset? With HPE’s Slingshot and the upcoming UEC, different applications are run with different network security keys (VNIs and JobIDs respectively). Two separately-launched applications will likely have different keys and be unable to intercommunicate. MPI is going to need PMI to tell it which key to use to talk to the processes that were launched separately. MPI will also need a privileged service to add the key to its network context. > > With Slingshot, applications launched under the same job share a JobVNI in addition to their individual job-step VNIs. Applications from separate jobs and users can use DRC2 to agree upon a VNI. We’ll need a generic attribute to pass this information in PMIx. > > Michael Raymond > HPE HPC Development Environment > 1 (612) 851-0500 > >> On Aug 18, 2025, at 20:37, Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >> >> Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> ZjQcmQRYFpfptBannerEnd >> #pfptBannervq9swly { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important; max-width: none !important; max-height: none !important } .pfptPrimaryButtonvq9swly:hover, .pfptPrimaryButtonvq9swly:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonvq9swly:active { background-color: #90a4ae !important; } Hi Dominik >> >> Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! >> >> Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a means of making it easier to extend in the future. I suspect the community at-large will discover additional use-cases that require new inputs, and it would be unfortunate if we had to create new APIs every time that happened. >> >> BTW: I have the resource usage code working in both PMIx and PRRTE now - getting ready to commit those changes. Just need to clean up a couple of things first. >> >> Ralph >> >> >>> On Aug 18, 2025, at 12:12 PM, Dominik Huber via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: >>> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> Hi all, >>> As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. >>> Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. >>> To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: >>> https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi… >>> >>> I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. >>> >>> Best regards, >>> >>> Dominik >>> >>> >>> >>> _______________________________________________ >>> mpiwg-sessions mailing list >>> mpiwg-sessions(a)lists.mpi-forum.org >>> https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi… >> >> >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> >> https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp…

1 0

Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion
by Raymond, Michael 19 Aug '25

19 Aug '25

1 0

Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion
by Ralph Castain 18 Aug '25

18 Aug '25

Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a means of making it easier to extend in the future. I suspect the community at-large will discover additional use-cases that require new inputs, and it would be unfortunate if we had to create new APIs every time that happened. BTW: I have the resource usage code working in both PMIx and PRRTE now - getting ready to commit those changes. Just need to clean up a couple of things first. Ralph > On Aug 18, 2025, at 12:12 PM, Dominik Huber via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hi all, > > As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. > Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. > To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: > https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi… <https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi…> > > I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. > > Best regards, > > Dominik > > > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi…

1 0

Draft Slides for Dynamic Processes WG Discussion
by Dominik Huber 18 Aug '25

18 Aug '25

Hi all, As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi… I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. Best regards, Dominik

1 0

Reminder no Sessions WG meeting today
by Pritchard Jr., Howard 28 Jul '25

28 Jul '25

Hi Folks, We will not have a Sessions WG meeting today per agreement of regular attendees a couple of weeks ago. Also, we will not be meeting on August 11. We will resume on August 25. Thanks, Howard ------- [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_3732073976]<https://urldefense.us/v3/__https://www.instagram.com/losalamosnatlab/__;!!G… >[signature_2732746159]<https://urldefense.us/v3/__https://twitter.com/LosAlamosNatLab__;!!G_uCfscf… >[signature_3748044306]<https://urldefense.us/v3/__https://www.linkedin.com/company/los-alamos-nati… >[signature_1238564370]<https://urldefense.us/v3/__https://www.facebook.com/LosAlamosNationalLab/__… >

1 0

Upcoming Sessions WG meetings
by Pritchard Jr., Howard 16 Jul '25

16 Jul '25

Hi All, At the last meeting we discussed our schedule for the next couple of months. Many of the regular attendees will be on vacation in August so we are canceling the meetings scheduled for 7/28 and 8/11. We will hold an off cadence meeting on 7/21 to continue discussion of Dominik’s slide deck and any other items. We will use the same TUM zoom link – see https://urldefense.us/v3/__https://github.com/mpiwg-sessions/mpi-standard/w… Thanks, Howard ------- [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_2831802778]<https://urldefense.us/v3/__https://www.instagram.com/losalamosnatlab/__;!!G… >[signature_1390913170]<https://urldefense.us/v3/__https://twitter.com/LosAlamosNatLab__;!!G_uCfscf… >[signature_1098369377]<https://urldefense.us/v3/__https://www.linkedin.com/company/los-alamos-nati… >[signature_479981520]<https://urldefense.us/v3/__https://www.facebook.com/LosAlamosNationalLab/__… >

1 0