August 2025 - mpiwg-sessions - mailman3.cels.anl.gov

Re: [mpiwg-sessions] [EXTERNAL] Re: Draft Slides for Dynamic Processes WG Discussion
by Happ, Sonja 22 Aug '25

22 Aug '25

Hi Dominik, thanks for the slides. They look good. I've added 3 comments (on slides 5, 9, and 11) with some suggestions. Best regards Sonja Dr. Sonja Happ Software Engineer ∙ Software & Architecture Development ParTec AG ∙ Possartstr. 20 ∙ D-81679 München ∙ https://urldefense.us/v3/__http://www.par-tec.com__;!!G_uCfscf7eWS!aQvAO7vJ… Registration Court: Amtsgericht München, HRB 263870 Executive Board: Bernhard Frohwitter (Chairman) Johann Kilger, Thomas Moschny, Ina Schmitz Chairman of the Supervisory Board: Prof. Dr. Reimund Neugebauer Privacy Policy: https://urldefense.us/v3/__http://www.par-tec.com/privacy__;!!G_uCfscf7eWS!… ________________________________ Von: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> im Auftrag von Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Gesendet: Donnerstag, 21. August 2025 18:35 An: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org>; Michael Raymond <mraymond(a)hpe.com> Cc: Pritchard Jr., Howard <howardp(a)lanl.gov> Betreff: Re: [mpiwg-sessions] [EXTERNAL] Re: Draft Slides for Dynamic Processes WG Discussion HI Dominik, I added a comment on slide 9. Otherwise, the slides look pretty good! Howard From: mpiwg-sessions <mpiwg-sessions-bounces@ lists. mpi-forum. org> on behalf of MPI Sessions working group <mpiwg-sessions@ lists. mpi-forum. org> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd HI Dominik, I added a comment on slide 9. Otherwise, the slides look pretty good! Howard From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Wednesday, August 20, 2025 at 6:25 AM To: Michael Raymond <mraymond(a)hpe.com>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Dominik Huber <domi.huber(a)tum.de> Subject: [EXTERNAL] Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion Hi Ralph and Michael, Thank you for the feedback. I added a draft of PMIx attributes to support the MPI API on slides 20 + 21. The current prototype uses PMIx_Allocation_request and PMIx_Notify_event for this purpose. Best regards, Dominik ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Ralph and Michael, Thank you for the feedback. I added a draft of PMIx attributes to support the MPI API on slides 20 + 21. The current prototype uses PMIx_Allocation_request and PMIx_Notify_event for this purpose. Best regards, Dominik ________________________________ Von: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> im Auftrag von Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Gesendet: Dienstag, 19. August 2025 15:47:23 An: Michael Raymond Cc: Ralph Castain; MPI Sessions working group Betreff: Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix. sec. ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix.sec.ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use // in environments where multiple authentication mechanisms // may be available. When returned in a callback function, a // string identifier of the credential type #define PMIX_CRYPTO_KEY “pmix.sec.key” // (pmix_byte_object_t) blob containing crypto key You can include those in the “spawn” request as part of the job-level information to be passed to all the new processes. Providing the child procs with access to the parent’s credentials can be done by including those credentials in the job-level info to spawn, or using PMIx_Connect or PMIx_Group_construct after spawn as they will distribute the parent’s job-level info. Note that OpenMPI has chosen the latter approach as it proved beneficial for the child procs to have a full picture of their parent job. The info is retrieved by individual procs with a simple PMIx_Get, passing the ID of the proc whose credential you are seeking (rank=wildcard since we assume it’s the same for all procs in the namespace) and the PMIX_CRYPTO_KEY key (if you need the type, you can ask for that too with PMIX_CRED_TYPE). I suppose it is possible (perhaps in the future) for a process/job to be assigned multiple security keys - e.g., if the system supported multiple parallel fabrics, each with its own security key. If that happens, we’ll have two choices: (a) define a new attribute for passing an array of structs containing keys and associated fabric ID so you know which key goes with which fabric; and/or (b) packing that info into the PMIX_CRYPTO_KEY byte object in a specified manner so it can be unpacked at the proc. We also have the PMIx_Get_credential API you can use to request and obtain a new credential. Of course, that requires that the host support it on the backend! As you probably recall, we had talked during the Aurora days about having PMIx assist with obtaining the credential from the system (or perhaps generating it algorithmically like Slurm does for VNI), but never quite got to that point. At this time, we would rely on the PMIx server’s upcall to the host to obtain the credential. In terms of injecting the key into the appropriate privileged service, there are a couple of options. One could use PMIx to connect to the privileged service and pass the key that way. I suspect system security folks would be leery of that no matter how carefully we tried to construct the connection validation handshake. Another option might be to create a setuid script that operates at the service-privileged level and communicates the key to the service. Might be more palatable to the security folks as the script could be tightly constrained as to its abilities and kept very simple. Or maybe have the vendors provide an appropriate library that PMIx can call to set the credential (thereby allowing the vendor to control the communication handshake). Or we could assign the responsibility for setting the network key to the host. If we pass the key in the spawn command, then the host could search for it and inject it into the services where required. Vendor environments would probably be able to do that as they usually have an accepted method for talking to a privileged service. Probably won’t work for mpirun, though, as that normally doesn’t run at privilege and therefore couldn’t do the injection. So if we want mpirun to support network credentials for spawn, then we probably need to either find an acceptable way to connect to the service, or get acceptance for a setuid script or library to span the gap. Happy to discuss this more over the phone or at a meeting, if people would like. Ralph On Aug 19, 2025, at 5:29 AM, Raymond, Michael <mraymond(a)hpe.com> wrote: On slide 11, “divers” works, but you may mean “diverse”. With the PMIx APIs, is there a way to get back and/or specify which network security key should be used with the new Pset? With HPE’s Slingshot and the upcoming UEC, different applications are run with different network security keys (VNIs and JobIDs respectively). Two separately-launched applications will likely have different keys and be unable to intercommunicate. MPI is going to need PMI to tell it which key to use to talk to the processes that were launched separately. MPI will also need a privileged service to add the key to its network context. With Slingshot, applications launched under the same job share a JobVNI in addition to their individual job-step VNIs. Applications from separate jobs and users can use DRC2 to agree upon a VNI. We’ll need a generic attribute to pass this information in PMIx. Michael Raymond HPE HPC Development Environment 1 (612) 851-0500 On Aug 18, 2025, at 20:37, Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd #pfptBannervq9swly { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important; max-width: none !important; max-height: none !important } .pfptPrimaryButtonvq9swly:hover, .pfptPrimaryButtonvq9swly:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonvq9swly:active { background-color: #90a4ae !important; } Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a means of making it easier to extend in the future. I suspect the community at-large will discover additional use-cases that require new inputs, and it would be unfortunate if we had to create new APIs every time that happened. BTW: I have the resource usage code working in both PMIx and PRRTE now - getting ready to commit those changes. Just need to clean up a couple of things first. Ralph On Aug 18, 2025, at 12:12 PM, Dominik Huber via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: This Message Is From an External Sender This message came from outside your organization. Hi all, As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi… <https://urldefense.com/v3/__https:/urldefense.us/v3/__https:/*docs.google.c… > I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. Best regards, Dominik _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi… <https://urldefense.com/v3/__https:/urldefense.us/v3/__https:/*lists.mpi-for… > _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]> https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp… <https://urldefense.com/v3/__https:/lists.mpi-forum.org/mailman/listinfo/mpi… >

1 0

Re: [mpiwg-sessions] [EXTERNAL] Re: Draft Slides for Dynamic Processes WG Discussion
by Pritchard Jr., Howard 21 Aug '25

21 Aug '25

HI Dominik, I added a comment on slide 9. Otherwise, the slides look pretty good! Howard From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Wednesday, August 20, 2025 at 6:25 AM To: Michael Raymond <mraymond(a)hpe.com>, MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Dominik Huber <domi.huber(a)tum.de> Subject: [EXTERNAL] Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion Hi Ralph and Michael, Thank you for the feedback. I added a draft of PMIx attributes to support the MPI API on slides 20 + 21. The current prototype uses PMIx_Allocation_request and PMIx_Notify_event for this purpose. Best regards, Dominik ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Ralph and Michael, Thank you for the feedback. I added a draft of PMIx attributes to support the MPI API on slides 20 + 21. The current prototype uses PMIx_Allocation_request and PMIx_Notify_event for this purpose. Best regards, Dominik ________________________________ Von: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> im Auftrag von Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Gesendet: Dienstag, 19. August 2025 15:47:23 An: Michael Raymond Cc: Ralph Castain; MPI Sessions working group Betreff: Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix. sec. ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix.sec.ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use // in environments where multiple authentication mechanisms // may be available. When returned in a callback function, a // string identifier of the credential type #define PMIX_CRYPTO_KEY “pmix.sec.key” // (pmix_byte_object_t) blob containing crypto key You can include those in the “spawn” request as part of the job-level information to be passed to all the new processes. Providing the child procs with access to the parent’s credentials can be done by including those credentials in the job-level info to spawn, or using PMIx_Connect or PMIx_Group_construct after spawn as they will distribute the parent’s job-level info. Note that OpenMPI has chosen the latter approach as it proved beneficial for the child procs to have a full picture of their parent job. The info is retrieved by individual procs with a simple PMIx_Get, passing the ID of the proc whose credential you are seeking (rank=wildcard since we assume it’s the same for all procs in the namespace) and the PMIX_CRYPTO_KEY key (if you need the type, you can ask for that too with PMIX_CRED_TYPE). I suppose it is possible (perhaps in the future) for a process/job to be assigned multiple security keys - e.g., if the system supported multiple parallel fabrics, each with its own security key. If that happens, we’ll have two choices: (a) define a new attribute for passing an array of structs containing keys and associated fabric ID so you know which key goes with which fabric; and/or (b) packing that info into the PMIX_CRYPTO_KEY byte object in a specified manner so it can be unpacked at the proc. We also have the PMIx_Get_credential API you can use to request and obtain a new credential. Of course, that requires that the host support it on the backend! As you probably recall, we had talked during the Aurora days about having PMIx assist with obtaining the credential from the system (or perhaps generating it algorithmically like Slurm does for VNI), but never quite got to that point. At this time, we would rely on the PMIx server’s upcall to the host to obtain the credential. In terms of injecting the key into the appropriate privileged service, there are a couple of options. One could use PMIx to connect to the privileged service and pass the key that way. I suspect system security folks would be leery of that no matter how carefully we tried to construct the connection validation handshake. Another option might be to create a setuid script that operates at the service-privileged level and communicates the key to the service. Might be more palatable to the security folks as the script could be tightly constrained as to its abilities and kept very simple. Or maybe have the vendors provide an appropriate library that PMIx can call to set the credential (thereby allowing the vendor to control the communication handshake). Or we could assign the responsibility for setting the network key to the host. If we pass the key in the spawn command, then the host could search for it and inject it into the services where required. Vendor environments would probably be able to do that as they usually have an accepted method for talking to a privileged service. Probably won’t work for mpirun, though, as that normally doesn’t run at privilege and therefore couldn’t do the injection. So if we want mpirun to support network credentials for spawn, then we probably need to either find an acceptable way to connect to the service, or get acceptance for a setuid script or library to span the gap. Happy to discuss this more over the phone or at a meeting, if people would like. Ralph On Aug 19, 2025, at 5:29 AM, Raymond, Michael <mraymond(a)hpe.com> wrote: On slide 11, “divers” works, but you may mean “diverse”. With the PMIx APIs, is there a way to get back and/or specify which network security key should be used with the new Pset? With HPE’s Slingshot and the upcoming UEC, different applications are run with different network security keys (VNIs and JobIDs respectively). Two separately-launched applications will likely have different keys and be unable to intercommunicate. MPI is going to need PMI to tell it which key to use to talk to the processes that were launched separately. MPI will also need a privileged service to add the key to its network context. With Slingshot, applications launched under the same job share a JobVNI in addition to their individual job-step VNIs. Applications from separate jobs and users can use DRC2 to agree upon a VNI. We’ll need a generic attribute to pass this information in PMIx. Michael Raymond HPE HPC Development Environment 1 (612) 851-0500 On Aug 18, 2025, at 20:37, Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd #pfptBannervq9swly { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important; max-width: none !important; max-height: none !important } .pfptPrimaryButtonvq9swly:hover, .pfptPrimaryButtonvq9swly:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonvq9swly:active { background-color: #90a4ae !important; } Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a means of making it easier to extend in the future. I suspect the community at-large will discover additional use-cases that require new inputs, and it would be unfortunate if we had to create new APIs every time that happened. BTW: I have the resource usage code working in both PMIx and PRRTE now - getting ready to commit those changes. Just need to clean up a couple of things first. Ralph On Aug 18, 2025, at 12:12 PM, Dominik Huber via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: This Message Is From an External Sender This message came from outside your organization. Hi all, As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi… <https://urldefense.com/v3/__https:/urldefense.us/v3/__https:/*docs.google.c… > I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. Best regards, Dominik _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi… <https://urldefense.com/v3/__https:/urldefense.us/v3/__https:/*lists.mpi-for… > _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]> https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp… <https://urldefense.com/v3/__https:/lists.mpi-forum.org/mailman/listinfo/mpi… >

1 0

Meet this coming Monday 25th August
by Pritchard Jr., Howard 21 Aug '25

21 Aug '25

Hi All, Let’s meet this coming Monday. A number of folks will presumably be out attending Europar, but there are some other topics we can discuss besides dynamic process sets. I’d like to resume the discussion on session attributes – see https://urldefense.us/v3/__https://github.com/mpiwg-sessions/mpi-standard/p… . Per recommendation of Sonja I added an example since we last discussed this topic. We can also discuss some planning for the upcoming MPI Forum. Thanks, Howard ------- [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_3406584976]<https://urldefense.us/v3/__https://www.instagram.com/losalamosnatlab/__;!!G… >[signature_3564112004]<https://urldefense.us/v3/__https://twitter.com/LosAlamosNatLab__;!!G_uCfscf… >[signature_2305798856]<https://urldefense.us/v3/__https://www.linkedin.com/company/los-alamos-nati… >[signature_3406533711]<https://urldefense.us/v3/__https://www.facebook.com/LosAlamosNationalLab/__… >

1 0

Planning for MPI Forum 9/29/25-10/1/25
by Pritchard Jr., Howard 21 Aug '25

21 Aug '25

Hello Sessions folks, We should do some planning for the upcoming MPI Forum taking place at Univ. North Carolina – Charlotte. A couple of weeks ago I responded to Wes saying the Sessions WG would like to meet in person and that we would have at least two topics to present to the larger Forum – one dynamic psets, the other session attributes. I might float the sessions attributes as a first reading since it doesn’t involve much in the way of conceptual changes. Pset topic should probably be a plenary format. I’m thinking for psets we would have had time to incorporate feedback from the presentation Dominik can hopefully do at non-voting meeting prior to the Forum. Who in the WG is planning to attend the Forum either in person or virtual? If we have virtual attendees from Europe I’ll ask Wes to allocate a morning time slot for our WG session breakout. Thanks, Howard ------- [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_2265338182]<https://urldefense.us/v3/__https://www.instagram.com/losalamosnatlab/__;!!G… >[signature_2279560789]<https://urldefense.us/v3/__https://twitter.com/LosAlamosNatLab__;!!G_uCfscf… >[signature_2293436884]<https://urldefense.us/v3/__https://www.linkedin.com/company/los-alamos-nati… >[signature_2347218470]<https://urldefense.us/v3/__https://www.facebook.com/LosAlamosNationalLab/__… >

1 0

Re: [mpiwg-sessions] Cancel 25th August sessions WG meeting?
by Happ, Sonja 20 Aug '25

20 Aug '25

I would be available next Monday. Dr. Sonja Happ Software Engineer ∙ Software & Architecture Development ParTec AG ∙ Possartstr. 20 ∙ D-81679 München ∙ https://urldefense.us/v3/__http://www.par-tec.com__;!!G_uCfscf7eWS!Y3Zu3ZyK… Registration Court: Amtsgericht München, HRB 263870 Executive Board: Bernhard Frohwitter (Chairman) Johann Kilger, Thomas Moschny, Ina Schmitz Chairman of the Supervisory Board: Prof. Dr. Reimund Neugebauer Privacy Policy: https://urldefense.us/v3/__http://www.par-tec.com/privacy__;!!G_uCfscf7eWS!… ________________________________ Von: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> im Auftrag von Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Gesendet: Dienstag, 19. August 2025 17:32 An: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Ralph Castain <rhc(a)pmix.org> Betreff: Re: [mpiwg-sessions] Cancel 25th August sessions WG meeting? Same here On Aug 19, 2025, at 9: 29 AM, Raymond, Michael via mpiwg-sessions <mpiwg-sessions@ lists. mpi-forum. org> wrote: This Message Is From an External Sender This message came from outside your organization. I’m fine either way. Michael ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Same here On Aug 19, 2025, at 9:29 AM, Raymond, Michael via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: This Message Is From an External Sender This message came from outside your organization. I’m fine either way. Michael Raymond HPE HPC Development Environment 1 (612) 851-0500 > On Aug 19, 2025, at 10:27, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > Hi All, > I think when we were discussing the WG’s August schedule we forgot about Europar. > So kind of a poll – who would be interested in meeting this coming Monday? > Howard > ------- > <image001.png>Howard Pritchard > Research Scientist > HPC-ENV > Los Alamos National Laboratory > howardp(a)lanl.gov<mailto:[email protected]> > <image002.png><image003.png><image004.png><image005.png> _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]> > https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp… _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]> https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi… <https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi…>

1 0

Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion
by Dominik Huber 20 Aug '25

20 Aug '25

Hi Ralph and Michael, Thank you for the feedback. I added a draft of PMIx attributes to support the MPI API on slides 20 + 21. The current prototype uses PMIx_Allocation_request and PMIx_Notify_event for this purpose. Best regards, Dominik ________________________________ Von: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> im Auftrag von Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> Gesendet: Dienstag, 19. August 2025 15:47:23 An: Michael Raymond Cc: Ralph Castain; MPI Sessions working group Betreff: Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix. sec. ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix.sec.ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use // in environments where multiple authentication mechanisms // may be available. When returned in a callback function, a // string identifier of the credential type #define PMIX_CRYPTO_KEY “pmix.sec.key” // (pmix_byte_object_t) blob containing crypto key You can include those in the “spawn” request as part of the job-level information to be passed to all the new processes. Providing the child procs with access to the parent’s credentials can be done by including those credentials in the job-level info to spawn, or using PMIx_Connect or PMIx_Group_construct after spawn as they will distribute the parent’s job-level info. Note that OpenMPI has chosen the latter approach as it proved beneficial for the child procs to have a full picture of their parent job. The info is retrieved by individual procs with a simple PMIx_Get, passing the ID of the proc whose credential you are seeking (rank=wildcard since we assume it’s the same for all procs in the namespace) and the PMIX_CRYPTO_KEY key (if you need the type, you can ask for that too with PMIX_CRED_TYPE). I suppose it is possible (perhaps in the future) for a process/job to be assigned multiple security keys - e.g., if the system supported multiple parallel fabrics, each with its own security key. If that happens, we’ll have two choices: (a) define a new attribute for passing an array of structs containing keys and associated fabric ID so you know which key goes with which fabric; and/or (b) packing that info into the PMIX_CRYPTO_KEY byte object in a specified manner so it can be unpacked at the proc. We also have the PMIx_Get_credential API you can use to request and obtain a new credential. Of course, that requires that the host support it on the backend! As you probably recall, we had talked during the Aurora days about having PMIx assist with obtaining the credential from the system (or perhaps generating it algorithmically like Slurm does for VNI), but never quite got to that point. At this time, we would rely on the PMIx server’s upcall to the host to obtain the credential. In terms of injecting the key into the appropriate privileged service, there are a couple of options. One could use PMIx to connect to the privileged service and pass the key that way. I suspect system security folks would be leery of that no matter how carefully we tried to construct the connection validation handshake. Another option might be to create a setuid script that operates at the service-privileged level and communicates the key to the service. Might be more palatable to the security folks as the script could be tightly constrained as to its abilities and kept very simple. Or maybe have the vendors provide an appropriate library that PMIx can call to set the credential (thereby allowing the vendor to control the communication handshake). Or we could assign the responsibility for setting the network key to the host. If we pass the key in the spawn command, then the host could search for it and inject it into the services where required. Vendor environments would probably be able to do that as they usually have an accepted method for talking to a privileged service. Probably won’t work for mpirun, though, as that normally doesn’t run at privilege and therefore couldn’t do the injection. So if we want mpirun to support network credentials for spawn, then we probably need to either find an acceptable way to connect to the service, or get acceptance for a setuid script or library to span the gap. Happy to discuss this more over the phone or at a meeting, if people would like. Ralph On Aug 19, 2025, at 5:29 AM, Raymond, Michael <mraymond(a)hpe.com> wrote: On slide 11, “divers” works, but you may mean “diverse”. With the PMIx APIs, is there a way to get back and/or specify which network security key should be used with the new Pset? With HPE’s Slingshot and the upcoming UEC, different applications are run with different network security keys (VNIs and JobIDs respectively). Two separately-launched applications will likely have different keys and be unable to intercommunicate. MPI is going to need PMI to tell it which key to use to talk to the processes that were launched separately. MPI will also need a privileged service to add the key to its network context. With Slingshot, applications launched under the same job share a JobVNI in addition to their individual job-step VNIs. Applications from separate jobs and users can use DRC2 to agree upon a VNI. We’ll need a generic attribute to pass this information in PMIx. Michael Raymond HPE HPC Development Environment 1 (612) 851-0500 On Aug 18, 2025, at 20:37, Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd #pfptBannervq9swly { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important; max-width: none !important; max-height: none !important } .pfptPrimaryButtonvq9swly:hover, .pfptPrimaryButtonvq9swly:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonvq9swly:active { background-color: #90a4ae !important; } Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a means of making it easier to extend in the future. I suspect the community at-large will discover additional use-cases that require new inputs, and it would be unfortunate if we had to create new APIs every time that happened. BTW: I have the resource usage code working in both PMIx and PRRTE now - getting ready to commit those changes. Just need to clean up a couple of things first. Ralph On Aug 18, 2025, at 12:12 PM, Dominik Huber via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: This Message Is From an External Sender This message came from outside your organization. Hi all, As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi… <https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi…> I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. Best regards, Dominik _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi… <https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi…> _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]> https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp…

1 0

Re: [mpiwg-sessions] Cancel 25th August sessions WG meeting?
by Ralph Castain 19 Aug '25

19 Aug '25

Same here > On Aug 19, 2025, at 9:29 AM, Raymond, Michael via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > This Message Is From an External Sender > This message came from outside your organization. > I’m fine either way. > > Michael Raymond > HPE HPC Development Environment > 1 (612) 851-0500 > > > On Aug 19, 2025, at 10:27, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Hi All, > > I think when we were discussing the WG’s August schedule we forgot about Europar. > > So kind of a poll – who would be interested in meeting this coming Monday? > > Howard > > ------- > > <image001.png>Howard Pritchard > > Research Scientist > > HPC-ENV > > Los Alamos National Laboratory > > howardp(a)lanl.gov <mailto:[email protected]> > > <image002.png><image003.png><image004.png><image005.png> _______________________________________________ > > mpiwg-sessions mailing list > > mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > > https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp… > > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi…

1 0

Re: [mpiwg-sessions] Cancel 25th August sessions WG meeting?
by Raymond, Michael 19 Aug '25

19 Aug '25

1 0

Cancel 25th August sessions WG meeting?
by Pritchard Jr., Howard 19 Aug '25

19 Aug '25

Hi All, I think when we were discussing the WG’s August schedule we forgot about Europar. So kind of a poll – who would be interested in meeting this coming Monday? Howard ------- [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> [signature_3189307141]<https://urldefense.us/v3/__https://www.instagram.com/losalamosnatlab/__;!!G… >[signature_3469584182]<https://urldefense.us/v3/__https://twitter.com/LosAlamosNatLab__;!!G_uCfscf… >[signature_4021162776]<https://urldefense.us/v3/__https://www.linkedin.com/company/los-alamos-nati… >[signature_393059152]<https://urldefense.us/v3/__https://www.facebook.com/LosAlamosNationalLab/__… >

1 0

Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion
by Ralph Castain 19 Aug '25

19 Aug '25

Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix.sec.ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use // in environments where multiple authentication mechanisms // may be available. When returned in a callback function, a // string identifier of the credential type #define PMIX_CRYPTO_KEY “pmix.sec.key” // (pmix_byte_object_t) blob containing crypto key You can include those in the “spawn” request as part of the job-level information to be passed to all the new processes. Providing the child procs with access to the parent’s credentials can be done by including those credentials in the job-level info to spawn, or using PMIx_Connect or PMIx_Group_construct after spawn as they will distribute the parent’s job-level info. Note that OpenMPI has chosen the latter approach as it proved beneficial for the child procs to have a full picture of their parent job. The info is retrieved by individual procs with a simple PMIx_Get, passing the ID of the proc whose credential you are seeking (rank=wildcard since we assume it’s the same for all procs in the namespace) and the PMIX_CRYPTO_KEY key (if you need the type, you can ask for that too with PMIX_CRED_TYPE). I suppose it is possible (perhaps in the future) for a process/job to be assigned multiple security keys - e.g., if the system supported multiple parallel fabrics, each with its own security key. If that happens, we’ll have two choices: (a) define a new attribute for passing an array of structs containing keys and associated fabric ID so you know which key goes with which fabric; and/or (b) packing that info into the PMIX_CRYPTO_KEY byte object in a specified manner so it can be unpacked at the proc. We also have the PMIx_Get_credential API you can use to request and obtain a new credential. Of course, that requires that the host support it on the backend! As you probably recall, we had talked during the Aurora days about having PMIx assist with obtaining the credential from the system (or perhaps generating it algorithmically like Slurm does for VNI), but never quite got to that point. At this time, we would rely on the PMIx server’s upcall to the host to obtain the credential. In terms of injecting the key into the appropriate privileged service, there are a couple of options. One could use PMIx to connect to the privileged service and pass the key that way. I suspect system security folks would be leery of that no matter how carefully we tried to construct the connection validation handshake. Another option might be to create a setuid script that operates at the service-privileged level and communicates the key to the service. Might be more palatable to the security folks as the script could be tightly constrained as to its abilities and kept very simple. Or maybe have the vendors provide an appropriate library that PMIx can call to set the credential (thereby allowing the vendor to control the communication handshake). Or we could assign the responsibility for setting the network key to the host. If we pass the key in the spawn command, then the host could search for it and inject it into the services where required. Vendor environments would probably be able to do that as they usually have an accepted method for talking to a privileged service. Probably won’t work for mpirun, though, as that normally doesn’t run at privilege and therefore couldn’t do the injection. So if we want mpirun to support network credentials for spawn, then we probably need to either find an acceptable way to connect to the service, or get acceptance for a setuid script or library to span the gap. Happy to discuss this more over the phone or at a meeting, if people would like. Ralph > On Aug 19, 2025, at 5:29 AM, Raymond, Michael <mraymond(a)hpe.com> wrote: > > On slide 11, “divers” works, but you may mean “diverse”. > > With the PMIx APIs, is there a way to get back and/or specify which network security key should be used with the new Pset? With HPE’s Slingshot and the upcoming UEC, different applications are run with different network security keys (VNIs and JobIDs respectively). Two separately-launched applications will likely have different keys and be unable to intercommunicate. MPI is going to need PMI to tell it which key to use to talk to the processes that were launched separately. MPI will also need a privileged service to add the key to its network context. > > With Slingshot, applications launched under the same job share a JobVNI in addition to their individual job-step VNIs. Applications from separate jobs and users can use DRC2 to agree upon a VNI. We’ll need a generic attribute to pass this information in PMIx. > > Michael Raymond > HPE HPC Development Environment > 1 (612) 851-0500 > >> On Aug 18, 2025, at 20:37, Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >> >> Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> ZjQcmQRYFpfptBannerEnd >> #pfptBannervq9swly { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important; max-width: none !important; max-height: none !important } .pfptPrimaryButtonvq9swly:hover, .pfptPrimaryButtonvq9swly:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonvq9swly:active { background-color: #90a4ae !important; } Hi Dominik >> >> Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! >> >> Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a means of making it easier to extend in the future. I suspect the community at-large will discover additional use-cases that require new inputs, and it would be unfortunate if we had to create new APIs every time that happened. >> >> BTW: I have the resource usage code working in both PMIx and PRRTE now - getting ready to commit those changes. Just need to clean up a couple of things first. >> >> Ralph >> >> >>> On Aug 18, 2025, at 12:12 PM, Dominik Huber via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: >>> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> Hi all, >>> As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. >>> Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. >>> To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: >>> https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfi… >>> >>> I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. >>> >>> Best regards, >>> >>> Dominik >>> >>> >>> >>> _______________________________________________ >>> mpiwg-sessions mailing list >>> mpiwg-sessions(a)lists.mpi-forum.org >>> https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi… >> >> >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> >> https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mp…

1 0