Hi all, Thank you for your comments. I have updated the slides accordingly. The only remaining part is the slide with concrete questions for the audience. Maybe we could discuss this in our Sessions WG meeting next Monday. Best regards, Dominik ________________________________ Von: Pritchard Jr., Howard <[email protected]> Gesendet: Donnerstag, 21. August 2025 18:35:08 An: MPI Sessions working group; Michael Raymond Cc: Dominik Huber Betreff: Re: [EXTERNAL] Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion HI Dominik, I added a comment on slide 9. Otherwise, the slides look pretty good! Howard From: mpiwg-sessions <[email protected]> on behalf of MPI Sessions working group <[email protected]> Reply-To: MPI Sessions working group <[email protected]> Date: Wednesday, August 20, 2025 at 6:25 AM To: Michael Raymond <[email protected]>, MPI Sessions working group <[email protected]> Cc: Dominik Huber <[email protected]> Subject: [EXTERNAL] Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion Hi Ralph and Michael, Thank you for the feedback. I added a draft of PMIx attributes to support the MPI API on slides 20 + 21. The current prototype uses PMIx_Allocation_request and PMIx_Notify_event for this purpose. Best regards, Dominik ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Ralph and Michael, Thank you for the feedback. I added a draft of PMIx attributes to support the MPI API on slides 20 + 21. The current prototype uses PMIx_Allocation_request and PMIx_Notify_event for this purpose. Best regards, Dominik ________________________________ Von: mpiwg-sessions <[email protected]> im Auftrag von Ralph Castain via mpiwg-sessions <[email protected]> Gesendet: Dienstag, 19. August 2025 15:47:23 An: Michael Raymond Cc: Ralph Castain; MPI Sessions working group Betreff: Re: [mpiwg-sessions] Draft Slides for Dynamic Processes WG Discussion Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix. sec. ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Michael Just a few thoughts. PMIx does have two security-related attributes: #define PMIX_CRED_TYPE “pmix.sec.ctype" // (char*) when passed in PMIx_Get_credential, a prioritized, // comma-delimited list of desired credential types for use // in environments where multiple authentication mechanisms // may be available. When returned in a callback function, a // string identifier of the credential type #define PMIX_CRYPTO_KEY “pmix.sec.key” // (pmix_byte_object_t) blob containing crypto key You can include those in the “spawn” request as part of the job-level information to be passed to all the new processes. Providing the child procs with access to the parent’s credentials can be done by including those credentials in the job-level info to spawn, or using PMIx_Connect or PMIx_Group_construct after spawn as they will distribute the parent’s job-level info. Note that OpenMPI has chosen the latter approach as it proved beneficial for the child procs to have a full picture of their parent job. The info is retrieved by individual procs with a simple PMIx_Get, passing the ID of the proc whose credential you are seeking (rank=wildcard since we assume it’s the same for all procs in the namespace) and the PMIX_CRYPTO_KEY key (if you need the type, you can ask for that too with PMIX_CRED_TYPE). I suppose it is possible (perhaps in the future) for a process/job to be assigned multiple security keys - e.g., if the system supported multiple parallel fabrics, each with its own security key. If that happens, we’ll have two choices: (a) define a new attribute for passing an array of structs containing keys and associated fabric ID so you know which key goes with which fabric; and/or (b) packing that info into the PMIX_CRYPTO_KEY byte object in a specified manner so it can be unpacked at the proc. We also have the PMIx_Get_credential API you can use to request and obtain a new credential. Of course, that requires that the host support it on the backend! As you probably recall, we had talked during the Aurora days about having PMIx assist with obtaining the credential from the system (or perhaps generating it algorithmically like Slurm does for VNI), but never quite got to that point. At this time, we would rely on the PMIx server’s upcall to the host to obtain the credential. In terms of injecting the key into the appropriate privileged service, there are a couple of options. One could use PMIx to connect to the privileged service and pass the key that way. I suspect system security folks would be leery of that no matter how carefully we tried to construct the connection validation handshake. Another option might be to create a setuid script that operates at the service-privileged level and communicates the key to the service. Might be more palatable to the security folks as the script could be tightly constrained as to its abilities and kept very simple. Or maybe have the vendors provide an appropriate library that PMIx can call to set the credential (thereby allowing the vendor to control the communication handshake). Or we could assign the responsibility for setting the network key to the host. If we pass the key in the spawn command, then the host could search for it and inject it into the services where required. Vendor environments would probably be able to do that as they usually have an accepted method for talking to a privileged service. Probably won’t work for mpirun, though, as that normally doesn’t run at privilege and therefore couldn’t do the injection. So if we want mpirun to support network credentials for spawn, then we probably need to either find an acceptable way to connect to the service, or get acceptance for a setuid script or library to span the gap. Happy to discuss this more over the phone or at a meeting, if people would like. Ralph On Aug 19, 2025, at 5:29 AM, Raymond, Michael <[email protected]> wrote: On slide 11, “divers” works, but you may mean “diverse”. With the PMIx APIs, is there a way to get back and/or specify which network security key should be used with the new Pset? With HPE’s Slingshot and the upcoming UEC, different applications are run with different network security keys (VNIs and JobIDs respectively). Two separately-launched applications will likely have different keys and be unable to intercommunicate. MPI is going to need PMI to tell it which key to use to talk to the processes that were launched separately. MPI will also need a privileged service to add the key to its network context. With Slingshot, applications launched under the same job share a JobVNI in addition to their individual job-step VNIs. Applications from separate jobs and users can use DRC2 to agree upon a VNI. We’ll need a generic attribute to pass this information in PMIx. Michael Raymond HPE HPC Development Environment 1 (612) 851-0500 On Aug 18, 2025, at 20:37, Ralph Castain via mpiwg-sessions <[email protected]<mailto:[email protected]>> wrote: Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd #pfptBannervq9swly { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important; max-width: none !important; max-height: none !important } .pfptPrimaryButtonvq9swly:hover, .pfptPrimaryButtonvq9swly:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonvq9swly:active { background-color: #90a4ae !important; } Hi Dominik Scanned through your slides and thought they looked good. Would definitely be nice to create an MPI-level abstraction for dynamic ops! Only comment I had was to perhaps provide the equivalent of PMIx attributes to the MPI API as a means of making it easier to extend in the future. I suspect the community at-large will discover additional use-cases that require new inputs, and it would be unfortunate if we had to create new APIs every time that happened. BTW: I have the resource usage code working in both PMIx and PRRTE now - getting ready to commit those changes. Just need to clean up a couple of things first. Ralph On Aug 18, 2025, at 12:12 PM, Dominik Huber via mpiwg-sessions <[email protected]> wrote: This Message Is From an External Sender This message came from outside your organization. Hi all, As discussed in our last meeting, we’re considering a presentation on the current state of our Dynamic Processes discussions at a tentative non-voting meeting in early September. Since many of us will be on vacation or at EuroPar in August, it may be difficult to hold WG meetings during this period. To keep things moving, I’ve uploaded a draft of our slides to Google Slides for collaboration and feedback: https://urldefense.us/v3/__https://docs.google.com/presentation/d/1N-eotOfiQ... <https://urldefense.com/v3/__https:/urldefense.us/v3/__https:/*docs.google.co... > I won’t be able to attend next week’s WG meeting (if one is planned), but the usual Zoom link should still work. Best regards, Dominik _______________________________________________ mpiwg-sessions mailing list [email protected] https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpiw... <https://urldefense.com/v3/__https:/urldefense.us/v3/__https:/*lists.mpi-foru... > _______________________________________________ mpiwg-sessions mailing list [email protected]<mailto:[email protected]> https://urldefense.com/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi... <https://urldefense.com/v3/__https:/lists.mpi-forum.org/mailman/listinfo/mpiw... >