March 2019 - mpiwg-tools - mailman3.cels.anl.gov

Meeting reminder: Mar 28, 2019
by Marc-Andre Hermanns 28 Mar '19

28 Mar '19

Hi everyone, This is a reminder of our regularly scheduled call today Marc 28 at 8 am Pacific / 11 am Eastern / 5pm CET. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: ------- - Feedback from Virtual Meeting yesterday - Discussion on pending changes Cheers, Marc-Andre -- Dr. Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Wilhelm-Johnen-Str. 52425 Jülich Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc email: hermanns(a)jara.rwth-aachen.de

1 0

Re: [mpiwg-tools] Meeting March 21, 2019
by Mohror, Kathryn 26 Mar '19

26 Mar '19

Hi John, Oops, I had a typo in the link text. I think it is fixed now. Please let me know if not. Thanks! Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org> On Behalf Of John DelSignore via mpiwg-tools Sent: Monday, March 25, 2019 6:33 AM To: mpiwg-tools(a)lists.mpi-forum.org Cc: John DelSignore <John.DelSignore(a)roguewave.com> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Kathryn, The PDF file containing the notes for the 2019-03-21 meeting seems to 0 length, or perhaps the link on the Wiki page is bad. Is it me or is there a problem on the Wiki? Cheers, John D. On 03/24/19 13:36, Mohror, Kathryn via mpiwg-tools wrote: Hi all, I added our notes from the last two meetings on debugger interfaces: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2019-02-28 https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2019-03-24 Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org><mailto:[email protected]> On Behalf Of Ahn, Dong H. via mpiwg-tools Sent: Thursday, March 21, 2019 10:17 AM To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Ahn, Dong H. <ahn1(a)llnl.gov><mailto:[email protected]> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Ralph and all, Definitely useful use case! I have not been able to call into this concall series, but I have a general input as I’m pretty familiar with these debug interfaces (as well as resource manager mechanisms). Generally speaking, I like any standard interface to be lean and mean with good separations of concerns so that many implementors can implement the interface. In that context, it seems it would be good to discuss what functionalities should be implemented at the tool’s side and what should be provided by the new MPIR2 standard. In this particular use case, a remote tool can easily ssh into the remote node and co-locate an agent to create a session like you are describing using whatever requisite security mechanism for the site. If the standard interface tries to solve this problem within itself, I fear it may lose good separation of concerns and the interface can quickly become unwieldy. But I do agree it would be good to capture this use case somewhere and factor that into when MPIR2 interface starts to emerge. Just my $0.02. Dong From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Ralph H Castain via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Reply-To: "mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>" <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Date: Thursday, March 21, 2019 at 9:37 AM To: "mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>" <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Ralph H Castain <rhc(a)open-mpi.org<mailto:[email protected]>> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Quick follow-up to today’s call. I’d like to suggest adding a bullet to the list of requirements for supporting remote tool connections - i.e., where the tool is executing on a node different from the process to which it is attached (RM or launcher). This is a really useful capability. For example, I am currently running a GUI-based tool on my desktop computer that is attached and controlling an application launcher on a cluster 200 miles away (I confess it is using PMIx). Yes, I could login to that machine, but then I either am limited to a text screen or must utilize secondary tools like VNC or X11. Much cleaner to just directly connect/control. This is where I was going with the ability to detach and remain running - however, I think some explicit language about remote operations might be useful. HTH Ralph On Mar 21, 2019, at 6:46 AM, Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, Thanks Dick and John for sending your use cases and straw men! Here are a couple more docs I found from our discussions in the past: 1. Straw man from Anh Vo (mpi_debugging_requirements-AV-05-10-2016) 2. Needs from Allinea debugger as described by Chris January (mpi_debugging_requirements (002)) 3. Notes from Forum meeting 2/27/2017: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2017-02-27 Talk to you all soon! Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Dirk Schubert via mpiwg-tools Sent: Thursday, March 21, 2019 3:43 AM To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Dirk Schubert <Dirk.Schubert(a)arm.com<mailto:[email protected]>> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Kathryn, The following use cases are important for Arm Forge (debugger DDT and profiler MAP). Use case 1: Launch an MPI job under the control of a tool. Input: The user's regular command line to start the MPI job, for example "mpirun -n 256 ./wave_c". Why? To make it as friction-less as possible to use a tool. NB: The tool does not want to parse or interpret the MPI starter arguments, as the tool has no knowledge of all possible MPI starter arguments. Requirements: * Hold MPI processes for the tool to attach to them and to release them when attached. Q: Where are processes held? In MPI_Init or _start? * Spawn tool daemon on nodes where job is running. Q: Co-spawn or separate? Q: One per node or one per process? Q: Environment? Same general environment as MPI processes instead of restricted environment (e.g. $HOME not set or chroot). * Acquire the process table (rank, hostname, pid and executable path) of spawned MPI processes. Q: Only global proctable or additionally local (per node) proctable? * [Optional] Modify the environment of the MPI processes before launching, such as prepending to LD_PRELOAD or LD_LIBRARY_PATH to inject preloads into the MPI processes (but not other processes, such as MPI daemons or tool daemons). Q: When? It would be great if modifying the environment could be done just before the MPI processes are forked/exec’ed. Why? For our profiler MAP we need to preload an MPI specific PMPI library, but for some “MPI” starter processes such as srun we don’t know the real MPI implementation that is used. Acquiring a partial process table (no PIDs) and inspecting the binaries pointed to by the executable paths, could allow us to detect the MPI implementation. * [Optional] Raise queue allocation request and granted events to allow the tool to disable startup timeouts temporarily while a queue allocation is in progress, as this can take a long time (applicable to srun). * [Optional] Scalable startup of tool daemons and shipping of preloads without touching the parallel file system on compute nodes. For example, pushing of daemon executable file (and dependencies plus preloads) to ram disk on compute nodes or Spindle? Q: Where files are pushed? A job specific temporary directory such as /tmp/mpi.job.1234/... or configurable? Q: How can the tool query the location of pushed files to reference them? Q: Who cleans up the files afterwards? The MPI job/starter itself or the tool? N: Files should be pushed with original file’s permissions. NB: Optional requirement = A tool could ultimately work without it, but I very much would like to see it supported. In PMIx this is the “indirect” use case and Ralph Castain and I have been working together since a while to make sure this use case will be supported by PMIx (excluding some optional requirements for now). Additional questions/notes: * How will the handshake work, such that the MPI starter knows it's running under a tool and when it's possible for a tool to interact with the MPI starter process? * The handshake must support cases where the MPI starter filename that is provided by the user is a wrapper around the real MPI starter executable, for example XALT. * In PMIx both problems are solved with PMIX_LAUNCHER_PAUSE_FOR_TOOL=1 and PMIX_LAUNCHER_RENDEZVOUS_FILE=<filename>. * Will only the tool or also the tool daemons interact with the MPI job? * Security, for example USER B shall not be able to attach to MPI job of USER A. * Anything to consider for MPMD or heterogeneous systems? * Must be scalable. * Must not require a debugger. Use case 2: Attach to a running MPI job with a tool. This use case is a subset of use case 1 with the following requirements: * Acquire the process table of spawned MPI processes of MPI job. * Spawn tool daemon on nodes where MPI job is running. * [Optional] Scalable startup of tool daemons without touching the file system. Thanks, Dirk Dirk Schubert | Arm | Staff Software Engineer dirk.schubert(a)arm.com<mailto:[email protected]> allinea is now part of Arm ________________________________ From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: 17 March 2019 14:46 To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Mohror, Kathryn Subject: [mpiwg-tools] Meeting March 21, 2019 Hey everybody, For our call this Thursday (3/21) we’ll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don’t think the EU has done so yet. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I’ll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, https://people.llnl.gov/kathryn Data Analysis Group<https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. <mpi_debugging_requirements-AV-05-10-2016.pdf><mpi_debugging_requirements (002).pdf>_______________________________________________ mpiwg-tools mailing list mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools _______________________________________________ mpiwg-tools mailing list mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools

1 0

Re: [mpiwg-tools] Meeting March 21, 2019
by John DelSignore 25 Mar '19

25 Mar '19

Hi Kathryn, The PDF file containing the notes for the 2019-03-21 meeting seems to 0 length, or perhaps the link on the Wiki page is bad. Is it me or is there a problem on the Wiki? Cheers, John D. On 03/24/19 13:36, Mohror, Kathryn via mpiwg-tools wrote: Hi all, I added our notes from the last two meetings on debugger interfaces: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2019-02-28 https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2019-03-24 Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org><mailto:[email protected]> On Behalf Of Ahn, Dong H. via mpiwg-tools Sent: Thursday, March 21, 2019 10:17 AM To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Ahn, Dong H. <ahn1(a)llnl.gov><mailto:[email protected]> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Ralph and all, Definitely useful use case! I have not been able to call into this concall series, but I have a general input as I’m pretty familiar with these debug interfaces (as well as resource manager mechanisms). Generally speaking, I like any standard interface to be lean and mean with good separations of concerns so that many implementors can implement the interface. In that context, it seems it would be good to discuss what functionalities should be implemented at the tool’s side and what should be provided by the new MPIR2 standard. In this particular use case, a remote tool can easily ssh into the remote node and co-locate an agent to create a session like you are describing using whatever requisite security mechanism for the site. If the standard interface tries to solve this problem within itself, I fear it may lose good separation of concerns and the interface can quickly become unwieldy. But I do agree it would be good to capture this use case somewhere and factor that into when MPIR2 interface starts to emerge. Just my $0.02. Dong From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Ralph H Castain via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Reply-To: "mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>" <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Date: Thursday, March 21, 2019 at 9:37 AM To: "mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>" <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Ralph H Castain <rhc(a)open-mpi.org<mailto:[email protected]>> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Quick follow-up to today’s call. I’d like to suggest adding a bullet to the list of requirements for supporting remote tool connections - i.e., where the tool is executing on a node different from the process to which it is attached (RM or launcher). This is a really useful capability. For example, I am currently running a GUI-based tool on my desktop computer that is attached and controlling an application launcher on a cluster 200 miles away (I confess it is using PMIx). Yes, I could login to that machine, but then I either am limited to a text screen or must utilize secondary tools like VNC or X11. Much cleaner to just directly connect/control. This is where I was going with the ability to detach and remain running - however, I think some explicit language about remote operations might be useful. HTH Ralph On Mar 21, 2019, at 6:46 AM, Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, Thanks Dick and John for sending your use cases and straw men! Here are a couple more docs I found from our discussions in the past: 1. Straw man from Anh Vo (mpi_debugging_requirements-AV-05-10-2016) 2. Needs from Allinea debugger as described by Chris January (mpi_debugging_requirements (002)) 3. Notes from Forum meeting 2/27/2017: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2017-02-27 Talk to you all soon! Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Dirk Schubert via mpiwg-tools Sent: Thursday, March 21, 2019 3:43 AM To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Dirk Schubert <Dirk.Schubert(a)arm.com<mailto:[email protected]>> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Kathryn, The following use cases are important for Arm Forge (debugger DDT and profiler MAP). Use case 1: Launch an MPI job under the control of a tool. Input: The user's regular command line to start the MPI job, for example "mpirun -n 256 ./wave_c". Why? To make it as friction-less as possible to use a tool. NB: The tool does not want to parse or interpret the MPI starter arguments, as the tool has no knowledge of all possible MPI starter arguments. Requirements: * Hold MPI processes for the tool to attach to them and to release them when attached. Q: Where are processes held? In MPI_Init or _start? * Spawn tool daemon on nodes where job is running. Q: Co-spawn or separate? Q: One per node or one per process? Q: Environment? Same general environment as MPI processes instead of restricted environment (e.g. $HOME not set or chroot). * Acquire the process table (rank, hostname, pid and executable path) of spawned MPI processes. Q: Only global proctable or additionally local (per node) proctable? * [Optional] Modify the environment of the MPI processes before launching, such as prepending to LD_PRELOAD or LD_LIBRARY_PATH to inject preloads into the MPI processes (but not other processes, such as MPI daemons or tool daemons). Q: When? It would be great if modifying the environment could be done just before the MPI processes are forked/exec’ed. Why? For our profiler MAP we need to preload an MPI specific PMPI library, but for some “MPI” starter processes such as srun we don’t know the real MPI implementation that is used. Acquiring a partial process table (no PIDs) and inspecting the binaries pointed to by the executable paths, could allow us to detect the MPI implementation. * [Optional] Raise queue allocation request and granted events to allow the tool to disable startup timeouts temporarily while a queue allocation is in progress, as this can take a long time (applicable to srun). * [Optional] Scalable startup of tool daemons and shipping of preloads without touching the parallel file system on compute nodes. For example, pushing of daemon executable file (and dependencies plus preloads) to ram disk on compute nodes or Spindle? Q: Where files are pushed? A job specific temporary directory such as /tmp/mpi.job.1234/... or configurable? Q: How can the tool query the location of pushed files to reference them? Q: Who cleans up the files afterwards? The MPI job/starter itself or the tool? N: Files should be pushed with original file’s permissions. NB: Optional requirement = A tool could ultimately work without it, but I very much would like to see it supported. In PMIx this is the “indirect” use case and Ralph Castain and I have been working together since a while to make sure this use case will be supported by PMIx (excluding some optional requirements for now). Additional questions/notes: * How will the handshake work, such that the MPI starter knows it's running under a tool and when it's possible for a tool to interact with the MPI starter process? * The handshake must support cases where the MPI starter filename that is provided by the user is a wrapper around the real MPI starter executable, for example XALT. * In PMIx both problems are solved with PMIX_LAUNCHER_PAUSE_FOR_TOOL=1 and PMIX_LAUNCHER_RENDEZVOUS_FILE=<filename>. * Will only the tool or also the tool daemons interact with the MPI job? * Security, for example USER B shall not be able to attach to MPI job of USER A. * Anything to consider for MPMD or heterogeneous systems? * Must be scalable. * Must not require a debugger. Use case 2: Attach to a running MPI job with a tool. This use case is a subset of use case 1 with the following requirements: * Acquire the process table of spawned MPI processes of MPI job. * Spawn tool daemon on nodes where MPI job is running. * [Optional] Scalable startup of tool daemons without touching the file system. Thanks, Dirk Dirk Schubert | Arm | Staff Software Engineer dirk.schubert(a)arm.com<mailto:[email protected]> allinea is now part of Arm ________________________________ From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: 17 March 2019 14:46 To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Mohror, Kathryn Subject: [mpiwg-tools] Meeting March 21, 2019 Hey everybody, For our call this Thursday (3/21) we’ll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don’t think the EU has done so yet. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I’ll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, https://people.llnl.gov/kathryn Data Analysis Group<https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. <mpi_debugging_requirements-AV-05-10-2016.pdf><mpi_debugging_requirements (002).pdf>_______________________________________________ mpiwg-tools mailing list mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools _______________________________________________ mpiwg-tools mailing list mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools

1 0

Re: [mpiwg-tools] Meeting March 21, 2019
by Mohror, Kathryn 24 Mar '19

24 Mar '19

Hi all, I added our notes from the last two meetings on debugger interfaces: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2019-02-28 https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2019-03-24 Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org> On Behalf Of Ahn, Dong H. via mpiwg-tools Sent: Thursday, March 21, 2019 10:17 AM To: mpiwg-tools(a)lists.mpi-forum.org Cc: Ahn, Dong H. <ahn1(a)llnl.gov> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Ralph and all, Definitely useful use case! I have not been able to call into this concall series, but I have a general input as I’m pretty familiar with these debug interfaces (as well as resource manager mechanisms). Generally speaking, I like any standard interface to be lean and mean with good separations of concerns so that many implementors can implement the interface. In that context, it seems it would be good to discuss what functionalities should be implemented at the tool’s side and what should be provided by the new MPIR2 standard. In this particular use case, a remote tool can easily ssh into the remote node and co-locate an agent to create a session like you are describing using whatever requisite security mechanism for the site. If the standard interface tries to solve this problem within itself, I fear it may lose good separation of concerns and the interface can quickly become unwieldy. But I do agree it would be good to capture this use case somewhere and factor that into when MPIR2 interface starts to emerge. Just my $0.02. Dong From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Ralph H Castain via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Reply-To: "mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>" <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Date: Thursday, March 21, 2019 at 9:37 AM To: "mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>" <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Ralph H Castain <rhc(a)open-mpi.org<mailto:[email protected]>> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Quick follow-up to today’s call. I’d like to suggest adding a bullet to the list of requirements for supporting remote tool connections - i.e., where the tool is executing on a node different from the process to which it is attached (RM or launcher). This is a really useful capability. For example, I am currently running a GUI-based tool on my desktop computer that is attached and controlling an application launcher on a cluster 200 miles away (I confess it is using PMIx). Yes, I could login to that machine, but then I either am limited to a text screen or must utilize secondary tools like VNC or X11. Much cleaner to just directly connect/control. This is where I was going with the ability to detach and remain running - however, I think some explicit language about remote operations might be useful. HTH Ralph On Mar 21, 2019, at 6:46 AM, Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, Thanks Dick and John for sending your use cases and straw men! Here are a couple more docs I found from our discussions in the past: 1. Straw man from Anh Vo (mpi_debugging_requirements-AV-05-10-2016) 2. Needs from Allinea debugger as described by Chris January (mpi_debugging_requirements (002)) 3. Notes from Forum meeting 2/27/2017: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2017-02-27 Talk to you all soon! Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Dirk Schubert via mpiwg-tools Sent: Thursday, March 21, 2019 3:43 AM To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Dirk Schubert <Dirk.Schubert(a)arm.com<mailto:[email protected]>> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Kathryn, The following use cases are important for Arm Forge (debugger DDT and profiler MAP). Use case 1: Launch an MPI job under the control of a tool. Input: The user's regular command line to start the MPI job, for example "mpirun -n 256 ./wave_c". Why? To make it as friction-less as possible to use a tool. NB: The tool does not want to parse or interpret the MPI starter arguments, as the tool has no knowledge of all possible MPI starter arguments. Requirements: * Hold MPI processes for the tool to attach to them and to release them when attached. Q: Where are processes held? In MPI_Init or _start? * Spawn tool daemon on nodes where job is running. Q: Co-spawn or separate? Q: One per node or one per process? Q: Environment? Same general environment as MPI processes instead of restricted environment (e.g. $HOME not set or chroot). * Acquire the process table (rank, hostname, pid and executable path) of spawned MPI processes. Q: Only global proctable or additionally local (per node) proctable? * [Optional] Modify the environment of the MPI processes before launching, such as prepending to LD_PRELOAD or LD_LIBRARY_PATH to inject preloads into the MPI processes (but not other processes, such as MPI daemons or tool daemons). Q: When? It would be great if modifying the environment could be done just before the MPI processes are forked/exec’ed. Why? For our profiler MAP we need to preload an MPI specific PMPI library, but for some “MPI” starter processes such as srun we don’t know the real MPI implementation that is used. Acquiring a partial process table (no PIDs) and inspecting the binaries pointed to by the executable paths, could allow us to detect the MPI implementation. * [Optional] Raise queue allocation request and granted events to allow the tool to disable startup timeouts temporarily while a queue allocation is in progress, as this can take a long time (applicable to srun). * [Optional] Scalable startup of tool daemons and shipping of preloads without touching the parallel file system on compute nodes. For example, pushing of daemon executable file (and dependencies plus preloads) to ram disk on compute nodes or Spindle? Q: Where files are pushed? A job specific temporary directory such as /tmp/mpi.job.1234/... or configurable? Q: How can the tool query the location of pushed files to reference them? Q: Who cleans up the files afterwards? The MPI job/starter itself or the tool? N: Files should be pushed with original file’s permissions. NB: Optional requirement = A tool could ultimately work without it, but I very much would like to see it supported. In PMIx this is the “indirect” use case and Ralph Castain and I have been working together since a while to make sure this use case will be supported by PMIx (excluding some optional requirements for now). Additional questions/notes: * How will the handshake work, such that the MPI starter knows it's running under a tool and when it's possible for a tool to interact with the MPI starter process? * The handshake must support cases where the MPI starter filename that is provided by the user is a wrapper around the real MPI starter executable, for example XALT. * In PMIx both problems are solved with PMIX_LAUNCHER_PAUSE_FOR_TOOL=1 and PMIX_LAUNCHER_RENDEZVOUS_FILE=<filename>. * Will only the tool or also the tool daemons interact with the MPI job? * Security, for example USER B shall not be able to attach to MPI job of USER A. * Anything to consider for MPMD or heterogeneous systems? * Must be scalable. * Must not require a debugger. Use case 2: Attach to a running MPI job with a tool. This use case is a subset of use case 1 with the following requirements: * Acquire the process table of spawned MPI processes of MPI job. * Spawn tool daemon on nodes where MPI job is running. * [Optional] Scalable startup of tool daemons without touching the file system. Thanks, Dirk Dirk Schubert | Arm | Staff Software Engineer dirk.schubert(a)arm.com<mailto:[email protected]> allinea is now part of Arm ________________________________ From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: 17 March 2019 14:46 To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Mohror, Kathryn Subject: [mpiwg-tools] Meeting March 21, 2019 Hey everybody, For our call this Thursday (3/21) we’ll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don’t think the EU has done so yet. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I’ll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, https://people.llnl.gov/kathryn Data Analysis Group<https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. <mpi_debugging_requirements-AV-05-10-2016.pdf><mpi_debugging_requirements (002).pdf>_______________________________________________ mpiwg-tools mailing list mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools

1 0

Re: [mpiwg-tools] Meeting March 21, 2019
by Ahn, Dong H. 21 Mar '19

21 Mar '19

Hi Ralph and all, Definitely useful use case! I have not been able to call into this concall series, but I have a general input as I’m pretty familiar with these debug interfaces (as well as resource manager mechanisms). Generally speaking, I like any standard interface to be lean and mean with good separations of concerns so that many implementors can implement the interface. In that context, it seems it would be good to discuss what functionalities should be implemented at the tool’s side and what should be provided by the new MPIR2 standard. In this particular use case, a remote tool can easily ssh into the remote node and co-locate an agent to create a session like you are describing using whatever requisite security mechanism for the site. If the standard interface tries to solve this problem within itself, I fear it may lose good separation of concerns and the interface can quickly become unwieldy. But I do agree it would be good to capture this use case somewhere and factor that into when MPIR2 interface starts to emerge. Just my $0.02. Dong From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org> on behalf of Ralph H Castain via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org> Reply-To: "mpiwg-tools(a)lists.mpi-forum.org" <mpiwg-tools(a)lists.mpi-forum.org> Date: Thursday, March 21, 2019 at 9:37 AM To: "mpiwg-tools(a)lists.mpi-forum.org" <mpiwg-tools(a)lists.mpi-forum.org> Cc: Ralph H Castain <rhc(a)open-mpi.org> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Quick follow-up to today’s call. I’d like to suggest adding a bullet to the list of requirements for supporting remote tool connections - i.e., where the tool is executing on a node different from the process to which it is attached (RM or launcher). This is a really useful capability. For example, I am currently running a GUI-based tool on my desktop computer that is attached and controlling an application launcher on a cluster 200 miles away (I confess it is using PMIx). Yes, I could login to that machine, but then I either am limited to a text screen or must utilize secondary tools like VNC or X11. Much cleaner to just directly connect/control. This is where I was going with the ability to detach and remain running - however, I think some explicit language about remote operations might be useful. HTH Ralph On Mar 21, 2019, at 6:46 AM, Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi all, Thanks Dick and John for sending your use cases and straw men! Here are a couple more docs I found from our discussions in the past: 1. Straw man from Anh Vo (mpi_debugging_requirements-AV-05-10-2016) 2. Needs from Allinea debugger as described by Chris January (mpi_debugging_requirements (002)) 3. Notes from Forum meeting 2/27/2017: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2017-02-27 Talk to you all soon! Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Dirk Schubert via mpiwg-tools Sent: Thursday, March 21, 2019 3:43 AM To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Dirk Schubert <Dirk.Schubert(a)arm.com<mailto:[email protected]>> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Kathryn, The following use cases are important for Arm Forge (debugger DDT and profiler MAP). Use case 1: Launch an MPI job under the control of a tool. Input: The user's regular command line to start the MPI job, for example "mpirun -n 256 ./wave_c". Why? To make it as friction-less as possible to use a tool. NB: The tool does not want to parse or interpret the MPI starter arguments, as the tool has no knowledge of all possible MPI starter arguments. Requirements: * Hold MPI processes for the tool to attach to them and to release them when attached. Q: Where are processes held? In MPI_Init or _start? * Spawn tool daemon on nodes where job is running. Q: Co-spawn or separate? Q: One per node or one per process? Q: Environment? Same general environment as MPI processes instead of restricted environment (e.g. $HOME not set or chroot). * Acquire the process table (rank, hostname, pid and executable path) of spawned MPI processes. Q: Only global proctable or additionally local (per node) proctable? * [Optional] Modify the environment of the MPI processes before launching, such as prepending to LD_PRELOAD or LD_LIBRARY_PATH to inject preloads into the MPI processes (but not other processes, such as MPI daemons or tool daemons). Q: When? It would be great if modifying the environment could be done just before the MPI processes are forked/exec’ed. Why? For our profiler MAP we need to preload an MPI specific PMPI library, but for some “MPI” starter processes such as srun we don’t know the real MPI implementation that is used. Acquiring a partial process table (no PIDs) and inspecting the binaries pointed to by the executable paths, could allow us to detect the MPI implementation. * [Optional] Raise queue allocation request and granted events to allow the tool to disable startup timeouts temporarily while a queue allocation is in progress, as this can take a long time (applicable to srun). * [Optional] Scalable startup of tool daemons and shipping of preloads without touching the parallel file system on compute nodes. For example, pushing of daemon executable file (and dependencies plus preloads) to ram disk on compute nodes or Spindle? Q: Where files are pushed? A job specific temporary directory such as /tmp/mpi.job.1234/... or configurable? Q: How can the tool query the location of pushed files to reference them? Q: Who cleans up the files afterwards? The MPI job/starter itself or the tool? N: Files should be pushed with original file’s permissions. NB: Optional requirement = A tool could ultimately work without it, but I very much would like to see it supported. In PMIx this is the “indirect” use case and Ralph Castain and I have been working together since a while to make sure this use case will be supported by PMIx (excluding some optional requirements for now). Additional questions/notes: * How will the handshake work, such that the MPI starter knows it's running under a tool and when it's possible for a tool to interact with the MPI starter process? * The handshake must support cases where the MPI starter filename that is provided by the user is a wrapper around the real MPI starter executable, for example XALT. * In PMIx both problems are solved with PMIX_LAUNCHER_PAUSE_FOR_TOOL=1 and PMIX_LAUNCHER_RENDEZVOUS_FILE=<filename>. * Will only the tool or also the tool daemons interact with the MPI job? * Security, for example USER B shall not be able to attach to MPI job of USER A. * Anything to consider for MPMD or heterogeneous systems? * Must be scalable. * Must not require a debugger. Use case 2: Attach to a running MPI job with a tool. This use case is a subset of use case 1 with the following requirements: * Acquire the process table of spawned MPI processes of MPI job. * Spawn tool daemon on nodes where MPI job is running. * [Optional] Scalable startup of tool daemons without touching the file system. Thanks, Dirk Dirk Schubert | Arm | Staff Software Engineer dirk.schubert(a)arm.com<mailto:[email protected]> allinea is now part of Arm ________________________________ From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: 17 March 2019 14:46 To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Mohror, Kathryn Subject: [mpiwg-tools] Meeting March 21, 2019 Hey everybody, For our call this Thursday (3/21) we’ll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don’t think the EU has done so yet. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I’ll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, https://people.llnl.gov/kathryn Data Analysis Group<https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. <mpi_debugging_requirements-AV-05-10-2016.pdf><mpi_debugging_requirements (002).pdf>_______________________________________________ mpiwg-tools mailing list mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools

1 0

Re: [mpiwg-tools] Meeting March 21, 2019
by Ralph H Castain 21 Mar '19

21 Mar '19

Quick follow-up to today’s call. I’d like to suggest adding a bullet to the list of requirements for supporting remote tool connections - i.e., where the tool is executing on a node different from the process to which it is attached (RM or launcher). This is a really useful capability. For example, I am currently running a GUI-based tool on my desktop computer that is attached and controlling an application launcher on a cluster 200 miles away (I confess it is using PMIx). Yes, I could login to that machine, but then I either am limited to a text screen or must utilize secondary tools like VNC or X11. Much cleaner to just directly connect/control. This is where I was going with the ability to detach and remain running - however, I think some explicit language about remote operations might be useful. HTH Ralph > On Mar 21, 2019, at 6:46 AM, Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org> wrote: > > Hi all, > > Thanks Dick and John for sending your use cases and straw men! > > Here are a couple more docs I found from our discussions in the past: > Straw man from Anh Vo (mpi_debugging_requirements-AV-05-10-2016) > Needs from Allinea debugger as described by Chris January (mpi_debugging_requirements (002)) > Notes from Forum meeting 2/27/2017: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2017-02-27 <https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2017-02-27> > > Talk to you all soon! > Kathryn > > From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> On Behalf Of Dirk Schubert via mpiwg-tools > Sent: Thursday, March 21, 2019 3:43 AM > To: mpiwg-tools(a)lists.mpi-forum.org <mailto:[email protected]> > Cc: Dirk Schubert <Dirk.Schubert(a)arm.com <mailto:[email protected]>> > Subject: Re: [mpiwg-tools] Meeting March 21, 2019 > > Hi Kathryn, > > > > The following use cases are important for Arm Forge (debugger DDT and profiler MAP). > > Use case 1: Launch an MPI job under the control of a tool. > > Input: The user's regular command line to start the MPI job, for example "mpirun -n 256 ./wave_c". Why? To make it as friction-less as possible to use a tool. > > NB: The tool does not want to parse or interpret the MPI starter arguments, as the tool has no knowledge of all possible MPI starter arguments. > > Requirements: > > Hold MPI processes for the tool to attach to them and to release them when attached. > Q: Where are processes held? In MPI_Init or _start? > Spawn tool daemon on nodes where job is running. > Q: Co-spawn or separate? > Q: One per node or one per process? > Q: Environment? Same general environment as MPI processes instead of restricted environment (e.g. $HOME not set or chroot). > Acquire the process table (rank, hostname, pid and executable path) of spawned MPI processes. > Q: Only global proctable or additionally local (per node) proctable? > [Optional] Modify the environment of the MPI processes before launching, such as prepending to LD_PRELOAD or LD_LIBRARY_PATH to inject preloads into the MPI processes (but not other processes, such as MPI daemons or tool daemons). > Q: When? It would be great if modifying the environment could be done just before the MPI processes are forked/exec’ed. Why? For our profiler MAP we need to preload an MPI specific PMPI library, but for some “MPI” starter processes such as srun we don’t know the real MPI implementation that is used. Acquiring a partial process table (no PIDs) and inspecting the binaries pointed to by the executable paths, could allow us to detect the MPI implementation. > [Optional] Raise queue allocation request and granted events to allow the tool to disable startup timeouts temporarily while a queue allocation is in progress, as this can take a long time (applicable to srun). > [Optional] Scalable startup of tool daemons and shipping of preloads without touching the parallel file system on compute nodes. For example, pushing of daemon executable file (and dependencies plus preloads) to ram disk on compute nodes or Spindle? > Q: Where files are pushed? A job specific temporary directory such as /tmp/mpi.job.1234/... or configurable? > Q: How can the tool query the location of pushed files to reference them? > Q: Who cleans up the files afterwards? The MPI job/starter itself or the tool? > N: Files should be pushed with original file’s permissions. > > NB: Optional requirement = A tool could ultimately work without it, but I very much would like to see it supported. > > In PMIx this is the “indirect” use case and Ralph Castain and I have been working together since a while to make sure this use case will be supported by PMIx (excluding some optional requirements for now). > > Additional questions/notes: > > How will the handshake work, such that the MPI starter knows it's running under a tool and when it's possible for a tool to interact with the MPI starter process? > The handshake must support cases where the MPI starter filename that is provided by the user is a wrapper around the real MPI starter executable, for example XALT. > In PMIx both problems are solved with PMIX_LAUNCHER_PAUSE_FOR_TOOL=1 and PMIX_LAUNCHER_RENDEZVOUS_FILE=<filename>. > Will only the tool or also the tool daemons interact with the MPI job? > Security, for example USER B shall not be able to attach to MPI job of USER A. > Anything to consider for MPMD or heterogeneous systems? > Must be scalable. > Must not require a debugger. > > Use case 2: Attach to a running MPI job with a tool. > > This use case is a subset of use case 1 with the following requirements: > > Acquire the process table of spawned MPI processes of MPI job. > Spawn tool daemon on nodes where MPI job is running. > [Optional] Scalable startup of tool daemons without touching the file system. > > Thanks, > > Dirk > > > > Dirk Schubert | Arm | Staff Software Engineer > dirk.schubert(a)arm.com <mailto:[email protected]> > allinea is now part of Arm > > > > From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org <mailto:[email protected]>> on behalf of Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org <mailto:[email protected]>> > Sent: 17 March 2019 14:46 > To: mpiwg-tools(a)lists.mpi-forum.org <mailto:[email protected]> > Cc: Mohror, Kathryn > Subject: [mpiwg-tools] Meeting March 21, 2019 > > Hey everybody, > > For our call this Thursday (3/21) we’ll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don’t think the EU has done so yet. > > > Webex Info: > https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php <https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php> > > Agenda: > > In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I’ll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. > > _________________________________________________________________ > Kathryn Mohror, kathryn(a)llnl.gov <mailto:[email protected]>, https://people.llnl.gov/kathryn <https://people.llnl.gov/kathryn> > Data Analysis Group <https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. > <mpi_debugging_requirements-AV-05-10-2016.pdf><mpi_debugging_requirements (002).pdf>_______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org <mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools <https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools>

1 0

Re: [mpiwg-tools] Meeting March 21, 2019
by Mohror, Kathryn 21 Mar '19

21 Mar '19

Hi all, Thanks Dick and John for sending your use cases and straw men! Here are a couple more docs I found from our discussions in the past: 1. Straw man from Anh Vo (mpi_debugging_requirements-AV-05-10-2016) 2. Needs from Allinea debugger as described by Chris January (mpi_debugging_requirements (002)) 3. Notes from Forum meeting 2/27/2017: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2017-02-27 Talk to you all soon! Kathryn From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org> On Behalf Of Dirk Schubert via mpiwg-tools Sent: Thursday, March 21, 2019 3:43 AM To: mpiwg-tools(a)lists.mpi-forum.org Cc: Dirk Schubert <Dirk.Schubert(a)arm.com> Subject: Re: [mpiwg-tools] Meeting March 21, 2019 Hi Kathryn, The following use cases are important for Arm Forge (debugger DDT and profiler MAP). Use case 1: Launch an MPI job under the control of a tool. Input: The user's regular command line to start the MPI job, for example "mpirun -n 256 ./wave_c". Why? To make it as friction-less as possible to use a tool. NB: The tool does not want to parse or interpret the MPI starter arguments, as the tool has no knowledge of all possible MPI starter arguments. Requirements: * Hold MPI processes for the tool to attach to them and to release them when attached. Q: Where are processes held? In MPI_Init or _start? * Spawn tool daemon on nodes where job is running. Q: Co-spawn or separate? Q: One per node or one per process? Q: Environment? Same general environment as MPI processes instead of restricted environment (e.g. $HOME not set or chroot). * Acquire the process table (rank, hostname, pid and executable path) of spawned MPI processes. Q: Only global proctable or additionally local (per node) proctable? * [Optional] Modify the environment of the MPI processes before launching, such as prepending to LD_PRELOAD or LD_LIBRARY_PATH to inject preloads into the MPI processes (but not other processes, such as MPI daemons or tool daemons). Q: When? It would be great if modifying the environment could be done just before the MPI processes are forked/exec'ed. Why? For our profiler MAP we need to preload an MPI specific PMPI library, but for some "MPI" starter processes such as srun we don't know the real MPI implementation that is used. Acquiring a partial process table (no PIDs) and inspecting the binaries pointed to by the executable paths, could allow us to detect the MPI implementation. * [Optional] Raise queue allocation request and granted events to allow the tool to disable startup timeouts temporarily while a queue allocation is in progress, as this can take a long time (applicable to srun). * [Optional] Scalable startup of tool daemons and shipping of preloads without touching the parallel file system on compute nodes. For example, pushing of daemon executable file (and dependencies plus preloads) to ram disk on compute nodes or Spindle? Q: Where files are pushed? A job specific temporary directory such as /tmp/mpi.job.1234/... or configurable? Q: How can the tool query the location of pushed files to reference them? Q: Who cleans up the files afterwards? The MPI job/starter itself or the tool? N: Files should be pushed with original file's permissions. NB: Optional requirement = A tool could ultimately work without it, but I very much would like to see it supported. In PMIx this is the "indirect" use case and Ralph Castain and I have been working together since a while to make sure this use case will be supported by PMIx (excluding some optional requirements for now). Additional questions/notes: * How will the handshake work, such that the MPI starter knows it's running under a tool and when it's possible for a tool to interact with the MPI starter process? * The handshake must support cases where the MPI starter filename that is provided by the user is a wrapper around the real MPI starter executable, for example XALT. * In PMIx both problems are solved with PMIX_LAUNCHER_PAUSE_FOR_TOOL=1 and PMIX_LAUNCHER_RENDEZVOUS_FILE=<filename>. * Will only the tool or also the tool daemons interact with the MPI job? * Security, for example USER B shall not be able to attach to MPI job of USER A. * Anything to consider for MPMD or heterogeneous systems? * Must be scalable. * Must not require a debugger. Use case 2: Attach to a running MPI job with a tool. This use case is a subset of use case 1 with the following requirements: * Acquire the process table of spawned MPI processes of MPI job. * Spawn tool daemon on nodes where MPI job is running. * [Optional] Scalable startup of tool daemons without touching the file system. Thanks, Dirk Dirk Schubert | Arm | Staff Software Engineer dirk.schubert(a)arm.com<mailto:[email protected]> allinea is now part of Arm ________________________________ From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]>> Sent: 17 March 2019 14:46 To: mpiwg-tools(a)lists.mpi-forum.org<mailto:[email protected]> Cc: Mohror, Kathryn Subject: [mpiwg-tools] Meeting March 21, 2019 Hey everybody, For our call this Thursday (3/21) we'll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don't think the EU has done so yet. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I'll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, https://people.llnl.gov/kathryn Data Analysis Group<https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 0

Re: [mpiwg-tools] Meeting March 21, 2019
by Dirk Schubert 21 Mar '19

21 Mar '19

Hi Kathryn, The following use cases are important for Arm Forge (debugger DDT and profiler MAP). Use case 1: Launch an MPI job under the control of a tool. Input: The user's regular command line to start the MPI job, for example "mpirun -n 256 ./wave_c". Why? To make it as friction-less as possible to use a tool. NB: The tool does not want to parse or interpret the MPI starter arguments, as the tool has no knowledge of all possible MPI starter arguments. Requirements: * Hold MPI processes for the tool to attach to them and to release them when attached. Q: Where are processes held? In MPI_Init or _start? * Spawn tool daemon on nodes where job is running. Q: Co-spawn or separate? Q: One per node or one per process? Q: Environment? Same general environment as MPI processes instead of restricted environment (e.g. $HOME not set or chroot). * Acquire the process table (rank, hostname, pid and executable path) of spawned MPI processes. Q: Only global proctable or additionally local (per node) proctable? * [Optional] Modify the environment of the MPI processes before launching, such as prepending to LD_PRELOAD or LD_LIBRARY_PATH to inject preloads into the MPI processes (but not other processes, such as MPI daemons or tool daemons). Q: When? It would be great if modifying the environment could be done just before the MPI processes are forked/exec’ed. Why? For our profiler MAP we need to preload an MPI specific PMPI library, but for some “MPI” starter processes such as srun we don’t know the real MPI implementation that is used. Acquiring a partial process table (no PIDs) and inspecting the binaries pointed to by the executable paths, could allow us to detect the MPI implementation. * [Optional] Raise queue allocation request and granted events to allow the tool to disable startup timeouts temporarily while a queue allocation is in progress, as this can take a long time (applicable to srun). * [Optional] Scalable startup of tool daemons and shipping of preloads without touching the parallel file system on compute nodes. For example, pushing of daemon executable file (and dependencies plus preloads) to ram disk on compute nodes or Spindle? Q: Where files are pushed? A job specific temporary directory such as /tmp/mpi.job.1234/... or configurable? Q: How can the tool query the location of pushed files to reference them? Q: Who cleans up the files afterwards? The MPI job/starter itself or the tool? N: Files should be pushed with original file’s permissions. NB: Optional requirement = A tool could ultimately work without it, but I very much would like to see it supported. In PMIx this is the “indirect” use case and Ralph Castain and I have been working together since a while to make sure this use case will be supported by PMIx (excluding some optional requirements for now). Additional questions/notes: * How will the handshake work, such that the MPI starter knows it's running under a tool and when it's possible for a tool to interact with the MPI starter process? * The handshake must support cases where the MPI starter filename that is provided by the user is a wrapper around the real MPI starter executable, for example XALT. * In PMIx both problems are solved with PMIX_LAUNCHER_PAUSE_FOR_TOOL=1 and PMIX_LAUNCHER_RENDEZVOUS_FILE=<filename>. * Will only the tool or also the tool daemons interact with the MPI job? * Security, for example USER B shall not be able to attach to MPI job of USER A. * Anything to consider for MPMD or heterogeneous systems? * Must be scalable. * Must not require a debugger. Use case 2: Attach to a running MPI job with a tool. This use case is a subset of use case 1 with the following requirements: * Acquire the process table of spawned MPI processes of MPI job. * Spawn tool daemon on nodes where MPI job is running. * [Optional] Scalable startup of tool daemons without touching the file system. Thanks, Dirk Dirk Schubert | Arm | Staff Software Engineer dirk.schubert(a)arm.com allinea is now part of Arm ________________________________ From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org> on behalf of Mohror, Kathryn via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org> Sent: 17 March 2019 14:46 To: mpiwg-tools(a)lists.mpi-forum.org Cc: Mohror, Kathryn Subject: [mpiwg-tools] Meeting March 21, 2019 Hey everybody, For our call this Thursday (3/21) we’ll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don’t think the EU has done so yet. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I’ll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, https://people.llnl.gov/kathryn Data Analysis Group<https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 0

Re: [mpiwg-tools] Meeting March 21, 2019
by John DelSignore 18 Mar '19

18 Mar '19

Hi Kathryn, I attached the slides I presented to this group in January of 2017. They are the so-called, "What I did on my XMas vacation" slides that recap work I did (with a lot of help from Ralph) to learn how PMIx might interact with a tool like TotalView. It also contains some thoughts about what the next generation of multi-process debugging interfaces might look like. Given the presentation's age and the fact that I slapped it together, I think it's fairly close to what's needed, but should serve only as a strawman. Slide 5 has a rough breakdown of the "layers" a tool needs. Refining that a bit, I think the layers / APIs are: * Job discovery. Not just for MPI. Allows the tool to find existing jobs. * Job submission, status, cancelation. Not just for MPI. Allows a tool to submit jobs to a batch/RM system, monitor job status, and cancel jobs. * Process discovery. Not just for MPI. Allows the tool to find existing or dynamically created processes in a job, and query proctable-like info (pids, nodes, and executables). * Tool daemon spawning. Not just for MPI. Allows a tool to spawn its daemon processes. * MPI task labeling. MPI-specific. Any time after process acquisition, the tool can discover a process's or thread's "role" as an MPI task, and "label" it accordingly. * MPI Message Queue Dumping. MPI-specific. Already have the MPI MQD API, not sure how or why we'd change it. * MPI Opaque Handles. MPI-specific. Open MPI / TotalView prototype (Jeff / Laust). Cheers, John D. From: mpiwg-tools <mpiwg-tools-bounces(a)lists.mpi-forum.org> On Behalf Of Mohror, Kathryn via mpiwg-tools Sent: Sunday, March 17, 2019 10:47 AM To: mpiwg-tools(a)lists.mpi-forum.org Cc: Mohror, Kathryn <mohror1(a)llnl.gov> Subject: [mpiwg-tools] Meeting March 21, 2019 Hey everybody, For our call this Thursday (3/21) we'll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don't think the EU has done so yet. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I'll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, https://people.llnl.gov/kathryn Data Analysis Group<https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA

1 0

Meeting March 21, 2019
by Mohror, Kathryn 17 Mar '19

17 Mar '19

Hey everybody, For our call this Thursday (3/21) we'll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don't think the EU has done so yet. Webex Info: https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php Agenda: In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I'll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this front. _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, https://people.llnl.gov/kathryn Data Analysis Group<https://computation.llnl.gov/casc/data-analysis-group> @ Lawrence Livermore National Laboratory, Livermore, CA, USA

1 0

No call on Mar 14, 2019
by Marc-Andre Hermanns 11 Mar '19

11 Mar '19

Dear all, Martin, Kathryn, and I will all be unavailable next Thursday, therefore we'll have to cancel this week's meeting. Cheers, Marc-Andre -- Dr. Marc-Andre Hermanns Jülich Aachen Research Alliance, High Performance Computing (JARA-HPC) Jülich Supercomputing Centre (JSC) Wilhelm-Johnen-Str. 52425 Jülich Germany Phone: +49 2461 61 2509 | +49 241 80 24381 Fax: +49 2461 80 6 99753 www.jara.org/jara-hpc email: hermanns(a)jara.rwth-aachen.de

1 0

Comments on MPI_T Events reading
by Martin Schulz 05 Mar '19

05 Mar '19

Hi all, Here are the comments from the MPI_T_Events reading. Martin Connected MPI processes -> line 43, page 622, get_info call -> need to tweak to include processes before init/after finalize same for line 26/27, page 623 -> requirement to return the same values line 8, page 626: add “during callback registration” Line 33, Page 626: rewrite “free-callback” Line 23, page 627: dropped handler >>callback<< function Line 30 (or so), page 627: need weasel language in case HW cannot even count dropped events -> best effort Line 43/44, page 627: wrong sentence, which allocation? Need to clarify Line 4, page 629: rewrite the sentence with “passes a pointer” Line 11, Page 629: space before “(pointer)” missing Line 16/17, Page 629: extent no longer returned Comment: why MPI_Count -> add what the value and behavior of a timestamp is -> or add forward timestamp at the bottom of page 629 Line 42, page 630, font for MPI not correct Line 44, page 630: MPI_T_SOURCE_GET_NUM -> need to pull description after the prototypes Line 7, Page 631, keep “num_sources” on one line Line 27, Page 631, add “source” / name of argument Line 39/40, line 17 and 20: page 631, missing type of argument in the prototype description Discussion on timestamps integer can roll-over if we don’t have enough bits 64 should be enough, but 32 bit can roll over add advice to users that problems can happen on true 32 bit platforms should specify what zero is, doesn’t have to be Linux zero time should we fix this to a particular time, so we have a fixed point for comparison but then the discussion turned away from that due to HW constraints still need to add definition of timestamps perhaps enough to say that things are comparable within a source Jeff added two small comments on GitHub -> need to look for that Comment om MPI_T_EVENT_GET_INFO (page 622, 37/38) who is allocating the info object and should we pass Info or a pointer if ignores, should we pass NULL or INFO_NULL should probably be NULL — Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Center (LRZ) Email: schulzm(a)in.tum.de

1 0