May 2010 - mpiwg-tools - mailman3.cels.anl.gov

Re: [Mpi3-tools] DRAFT of the MPIR Process Acquisition Interface document
by Greg Watson 15 May '10

15 May '10

Some comments/suggestions regarding the tool daemon launch extension below. I'm using "server process" to refer to the tool daemon server process and "MPI process" to refer to the MPI application process. 1. A number of debuggers and tools use the MPI runtime to launch their server processes rather than having to implement a second parallel launch mechanism. In lieu of standard infrastructure for launching tools, it makes sense to use the MPI runtime if possible. This approach is currently supported by Open MPI, MPICH2, PE and probably others as well. 2. The current extension relies on reading/writing memory in the starter process. This is adequate (although complicated) for debuggers, but does not work with other sorts of tools. In order to address this, I would like to see these features available from the command line as well, and would suggest a requirement that the tool daemon variables be able to be specified on the command line of the starter process. 3. The extension provides no control of where or how many server processes are launched. I presume the intention is one server process per MPI process, but this is not specified, and is probably not desirable on shared memory and other types of architectures. At a minimum, a "one server process per node" variable is desirable, such as "MPIR_one_server_per_node" or something similar. However, it may be desirable to make this completely general by allowing the use of the same process allocation mechanism provided by the MPI implementation. 4. The extension does not provide any mechanism for server processes to identify themselves or the MPI process(es) they are interested in. I would suggest a requirement that the each server command line or environment be supplied with (a) the MPI rank(s) of the MPI process(es) associated with each server; and (b) the PID's of the MPI process(es) if launched by the MPI runtime. 5. Some tools would prefer to start the MPI processes rather than attach to existing processes (which implies a ptrace requirement). I would suggest the optional variable "MPIR_external_spawn" be specified that indicates to the MPI runtime that the processes will be spawned externally. For this mode of launch, the identification information provided in (3) would be used by the MPI implementation to complete the MPI initialization, as well as the by tool. This variable would be available only on MPI implementations that support this startup mechanism. As mentioned, a number of MPI implementations already provide most of these features, so the burden of adding this support should not be great. Providing a standard mechanism for tool daemon launch would go a long way to addressing some of the tool infrastructure problems that affect most systems today. Regards, Greg On May 13, 2010, at 12:42 PM, John DelSignore wrote: > Hi all, > > Attached is the second DRAFT of the MPIR Process Acquisition Interface document for discussion during the 5/17/10 concall. > > I also included a copy with change tracking turned on so that you can see the changes. I incorporated most of the comments I received from Bill Gropp and Jeff Squyres on the first draft. > > TO DO items: > * Review this draft to see if people like it any better than the first draft. > * The document master is still in Word, and should to get converted to LaTex. I'd rather someone else do that, not me. > * This document describes the MPIR Process Acquisition Interface, but has a placeholder for the MPIR Message Queue Display Interface. I think all of the MQD stuff should be contained in a separate document, and this document should be restructured accordingly. > * The MQD stuff needs to get written up. > > Cheers, John D. > > <MPIR Process Acquisition Interface 2010-05-13.pdf><MPIR Process Acquisition Interface 2010-05-13 diffs.pdf>_______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools

1 0

[Mpi3-tools] Next WG Meeting 5/17: MPIR Discussion
by Martin Schulz 14 May '10

14 May '10

Hi all, Just as a reminder - the next MPI3 tools WG meeting will be on Monday 5/17 at 8am PDT, 11am EDT, 5pm MSZ. The main agenda point will be the discussion of the MPIR document that John sent around yesterday. The meeting link is: https://cisco.webex.com/cisco/j.php?ED=139132487&UID=0&PW=NNTM1ODViNjFk&RT=… Password: tools Talk to you on Monday, Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] DRAFT of the MPIR Process Acquisition Interface document
by John DelSignore 14 May '10

14 May '10

David Lecomber wrote: > Hi John, > > Thanks for circulating. > > I have one query - for the Tool Daemon Launch Extension as described, we > get a binary executed out on the nodes with arguments.. Could we/should > we push for more? Remember, this paper attempts to describe where we are, not where we want to be. It describes current practice. "Know from whence you came. If you know whence you came, there are absolutely no limitations to where you can go." -- James Baldwin > My preference - and I think other other tools would benefit - would be > for having arbitrary files shipped out to compute nodes, as well as > optionally executing these files. Sure, that sounds useful, but the MPIR interface as it stands now has no provision for that, and I'm not aware of any vendor extension to MPIR that provides that functionality. So... if that's what is wanted, someone will need to write a separate proposal, implement it, and get it through the standardization proecss. > This would allow rapid sending of shared libraries, topology files, etc. > - which could really help scalability in the case where a system vendor > has specific capability for this in their existing tree network. You must be thinking of the capability on Cray XT systems. > In the case of a large but ordinary cluster, for example, it would be > preferable not to take shared libs and other daemons or config files > from a shared filesystem, but for the remote nodes to be able to read > them from their own /tmp or equivalent local storage having had them > delivered by the MPI's own daemons. Again, sounds like what Cray XT does. I'm not sure if other system support a similar arrangement. But one thing to be wary of, at least on Cray, is that "/tmp" on the compute node is a memory disk which transparently steals memory from the user's application. So, the bigger the files in /tmp the less memory available for the application, and users don't like to "share" memory (no pun intended). So, there are limits to how much stuff can be copied to the compute nodes. > Of course, you could do all this already by appending to your binary > that file you want to send out with it, but that does feel a bit dirty! Or you can use a tool like makeself.sh to create a self extracting .run file. Cheers, John D. > I'll be on the call next week I hope - but thought I'd just raise the > question now, and let folks ponder it first. > > Best wishes > David >

1 0

Re: [Mpi3-tools] DRAFT of the MPIR Process Acquisition Interface document
by David Lecomber 14 May '10

14 May '10

Hi John, Thanks for circulating. I have one query - for the Tool Daemon Launch Extension as described, we get a binary executed out on the nodes with arguments.. Could we/should we push for more? My preference - and I think other other tools would benefit - would be for having arbitrary files shipped out to compute nodes, as well as optionally executing these files. This would allow rapid sending of shared libraries, topology files, etc. - which could really help scalability in the case where a system vendor has specific capability for this in their existing tree network. In the case of a large but ordinary cluster, for example, it would be preferable not to take shared libs and other daemons or config files from a shared filesystem, but for the remote nodes to be able to read them from their own /tmp or equivalent local storage having had them delivered by the MPI's own daemons. Of course, you could do all this already by appending to your binary that file you want to send out with it, but that does feel a bit dirty! I'll be on the call next week I hope - but thought I'd just raise the question now, and let folks ponder it first. Best wishes David -- David Lecomber, CTO, Allinea Software +44 7980 863033 Allinea Software Limited: Registered in England and Wales No: 6871298 The Innovation Centre, Warwick Technology Park, Warwick, CV34 6UW, UK

1 0

[Mpi3-tools] DRAFT of the MPIR Process Acquisition Interface document
by John DelSignore 13 May '10

13 May '10

Hi all, Attached is the second DRAFT of the MPIR Process Acquisition Interface document for discussion during the 5/17/10 concall. I also included a copy with change tracking turned on so that you can see the changes. I incorporated most of the comments I received from Bill Gropp and Jeff Squyres on the first draft. TO DO items: * Review this draft to see if people like it any better than the first draft. * The document master is still in Word, and should to get converted to LaTex. I'd rather someone else do that, not me. * This document describes the MPIR Process Acquisition Interface, but has a placeholder for the MPIR Message Queue Display Interface. I think all of the MQD stuff should be contained in a separate document, and this document should be restructured accordingly. * The MQD stuff needs to get written up. Cheers, John D.

1 0