January 2009 - mpiwg-tools - mailman3.cels.anl.gov

Re: [Mpi3-tools] Notes from today's telecon
by Martin Schulz 14 Jan '09

14 Jan '09

Hi Bronis, all, Thanks for the extensive notes - they are now available on the Wiki. Martin At 09:00 AM 1/12/2009, Bronis R. de Supinski wrote: >All: > >Here are my notes from today's telecon. > >Participants: > >Dong Ahn (LLNL) >Chih-Ping Chen (Intel) >John DelSignore (TotalView Tech) >Bronis de Supinski (LLNL) >Chris Gottbrath (TotalView Tech) >Marty Itzkowitz (Sun) >Bob Moench (Cray) >Phil Roth (ORNL) >Martin Schulz (LLNL) >Jeff Squyres (Cisco) > >Notes: > >Chris has added stuff to the variables section and >some other edits to the evolving MPIR document: > >https:// svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools/mpir-current > >For now, they are emphasized by being preceded by >Chris's name. > >Still need to clean up the document, as per John's >email -- we need things like types for example. It >is still pretty incomplete and has a lot of ambiguities. >TotalView will try to fill it in since they have a >lot of the historical knowledge of what has happened. >They see lots of different implementations and styles >for using the interface. That will help us understand >what we what do going forward, including extending it >for MPI-2 constructs like dynamic processes. > >What form do we want to maintain the document in? The >simplest thing is to leave it as a document on the Wiki. >We will refer to that in the MPI-2.2 standard. The plan >is to try to get a revised and improved interface into >MPI-3. John's model is based on Dwarf -- have an author >and an editor and a list of contributors; we probably >don't need an editor given the shorter term nature of >this document. John will try to come up with a Word >document that captures the historical perspective. >We could use the MPI subversion repo. The revised >plan is that John will get an initial draft together, >either as a text document or in Word and then Martin >will transform it into LaTex using the MPI standard >style sheets. John will check with his management in >order to determine the time frame in which it can be >completed. He esitmates a month or two. The overall >goal is to have the final version done by the time >MPI-2.2 is fully approved (September 2009?). > >Chris presented the Euro PVM/MPI paper on their proposal. >The paper can be accessed through this link: > >http:// www. >open-mpi.org/papers/euro-pvmmpi-2006-mpi2-debugging/euro-pvmmpi-2006-mpi2-d… > >Tis interface supports dynamic processes. It was also >designed to handle large scales by allowing the proctable >information to be distributed. Chris sent his slides >to the tools list; someone will make sure they are made >available on the Wiki. > >Subset attach is an important consideration, as is the >naming of nodes, which becomes more complicated with >dynamic processes. Must support Spawn, Connect/Acept >and Join. Overhead needs to be low enough that MPI >can support it during normal job operations so that >user can attach to a running job ("on by default"). > >Distributes the process table so debugger (may) need to >reassemble it. Dong: Does this imply that debugging >launching will be hierarchical? The process of the >debugger beoming attached to processes may be multi-step, >which may not need to imply hierarchical launching. >Can the debugger use resource manager provided >functionality to do bulk launching? Ability to distribute >the proctable does not preclude the possibility of >having a centralized table. It is important, perhaps >even more important, to have the appropriate support >in the runtime system, whether it is the MPI runtime >or the job manager (the tool doesn't care where it >lives as long as the facilities are there). Launching >the servers, establishing communication and finding >the topology of the machine is currently ad hoc. A >good thing would be for the standard to encompass >these aspects. > >The goal should be to define an API and leave the >implementation open. This is consistent with the >MPI standard in general. It needs to be more of an >abstraction that the tool calls for the information. > >An original design constraint was that the interface >cannot require the target to make progress. A DLL >that the tool can load and call should be able to >provide the needed abstractions. John thinks we may >be going down the wrong path if we consider what >Chris is presenting an interface since it might >be easier not to require the table be distributed. >The original interface had a problem in that it >dictated the implementation a little too much. > >Interface should give information on what the job >looks like. John thinks it also should give services >like spawning the daemons and establishing communication >between them. That would provide an interface we could >point at that would mean implementers can support >their tools in a scalable fashion. > >Chris's presentation provides a good implementation >that can help us design the API that we need. Chris >agrees that we do not yet have the API written down. >MIght be useful to have two interfaces: one that >targets a small scale system in which a central RM >manages all the dynamic process operations and can >store all of the information in one place; and one >that is more scalable and supports the distributed >table in the Euor PVM/MPI paper. > >Synchronization is an important consideration -- allow >user to debug job early in a well defined state. > >Prcoess table entries -- probably things that need >to be accessible through any API. > >Process naming is an important consideration -- you >want to have a notion of repeatable naming for scripting. >If you run the program twice in the same way and it >is deterministic then you should beable to interact >with it through the debugger in the same way. With >dynamic processes, there is no universal name space. >There are several MPI_COMM_WORLDs; a notion of >MPI_COMM_UNIVERSE would be useful but is not consistent >with the design of MPI dynamic processes. Thus, a >hierarchical naming scheme is probably necessary. > >Different levels of notification -- user may want to >know about all spawns but they might not. The notification >mechanism should minimize the impact on MPI processes. > >TotalView was developing an implementation in collaboration >with MPICH and OpenMPI. TotalView did the first stages >of its implementation (not completed but workable). No >one has implemented it on the MPI side yet. > > > >_______________________________________________ >Mpi3-tools mailing list >Mpi3-tools(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

[Mpi3-tools] Notes from today's telecon
by Bronis R. de Supinski 12 Jan '09

12 Jan '09

All: Here are my notes from today's telecon. Participants: Dong Ahn (LLNL) Chih-Ping Chen (Intel) John DelSignore (TotalView Tech) Bronis de Supinski (LLNL) Chris Gottbrath (TotalView Tech) Marty Itzkowitz (Sun) Bob Moench (Cray) Phil Roth (ORNL) Martin Schulz (LLNL) Jeff Squyres (Cisco) Notes: Chris has added stuff to the variables section and some other edits to the evolving MPIR document: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools/mpir-current For now, they are emphasized by being preceded by Chris's name. Still need to clean up the document, as per John's email -- we need things like types for example. It is still pretty incomplete and has a lot of ambiguities. TotalView will try to fill it in since they have a lot of the historical knowledge of what has happened. They see lots of different implementations and styles for using the interface. That will help us understand what we what do going forward, including extending it for MPI-2 constructs like dynamic processes. What form do we want to maintain the document in? The simplest thing is to leave it as a document on the Wiki. We will refer to that in the MPI-2.2 standard. The plan is to try to get a revised and improved interface into MPI-3. John's model is based on Dwarf -- have an author and an editor and a list of contributors; we probably don't need an editor given the shorter term nature of this document. John will try to come up with a Word document that captures the historical perspective. We could use the MPI subversion repo. The revised plan is that John will get an initial draft together, either as a text document or in Word and then Martin will transform it into LaTex using the MPI standard style sheets. John will check with his management in order to determine the time frame in which it can be completed. He esitmates a month or two. The overall goal is to have the final version done by the time MPI-2.2 is fully approved (September 2009?). Chris presented the Euro PVM/MPI paper on their proposal. The paper can be accessed through this link: http://www.open-mpi.org/papers/euro-pvmmpi-2006-mpi2-debugging/euro-pvmmpi-… Tis interface supports dynamic processes. It was also designed to handle large scales by allowing the proctable information to be distributed. Chris sent his slides to the tools list; someone will make sure they are made available on the Wiki. Subset attach is an important consideration, as is the naming of nodes, which becomes more complicated with dynamic processes. Must support Spawn, Connect/Acept and Join. Overhead needs to be low enough that MPI can support it during normal job operations so that user can attach to a running job ("on by default"). Distributes the process table so debugger (may) need to reassemble it. Dong: Does this imply that debugging launching will be hierarchical? The process of the debugger beoming attached to processes may be multi-step, which may not need to imply hierarchical launching. Can the debugger use resource manager provided functionality to do bulk launching? Ability to distribute the proctable does not preclude the possibility of having a centralized table. It is important, perhaps even more important, to have the appropriate support in the runtime system, whether it is the MPI runtime or the job manager (the tool doesn't care where it lives as long as the facilities are there). Launching the servers, establishing communication and finding the topology of the machine is currently ad hoc. A good thing would be for the standard to encompass these aspects. The goal should be to define an API and leave the implementation open. This is consistent with the MPI standard in general. It needs to be more of an abstraction that the tool calls for the information. An original design constraint was that the interface cannot require the target to make progress. A DLL that the tool can load and call should be able to provide the needed abstractions. John thinks we may be going down the wrong path if we consider what Chris is presenting an interface since it might be easier not to require the table be distributed. The original interface had a problem in that it dictated the implementation a little too much. Interface should give information on what the job looks like. John thinks it also should give services like spawning the daemons and establishing communication between them. That would provide an interface we could point at that would mean implementers can support their tools in a scalable fashion. Chris's presentation provides a good implementation that can help us design the API that we need. Chris agrees that we do not yet have the API written down. MIght be useful to have two interfaces: one that targets a small scale system in which a central RM manages all the dynamic process operations and can store all of the information in one place; and one that is more scalable and supports the distributed table in the Euor PVM/MPI paper. Synchronization is an important consideration -- allow user to debug job early in a well defined state. Prcoess table entries -- probably things that need to be accessible through any API. Process naming is an important consideration -- you want to have a notion of repeatable naming for scripting. If you run the program twice in the same way and it is deterministic then you should beable to interact with it through the debugger in the same way. With dynamic processes, there is no universal name space. There are several MPI_COMM_WORLDs; a notion of MPI_COMM_UNIVERSE would be useful but is not consistent with the design of MPI dynamic processes. Thus, a hierarchical naming scheme is probably necessary. Different levels of notification -- user may want to know about all spawns but they might not. The notification mechanism should minimize the impact on MPI processes. TotalView was developing an implementation in collaboration with MPICH and OpenMPI. TotalView did the first stages of its implementation (not completed but workable). No one has implemented it on the MPI side yet.

1 0

[Mpi3-tools] MPI 2 dynamic process talk from 2006 Euro PVM MPI
by Chris Gottbrath 12 Jan '09

12 Jan '09

Chris Gottbrath, 508-652-7735 Director of Product Management, TotalView Technologies chris.gottbrath(a)totalviewtech.com -- Learn how to radically simplify your debugging: http://www.totalviewtech.com/support/white_papers.html?id=163

1 0

Re: [Mpi3-tools] Next TelCon 1/12 8am PT, 11am ET, 5pm MEZ
by Martin Schulz 11 Jan '09

11 Jan '09

Hi John, >* I think the current MPIR description on the Wiki needs a lot more >detail. I think it needs to include a "C" language description that >shows the declarations of the structure types, variables, and >functions. It's missing type and variable declarations. At the next Agreed. We need to add that. Do you agree with the general workflow, though? Is this how it works in TV? > higher level of detail, it needs to elaborate on the semantics of > each of the variables and functions, both from the debugger's and > MPI perspective; it currently very thin. At the highest level of > detail, it should talk about the different "modes" of useing the > interface. For example, the "rank 0 process launches" model used by > MPICH 1, vs. the "separate mpirun starter process launches" model > used by most other MPI implementations. Shouldn't the interface hide this from the tool? The only piece of information that should be needed is the initial process the tool needs to attach to. Enumerating the modes could result in leaving out implementations that we are not aware of. > Also, there are special conventions and used on various MPI > implementations. For example, Blue Gene (and now Open MPI) support > the notion of co-spawning tool daemons, and several implementations > support the notion of partial (scalable subset) attach. Yes, we need to document those. We talked about this during the forum, but nobody there had enough information at hand to list all of these extensions. We know about the BG/L ones, but we weren't sure about what else is there. If you have this information, it would be great if you could add this to the Wiki. >* On Monday, I think we need to figure out who's going to be the >principle author of the document and who is going to edit/maintain >the document. If there is no other volunteer (is there :)?), I can be the central person to collect the information and to write and maintain the document. However, the information must come from the MPI implementors as well as the tools that already use the interface. I also contacted the O|SS developers, who are using the interface, but most of the usage is probably in TV and hence it would be great if someone from TVT could document all the special operations you had to implement for different MPI implementations. Martin >Cheers, John D. > > >Martin Schulz wrote: > > Hi all, > > > > Just as a reminder: we have our next MPI Tools WG TelCon on Monday > > 1/12 at the usual time at 8am PT, 11am ET, 5pm MEZ. Here is the > > dial-in information: > > > >> Toll Free Dial In Number for Offsite Participants: 866-914-3976 > >> International Access/Caller Paid Dial In Number: 925-424-8105 > >> Dial In Number for LLNL Onsite Participants: 4-8105 > >> > >> Access Security Code: 860609 # > > > > Unfortunately, I won't be able to be there for the whole TelCon > > since I will try to call-in from on the road. Bronis has agreed > > to lead the TelCon instead. The main topics will be to continue > > the discussion of the current state of the MPIR interface as well > > as a presentation by Chris on the MPIR-2 interface, which includes > > dynamic process support. > > > > Please also take a look at the current MPIR description on > > the Wiki and suggest changes or corrections. The plan is to > > convert the Wiki in the coming weeks into a standalone document > > that we can reference in the MPI 2.2 standard. In particular, > > we still need to document existing, platform specific extensions > > to MPIR. If you are aware of such an extension, please add it to > > the Wiki page. > > > > Thanks, > > > > Martin > > > > > > > > _______________________________________________________________________ > > Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 > > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > _______________________________________________ > > Mpi3-tools mailing list > > Mpi3-tools(a)lists.mpi-forum.org > > http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools > > > > >_______________________________________________ >Mpi3-tools mailing list >Mpi3-tools(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Next TelCon 1/12 8am PT, 11am ET, 5pm MEZ
by Jeff Squyres 10 Jan '09

10 Jan '09

There's a link on the front page of the wiki for getting new accounts: https://svn.mpi-forum.org/wiki/start On Jan 10, 2009, at 2:16 PM, John DelSignore wrote: > Hi Jeff, > > Thanks for the help... > > I guess I lost my username too, since I tried every permutation of > my email address and username I could think of, and it tells me > "Sorry, we can't find this user in our database." > > Or, maybe I never had a username, since I'm usually pretty good at > tracking account information, and I have no record of having an MPI3 > Wiki account. > > Thanks, John D. > > > Jeff Squyres wrote: >> On Jan 10, 2009, at 9:20 AM, John DelSignore wrote: >> >>> Hi Martin, >>> >>> A few things... >>> >>> * I should probably know this, but I probably misplaced the >>> information. How do edit the Wiki page? I tried logging in with my >>> MPI3 Tools password, but it failed. >> >> If you have forgotten your wiki password, use this link (it's on the >> front page of the wiki as well): >> >> https://svn.mpi-forum.org/wiki/start?do=resendpwd >> >> I apologize; we do have somewhat of a weird wiki setup. The >> usernames/passwords are maintained by a separate system (not the wiki >> itself). It's a little confusing. >> >> --Jeff Squyres >> Cisco Systems >> >> _______________________________________________ >> Mpi3-tools mailing list >> Mpi3-tools(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >> >> > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi3-tools] Next TelCon 1/12 8am PT, 11am ET, 5pm MEZ
by John DelSignore 10 Jan '09

10 Jan '09

Hi Jeff, Thanks for the help... I guess I lost my username too, since I tried every permutation of my email address and username I could think of, and it tells me "Sorry, we can't find this user in our database." Or, maybe I never had a username, since I'm usually pretty good at tracking account information, and I have no record of having an MPI3 Wiki account. Thanks, John D. Jeff Squyres wrote: > On Jan 10, 2009, at 9:20 AM, John DelSignore wrote: > >> Hi Martin, >> >> A few things... >> >> * I should probably know this, but I probably misplaced the >> information. How do edit the Wiki page? I tried logging in with my >> MPI3 Tools password, but it failed. > > If you have forgotten your wiki password, use this link (it's on the > front page of the wiki as well): > > https://svn.mpi-forum.org/wiki/start?do=resendpwd > > I apologize; we do have somewhat of a weird wiki setup. The > usernames/passwords are maintained by a separate system (not the wiki > itself). It's a little confusing. > > --Jeff Squyres > Cisco Systems > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools > >

1 0

Re: [Mpi3-tools] Next TelCon 1/12 8am PT, 11am ET, 5pm MEZ
by Jeff Squyres 10 Jan '09

10 Jan '09

On Jan 10, 2009, at 9:20 AM, John DelSignore wrote: > Hi Martin, > > A few things... > > * I should probably know this, but I probably misplaced the > information. How do edit the Wiki page? I tried logging in with my > MPI3 Tools password, but it failed. If you have forgotten your wiki password, use this link (it's on the front page of the wiki as well): https://svn.mpi-forum.org/wiki/start?do=resendpwd I apologize; we do have somewhat of a weird wiki setup. The usernames/ passwords are maintained by a separate system (not the wiki itself). It's a little confusing. -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi3-tools] Next TelCon 1/12 8am PT, 11am ET, 5pm MEZ
by John DelSignore 10 Jan '09

10 Jan '09

Hi Martin, A few things... * I should probably know this, but I probably misplaced the information. How do edit the Wiki page? I tried logging in with my MPI3 Tools password, but it failed. * I think the current MPIR description on the Wiki needs a lot more detail. I think it needs to include a "C" language description that shows the declarations of the structure types, variables, and functions. It's missing type and variable declarations. At the next higher level of detail, it needs to elaborate on the semantics of each of the variables and functions, both from the debugger's and MPI perspective; it currently very thin. At the highest level of detail, it should talk about the different "modes" of useing the interface. For example, the "rank 0 process launches" model used by MPICH 1, vs. the "separate mpirun starter process launches" model used by most other MPI implementations. Also, there are special conventions and used on various MPI implementations. For example, Blue Gene (and now Open MPI) support the notion of co-spawning tool daemons, and several implementations support the notion of partial (scalable subset) attach. * On Monday, I think we need to figure out who's going to be the principle author of the document and who is going to edit/maintain the document. Cheers, John D. Martin Schulz wrote: > Hi all, > > Just as a reminder: we have our next MPI Tools WG TelCon on Monday > 1/12 at the usual time at 8am PT, 11am ET, 5pm MEZ. Here is the > dial-in information: > >> Toll Free Dial In Number for Offsite Participants: 866-914-3976 >> International Access/Caller Paid Dial In Number: 925-424-8105 >> Dial In Number for LLNL Onsite Participants: 4-8105 >> >> Access Security Code: 860609 # > > Unfortunately, I won't be able to be there for the whole TelCon > since I will try to call-in from on the road. Bronis has agreed > to lead the TelCon instead. The main topics will be to continue > the discussion of the current state of the MPIR interface as well > as a presentation by Chris on the MPIR-2 interface, which includes > dynamic process support. > > Please also take a look at the current MPIR description on > the Wiki and suggest changes or corrections. The plan is to > convert the Wiki in the coming weeks into a standalone document > that we can reference in the MPI 2.2 standard. In particular, > we still need to document existing, platform specific extensions > to MPIR. If you are aware of such an extension, please add it to > the Wiki page. > > Thanks, > > Martin > > > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools > >

1 0

[Mpi3-tools] Next TelCon 1/12 8am PT, 11am ET, 5pm MEZ
by Martin Schulz 10 Jan '09

10 Jan '09

Hi all, Just as a reminder: we have our next MPI Tools WG TelCon on Monday 1/12 at the usual time at 8am PT, 11am ET, 5pm MEZ. Here is the dial-in information: > Toll Free Dial In Number for Offsite Participants: 866-914-3976 > International Access/Caller Paid Dial In Number: 925-424-8105 > Dial In Number for LLNL Onsite Participants: 4-8105 > > Access Security Code: 860609 # Unfortunately, I won't be able to be there for the whole TelCon since I will try to call-in from on the road. Bronis has agreed to lead the TelCon instead. The main topics will be to continue the discussion of the current state of the MPIR interface as well as a presentation by Chris on the MPIR-2 interface, which includes dynamic process support. Please also take a look at the current MPIR description on the Wiki and suggest changes or corrections. The plan is to convert the Wiki in the coming weeks into a standalone document that we can reference in the MPI 2.2 standard. In particular, we still need to document existing, platform specific extensions to MPIR. If you are aware of such an extension, please add it to the Wiki page. Thanks, Martin _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Tools WG teleconf today?
by Jeff Squyres 05 Jan '09

05 Jan '09

I blame Outlook. :-) See everyone on the 12th... On Jan 5, 2009, at 11:19 AM, Bettina Krammer wrote: > Hello, > > Jeff Squyres wrote: >> I dialed in late (11:10am US Eastern) and am the only one on the >> line. Is there no teleconf today? >> > Martin Schulz wrote: >> >> >> The next TelCon dates are: >> >> 1/12/09 - (hopefully final) discussion on the MPIR-1 interface with >> all comments included plus a quick overview over MPIR-2 >> ideas by Chris Gottbrath >> 2/2/09 - MPI state and information query interfaces > > Bettina > > -- > Bettina Krammer > Technical Director of LRC ITACA > Laboratoire PRiSM > Universitï¿½ de Versailles Saint-Quentin-en-Yvelines > 45, Av. des Etats-Unis > 78000 Versailles > Tel: (+33) 139 253 615 > Email: bettina.krammer(a)prism.uvsq.fr > Web: http://www.prism.uvsq.fr > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- Jeff Squyres Cisco Systems

1 0