December 2008 - mpiwg-tools - mailman3.cels.anl.gov

[Mpi3-tools] Meeting notes 12/22
by Martin Schulz 22 Dec '08

22 Dec '08

Hi all, The meeting notes for today's (short) meeting are available on the WiKi. The main topic was an update on the MPIR discussion we had during the meeting. In general, we noticed that we need more input from other parties (tool and MPI developers) to ensure that we are capturing all details. If you have contact to people that have an interest/knowledge in this issue, but are not participating already (e.g., other development groups in your organization), please approach them and ask them for feedback. It would be great if we could wrap up this MPIR-1 discussion as soon as possible. The next TelCon dates are: 1/12/09 - (hopefully final) discussion on the MPIR-1 interface with all comments included plus a quick overview over MPIR-2 ideas by Chris Gottbrath 2/2/09 - MPI state and information query interfaces Happy Holidays! Martin _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Meeting notes from 12/15 and next meetings (12/22 and 1/12)
by Chris Gottbrath 22 Dec '08

22 Dec '08

Martin, I can be on the call today but I'm not prepared to answer technical questions from TotalView Tech's perspective about the interface. What I can do is collect questions and have responses for the next meeting. Cheers, Chris Chris Gottbrath, 508-652-7735 Director of Product Management, TotalView Technologies chris.gottbrath(a)totalviewtech.com -- Learn how to radically simplify your debugging: http://www.totalviewtech.com/support/white_papers.html?id=163 On Dec 20, 2008, at 3:52 PM, Martin Schulz wrote: > Hi all, > > We had a tools WG meeting during the MPI forum meeting last week > at Menlo Park. The notes and slides from this meeting are now > available on the Tools WG Wiki. > > The main part was a discussion on how the MPIR interface is > currently implemented and used. The outcome of this discussion > is also on the Wiki, but needs further reviews, especially from > the tools community (in particular debugger developers). > > We will try to hold a meeting next Monday 12/22 at 8am PT, > 11am ET, 5pm MEZ to give people the chance to comment on this > MPIR discussion and to refine the document. Since not many > people might be around, we will hold off on any other topic > until the new year. The next regular TelCon will be on Monday > 1/12 at the same time and the next topic (after the MPIR > interface) will be a discussion on Marty's MPI state proposal > as well as the performance information interface ideas. > > Also, if anyone can't make it on Monday, please send comments > via email. It would be great if we could come to a conclusion > on the MPIR document rather sooner than later. > > Thanks (and in case you can't make it Monday: Happy Holidays!), > > Martin > > > PS: The Wiki is at: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools > > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools

1 0

Re: [Mpi3-tools] Meeting notes from 12/15 and next meetings (12/22 and 1/12)
by Martin Schulz 22 Dec '08

22 Dec '08

At 07:41 AM 12/22/2008, Marty Itzkowitz wrote: >Have we shifted phase, so that we meet every two weeks starting >on 1/12? > >I assume so. Yes, we have - at least for this time. Otherwise, due to various holidays, our next meeting would have been in February. If this doesn't work for people, we can try to shift back after January. Martin > Marty > >On 12/20/08 12:52 PM, Martin Schulz wrote: >>Hi all, >> >>We had a tools WG meeting during the MPI forum meeting last week >>at Menlo Park. The notes and slides from this meeting are now >>available on the Tools WG Wiki. >> >>The main part was a discussion on how the MPIR interface is >>currently implemented and used. The outcome of this discussion >>is also on the Wiki, but needs further reviews, especially from >>the tools community (in particular debugger developers). >> >>We will try to hold a meeting next Monday 12/22 at 8am PT, >>11am ET, 5pm MEZ to give people the chance to comment on this >>MPIR discussion and to refine the document. Since not many >>people might be around, we will hold off on any other topic >>until the new year. The next regular TelCon will be on Monday >>1/12 at the same time and the next topic (after the MPIR >>interface) will be a discussion on Marty's MPI state proposal >>as well as the performance information interface ideas. >> >>Also, if anyone can't make it on Monday, please send comments >>via email. It would be great if we could come to a conclusion >>on the MPIR document rather sooner than later. >> >>Thanks (and in case you can't make it Monday: Happy Holidays!), >> >>Martin >> >> >>PS: The Wiki is at: >> >>https:// svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools >> >> >>_______________________________________________________________________ >>Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 >>CASC @ Lawrence Livermore National Laboratory, Livermore, USA >>_______________________________________________ >>Mpi3-tools mailing list >>Mpi3-tools(a)lists.mpi-forum.org >>http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >_______________________________________________ >Mpi3-tools mailing list >Mpi3-tools(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Meeting notes from 12/15 and next meetings (12/22 and 1/12)
by Marty Itzkowitz 22 Dec '08

22 Dec '08

Have we shifted phase, so that we meet every two weeks starting on 1/12? I assume so. Marty On 12/20/08 12:52 PM, Martin Schulz wrote: > Hi all, > > We had a tools WG meeting during the MPI forum meeting last week > at Menlo Park. The notes and slides from this meeting are now > available on the Tools WG Wiki. > > The main part was a discussion on how the MPIR interface is > currently implemented and used. The outcome of this discussion > is also on the Wiki, but needs further reviews, especially from > the tools community (in particular debugger developers). > > We will try to hold a meeting next Monday 12/22 at 8am PT, > 11am ET, 5pm MEZ to give people the chance to comment on this > MPIR discussion and to refine the document. Since not many > people might be around, we will hold off on any other topic > until the new year. The next regular TelCon will be on Monday > 1/12 at the same time and the next topic (after the MPIR > interface) will be a discussion on Marty's MPI state proposal > as well as the performance information interface ideas. > > Also, if anyone can't make it on Monday, please send comments > via email. It would be great if we could come to a conclusion > on the MPIR document rather sooner than later. > > Thanks (and in case you can't make it Monday: Happy Holidays!), > > Martin > > > PS: The Wiki is at: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools > > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools

1 0

[Mpi3-tools] Meeting notes from 12/15 and next meetings (12/22 and 1/12)
by Martin Schulz 20 Dec '08

20 Dec '08

Hi all, We had a tools WG meeting during the MPI forum meeting last week at Menlo Park. The notes and slides from this meeting are now available on the Tools WG Wiki. The main part was a discussion on how the MPIR interface is currently implemented and used. The outcome of this discussion is also on the Wiki, but needs further reviews, especially from the tools community (in particular debugger developers). We will try to hold a meeting next Monday 12/22 at 8am PT, 11am ET, 5pm MEZ to give people the chance to comment on this MPIR discussion and to refine the document. Since not many people might be around, we will hold off on any other topic until the new year. The next regular TelCon will be on Monday 1/12 at the same time and the next topic (after the MPIR interface) will be a discussion on Marty's MPI state proposal as well as the performance information interface ideas. Also, if anyone can't make it on Monday, please send comments via email. It would be great if we could come to a conclusion on the MPIR document rather sooner than later. Thanks (and in case you can't make it Monday: Happy Holidays!), Martin PS: The Wiki is at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] MPI State Profiling Proposal
by Martin Schulz 15 Dec '08

15 Dec '08

Hi Marty, all, I added this to the wiki - it is now at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools/state Martin At 02:16 PM 12/12/2008, you wrote: >I'm attaching a draft of the MPI State Profiling description. Can someone >please add it in the right place to the website in time for the >meeting Monday? > > Thanks, > Marty > > > > >MPI State Profiling > > > > > >Marty Itzkowitz > > > > > >Sun Microsystems > > > > > >Introduction > > > >Most current profiling for MPI applications is based on tracing API >calls into the MPI runtime library with, for example, VampirTrace. >The data collected shows API calls and the messages that are sent >and received. In general, all trace data from all ranks is needed in >order to be able to match sends and receives, and that causes high >data volume and other scalability issues. > >We are here proposing an additional means of profiling MPI >applications based on statistical sampling of the MPI runtime state. >It has improved scalability properties relative to MPI API tracing, >and supports selectivity. > >The proposal described here has been implemented in Sun's >ClusterTools 8.1 product, based on Open MPI. Sun will be happy to >contribute the code back to the Open MPI tree. > > > >MPI State Profiling > > > >The basic idea is to have the MPI runtime maintain a per-thread >state variable, and provide an asynchronous-signal-safe API for a >thread to read its own state. In the prototype implementation, there >are three states: >Not in MPI >MPI-Work (In the MPI runtime, and working) >MPI-Stall (In the MPI runtime, but stalled for some reason) >The interface, as prototyped, only has one state for MPI-Stall, no >matter why the runtime is stalled. It would be a simple extension to >define multiple MPI-Stall states, one for each reason it is stalled. > >MPI-Stall Time is accumulated when the thread is stalled within MPI, >whether the thread is busy-waiting (when the process is also >accumulating User CPU Time), as well as when the process is >sleep-waiting (when the process is not accumulating User CPU Time). >In other words, MPI-Stall time is a statistical measure of when the >process is blocked inside the MPI library. > >Note that MPI-Stall time is not a measure of time spent in the >MPI_Wait call. Some of the time in that call is MPI-Work time, and >some of the time spent in other calls may be MPI-Stall time. > >MPI-Work means that the runtime is doing something other than >waiting. It is not necessarily directly-useful work from the point >of view of the application -- it may be what the user would think of >as overhead. > >A profiler can then read the state with each profile tick, and >accumulate a statistical measure of time spent in MPI-Work and time >spent in MPI-Stall. If callstacks are recorded at the same time as >the state is captured, that MPI state can be attributed to the >frames in the callstack, allowing the user to see which MPI calls >are spending the most time stalled, and how the user got to that call. > > > >Runtime Implementation > > > >Implementation requires three things: the maintenance of the state, >an API to read it, and a rendezvous mechanism for the profiler to >find the API. > >Maintenance of the state is straightforward. All states start out in >the Not-in-MPI state. Whenever an API call is entered, the state for >the calling thread is changed to MPI-Work; whenever an API call >returns, the state is changed back to Not-in-MPI. During the >processing, whenever the OMPI progress engine determines that no >progress is being made on any outstanding requests, the state is >changed to MPI-Stall; Whenever the progress engine detects progress >being made, the state is changed back to MPI-Work. The work to wait >transition points are very implementation-specific, and must be done >carefully to avoid high overhead. > >The API is quite simple. There is only one function: >OMPI_state_t (*mpi_collector_api)(void*) > >The return value is an enum representing the current state for the >calling thread. That function must be asynchronous-signal-safe, so >that it can be called from a signal handler processing a profile tick. > >Rendezvous is also simple. The profile code asks the runtime linker >if the API described above is present in the process address space. >If so, the functionality is available. If not, the functionality is >not available. > > > >Using the API > > > >In the prototype, the Sun Studio Profiler invokes the API if it is >available to ask for the data on every clock-profiling tick. >MPI-Work time is recorded as such, unless the thread is not on CPU, >in which case it is converted to MPI-Stall time. On Linux, ticks do >not occur when the thread is not on CPU, so the technique is valid >only for busy-waits. > >Overhead in maintaining state is a important consideration. There is >relatively little overhead in Sun's implementation for most MPI >operations, but there is some performance cost. The worst case we've >seen is in the latency for short messages, where the instrumented >runtime shows about ~100-200 ns. increase, or about 10%. For that >reason, Sun chose to ship both instrumented and uninstrumented >libraries. We also provide a flag to mpirun that automatically >switches to the instrumented libraries. > > > >Scalability Differences between API Tracing and Clock-Profiling > > > >MPI can be used for very large scale jobs, using hundreds, if not >thousands, of processors and MPI processes. In such cases, >scalability of the performance measurements becomes significant. > >MPI API tracing records data proportional to the number of calls and >messages, collected from all ranks, and suffers dilation in the same >proportion. If the data volume becomes too great, the time to match >the sends and the receives becomes unacceptable. Rank-selective >profiling of MPI API tracing data cannot be done without giving up >on matching sends and receives: if some data from some ranks are >missing, that matching cannot be done. Likewise, time-selective API >tracing presents problems at the boundaries where part of a >transaction may be present, and part absent. > >On the other hand, the data volume and overhead from clock-profiling >depends on the total time the processes run and the profiling >frequency. Since the data is a statistical sample, rank-selective >data collection is reasonable as long as the selected ranks are >representative. Likewise time-selective data collection for any >representative interval is reasonable. Furthermore, data volume and >distortion may also be managed by changing the profiling frequency. > >_______________________________________________ >Mpi3-tools mailing list >Mpi3-tools(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

[Mpi3-tools] MPI State Profiling Proposal
by Marty.Itzkowitz＠Sun.COM 12 Dec '08

12 Dec '08

I'm attaching a draft of the MPI State Profiling description. Can someone please add it in the right place to the website in time for the meeting Monday? Thanks, Marty

1 0

Re: [Mpi3-tools] Meeting notes and Wiki
by Jeff Squyres 10 Dec '08

10 Dec '08

I added my three items to the wiki: - DLL discovery API proposal - MPI handle debugging proposal - Title/authors/abstract/link to the paper on MPI-2 dynamic process acquisition I also moved all the MPI3Tools wiki links into the MPI3Tools/ directory, just so that we keep this WG's files out of global wiki namespace (since we share the wiki with all the other WG's). https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools Enjoy. On Dec 8, 2008, at 5:48 PM, Martin Schulz wrote: > Hi all, > > I added the meeting notes from todays meeting to the MPI tools WG > Wiki page and also modified the structure of the main page based > on our discussion today. Please feel free to add anything I might > have missed from the discussion. Also, I created links to new > pages for each area - if you have information to contribute, please > start filling in these new pages. > > Thanks, > > Martin > > PS: The Wiki page is at: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools > > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi3-tools] Meeting notes and Wiki
by William Gropp 09 Dec '08

09 Dec '08

I didn't mention this yesterday, but just to be clear, there never was a formal "MPIR" spec for the process discovery - there was a hack that was created for the first prototype implementation with MPICH and the ch_p4 device, but this was never intended to be a standard. Unfortunately, like so many prototypes, since there wasn't a standard, this part of the implementation was reversed engineered into other implementations. Bill On Dec 8, 2008, at 4:48 PM, Martin Schulz wrote: > Hi all, > > I added the meeting notes from todays meeting to the MPI tools WG > Wiki page and also modified the structure of the main page based > on our discussion today. Please feel free to add anything I might > have missed from the discussion. Also, I created links to new > pages for each area - if you have information to contribute, please > start filling in these new pages. > > Thanks, > > Martin > > PS: The Wiki page is at: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools > > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

[Mpi3-tools] Meeting notes and Wiki
by Martin Schulz 08 Dec '08

08 Dec '08

Hi all, I added the meeting notes from todays meeting to the MPI tools WG Wiki page and also modified the structure of the main page based on our discussion today. Please feel free to add anything I might have missed from the discussion. Also, I created links to new pages for each area - if you have information to contribute, please start filling in these new pages. Thanks, Martin PS: The Wiki page is at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0