- mpiwg-tools - mailman3.cels.anl.gov

[Mpi3-tools] Meeting notes from 12/15 and next meetings (12/22 and 1/12)
by Martin Schulz 20 Dec '08

20 Dec '08

Hi all, We had a tools WG meeting during the MPI forum meeting last week at Menlo Park. The notes and slides from this meeting are now available on the Tools WG Wiki. The main part was a discussion on how the MPIR interface is currently implemented and used. The outcome of this discussion is also on the Wiki, but needs further reviews, especially from the tools community (in particular debugger developers). We will try to hold a meeting next Monday 12/22 at 8am PT, 11am ET, 5pm MEZ to give people the chance to comment on this MPIR discussion and to refine the document. Since not many people might be around, we will hold off on any other topic until the new year. The next regular TelCon will be on Monday 1/12 at the same time and the next topic (after the MPIR interface) will be a discussion on Marty's MPI state proposal as well as the performance information interface ideas. Also, if anyone can't make it on Monday, please send comments via email. It would be great if we could come to a conclusion on the MPIR document rather sooner than later. Thanks (and in case you can't make it Monday: Happy Holidays!), Martin PS: The Wiki is at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] MPI State Profiling Proposal
by Martin Schulz 15 Dec '08

15 Dec '08

Hi Marty, all, I added this to the wiki - it is now at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools/state Martin At 02:16 PM 12/12/2008, you wrote: >I'm attaching a draft of the MPI State Profiling description. Can someone >please add it in the right place to the website in time for the >meeting Monday? > > Thanks, > Marty > > > > >MPI State Profiling > > > > > >Marty Itzkowitz > > > > > >Sun Microsystems > > > > > >Introduction > > > >Most current profiling for MPI applications is based on tracing API >calls into the MPI runtime library with, for example, VampirTrace. >The data collected shows API calls and the messages that are sent >and received. In general, all trace data from all ranks is needed in >order to be able to match sends and receives, and that causes high >data volume and other scalability issues. > >We are here proposing an additional means of profiling MPI >applications based on statistical sampling of the MPI runtime state. >It has improved scalability properties relative to MPI API tracing, >and supports selectivity. > >The proposal described here has been implemented in Sun's >ClusterTools 8.1 product, based on Open MPI. Sun will be happy to >contribute the code back to the Open MPI tree. > > > >MPI State Profiling > > > >The basic idea is to have the MPI runtime maintain a per-thread >state variable, and provide an asynchronous-signal-safe API for a >thread to read its own state. In the prototype implementation, there >are three states: >Not in MPI >MPI-Work (In the MPI runtime, and working) >MPI-Stall (In the MPI runtime, but stalled for some reason) >The interface, as prototyped, only has one state for MPI-Stall, no >matter why the runtime is stalled. It would be a simple extension to >define multiple MPI-Stall states, one for each reason it is stalled. > >MPI-Stall Time is accumulated when the thread is stalled within MPI, >whether the thread is busy-waiting (when the process is also >accumulating User CPU Time), as well as when the process is >sleep-waiting (when the process is not accumulating User CPU Time). >In other words, MPI-Stall time is a statistical measure of when the >process is blocked inside the MPI library. > >Note that MPI-Stall time is not a measure of time spent in the >MPI_Wait call. Some of the time in that call is MPI-Work time, and >some of the time spent in other calls may be MPI-Stall time. > >MPI-Work means that the runtime is doing something other than >waiting. It is not necessarily directly-useful work from the point >of view of the application -- it may be what the user would think of >as overhead. > >A profiler can then read the state with each profile tick, and >accumulate a statistical measure of time spent in MPI-Work and time >spent in MPI-Stall. If callstacks are recorded at the same time as >the state is captured, that MPI state can be attributed to the >frames in the callstack, allowing the user to see which MPI calls >are spending the most time stalled, and how the user got to that call. > > > >Runtime Implementation > > > >Implementation requires three things: the maintenance of the state, >an API to read it, and a rendezvous mechanism for the profiler to >find the API. > >Maintenance of the state is straightforward. All states start out in >the Not-in-MPI state. Whenever an API call is entered, the state for >the calling thread is changed to MPI-Work; whenever an API call >returns, the state is changed back to Not-in-MPI. During the >processing, whenever the OMPI progress engine determines that no >progress is being made on any outstanding requests, the state is >changed to MPI-Stall; Whenever the progress engine detects progress >being made, the state is changed back to MPI-Work. The work to wait >transition points are very implementation-specific, and must be done >carefully to avoid high overhead. > >The API is quite simple. There is only one function: >OMPI_state_t (*mpi_collector_api)(void*) > >The return value is an enum representing the current state for the >calling thread. That function must be asynchronous-signal-safe, so >that it can be called from a signal handler processing a profile tick. > >Rendezvous is also simple. The profile code asks the runtime linker >if the API described above is present in the process address space. >If so, the functionality is available. If not, the functionality is >not available. > > > >Using the API > > > >In the prototype, the Sun Studio Profiler invokes the API if it is >available to ask for the data on every clock-profiling tick. >MPI-Work time is recorded as such, unless the thread is not on CPU, >in which case it is converted to MPI-Stall time. On Linux, ticks do >not occur when the thread is not on CPU, so the technique is valid >only for busy-waits. > >Overhead in maintaining state is a important consideration. There is >relatively little overhead in Sun's implementation for most MPI >operations, but there is some performance cost. The worst case we've >seen is in the latency for short messages, where the instrumented >runtime shows about ~100-200 ns. increase, or about 10%. For that >reason, Sun chose to ship both instrumented and uninstrumented >libraries. We also provide a flag to mpirun that automatically >switches to the instrumented libraries. > > > >Scalability Differences between API Tracing and Clock-Profiling > > > >MPI can be used for very large scale jobs, using hundreds, if not >thousands, of processors and MPI processes. In such cases, >scalability of the performance measurements becomes significant. > >MPI API tracing records data proportional to the number of calls and >messages, collected from all ranks, and suffers dilation in the same >proportion. If the data volume becomes too great, the time to match >the sends and the receives becomes unacceptable. Rank-selective >profiling of MPI API tracing data cannot be done without giving up >on matching sends and receives: if some data from some ranks are >missing, that matching cannot be done. Likewise, time-selective API >tracing presents problems at the boundaries where part of a >transaction may be present, and part absent. > >On the other hand, the data volume and overhead from clock-profiling >depends on the total time the processes run and the profiling >frequency. Since the data is a statistical sample, rank-selective >data collection is reasonable as long as the selected ranks are >representative. Likewise time-selective data collection for any >representative interval is reasonable. Furthermore, data volume and >distortion may also be managed by changing the profiling frequency. > >_______________________________________________ >Mpi3-tools mailing list >Mpi3-tools(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

[Mpi3-tools] MPI State Profiling Proposal
by Marty.Itzkowitz＠Sun.COM 12 Dec '08

12 Dec '08

I'm attaching a draft of the MPI State Profiling description. Can someone please add it in the right place to the website in time for the meeting Monday? Thanks, Marty

1 0

Re: [Mpi3-tools] Meeting notes and Wiki
by Jeff Squyres 10 Dec '08

10 Dec '08

I added my three items to the wiki: - DLL discovery API proposal - MPI handle debugging proposal - Title/authors/abstract/link to the paper on MPI-2 dynamic process acquisition I also moved all the MPI3Tools wiki links into the MPI3Tools/ directory, just so that we keep this WG's files out of global wiki namespace (since we share the wiki with all the other WG's). https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools Enjoy. On Dec 8, 2008, at 5:48 PM, Martin Schulz wrote: > Hi all, > > I added the meeting notes from todays meeting to the MPI tools WG > Wiki page and also modified the structure of the main page based > on our discussion today. Please feel free to add anything I might > have missed from the discussion. Also, I created links to new > pages for each area - if you have information to contribute, please > start filling in these new pages. > > Thanks, > > Martin > > PS: The Wiki page is at: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools > > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi3-tools] Meeting notes and Wiki
by William Gropp 09 Dec '08

09 Dec '08

I didn't mention this yesterday, but just to be clear, there never was a formal "MPIR" spec for the process discovery - there was a hack that was created for the first prototype implementation with MPICH and the ch_p4 device, but this was never intended to be a standard. Unfortunately, like so many prototypes, since there wasn't a standard, this part of the implementation was reversed engineered into other implementations. Bill On Dec 8, 2008, at 4:48 PM, Martin Schulz wrote: > Hi all, > > I added the meeting notes from todays meeting to the MPI tools WG > Wiki page and also modified the structure of the main page based > on our discussion today. Please feel free to add anything I might > have missed from the discussion. Also, I created links to new > pages for each area - if you have information to contribute, please > start filling in these new pages. > > Thanks, > > Martin > > PS: The Wiki page is at: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools > > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

[Mpi3-tools] Meeting notes and Wiki
by Martin Schulz 08 Dec '08

08 Dec '08

Hi all, I added the meeting notes from todays meeting to the MPI tools WG Wiki page and also modified the structure of the main page based on our discussion today. Please feel free to add anything I might have missed from the discussion. Also, I created links to new pages for each area - if you have information to contribute, please start filling in these new pages. Thanks, Martin PS: The Wiki page is at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

[Mpi3-tools] Reminder: MPI Tools WG TelCon tomorrow (Monday 12/8/08)
by Martin Schulz 07 Dec '08

07 Dec '08

Hi all, Just as a reminder: we are holding our first MPI Forum Tools WG TelCon tomorrow, Monday 12/8/08, at 8am PT, 11am ET, 5pm MEZ. Here is the dial-in information: > Toll Free Dial In Number for Offsite Participants: 866-914-3976 > International Access/Caller Paid Dial In Number: 925-424-8105 > Dial In Number for LLNL Onsite Participants: 4-8105 > > Access Security Code: 860609 # So far the goals of this group (based on the discussion at the MPI Forum meeting in Dublin) are to define three standardization proposals: - A performance query interface to give tools access to MPI internal performance information (without restricting implementation freedom) - A process acquisition interface allowing tools to identify and track all processes of one MPI job (based on the existing MPIR API). - A debugging interface giving debuggers an easy way to investigate MPI state If there are other topics that should be included in this WG, please bring them up tomorrow or send email to this list. To get started tomorrow, I would suggest to go over these topics, see where we are and how to proceed with each and then to prioritize the order in which we want to address them. If anyone has additional suggestions for the agenda, please let me know. Thanks and talk to you all tomorrow, Martin _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

[Mpi3-tools] First MPI tools TelCon - 8/12, 8am PT, 11am ET, 5pm CET
by Martin Schulz 25 Nov '08

25 Nov '08

Hi all, Based on the feedback that I got, I think Monday is the best day that fits for most (I hope this is correct and doesn't exclude many, but finding a day that fits everybody was - as expected - not possible). As for the time, I think we should stick with 8am Pacific, 11am Eastern, and 5pm Central European Time. This will allow more people from Europe to participate. Also, I doubt that we will be able to provide a local European dial-in number for easy after hours access. At least we at LLNL can't do this - if someone else has the ability to set this up and is willing to host these TelCons, that would of course be great; please let me know. Hence, I suggest that we have our first TelCon for the MPI tools WG on Monday December 8th (a week after Thanksgiving to provide some time for everyone to get this onto their calendars) and then continue (for now) on a bi-weekly basis. We can use the following number: > Toll Free Dial In Number for Offsite Participants: 866-914-3976 > International Access/Caller Paid Dial In Number: 925-424-8105 > Dial In Number for LLNL Onsite Participants: 4-8105 > > Access Security Code: 860609 # I hope this works for everyone - talk to you soon. Happy Thanksgiving, Martin _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Welcome to MPI-Forum WG on Tools / Time for TelCon
by Marty Itzkowitz 25 Nov '08

25 Nov '08

What are the proposed dates and times Pacific Time? Thanks, Marty On 11/25/08 9:05 AM, Hubert Ritzdorf wrote: > Hi, > > 5pm CET is fine for me. 8pm-10pm also. > Any day except Friday 5th December and Thursday 11th. > > Hubert > > Marc-Andre Hermanns wrote: >> Hi Martin, Hi all, >> >> It would be interesting to know who -- other than me -- is calling from >> Europe? >> >> In the first week in December, for me the best day would be Tuesday >> 2nd, if the call is to be at 5 pm CET (as I stay late that day anyway). >> I could make Thursday work that week, too, I think, but I am not sure >> yet. >> >> If I am the only one calling from CET, I could offer another window of >> 8pm to 10 pm CET (any day). I would be calling from home then, a local >> (German) dial-in number would be great, as I can call those without >> additional costs. >> >> Friday is a bad day for me (for the 5pm CET slot), as we finish work >> early those days (4 pm CET) and I have a family appointment after that, >> which is even tight to meet with the 4 pm. The 8-10 pm slot would then >> be available again (assuming the local dial-in ;)) >> >> Best regards, >> Marc-Andre >> >> >>> I would suggest to first meet the week after >>> Thanksgiving (the week starting 12/1) and then continue on a >>> bi-weekly basis. If people think we should go with a weekly >>> meeting, I would be open for this as well. >>> >> >> At first I would also suggest bi-weekly. Maybe, when we get closer to >> any (to-be-defined) milestones >> >> Best regards, >> Marc-Andre >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Mpi3-tools mailing list >> Mpi3-tools(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >> > ------------------------------------------------------------------------ > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >

1 0

Re: [Mpi3-tools] Welcome to MPI-Forum WG on Tools / Time for TelCon
by Hubert Ritzdorf 25 Nov '08

25 Nov '08

Hi, 5pm CET is fine for me. 8pm-10pm also. Any day except Friday 5th December and Thursday 11th. Hubert Marc-Andre Hermanns wrote: > Hi Martin, Hi all, > > It would be interesting to know who -- other than me -- is calling from > Europe? > > In the first week in December, for me the best day would be Tuesday > 2nd, if the call is to be at 5 pm CET (as I stay late that day anyway). > I could make Thursday work that week, too, I think, but I am not sure yet. > > If I am the only one calling from CET, I could offer another window of > 8pm to 10 pm CET (any day). I would be calling from home then, a local > (German) dial-in number would be great, as I can call those without > additional costs. > > Friday is a bad day for me (for the 5pm CET slot), as we finish work > early those days (4 pm CET) and I have a family appointment after that, > which is even tight to meet with the 4 pm. The 8-10 pm slot would then > be available again (assuming the local dial-in ;)) > > Best regards, > Marc-Andre > > >> I would suggest to first meet the week after >> Thanksgiving (the week starting 12/1) and then continue on a >> bi-weekly basis. If people think we should go with a weekly >> meeting, I would be open for this as well. >> > > At first I would also suggest bi-weekly. Maybe, when we get closer to > any (to-be-defined) milestones > > Best regards, > Marc-Andre > > ------------------------------------------------------------------------ > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >

1 0