July 2009 - mpiwg-tools - mailman3.cels.anl.gov

[Mpi3-tools] Reminder: Meeting 7/28, 7am PDT, 10am EDT
by Martin Schulz 28 Jul '09

28 Jul '09

Hi all, Just as a reminder - we will have a face-to-face meeting for the MPI tools WG on Tuesday 7/28 during the MPI forum meeting at 9am CDT (7am PDT, one hour earlier than usual). The time slot for this meeting is for 1h 45min. We will also have a conference line for people who would like dial in. The number is as usual: * Toll Free Dial In Number for Offsite Participants: 866-914-3976 * International Access/Caller Paid Dial In Number: 925-424-8105 * Dial In Number for LLNL Onsite Participants: 4-8105 * Access Security Code: 860609 # As usual during the face-to-face meetings, we might need a little time to set up at the beginning and hence might be delayed for a few minutes. So far, the agenda has two main points (please let me know if someone would like to add a topic): - Report on the progress of the debug APIs (Jeff) - Discussion on a possible API for the performance interface (Martin) Slides for the latter are on the tools WG Wiki at: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/MPI3Tools/slid… Martin _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Notes online & Next TelCon 7/13
by Martin Schulz 13 Jul '09

13 Jul '09

HI Marty, On Jul 13, 2009, at 7:39 AM, Marty Itzkowitz wrote: > Is the concall on for this morning? What phone number? Yes, at the usual number: > Toll Free Dial In Number for Offsite Participants: 866-914-3976 > International Access/Caller Paid Dial In Number: 925-424-8105 > Dial In Number for LLNL Onsite Participants: 4-8105 > > Access Security Code: 860609 # Martin > > Thanks, > Marty > > On 7/6/09 3:12 PM, Martin Schulz wrote: >> Hi all, >> >> The notes from the PIMPI discussion last week are now online (sorry >> for the delay). The >> next MPI3 tools WG TelCon is scheduled for next week on 7/13 at 8am >> PDT as usual. >> The main agenda item so far is to continue the discussion of the >> PIMPI proposal. If >> you have anything that you would like to bring up, please let me >> know (Jeff, any >> updates on the debug APIs you want to talk about?). >> >> On 7/27 we will skip the TelCon due to the MPI Forum meeting that >> week. At the forum >> we will have a face-to-face tools WG meeting, which is scheduled >> for Tuesday 7/28 at >> 9am CDT, which one hour earlier than usual. We will setup a TelCon >> for this meeting >> using the usual number. >> >> Martin >> >> >> _______________________________________________________________________ >> Martin Schulz, schulzm(a)llnl.gov, http://* people.llnl.gov/schulz6 >> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >> >> >> >> >> >> _______________________________________________ >> Mpi3-tools mailing list >> Mpi3-tools(a)lists.mpi-forum.org >> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools > _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Notes online & Next TelCon 7/13
by Marty Itzkowitz 13 Jul '09

13 Jul '09

Is the concall on for this morning? What phone number? Thanks, Marty On 7/6/09 3:12 PM, Martin Schulz wrote: > Hi all, > > The notes from the PIMPI discussion last week are now online (sorry > for the delay). The > next MPI3 tools WG TelCon is scheduled for next week on 7/13 at 8am > PDT as usual. > The main agenda item so far is to continue the discussion of the PIMPI > proposal. If > you have anything that you would like to bring up, please let me know > (Jeff, any > updates on the debug APIs you want to talk about?). > > On 7/27 we will skip the TelCon due to the MPI Forum meeting that > week. At the forum > we will have a face-to-face tools WG meeting, which is scheduled for > Tuesday 7/28 at > 9am CDT, which one hour earlier than usual. We will setup a TelCon for > this meeting > using the usual number. > > Martin > > > _______________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools

1 0

Re: [Mpi3-tools] lightweight event log
by Marc-Andre Hermanns 08 Jul '09

08 Jul '09

Hi Martin, >>> Note, though, that you (as a tool writer) can always combine this >>> logging interface with a PMPI logger and can grab any cleanly defined >>> logging information from there as well. >> Understood. What is still unclear to me, however, is how to >> incorporate the log information at runtime. Usually, I enter the >> wrapper create a timestamped enter-record, do whatever I want/need >> to do, call the PMPI call and create a timestamped exit-record. >> Now, in between I get another timestamped buffer that can only be >> interpreted later on? Then I could also just save the 'blob' with >> my trace and interpret the contents later on. > Yes, that is an option. During the pre-call wrapper, you can enable > the fine grain logging and in the post-call wrapper you can then > access the log buffer with all events that happened in between. > Having an option at that time to save that buffer is a good idea, but > would require the post processing tool to link against the same MPI > on the same machine, since you will get only then the accessor > functions that can interpret the buffer. For a tool like ours it would not be a problem, as we distribute in source anyway, yet, I understand that this is not an option for every tool vendor. I think it should be doable, though, to have 'adapters' for a tool distributed in binary form, that is compiled at installation time, which does not contain any 'business logic' but just the interface to the MPI library. >>> I am not sure if it's worth having the user pick what should be >>> recorded and what not, since this will be very limited anyway. >> The user in this case is not necessarily the end-user but a tool. >> As you said above, a tool might want to combine these logs with the >> information gathered in a wrapper. The tool then might have an >> interest in defining a-priori what it wants to gather, to minimize >> overhead. However, you might be right that this flexibility might >> bear more overhead than a fixed set of predefined sets of logging >> variables. > > This is a question for the MPI implementors about what they prefer or > think is faster. It also depends on how many things/points an > implementation will log. Basically it is a tradeoff between recording > all information and testing which information should be recorded. I I think anything that can be done to reduce trace size is worth to be discussed. > think some MPIs like MPICH already do something like this internally > in debug builds, so it would be good to get some feedback from them > on this issue. Let's see what the discussion brings next time. I am still not sure whether I can make it to the next call. I have a workshop outside of Juelich on Monday till 5:30 PM CEST. I will try to get internet access and get out a little early to join in time, but I cannot promise anything at this moment. >> Do I understand it correctly that the internal data should be >> converted on reading the buffer, or is a post-mortem interface >> possibly the better choice? This way any conversion can be delayed >> until we are no longer in measurement. > > I think either way is fine. As mentioned above, we can add a > mechanism to simply store the whole buffer in raw form into a buffer > provided by the tool and then allow later inspection of it. However, > as also said above, the tool will have to link against that > particular MPI version on that particular platform to allow a correct > interpretation of the data. This should be doable, though, at least > for some post-processing converter. What I didn't think of is that not all tools necessarily build against the MPI library. Thus you are right in that we might need both interfaces, one for the in-measurement conversion and one for post-mortem conversion. I think any post-processing conversion needs to be done on-the-fly by the tools to avoid re-writing the trace data. Best regards, Marc-Andre -- Marc-Andre Hermanns Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH D-52425 Juelich Germany Phone : +49-2461-61-2054 Fax : +49-2461-61-6656 eMail : m.a.hermanns(a)fz-juelich.de WWW : http://www.fz-juelich.de/jsc/ JSC is the coordinator of the John von Neumann Institute for Computing and member of the Gauss Centre for Supercomputing Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr. Harald Bolt, Prof. Dr. Sebastian M. Schmidt

1 0

Re: [Mpi3-tools] lightweight event log
by Martin Schulz 07 Jul '09

07 Jul '09

Hi Marc-Andre, On Jul 3, 2009, at 12:30 AM, Marc-Andre Hermanns wrote: > Hi Martin, > >> Note, though, that you (as a tool writer) can always combine this >> logging interface with a PMPI logger and can grab any cleanly defined >> logging information from there as well. > > Understood. What is still unclear to me, however, is how to > incorporate > the log information at runtime. Usually, I enter the wrapper create a > timestamped enter-record, do whatever I want/need to do, call the PMPI > call and create a timestamped exit-record. Now, in between I get > another > timestamped buffer that can only be interpreted later on? Then I could > also just save the 'blob' with my trace and interpret the contents > later on. Yes, that is an option. During the pre-call wrapper, you can enable the fine grain logging and in the post-call wrapper you can then access the log buffer with all events that happened in between. Having an option at that time to save that buffer is a good idea, but would require the post processing tool to link against the same MPI on the same machine, since you will get only then the accessor functions that can interpret the buffer. > > I can envision recording timespans with this to identify where the > communication stalled or was making progress. In relation to the same > data on other processes, this might become handy in reasoning about > the > communication pattern used. Yes, that's a good use case. > >>>> 2) it is lightweight for two reasons: >>>> (1) the data logged may be as small and simple as the >>>> implementation chooses and (2) the data may be logged in an >>>> internal, time and space efficient format. A separate set of >>>> routines are provided to interpret the logged data. For example, >>>> the timestamps may be stored in an internal format (such as the >>>> bits from a clock register) and converted to seconds only when >>>> the user evaluates the logged data. > >>> Combining your answers to point 1 and 2, I envision to >>> pre-construct memory-layouts (like or even with MPI datatypes) to >>> the relevant information, so that a tool can basically first tell >>> the implementation what it want to be logged, then define handles >>> for the data layout, and >> >> I am not sure if it's worth having the user pick what should be >> recorded and what not, since this will be very limited anyway. > > The user in this case is not necessarily the end-user but a tool. As > you > said above, a tool might want to combine these logs with the > information > gathered in a wrapper. The tool then might have an interest in > defining > a-priori what it wants to gather, to minimize overhead. However, you > might be right that this flexibility might bear more overhead than a > fixed set of predefined sets of logging variables. This is a question for the MPI implementors about what they prefer or think is faster. It also depends on how many things/points an implementation will log. Basically it is a tradeoff between recording all information and testing which information should be recorded. I think some MPIs like MPICH already do something like this internally in debug builds, so it would be good to get some feedback from them on this issue. > > Do I understand it correctly that the internal data should be > converted > on reading the buffer, or is a post-mortem interface possibly the > better > choice? This way any conversion can be delayed until we are no > longer in > measurement. I think either way is fine. As mentioned above, we can add a mechanism to simply store the whole buffer in raw form into a buffer provided by the tool and then allow later inspection of it. However, as also said above, the tool will have to link against that particular MPI version on that particular platform to allow a correct interpretation of the data. This should be doable, though, at least for some post-processing converter. Martin > > Best regards, > Marc-Andre > -- > Marc-Andre Hermanns > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > D-52425 Juelich > Germany > > Phone : +49-2461-61-2054 > Fax : +49-2461-61-6656 > eMail : m.a.hermanns(a)fz-juelich.de > WWW : http://www.fz-juelich.de/jsc/ > > JSC is the coordinator of the > John von Neumann Institute for Computing > and member of the > Gauss Centre for Supercomputing > > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Dr. Ulrich Krafft (stellv. Vorsitzender), > Prof. Dr. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

[Mpi3-tools] Notes online & Next TelCon 7/13
by Martin Schulz 06 Jul '09

06 Jul '09

Hi all, The notes from the PIMPI discussion last week are now online (sorry for the delay). The next MPI3 tools WG TelCon is scheduled for next week on 7/13 at 8am PDT as usual. The main agenda item so far is to continue the discussion of the PIMPI proposal. If you have anything that you would like to bring up, please let me know (Jeff, any updates on the debug APIs you want to talk about?). On 7/27 we will skip the TelCon due to the MPI Forum meeting that week. At the forum we will have a face-to-face tools WG meeting, which is scheduled for Tuesday 7/28 at 9am CDT, which one hour earlier than usual. We will setup a TelCon for this meeting using the usual number. Martin _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] lightweight event log
by Marc-Andre Hermanns 03 Jul '09

03 Jul '09

Hi Martin, > Note, though, that you (as a tool writer) can always combine this > logging interface with a PMPI logger and can grab any cleanly defined > logging information from there as well. Understood. What is still unclear to me, however, is how to incorporate the log information at runtime. Usually, I enter the wrapper create a timestamped enter-record, do whatever I want/need to do, call the PMPI call and create a timestamped exit-record. Now, in between I get another timestamped buffer that can only be interpreted later on? Then I could also just save the 'blob' with my trace and interpret the contents later on. I can envision recording timespans with this to identify where the communication stalled or was making progress. In relation to the same data on other processes, this might become handy in reasoning about the communication pattern used. >>> 2) it is lightweight for two reasons: >>> (1) the data logged may be as small and simple as the >>> implementation chooses and (2) the data may be logged in an >>> internal, time and space efficient format. A separate set of >>> routines are provided to interpret the logged data. For example, >>> the timestamps may be stored in an internal format (such as the >>> bits from a clock register) and converted to seconds only when >>> the user evaluates the logged data. >> Combining your answers to point 1 and 2, I envision to >> pre-construct memory-layouts (like or even with MPI datatypes) to >> the relevant information, so that a tool can basically first tell >> the implementation what it want to be logged, then define handles >> for the data layout, and > > I am not sure if it's worth having the user pick what should be > recorded and what not, since this will be very limited anyway. The user in this case is not necessarily the end-user but a tool. As you said above, a tool might want to combine these logs with the information gathered in a wrapper. The tool then might have an interest in defining a-priori what it wants to gather, to minimize overhead. However, you might be right that this flexibility might bear more overhead than a fixed set of predefined sets of logging variables. Do I understand it correctly that the internal data should be converted on reading the buffer, or is a post-mortem interface possibly the better choice? This way any conversion can be delayed until we are no longer in measurement. Best regards, Marc-Andre -- Marc-Andre Hermanns Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH D-52425 Juelich Germany Phone : +49-2461-61-2054 Fax : +49-2461-61-6656 eMail : m.a.hermanns(a)fz-juelich.de WWW : http://www.fz-juelich.de/jsc/ JSC is the coordinator of the John von Neumann Institute for Computing and member of the Gauss Centre for Supercomputing Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr. Harald Bolt, Prof. Dr. Sebastian M. Schmidt

1 0

Re: [Mpi3-tools] lightweight event log
by Martin Schulz 02 Jul '09

02 Jul '09

Hi Marc-Andre, On Jul 2, 2009, at 1:58 AM, Marc-Andre Hermanns wrote: > Hi Bill, > >> The idea is that the implementation may be instrumented with the >> ability to record timestamped events. Like the "variable" interface, >> the idea here is to provide a way to expose whatever the >> implementation is prepared to make available, such as information >> that >> the developers and tuners of that implementation have found valuable. >> So in answer to your questoins: >> >> 1) what will be logged? Probably timestamps, some indication of >> location, and some context (like message length). I would not expect >> all routine parameters to be logged, though an implementation could >> choose to do thiat > > Sometimes tools need all parameter information, though. Especially for > higher level analysis, one might need more than the above information. The idea is to log low-level events below the MPI call interface. Hence, we can't assume a fixed list of parameters anymore. Each implementation can have its own set of information here. What Bill is suggesting is basically only a way for a tool to gather some context around the event and we will try to suggest some commonly existing parameters, but as with the performance variables, we don't want to demand any particular structure. Note, though, that you (as a tool writer) can always combine this logging interface with a PMPI logger and can grab any cleanly defined logging information from there as well. > >> 2) it is lightweight for two reasons: (1) the data logged may be as >> small and simple as the implementation chooses and (2) the data may >> be >> logged in an internal, time and space efficient format. A separate >> set of routines are provided to interpret the logged data. For >> example, the timestamps may be stored in an internal format (such as >> the bits from a clock register) and converted to seconds only when >> the >> user evaluates the logged data. > > Combining your answers to point 1 and 2, I envision to pre-construct > memory-layouts (like or even with MPI datatypes) to the relevant > information, so that a tool can basically first tell the > implementation > what it want to be logged, then define handles for the data layout, > and I am not sure if it's worth having the user pick what should be recorded and what not, since this will be very limited anyway. > finally, retrieve the desired information through the defined > handles at > the location where they are needed. > > I admit I didn't evaluate the implementation overhead for something > like > that yet, and I am aware that flexibility usually comes with a > runtime cost. > >> 3) A performance tool can examine the log files but (as part of the >> light-weight part) there is no option to invoke a method or routine >> at >> every point a log record could be created. >> >> "Users" here means users with some degree of sophistication. >> Basically, if you can use PAPI and really understand what the various >> counters mean, then you should be able to use this interface. > > I am uncertain whether this logging facility would/should be > complementing or competing with existing tracing tools. > > Some MPI implementations already have tracing facilities. To my > knowledge mpich2 has MPE and OpenMPI has vampirtrace. Is this new > logging facility to be provided by these or is it envisioned to be a > new > infrastructure. Would this not create competition between measurement > infrastructures within MPI implementations? No, it is certainly not intended to be an alternative to existing MPI tracers. This logging facility would run at a much lower level and only record selected MPI internal events, mostly for debugging or very low-level performance analysis in conjunction with a high-level tracer such as VT. > > Martin mentioned a ring buffer to be queried? What happens when the > buffer is full? If some logging information is lost, what can the user > derive from an incomplete log? The user will be able to size the buffer and query whether there was an overflow, but if there is too much data, it will be lost. The user is responsible for querying this on time. Anything else has the danger of significantly increasing the critical path. Martin > > Best regards, > Marc-Andre > >> On Jun 29, 2009, at 7:42 AM, Marc-Andre Hermanns wrote: >> >>> Hi Martin, >>> >>> I am interested in how this logging facility is meant. Do you have a >>> more detailed description of it somewhere? >>> >>> - What do you anticipate to be logged (despite of timestamps)? >>> - What will make it lightweight? >>> - Is there any way that a performance tool might hook into? >>> >>> "[...] Users can then read and evaluate this event log at a later >>> time. >>> [...]" >>> >>> Who are these users? Application developers or tool developers? >>> >>> Best regards, >>> Marc-Andre >>> >>> Martin Schulz wrote: >>>> Hi all, >>>> >>>> As discussed during the last meeting, our next MPI 3 tools WG >>>> TelCon >>>> is on 6/29 >>>> at the usual time (8am PDT, 11am EDT, 5pm MSZ). The dial-in numbers >>>> are as >>>> usual: >>>> >>>> * Toll Free Dial In Number for Offsite Participants: >>>> 866-914-3976 >>>> * International Access/Caller Paid Dial In Number: 925-424-8105 >>>> * Dial In Number for LLNL Onsite Participants: 4-8105 >>>> * Access Security Code: 860609 # >>>> >>>> So far, the agenda is to talk about a proposal for a general >>>> performance information >>>> interface. More information on this proposal is at: >>>> >>>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools/pimpi >>>> >>>> If anybody wants to bring up another agenda item, please let me >>>> know. >>>> >>>> Martin >>>> >>>> _______________________________________________________________________ >>>> Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 >>>> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Mpi3-tools mailing list >>>> Mpi3-tools(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >>> >>> -- >>> Marc-Andre Hermanns >>> Juelich Supercomputing Centre >>> Institute for Advanced Simulation >>> Forschungszentrum Juelich GmbH >>> D-52425 Juelich >>> Germany >>> >>> Phone : +49-2461-61-2054 >>> Fax : +49-2461-61-6656 >>> eMail : m.a.hermanns(a)fz-juelich.de >>> WWW : http://www.fz-juelich.de/jsc/ >>> >>> JSC is the coordinator of the >>> John von Neumann Institute for Computing >>> and member of the >>> Gauss Centre for Supercomputing >>> >>> Sitz der Gesellschaft: Juelich >>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>> Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe >>> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), >>> Dr. Ulrich Krafft (stellv. Vorsitzender), >>> Prof. Dr. Harald Bolt, >>> Prof. Dr. Sebastian M. Schmidt >>> <smime.p7s><ATT00001.txt> >> >> William Gropp >> Deputy Director for Research >> Institute for Advanced Computing Applications and Technologies >> Paul and Cynthia Saylor Professor of Computer Science >> University of Illinois Urbana-Champaign >> >> >> >> >> _______________________________________________ >> Mpi3-tools mailing list >> Mpi3-tools(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools > > > -- > Marc-Andre Hermanns > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > D-52425 Juelich > Germany > > Phone : +49-2461-61-2054 > Fax : +49-2461-61-6656 > eMail : m.a.hermanns(a)fz-juelich.de > WWW : http://www.fz-juelich.de/jsc/ > > JSC is the coordinator of the > John von Neumann Institute for Computing > and member of the > Gauss Centre for Supercomputing > > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Dr. Ulrich Krafft (stellv. Vorsitzender), > Prof. Dr. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools _______________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] lightweight event log
by Marc-Andre Hermanns 02 Jul '09

02 Jul '09

Hi Bill, > The idea is that the implementation may be instrumented with the > ability to record timestamped events. Like the "variable" interface, > the idea here is to provide a way to expose whatever the > implementation is prepared to make available, such as information that > the developers and tuners of that implementation have found valuable. > So in answer to your questoins: > > 1) what will be logged? Probably timestamps, some indication of > location, and some context (like message length). I would not expect > all routine parameters to be logged, though an implementation could > choose to do thiat Sometimes tools need all parameter information, though. Especially for higher level analysis, one might need more than the above information. > 2) it is lightweight for two reasons: (1) the data logged may be as > small and simple as the implementation chooses and (2) the data may be > logged in an internal, time and space efficient format. A separate > set of routines are provided to interpret the logged data. For > example, the timestamps may be stored in an internal format (such as > the bits from a clock register) and converted to seconds only when the > user evaluates the logged data. Combining your answers to point 1 and 2, I envision to pre-construct memory-layouts (like or even with MPI datatypes) to the relevant information, so that a tool can basically first tell the implementation what it want to be logged, then define handles for the data layout, and finally, retrieve the desired information through the defined handles at the location where they are needed. I admit I didn't evaluate the implementation overhead for something like that yet, and I am aware that flexibility usually comes with a runtime cost. > 3) A performance tool can examine the log files but (as part of the > light-weight part) there is no option to invoke a method or routine at > every point a log record could be created. > > "Users" here means users with some degree of sophistication. > Basically, if you can use PAPI and really understand what the various > counters mean, then you should be able to use this interface. I am uncertain whether this logging facility would/should be complementing or competing with existing tracing tools. Some MPI implementations already have tracing facilities. To my knowledge mpich2 has MPE and OpenMPI has vampirtrace. Is this new logging facility to be provided by these or is it envisioned to be a new infrastructure. Would this not create competition between measurement infrastructures within MPI implementations? Martin mentioned a ring buffer to be queried? What happens when the buffer is full? If some logging information is lost, what can the user derive from an incomplete log? Best regards, Marc-Andre > On Jun 29, 2009, at 7:42 AM, Marc-Andre Hermanns wrote: > >> Hi Martin, >> >> I am interested in how this logging facility is meant. Do you have a >> more detailed description of it somewhere? >> >> - What do you anticipate to be logged (despite of timestamps)? >> - What will make it lightweight? >> - Is there any way that a performance tool might hook into? >> >> "[...] Users can then read and evaluate this event log at a later >> time. >> [...]" >> >> Who are these users? Application developers or tool developers? >> >> Best regards, >> Marc-Andre >> >> Martin Schulz wrote: >>> Hi all, >>> >>> As discussed during the last meeting, our next MPI 3 tools WG TelCon >>> is on 6/29 >>> at the usual time (8am PDT, 11am EDT, 5pm MSZ). The dial-in numbers >>> are as >>> usual: >>> >>> * Toll Free Dial In Number for Offsite Participants: 866-914-3976 >>> * International Access/Caller Paid Dial In Number: 925-424-8105 >>> * Dial In Number for LLNL Onsite Participants: 4-8105 >>> * Access Security Code: 860609 # >>> >>> So far, the agenda is to talk about a proposal for a general >>> performance information >>> interface. More information on this proposal is at: >>> >>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools/pimpi >>> >>> If anybody wants to bring up another agenda item, please let me know. >>> >>> Martin >>> >>> _______________________________________________________________________ >>> Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 >>> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Mpi3-tools mailing list >>> Mpi3-tools(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >> >> -- >> Marc-Andre Hermanns >> Juelich Supercomputing Centre >> Institute for Advanced Simulation >> Forschungszentrum Juelich GmbH >> D-52425 Juelich >> Germany >> >> Phone : +49-2461-61-2054 >> Fax : +49-2461-61-6656 >> eMail : m.a.hermanns(a)fz-juelich.de >> WWW : http://www.fz-juelich.de/jsc/ >> >> JSC is the coordinator of the >> John von Neumann Institute for Computing >> and member of the >> Gauss Centre for Supercomputing >> >> Sitz der Gesellschaft: Juelich >> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >> Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe >> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), >> Dr. Ulrich Krafft (stellv. Vorsitzender), >> Prof. Dr. Harald Bolt, >> Prof. Dr. Sebastian M. Schmidt >> <smime.p7s><ATT00001.txt> > > William Gropp > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- Marc-Andre Hermanns Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH D-52425 Juelich Germany Phone : +49-2461-61-2054 Fax : +49-2461-61-6656 eMail : m.a.hermanns(a)fz-juelich.de WWW : http://www.fz-juelich.de/jsc/ JSC is the coordinator of the John von Neumann Institute for Computing and member of the Gauss Centre for Supercomputing Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr. Harald Bolt, Prof. Dr. Sebastian M. Schmidt

1 0

Re: [Mpi3-tools] lightweight event log
by William Gropp 01 Jul '09

01 Jul '09

Hi Marc-Andre, The idea is that the implementation may be instrumented with the ability to record timestamped events. Like the "variable" interface, the idea here is to provide a way to expose whatever the implementation is prepared to make available, such as information that the developers and tuners of that implementation have found valuable. So in answer to your questoins: 1) what will be logged? Probably timestamps, some indication of location, and some context (like message length). I would not expect all routine parameters to be logged, though an implementation could choose to do thiat 2) it is lightweight for two reasons: (1) the data logged may be as small and simple as the implementation chooses and (2) the data may be logged in an internal, time and space efficient format. A separate set of routines are provided to interpret the logged data. For example, the timestamps may be stored in an internal format (such as the bits from a clock register) and converted to seconds only when the user evaluates the logged data. 3) A performance tool can examine the log files but (as part of the light-weight part) there is no option to invoke a method or routine at every point a log record could be created. "Users" here means users with some degree of sophistication. Basically, if you can use PAPI and really understand what the various counters mean, then you should be able to use this interface. Bill On Jun 29, 2009, at 7:42 AM, Marc-Andre Hermanns wrote: > Hi Martin, > > I am interested in how this logging facility is meant. Do you have a > more detailed description of it somewhere? > > - What do you anticipate to be logged (despite of timestamps)? > - What will make it lightweight? > - Is there any way that a performance tool might hook into? > > "[...] Users can then read and evaluate this event log at a later > time. > [...]" > > Who are these users? Application developers or tool developers? > > Best regards, > Marc-Andre > > Martin Schulz wrote: >> Hi all, >> >> As discussed during the last meeting, our next MPI 3 tools WG TelCon >> is on 6/29 >> at the usual time (8am PDT, 11am EDT, 5pm MSZ). The dial-in numbers >> are as >> usual: >> >> * Toll Free Dial In Number for Offsite Participants: 866-914-3976 >> * International Access/Caller Paid Dial In Number: 925-424-8105 >> * Dial In Number for LLNL Onsite Participants: 4-8105 >> * Access Security Code: 860609 # >> >> So far, the agenda is to talk about a proposal for a general >> performance information >> interface. More information on this proposal is at: >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools/pimpi >> >> If anybody wants to bring up another agenda item, please let me know. >> >> Martin >> >> _______________________________________________________________________ >> Martin Schulz, schulzm(a)llnl.gov, http:// people.llnl.gov/schulz6 >> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >> >> >> >> >> >> _______________________________________________ >> Mpi3-tools mailing list >> Mpi3-tools(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools > > > -- > Marc-Andre Hermanns > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > D-52425 Juelich > Germany > > Phone : +49-2461-61-2054 > Fax : +49-2461-61-6656 > eMail : m.a.hermanns(a)fz-juelich.de > WWW : http://www.fz-juelich.de/jsc/ > > JSC is the coordinator of the > John von Neumann Institute for Computing > and member of the > Gauss Centre for Supercomputing > > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Dr. Ulrich Krafft (stellv. Vorsitzender), > Prof. Dr. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > <smime.p7s><ATT00001.txt> William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0