December 2013 - mpiwg-tools - mailman3.cels.anl.gov

Re: [mpiwg-tools] MPI_T usage question: other sensors
by Schulz, Martin 19 Dec '13

19 Dec '13

On Dec 19, 2013, at 1:43 AM, Michael Knobloch <m.knobloch(a)fz-juelich.de> wrote: > Hi all, > > > On 19.12.2013 02:19, Schulz, Martin wrote: >> Hi all, >> >> I agree that a single interface to extract any kind of information from a system (MPI and beyond) would be very helpful and make our tools simpler. As Michael, though, I always thought of PAPI being that unifying interface - it should be very easy to write an MPI_T component to PAPI (in fact, I talked with Dan Terpstra from the PAPI team a few years ago when MPI_T was still under design and he thought that should work) and with that offer all MPI_T counters through PAPI (probably not the control variables, though). This should also allow us to provide "standardized" names for common performance variables (similar to what PAPI does for HW counters). >> > > I'm not so sure that writing a PAPI component for MPI_T will be that > simple. It's a while back since I wrote my last PAPI component, but if I > remember correctly such a component requires the names and types of > counters at compile time, which is something that the MPI_T interface > cannot provide. I mean I have no problem to build a PAPI component for > each MPI3 library out there (not that there are too many) or even a > generalized one which queries such information at compile time (although > this might be difficult on frontend-backend archtitectures), but the > standard states that I cannot assume the same set of counters even > between two runs or on different processes (which is horrible btw.). Hmm, that would not be good - when I talked with Dan a few years back, we already had this very dynamic query scheme and he didn't pick this up as a problem, so I thought we were good. However, he may also just >> The situation changes a bit, though, if MPI implementations query other information, like temperature or power, for themselves (Jeff, was this why you were asking?). In this case, the MPI implementation should use a standardized interface like PAPI itself (to avoid conflicts with tools), which could lead us to some strange circular SW dependencies. >> > > I don't even see how that could work in a meaningful way. Querying MPI_T > variables is process local, while several counters (especially such as > power and temperature) are shared between several processes, some even > with restricted access (i.e. only one process might query the counter). This is probably similar to the support for UNCORE events that is needed already for hardware counters - there isn't a clean solution for that either. This may even get more complicated if we want data from components that are not nodes (e.g., switches in the network). It's not clear how they get mapped back. > So supporting this is either a huge implementation work or using an > existing interface, i.e. PAPI. And I completely agree with Martin that > we should try to avoid circular dependencies, especially if there's no > real benefit for the user. > > -Michael Martin > -- > Michael Knobloch > Institute for Advanced Simulation (IAS) > Jülich Supercomputing Centre (JSC) > Telefon: +49 2461 61-3546 > Telefax: +49 2461 61-6656 > E-Mail: m.knobloch(a)fz-juelich.de > Internet: http://www.fz-juelich.de/jsc > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [mpiwg-tools] MPI_T usage question: other sensors
by Schulz, Martin 19 Dec '13

19 Dec '13

Hi Jeff, On Dec 19, 2013, at 3:29 AM, "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> wrote: > In Dec 18, 2013, at 8:19 PM, Schulz, Martin <schulzm(a)llnl.gov> wrote: > >> I agree that a single interface to extract any kind of information from a system (MPI and beyond) would be very helpful and make our tools simpler. As Michael, though, I always thought of PAPI being that unifying interface - it should be very easy to write an MPI_T component to PAPI (in fact, I talked with Dan Terpstra from the PAPI team a few years ago when MPI_T was still under design and he thought that should work) and with that offer all MPI_T counters through PAPI (probably not the control variables, though). This should also allow us to provide "standardized" names for common performance variables (similar to what PAPI does for HW counters). >> >> The situation changes a bit, though, if MPI implementations query other information, like temperature or power, for themselves (Jeff, was this why you were asking?). In this case, the MPI implementation should use a standardized interface like PAPI itself (to avoid conflicts with tools), which could lead us to some strange circular SW dependencies. > > Yes, this is what I was asking. > > If PAPI is going to be the standardized interface for tools to use, that's fine. I would say not only for tools - if you wanted to get access to this information within MPI, you should probably consider PAPI as well, since it is (to my knowledge) the only reasonably portable way to get to this kind of information. > > MPI's could certainly hide a private copy of PAPI in their internals (done correctly, it would avoid any symbol conflicts with application-linked in PAPI instances), but it sounds like it would be kinda wasteful. Hiding the symbols should work, but I am also worried about the underlying mechanisms to query data aren't mutually exclusive, i.e., I am not sure if two independent copies of PAPI can run concurrently (for any type of data that is being accessed). > Has anyone talked to the PAPI people recently to see if they're still interested in hooking up to MPI_T? Not lately - I am sure they are interested in it, but I doubt they will do it themselves. If someone else does it, they will most likely host the plugin and offer it as part of their distribution. Martin > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [mpiwg-tools] MPI_T usage question: other sensors
by Jeff Squyres (jsquyres) 19 Dec '13

19 Dec '13

In Dec 18, 2013, at 8:19 PM, Schulz, Martin <schulzm(a)llnl.gov> wrote: > I agree that a single interface to extract any kind of information from a system (MPI and beyond) would be very helpful and make our tools simpler. As Michael, though, I always thought of PAPI being that unifying interface - it should be very easy to write an MPI_T component to PAPI (in fact, I talked with Dan Terpstra from the PAPI team a few years ago when MPI_T was still under design and he thought that should work) and with that offer all MPI_T counters through PAPI (probably not the control variables, though). This should also allow us to provide "standardized" names for common performance variables (similar to what PAPI does for HW counters). > > The situation changes a bit, though, if MPI implementations query other information, like temperature or power, for themselves (Jeff, was this why you were asking?). In this case, the MPI implementation should use a standardized interface like PAPI itself (to avoid conflicts with tools), which could lead us to some strange circular SW dependencies. Yes, this is what I was asking. If PAPI is going to be the standardized interface for tools to use, that's fine. MPI's could certainly hide a private copy of PAPI in their internals (done correctly, it would avoid any symbol conflicts with application-linked in PAPI instances), but it sounds like it would be kinda wasteful. Has anyone talked to the PAPI people recently to see if they're still interested in hooking up to MPI_T? -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-tools] MPI_T usage question: other sensors
by Michael Knobloch 19 Dec '13

19 Dec '13

Hi all, On 19.12.2013 02:19, Schulz, Martin wrote: > Hi all, > > I agree that a single interface to extract any kind of information from a system (MPI and beyond) would be very helpful and make our tools simpler. As Michael, though, I always thought of PAPI being that unifying interface - it should be very easy to write an MPI_T component to PAPI (in fact, I talked with Dan Terpstra from the PAPI team a few years ago when MPI_T was still under design and he thought that should work) and with that offer all MPI_T counters through PAPI (probably not the control variables, though). This should also allow us to provide "standardized" names for common performance variables (similar to what PAPI does for HW counters). > I'm not so sure that writing a PAPI component for MPI_T will be that simple. It's a while back since I wrote my last PAPI component, but if I remember correctly such a component requires the names and types of counters at compile time, which is something that the MPI_T interface cannot provide. I mean I have no problem to build a PAPI component for each MPI3 library out there (not that there are too many) or even a generalized one which queries such information at compile time (although this might be difficult on frontend-backend archtitectures), but the standard states that I cannot assume the same set of counters even between two runs or on different processes (which is horrible btw.). > The situation changes a bit, though, if MPI implementations query other information, like temperature or power, for themselves (Jeff, was this why you were asking?). In this case, the MPI implementation should use a standardized interface like PAPI itself (to avoid conflicts with tools), which could lead us to some strange circular SW dependencies. > I don't even see how that could work in a meaningful way. Querying MPI_T variables is process local, while several counters (especially such as power and temperature) are shared between several processes, some even with restricted access (i.e. only one process might query the counter). So supporting this is either a huge implementation work or using an existing interface, i.e. PAPI. And I completely agree with Martin that we should try to avoid circular dependencies, especially if there's no real benefit for the user. -Michael -- Michael Knobloch Institute for Advanced Simulation (IAS) Jülich Supercomputing Centre (JSC) Telefon: +49 2461 61-3546 Telefax: +49 2461 61-6656 E-Mail: m.knobloch(a)fz-juelich.de Internet: http://www.fz-juelich.de/jsc ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Re: [mpiwg-tools] MPI_T usage question: other sensors
by Schulz, Martin 18 Dec '13

18 Dec '13

Hi all, I agree that a single interface to extract any kind of information from a system (MPI and beyond) would be very helpful and make our tools simpler. As Michael, though, I always thought of PAPI being that unifying interface - it should be very easy to write an MPI_T component to PAPI (in fact, I talked with Dan Terpstra from the PAPI team a few years ago when MPI_T was still under design and he thought that should work) and with that offer all MPI_T counters through PAPI (probably not the control variables, though). This should also allow us to provide "standardized" names for common performance variables (similar to what PAPI does for HW counters). The situation changes a bit, though, if MPI implementations query other information, like temperature or power, for themselves (Jeff, was this why you were asking?). In this case, the MPI implementation should use a standardized interface like PAPI itself (to avoid conflicts with tools), which could lead us to some strange circular SW dependencies. Martin On Dec 18, 2013, at 11:05 AM, Michael Knobloch <m.knobloch(a)fz-juelich.de> wrote: > Hi all, > > On 12/18/2013 05:09 PM, Marc-Andre Hermanns wrote: >> Hi Jeff, >> >> as a tools provider I would also like MPI_T to expose other sensors, as >> this means I don't have to support yet another metric interface. Not >> stacking a multitude of different metric interfaces on top of each other >> might also cut down on the overhead as you said. >> > > Coming from the same tools group I unfortunately have to disagree with > Marc-Andre on that. Most of the tools out there (if not all relevant for > HPC) already have good support for PAPI, which does a great job of > providing a portable API to counters of all kind. And it's not an easy > job to do, so I'm not sure every MPI implementation would want to copy > that work. And that would be necessary if we'd only want to use MPI_T as > a single counter source, otherwise we'd still have to maintain both (in > fact, we'd for sure have to, as there are other parallel programming > paradigms out there as well which tools have to support). > So what I'd prefer is a PAPI component exposing the MPI relevant > counters via PAPI, that way the tools could benefit from that with very > little additional work. > >> But could an implementation not expose such info right now? I thought >> the whole idea of not standardizing anything about counters, their names >> and semantics, is that implementations can expose whatever they care about. >> > > Here I agree, from what I read in the current standard the interface > doesn't guarantee me anything. So at least some kind of > high-level/low-level interface as in PAPI would certainly be desirable. > > -Michael > -- > Michael Knobloch > Institute for Advanced Simulation (IAS) > Jülich Supercomputing Centre (JSC) > Telefon: +49 2461 61-3546 > Telefax: +49 2461 61-6656 > E-Mail: m.knobloch(a)fz-juelich.de > Internet: http://www.fz-juelich.de/jsc > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-tools ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [mpiwg-tools] MPI_T usage question: other sensors
by Michael Knobloch 18 Dec '13

18 Dec '13

Hi all, On 12/18/2013 05:09 PM, Marc-Andre Hermanns wrote: > Hi Jeff, > > as a tools provider I would also like MPI_T to expose other sensors, as > this means I don't have to support yet another metric interface. Not > stacking a multitude of different metric interfaces on top of each other > might also cut down on the overhead as you said. > Coming from the same tools group I unfortunately have to disagree with Marc-Andre on that. Most of the tools out there (if not all relevant for HPC) already have good support for PAPI, which does a great job of providing a portable API to counters of all kind. And it's not an easy job to do, so I'm not sure every MPI implementation would want to copy that work. And that would be necessary if we'd only want to use MPI_T as a single counter source, otherwise we'd still have to maintain both (in fact, we'd for sure have to, as there are other parallel programming paradigms out there as well which tools have to support). So what I'd prefer is a PAPI component exposing the MPI relevant counters via PAPI, that way the tools could benefit from that with very little additional work. > But could an implementation not expose such info right now? I thought > the whole idea of not standardizing anything about counters, their names > and semantics, is that implementations can expose whatever they care about. > Here I agree, from what I read in the current standard the interface doesn't guarantee me anything. So at least some kind of high-level/low-level interface as in PAPI would certainly be desirable. -Michael -- Michael Knobloch Institute for Advanced Simulation (IAS) Jülich Supercomputing Centre (JSC) Telefon: +49 2461 61-3546 Telefax: +49 2461 61-6656 E-Mail: m.knobloch(a)fz-juelich.de Internet: http://www.fz-juelich.de/jsc ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Re: [mpiwg-tools] MPI_T usage question: other sensors
by Marc-Andre Hermanns 18 Dec '13

18 Dec '13

Hi Jeff, as a tools provider I would also like MPI_T to expose other sensors, as this means I don't have to support yet another metric interface. Not stacking a multitude of different metric interfaces on top of each other might also cut down on the overhead as you said. But could an implementation not expose such info right now? I thought the whole idea of not standardizing anything about counters, their names and semantics, is that implementations can expose whatever they care about. Cheers, Marc-Andre On 17.12.13 21:39, Jeff Squyres (jsquyres) wrote: > We talked briefly in the meeting last week about using MPI_T to gather non-MPI information, such as temperatures, fan speeds, CPU counters, ...etc. > > I was originally in favor of such things, but I think I got talked down by the argument of "it seems like there are already other tools to do this; why would we [also] expose these kinds of values through MPI_T?" > > Which I think I agree with. > > But I had a thought on the way home from Chicago as to why we would want to do this: using a single tool to gather these values could give a more consistent set of numbers. I.e., if you're gathering all the values through a single interface (e.g., MPI_T) you can get a consistent view of *all* sensor values, to include MPI introspective sensors (average queue depths, number of sends, ...etc.). If you're using multiple tools to gather these values, there's variability introduced by using multiple difference interfaces, each with different costs of reading values, etc. > > Is this a worthwhile observation? Or should MPI_T just stick to gathering MPI-related information? > -- Marc-Andre Hermanns German Research School for Simulation Sciences GmbH c/o Laboratory for Parallel Programming 52062 Aachen | Germany Tel +49 241 80 99753 Fax +49 241 80 6 99753 Web www.grs-sim.de Members: Forschungszentrum Jülich GmbH | RWTH Aachen University Registered in the commercial register of the local court of Düren (Amtsgericht Düren) under registration number HRB 5268 Registered office: Jülich Executive board: Prof. Marek Behr, Ph.D | Prof. Dr. Sebastian M. Schmidt

1 0

MPI_T usage question: other sensors
by Jeff Squyres (jsquyres) 17 Dec '13

17 Dec '13

We talked briefly in the meeting last week about using MPI_T to gather non-MPI information, such as temperatures, fan speeds, CPU counters, ...etc. I was originally in favor of such things, but I think I got talked down by the argument of "it seems like there are already other tools to do this; why would we [also] expose these kinds of values through MPI_T?" Which I think I agree with. But I had a thought on the way home from Chicago as to why we would want to do this: using a single tool to gather these values could give a more consistent set of numbers. I.e., if you're gathering all the values through a single interface (e.g., MPI_T) you can get a consistent view of *all* sensor values, to include MPI introspective sensors (average queue depths, number of sends, ...etc.). If you're using multiple tools to gather these values, there's variability introduced by using multiple difference interfaces, each with different costs of reading values, etc. Is this a worthwhile observation? Or should MPI_T just stick to gathering MPI-related information? -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-tools] Impromptu WG time
by Kathryn Mohror 11 Dec '13

11 Dec '13

Hi all, Here are the notes from this meeting: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools/notes-2013-12-1… Kathryn On Dec 10, 2013, at 11:38 AM, Kathryn Mohror <kathryn(a)llnl.gov> wrote: > The webex is up now. > > Kathryn > > On Dec 10, 2013, at 11:17 AM, Kathryn Mohror <kathryn(a)llnl.gov> wrote: > >> Hi all, >> >> If anyone wants to join our impromptu WG meeting, we are starting relatively soon. You can join by the webex link: >> https://cisco.webex.com/ciscosales/j.php?ED=247028927&UID=0&PW=NODI0MWJkOTA… >> >> Kathryn >> ______________________________________________________________ >> Kathryn Mohror, kathryn(a)llnl.gov, http://people.llnl.gov/mohror1 >> CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA >> >> >> >> >> _______________________________________________ >> mpiwg-tools mailing list >> mpiwg-tools(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-tools > > ______________________________________________________________ > Kathryn Mohror, kathryn(a)llnl.gov, http://people.llnl.gov/mohror1 > CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA > > > > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-tools _________________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov, http://scalability.llnl.gov/ Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA

1 0

Re: [mpiwg-tools] Impromptu WG time
by Kathryn Mohror 10 Dec '13

10 Dec '13

The webex is up now. Kathryn On Dec 10, 2013, at 11:17 AM, Kathryn Mohror <kathryn(a)llnl.gov> wrote: > Hi all, > > If anyone wants to join our impromptu WG meeting, we are starting relatively soon. You can join by the webex link: > https://cisco.webex.com/ciscosales/j.php?ED=247028927&UID=0&PW=NODI0MWJkOTA… > > Kathryn > ______________________________________________________________ > Kathryn Mohror, kathryn(a)llnl.gov, http://people.llnl.gov/mohror1 > CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA > > > > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-tools ______________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov, http://people.llnl.gov/mohror1 CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA

1 0