April 2012 - mpiwg-tools - mailman3.cels.anl.gov

[Mpi3-tools] Next MPI-3 Tools WG Meeting & Discussions with the FT group
by Schulz, Martin 27 Apr '12

27 Apr '12

Hi all, Just as a reminder: the next MPI-3 tools WG meeting is scheduled for Monday April 30th at 8am PDT, 11am EDT, and 5pm MSZ. The dial-in information is, as always, on the MPI-3 tools Wiki at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools For this next meeting, we'll be joined by the FT working group - they will give an overview of the current state of the FT proposal, followed by a discussion on whether this proposal has any impact on tools that would have to be implemented on top of this interface (both through debugging interfaces and the MPI profiling interface). Talk to you on Monday, Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by George Bosilca 24 Apr '12

24 Apr '12

MCW is only meaningful in a static world. I'm not a tool expert, but I do not recall hearing about any tools that can cope with a dynamic environment where multiple MCW coexist in the context of the same application (and such application do exist). What would be a "reasonable" unique identifier in this context? george. On Apr 24, 2012, at 18:01 , Jeffrey Squyres wrote: > On Apr 24, 2012, at 5:36 PM, Dries Kimpe wrote: > >>> Any kind of unique value would allow the tool to have a unique, safe filename. >> >> What about a tmpnam for MPI then :-) >> >> No need to bring details like rank->thread->process mapping into this if >> all you want is a guaranteed unique filename among the MPI ranks of the >> application. > > > You do want it to be able to be mapped back to individual MPI (and/or OS) processes, though. MCW rank is a pretty easy way to identify things. > > If it weren't for this desire, PID would be sufficient. But users find it to be a PITA to map PID back to whatever MCW rank it was. That's why tools are looking for MPI-specific information that is meaningful to the user. > > FWIW: Also add Valgrind to the list of tools that Rayson mentioned (there's an option to suffix the valgrind output files with the MCW rank). > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by Rayson Ho 24 Apr '12

24 Apr '12

On Tue, Apr 24, 2012 at 5:36 PM, Dries Kimpe <dkimpe(a)mcs.anl.gov> wrote: > No need to bring details like rank->thread->process mapping into this if > all you want is a guaranteed unique filename among the MPI ranks of the > application. Hi Dries, We are trying to find an environment variable that would also allow us to locate a task, and the MPI rank seems to be the logical choice. My previous email has URLs to other projects that reference the MPI rank environment variable. Having the rank env. var. (or something logical or meaningful that is unique and can represent a task) that would allow us to link the profile back to the code is the cleanest IMO, as the profiling tools can't link against any of the MPI libraries. So we can't issue an MPI call to find the rank in the COMM World. BTW, many of those tools are loaded into the executable via LD_PRELOAD, and that means before the first call to MPI_Init(). Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ > > ï¿½Dries > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by Jeffrey Squyres 24 Apr '12

24 Apr '12

On Apr 24, 2012, at 5:36 PM, Dries Kimpe wrote: >> Any kind of unique value would allow the tool to have a unique, safe filename. > > What about a tmpnam for MPI then :-) > > No need to bring details like rank->thread->process mapping into this if > all you want is a guaranteed unique filename among the MPI ranks of the > application. You do want it to be able to be mapped back to individual MPI (and/or OS) processes, though. MCW rank is a pretty easy way to identify things. If it weren't for this desire, PID would be sufficient. But users find it to be a PITA to map PID back to whatever MCW rank it was. That's why tools are looking for MPI-specific information that is meaningful to the user. FWIW: Also add Valgrind to the list of tools that Rayson mentioned (there's an option to suffix the valgrind output files with the MCW rank). -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by Dries Kimpe 24 Apr '12

24 Apr '12

* Jeffrey Squyres <jsquyres(a)cisco.com> [2012-04-24 16:52:55]: > On Apr 24, 2012, at 4:24 PM, William Gropp wrote: > > Yes, an MPI process need not be an OS process, and there is in fact a long history of such implementations, starting (as far as I know) with TMPI. An easy out is to make this optional and only apply when there is such a one-to-one mapping of MPI and OS processes. > That would defeat the use-case that I proposed (tools needing a unique ID for the process). Other options: > - provide a comma-delimited list of MCW ranks in the OS process > - provide the lowest (or highest? or median? or ...?) MCW rank in the OS process > - ...some other unique value > Any kind of unique value would allow the tool to have a unique, safe filename. What about a tmpnam for MPI then :-) No need to bring details like rank->thread->process mapping into this if all you want is a guaranteed unique filename among the MPI ranks of the application. Dries

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by Rayson Ho 24 Apr '12

24 Apr '12

On Tue, Apr 24, 2012 at 4:52 PM, Jeffrey Squyres <jsquyres(a)cisco.com> wrote: > Any kind of unique value would allow the tool to have a unique, safe filename. > > The MPI (or runtime) is in a good position to provide this unique value because it has global knowledge of the entire MPI job, where the tool does/may not. Thanks Jeff for starting the discussion! Not only the Google gperftools would find having this unique value useful (currently, we plan to use it mainly for the output filename), there are also other tools that actually use the MPI rank to match the profiles to the MPI tasks: Sun Studio Performance Tools' "collect": http://developers.sun.com/sunstudio/documentation/techart/mpi_apps/ Or DTrace - for the user to relate the profiles to the MPI tasks: http://developers.sun.com/solaris/articles/dtrace_article.html#howto Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by Rayson Ho 24 Apr '12

24 Apr '12

On Tue, Apr 24, 2012 at 4:24 PM, William Gropp <wgropp(a)illinois.edu> wrote: > Yes, an MPI process need not be an OS process, and there is in fact a long > history of such implementations, starting (as far as I know) with TMPI. ï¿½An > easy out is to make this optional and only apply when there is such a > one-to-one mapping of MPI and OS processes. I believe in those "threaded MPI task" environments, internally inside the MPI library there is still a way to tell a process (and the thread group inside that process) on particular host where to start from. I think it is more straightforward than communicating the RANK during MPI_Init() from an MPI implementer point of view. So whoever starts the MPI tasks on a node should be able to set something unique in the environment... And BTW, thanks William & Rob Ross for sending me the MPITypes library 2 years ago. I played with it for a while and I will eventually go back to it! Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ > > Bill > > Williamï¿½Gropp > Director, Parallel Computing Institute > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > On Apr 24, 2012, at 12:46 PM, John DelSignore wrote: > > > I'm not familiar with any implementations where there are multiple MPI > processes per OS process. I did read Snir's proposal, but I can't say that > I'm less confused about what it implies relative to an MPI process's rank in > MPI_COMM_WORLD. Are you saying that a single OS process may have contain > multiple MPI processes where each MPI process has a unique rank in > MPI_COMM_WORLD? And therefore, if we try to use the environment to identify > an OS process's rank in MPI_COMM_WORLD, it would have to be more like a list > of ranks rather than a single value? > > > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by John DelSignore 24 Apr '12

24 Apr '12

Then it sounds like we have bigger problems, like the MPIR proctable is inadequate to represent this configuration. Jeffrey Squyres wrote: > On Apr 24, 2012, at 3:46 PM, John DelSignore wrote: > >> I'm not familiar with any implementations where there are multiple MPI processes per OS process. I did read Snir's proposal, but I can't say that I'm less confused about what it implies relative to an MPI process's rank in MPI_COMM_WORLD. Are you saying that a single OS process may have contain multiple MPI processes where each MPI process has a unique rank in MPI_COMM_WORLD? And therefore, if we try to use the environment to identify an OS process's rank in MPI_COMM_WORLD, it would have to be more like a list of ranks rather than a single value? > > > Yes. > > And I believe IBM wants to make a comeback with this model (threads that have different MPI_COMM_WORLD rank values, for example). >

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by Jeffrey Squyres 24 Apr '12

24 Apr '12

On Apr 24, 2012, at 4:24 PM, William Gropp wrote: > Yes, an MPI process need not be an OS process, and there is in fact a long history of such implementations, starting (as far as I know) with TMPI. An easy out is to make this optional and only apply when there is such a one-to-one mapping of MPI and OS processes. That would defeat the use-case that I proposed (tools needing a unique ID for the process). Other options: - provide a comma-delimited list of MCW ranks in the OS process - provide the lowest (or highest? or median? or ...?) MCW rank in the OS process - ...some other unique value Any kind of unique value would allow the tool to have a unique, safe filename. The MPI (or runtime) is in a good position to provide this unique value because it has global knowledge of the entire MPI job, where the tool does/may not. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-tools] Standardized MPI_COMM_WORLD rank environment variable?
by Jeffrey Squyres 24 Apr '12

24 Apr '12

On Apr 24, 2012, at 3:46 PM, John DelSignore wrote: > I'm not familiar with any implementations where there are multiple MPI processes per OS process. I did read Snir's proposal, but I can't say that I'm less confused about what it implies relative to an MPI process's rank in MPI_COMM_WORLD. Are you saying that a single OS process may have contain multiple MPI processes where each MPI process has a unique rank in MPI_COMM_WORLD? And therefore, if we try to use the environment to identify an OS process's rank in MPI_COMM_WORLD, it would have to be more like a list of ranks rather than a single value? Yes. And I believe IBM wants to make a comeback with this model (threads that have different MPI_COMM_WORLD rank values, for example). -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0