April 2013 - mpiwg-tools - mailman3.cels.anl.gov

Re: [Mpi3-tools] [MPI3 Fortran] Issues with Fortran bindings for MPI_Pcontrol
by Schulz, Martin 16 Apr '13

16 Apr '13

On Apr 16, 2013, at 2:11 AM, "N.M. Maclaren" <nmm1(a)cam.ac.uk> wrote: > On Apr 15 2013, Bill Long wrote: >> >> On 4/15/13 4:38 PM, Schulz, Martin wrote: >>> As a first simple approach, this could be done with >>> >>> MPI_Pcontrol_new(MPI_Info info) >> >> The original question seemed specific to the Fortran interface for MPI_PCONTROL. Would it be possible to keep the C interface, either as is or modified as suggested above, but delete the Fortran interface for this routine? That solves the immediate issue. >> >> I would argue that the few people who use this routine are "expert users" who would have little trouble writing their own MY_PCONTROL, written in C but callable from Fortran that had whatever interoperable arguments were desired, and internally called the C MPI_Pcontrol function. > > Or not even bother with MPI_Pcontrol, because it's less hassle to > implement the whole thing from scratch and provide their own interfaces. > I agree - this isn't likely to impact more than 0.1% of users, who are > quite capable of resolving any difficulties. I don't think that's quite true - Pcontrol is useful in many cases and is used by profilers (that are advertised as easy to use). It is certainly a bit more advanced usage, but in many cases we tell users where to put it, so they can run their code and get data, but it has to be easy for them to add it or they won't do it. Having to go through C in Fortran codes is not really a workable solution in this case. That's also how we ran into the problem that we are discussing: we had a user new to performance analysis during a tutorial and he wanted to profile a particular snapshot and so we asked him to add the calls. Due to the ierr issue, this crashed the code. The only way to make only having a C call work would be a separate library, but then that defeats the purpose of adding something that can just stay in the code without having to link to special libraries (which was the main reason for adding MPI_Pcontrol). MPI_Pcontrol is (unlike MPI_T) intended to be used in the application code and hence should have bindings in all languages supported by MPI. Martin > > > Regards, > Nick Maclaren. > ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Issues with Fortran bindings for MPI_Pcontrol
by Marc-Andre Hermanns 16 Apr '13

16 Apr '13

Jeff, > I certainly don't mean deleting PMPI from MPI-4. I mean putting > something better in MPI-4 that, over time, can replace PMPI (so that > PMPI *can* be deleted in some future version). I don't really understand why the issues with Pcontrol has an influence on whether the PMPI should be surpassed by anything else? Isn't it for use of the profiling tool only? Scalasca does not use Pcontrol at the moment. Could anyone give me an example of a tool using it, and some context in how it is used to understand what kind of interface is needed? Cheers, Marc-Andre -- Marc-Andre Hermanns German Research School for Simulation Sciences GmbH c/o Laboratory for Parallel Programming 52062 Aachen | Germany Tel +49 241 80 99753 Fax +49 241 80 6 99753 Web www.grs-sim.de Members: Forschungszentrum Jï¿½lich GmbH | RWTH Aachen University Registered in the commercial register of the local court of Dï¿½ren (Amtsgericht Dï¿½ren) under registration number HRB 5268 Registered office: Jï¿½lich Executive board: Prof. Marek Behr, Ph.D | Prof. Dr. Sebastian M. Schmidt

1 0

Re: [Mpi3-tools] Issues with Fortran bindings for MPI_Pcontrol
by Jeff Squyres (jsquyres) 15 Apr '13

15 Apr '13

On Apr 15, 2013, at 2:38 PM, "Schulz, Martin" <schulzm(a)llnl.gov> wrote: >> I'm really in favor of ditching the PMPI layer and putting something better there. > > I wouldn't go that far - the PMPI interface idea has been very successful and extremely useful and has lead to significant tool support. I think getting rid of it would be a huge setback. I'm sorry; I was typing hastily while closing up my laptop and boarding a plane. I certainly don't mean deleting PMPI from MPI-4. I mean putting something better in MPI-4 that, over time, can replace PMPI (so that PMPI *can* be deleted in some future version). -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-tools] Issues with Fortran bindings for MPI_Pcontrol
by Jeff Hammond 15 Apr '13

15 Apr '13

I assume JeffS meant something other than ditching PMPI, because I think that's the worst idea since Windows Me. Hamm On Mon, Apr 15, 2013 at 4:38 PM, Schulz, Martin <schulzm(a)llnl.gov> wrote: > Hi Jeff, > > On Apr 15, 2013, at 3:03 PM, "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> > wrote: > >> On Apr 14, 2013, at 9:01 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> >>> It seems better to correct the text than to break the existing >>> definition of MPI_PCONTROL. What error code would be returned >>> anyways? >> >> >> I think the point is that we don't know what could/would be returned by Pcontrol, because Pcontrol can be subsumed by a profiling library, and therefore it can do whatever it wants. >> >> The issue here is that the C API is varargs, and therefore allows you to pass in whatever you want to Pcontrol (at least in C), and therefore a tool (or the underlying C MPI API) can document what arguments it expects. >> >> There's (at least) two problems with this: >> >> 1. If a profiling library and the MPI library have differing arguments that they expect, even though varargs makes the different signatures compile-time safe, there could still be run-time errors (this is a contradiction that I've never understood in the definition of Pcontrol). >> >> 2. Fortran has no varargs. Prior to MPI 2.2, we actually did have a "..." in the Fortran prototype to indicate that you could do whatever you wanted for this signature. And that was fine in the days before strict dummy argument type checking. But now we mandate struct signature checking, and ... had to go, because it's not legal Fortran. And now the text doesn't say "Fortran implementations can add whatever parameters they want" (which has problems, anyway). >> >> I'm really in favor of ditching the PMPI layer and putting something better there. > > I wouldn't go that far - the PMPI interface idea has been very successful and extremely useful and has lead to significant tool support. I think getting rid of it would be a huge setback. > > I do agree, however, that the current Pcontrol is problematic, especially with the more and more creative uses of the PMPI interface. In most cases tools only use the single integer and in some of our tools we started using one additional argument (often a pointer) to transfer extra information. Hence, stepping back from the vargs in C is likely not a problem for any tool. > > The bigger issue is that the current scheme basically makes the Pcontrol calls tool specific (i.e., they must be added targeted for a specific tool). This was fine when Pcontrol was mainly used for turning on and off profiling, but we are now using that interface to communicate application information to the tool (e.g., phase boundaries, load balance data, ï¿½). These are all things that should stay in the code for easier maintenance and hence an application would have to be able to specify a target tool and a tool must be able to filter calls not intended for it. We started doing this using magic numbers as the arguments, but this is obviously far from ideal. Better would be some kind of name space that tools put themselves into and pass that as an argument. > > As a first simple approach, this could be done with > > MPI_Pcontrol_new(MPI_Info info) > > but the need for string parsing on every MPI_Pcontrol call could be a problem. > > Martin > > > > >> >> -- >> Jeff Squyres >> jsquyres(a)cisco.com >> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >> > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [Mpi3-tools] Issues with Fortran bindings for MPI_Pcontrol
by Schulz, Martin 15 Apr '13

15 Apr '13

Hi Jeff, On Apr 15, 2013, at 3:03 PM, "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> wrote: > On Apr 14, 2013, at 9:01 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > >> It seems better to correct the text than to break the existing >> definition of MPI_PCONTROL. What error code would be returned >> anyways? > > > I think the point is that we don't know what could/would be returned by Pcontrol, because Pcontrol can be subsumed by a profiling library, and therefore it can do whatever it wants. > > The issue here is that the C API is varargs, and therefore allows you to pass in whatever you want to Pcontrol (at least in C), and therefore a tool (or the underlying C MPI API) can document what arguments it expects. > > There's (at least) two problems with this: > > 1. If a profiling library and the MPI library have differing arguments that they expect, even though varargs makes the different signatures compile-time safe, there could still be run-time errors (this is a contradiction that I've never understood in the definition of Pcontrol). > > 2. Fortran has no varargs. Prior to MPI 2.2, we actually did have a "..." in the Fortran prototype to indicate that you could do whatever you wanted for this signature. And that was fine in the days before strict dummy argument type checking. But now we mandate struct signature checking, and ... had to go, because it's not legal Fortran. And now the text doesn't say "Fortran implementations can add whatever parameters they want" (which has problems, anyway). > > I'm really in favor of ditching the PMPI layer and putting something better there. I wouldn't go that far - the PMPI interface idea has been very successful and extremely useful and has lead to significant tool support. I think getting rid of it would be a huge setback. I do agree, however, that the current Pcontrol is problematic, especially with the more and more creative uses of the PMPI interface. In most cases tools only use the single integer and in some of our tools we started using one additional argument (often a pointer) to transfer extra information. Hence, stepping back from the vargs in C is likely not a problem for any tool. The bigger issue is that the current scheme basically makes the Pcontrol calls tool specific (i.e., they must be added targeted for a specific tool). This was fine when Pcontrol was mainly used for turning on and off profiling, but we are now using that interface to communicate application information to the tool (e.g., phase boundaries, load balance data, ï¿½). These are all things that should stay in the code for easier maintenance and hence an application would have to be able to specify a target tool and a tool must be able to filter calls not intended for it. We started doing this using magic numbers as the arguments, but this is obviously far from ideal. Better would be some kind of name space that tools put themselves into and pass that as an argument. As a first simple approach, this could be done with MPI_Pcontrol_new(MPI_Info info) but the need for string parsing on every MPI_Pcontrol call could be a problem. Martin > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Issues with Fortran bindings for MPI_Pcontrol
by Mohror, Kathryn 15 Apr '13

15 Apr '13

Hello all, Just as a heads up, we will be discussing issues with the new Fortran bindings and the profiling interface during this week's tools wg call (Thursday 8:00 AM PDT). Craig Rasmussen and Rolf Rabenseifner have graciously agreed to join the call to explain to us what the specific problems are and what might be done to address them. It would be great if others want to join and help out with the discussion. I'll send out a reminder later in the week. Kathryn On Apr 15, 2013, at 1:03 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com<mailto:[email protected]>> wrote: On Apr 14, 2013, at 9:01 PM, Jeff Hammond <jhammond(a)alcf.anl.gov<mailto:[email protected]>> wrote: It seems better to correct the text than to break the existing definition of MPI_PCONTROL. What error code would be returned anyways? I think the point is that we don't know what could/would be returned by Pcontrol, because Pcontrol can be subsumed by a profiling library, and therefore it can do whatever it wants. The issue here is that the C API is varargs, and therefore allows you to pass in whatever you want to Pcontrol (at least in C), and therefore a tool (or the underlying C MPI API) can document what arguments it expects. There's (at least) two problems with this: 1. If a profiling library and the MPI library have differing arguments that they expect, even though varargs makes the different signatures compile-time safe, there could still be run-time errors (this is a contradiction that I've never understood in the definition of Pcontrol). 2. Fortran has no varargs. Prior to MPI 2.2, we actually did have a "..." in the Fortran prototype to indicate that you could do whatever you wanted for this signature. And that was fine in the days before strict dummy argument type checking. But now we mandate struct signature checking, and ... had to go, because it's not legal Fortran. And now the text doesn't say "Fortran implementations can add whatever parameters they want" (which has problems, anyway). I'm really in favor of ditching the PMPI layer and putting something better there. -- Jeff Squyres jsquyres(a)cisco.com<mailto:[email protected]> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ _______________________________________________ Mpi3-tools mailing list Mpi3-tools(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools ______________________________________________________________ Kathryn Mohror, kathryn(a)llnl.gov<mailto:[email protected]>, http://people.llnl.gov/mohror1 CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA

1 0

Re: [Mpi3-tools] Issues with Fortran bindings for MPI_Pcontrol
by Jeff Squyres (jsquyres) 15 Apr '13

15 Apr '13

On Apr 14, 2013, at 9:01 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > It seems better to correct the text than to break the existing > definition of MPI_PCONTROL. What error code would be returned > anyways? I think the point is that we don't know what could/would be returned by Pcontrol, because Pcontrol can be subsumed by a profiling library, and therefore it can do whatever it wants. The issue here is that the C API is varargs, and therefore allows you to pass in whatever you want to Pcontrol (at least in C), and therefore a tool (or the underlying C MPI API) can document what arguments it expects. There's (at least) two problems with this: 1. If a profiling library and the MPI library have differing arguments that they expect, even though varargs makes the different signatures compile-time safe, there could still be run-time errors (this is a contradiction that I've never understood in the definition of Pcontrol). 2. Fortran has no varargs. Prior to MPI 2.2, we actually did have a "..." in the Fortran prototype to indicate that you could do whatever you wanted for this signature. And that was fine in the days before strict dummy argument type checking. But now we mandate struct signature checking, and ... had to go, because it's not legal Fortran. And now the text doesn't say "Fortran implementations can add whatever parameters they want" (which has problems, anyway). I'm really in favor of ditching the PMPI layer and putting something better there. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-tools] [MPI3 Fortran] Issues with Fortran bindings for MPI_Pcontrol
by N.M. Maclaren 15 Apr '13

15 Apr '13

On Apr 14 2013, Schulz, Martin wrote: > > We ran into this, when one developer added Pcontrol calls with ierr > arguments, which led to a seg fault and then he pointed to the statement > that all MPI routines require that argument. > > We (Jeff S., Dave G., Bill, and I) had a brief discussion on this and it > turns out that MPICH and Open MPI actually implement this differently: > MPICH includes ierr (following the general rule) and Open MPI does not > (following the specified prototype). This does cause problems for codes > that are used on both implementations (as we had in our case that led to > this discussion) and hence breaks portability. > > The two options seem to be to either change the prototype or allow for > exceptions to the general rule and there are obvious pros and cons for > either option. That's why we thought it would be better to get both > working groups involved. > >Any comments or other good ideas on how to fix this? My main comment isn't particularly helpful for a solution, and is probably well-known. When I first looked at MPI_Pcontrol, I tried to work out what it could be used for, and failed. If anyone is writing a PMPI-based tool and needs programmatic control, it is both unlikely to be just a single integer and is absolutely trivial to provide extra tool-dependent functions. Because MPI_Pcontrol had no defined semantics in MPI 2.2, it couldn't be used by any program that wants to interoperate with arbitrary PMPI-based tools. So there is unlikely to be a significant amount of old code using it, and changes should not be a big deal. I would favour restoring IERR as an optional argument. Regards, Nick Maclaren.

1 0

Re: [Mpi3-tools] Issues with Fortran bindings for MPI_Pcontrol
by Jeff Hammond 14 Apr '13

14 Apr '13

It seems better to correct the text than to break the existing definition of MPI_PCONTROL. What error code would be returned anyways? Jeff On Sun, Apr 14, 2013 at 3:40 PM, Schulz, Martin <schulz6(a)llnl.gov> wrote: > Hi all, > > I am sending this to both the Fortran list and the tools list, since this affects both (sorry if some of you get this twice). > > The MPI standard currently has some inconsistencies with respect to how the MPI_Pcontrol call is defined. The prototype in the tools chapter does not have the usual ierr return code (which is an historical artifact from the use of variable arguments up to MPI 2.2). In particular, it is now stated as: > > MPI_Pcontrol(level) BIND(C) > INTEGER, INTENT(IN) :: level > > MPI_PCONTROL(LEVEL) > INTEGER LEVEL > > This is, however, in contradiction to the text on page 19: > > "All MPI Fortran subroutines have a return code in the last argument. With USE mpi_f08, this last argument is declared as OPTIONAL, except for user-defined callback func- tions (e.g., COMM_COPY_ATTR_FUNCTION) and their predefined callbacks (e.g., MPI_NULL_COPY_FN). A few MPI operations which are functions do not have the return code argument." > > MPI_Pcontrol is not a function and hence shouldn't be covered by the last sentence. > > We ran into this, when one developer added Pcontrol calls with ierr arguments, which led to a seg fault and then he pointed to the statement that all MPI routines require that argument. > > We (Jeff S., Dave G., Bill, and I) had a brief discussion on this and it turns out that MPICH and Open MPI actually implement this differently: MPICH includes ierr (following the general rule) and Open MPI does not (following the specified prototype). This does cause problems for codes that are used on both implementations (as we had in our case that led to this discussion) and hence breaks portability. > > The two options seem to be to either change the prototype or allow for exceptions to the general rule and there are obvious pros and cons for either option. That's why we thought it would be better to get both working groups involved. > > Any comments or other good ideas on how to fix this? > > Thanks! > ` > Martin > > > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [Mpi3-tools] [MPI3 Fortran] Issues with Fortran bindings for MPI_Pcontrol
by Bill Long 14 Apr '13

14 Apr '13

On 4/14/13 3:40 PM, Schulz, Martin wrote: > Hi all, > > I am sending this to both the Fortran list and the tools list, since this affects both (sorry if some of you get this twice). > > The MPI standard currently has some inconsistencies with respect to how the MPI_Pcontrol call is defined. The prototype in the tools chapter does not have the usual ierr return code (which is an historical artifact from the use of variable arguments up to MPI 2.2). In particular, it is now stated as: > > MPI_Pcontrol(level) BIND(C) > INTEGER, INTENT(IN) :: level > > MPI_PCONTROL(LEVEL) > INTEGER LEVEL > > This is, however, in contradiction to the text on page 19: > > "All MPI Fortran subroutines have a return code in the last argument. With USE mpi_f08, this last argument is declared as OPTIONAL, except for user-defined callback func- tions (e.g., COMM_COPY_ATTR_FUNCTION) and their predefined callbacks (e.g., MPI_NULL_COPY_FN). A few MPI operations which are functions do not have the return code argument." > > MPI_Pcontrol is not a function and hence shouldn't be covered by the last sentence. A view from someone who has not been involved on either side of the argument: The C prototype is int MPI_Pcontrol(const int level, ...) which says to me that MPI_Pcontrol is a function that returns a result value. Absent indication otherwise, the int results from the C MPI_* functions are assumed to be error codes. So, it seems to me that the Fortran interfaces should have included the IERROR argument, and the omission was just an oversight. Alternatively, the documentation for MPI_Pcontrol should explicitly say that the return value is not an error code, and has no meaning. In the later case, the Fortran wrapper would just discard the return value. Cheers, Bill > > We ran into this, when one developer added Pcontrol calls with ierr arguments, which led to a seg fault and then he pointed to the statement that all MPI routines require that argument. > > We (Jeff S., Dave G., Bill, and I) had a brief discussion on this and it turns out that MPICH and Open MPI actually implement this differently: MPICH includes ierr (following the general rule) and Open MPI does not (following the specified prototype). This does cause problems for codes that are used on both implementations (as we had in our case that led to this discussion) and hence breaks portability. > > The two options seem to be to either change the prototype or allow for exceptions to the general rule and there are obvious pros and cons for either option. That's why we thought it would be better to get both working groups involved. > > Any comments or other good ideas on how to fix this? > > Thanks! > ` > Martin > > > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101

1 0