March 2013 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Jeff Squyres (jsquyres) 21 Mar '13

21 Mar '13

On Mar 21, 2013, at 8:44 AM, "Schulz, Martin" <schulzm(a)llnl.gov> wrote: > I feared that this would be the reaction and that were also my original reservations. From a tools perspective this would indeed be the easiest and cleanest solution, but I can see the downside. Implementation effort for rewriting aside (which I realize is important), do you expect a large performance impact, though? I can't speak for NEC. The reason we did it this was in OMPI was not for performance -- it was for implementation-specific reasons. That is, there are some cases where we want to do something different for the Fortran implementation vs. the C implementation. One obvious example of this is for just about any function involving callbacks. MPICH may not have this issue because their handles are always integers, but for OMPI, C handles=pointers/F handles=integers, so we have to do some swizzling/de-swizzling for the conversion. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Schulz, Martin 21 Mar '13

21 Mar '13

Hi Jeff, all, On Mar 21, 2013, at 8:38 AM, "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> wrote: > On Mar 21, 2013, at 8:26 AM, Craig Rasmussen <rasmus(a)cas.uoregon.edu> wrote: > >> I talked with Martin yesterday and he would be happy to have the requirement in the standard that the Fortran wrappers MUST call the MPI C interface. This would much simplify things for the tools community. (I had earlier thought he would object to this solution.) > > No! This is NOT a good idea to mandate. > > Open MPI currently has a bunch of Fortran functions that do NOT call the corresponding MPI_<foo> C function. Instead, they call some other back-end, internal function. > > I believe the NEC Fortran MPI API may do similar things. I feared that this would be the reaction and that were also my original reservations. From a tools perspective this would indeed be the easiest and cleanest solution, but I can see the downside. Implementation effort for rewriting aside (which I realize is important), do you expect a large performance impact, though? Martin > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Jeff Squyres (jsquyres) 21 Mar '13

21 Mar '13

On Mar 21, 2013, at 8:26 AM, Craig Rasmussen <rasmus(a)cas.uoregon.edu> wrote: > I talked with Martin yesterday and he would be happy to have the requirement in the standard that the Fortran wrappers MUST call the MPI C interface. This would much simplify things for the tools community. (I had earlier thought he would object to this solution.) No! This is NOT a good idea to mandate. Open MPI currently has a bunch of Fortran functions that do NOT call the corresponding MPI_<foo> C function. Instead, they call some other back-end, internal function. I believe the NEC Fortran MPI API may do similar things. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Craig Rasmussen 21 Mar '13

21 Mar '13

On Mar 20, 2013, at 12:29 PM, Hubert Ritzdorf wrote: > Hi, > > I think that Bill's proposal kills the Fortran Profiling Interface. > > If the user defines a > > subroutine MPI_Test (....) > > do something > > call PMPI _Test() > > do something > > end subroutine MPI_Test > > then this function will never be executed because the Fortrran compiler may have already > inlined the MPI_Test subroutine in the original code or the module version of MPI_Test is used. > I believe this is the case. But the user could implement something similar using a BIND(C) interface in Fortran. -craig

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Craig Rasmussen 21 Mar '13

21 Mar '13

On Mar 20, 2013, at 10:09 AM, Rolf Rabenseifner wrote: > This mail contains a proposal how we can solve the current > Fortran problems and possibly making the tools people also happy. > > Implications and proposal: > - We should rewrite the linker names section MPI-3.0 17.1.5 > with following goals: > - Allow additionally to define that the PMPI is provided > only on the MPI C interface and the Fortran MPI routines > internally call the C interface; I talked with Martin yesterday and he would be happy to have the requirement in the standard that the Fortran wrappers MUST call the MPI C interface. This would much simplify things for the tools community. (I had earlier thought he would object to this solution.) Rolf, I'm not sure what you mean by additionally. It seems to me the easiest solution is to drop BIND(C) entirely from the MPI-3 use_mpi_f08 API entirely. This would make things much simpler for both the MPI implementations and the tools community. There would mean that the PMPI Fortran routines can go away. The objection to this proposal will like come from vendors who implement MPI routines in Fortran for performance. These implementations would require a Fortran wrapper that makes a call to the C interface via BIND(C). Note that this C interface can still be written in Fortran using BIND(C). In addition, as Bill points out, this is not necessarily a performance problem because the compiler could inline the wrapper call. > again, a macro definition in mpi.h tells whether this > is chosen. > Question: Would this cause a problem when linking a > C or Fortran user application with some own > MPI_Test etc. profiling routines that internally call > PMPI_Test? It would mean that a Fortran user would have to implement their own MPI_Test routine in C or in Fortran using BIND(C). However, this would require some sophistication on the part of the Fortran programmer. > - If Fortran profiling is done through the MPI C binding > then there are no rules about the Fortran linker names > needed, and therefore, one may put the thin Fortran > wrappers into a Fortran module. > - We can define BIND(C) for MPI routine groups as a > future option that is currently not used. > This would reduce the current effort for the tools people. > Is there anybody who wants to implement one of the Fortran > routine groups with BIND(C)? > > I would expect, that this is the best to solve the problem > that #364 prohibits BIND(C) for some routines and implementors > want to do the same method for all routines. I suppose the _f08 linker names can't go away for now but presumably later (MPI 4?) they could go away? > > Don't forget that BIND(C) and TR29113 is still important > for implementing these thin wrappers internally. > > Without using BIND(C) on the official mpi_f08 and mpi module > interface, some features of TR29113 are not used: > - the extension for interoperabple CHARACTER(LEN=*), > - the extension for OPTIONAL dummy arguments. > > Other TR29113 features are heavily used: > - The DIMENSION(..), TYPE(*) is needed for some internal > C-written MPI_Isend_desc. > - The ASYNCHRONOUS usage for communication buffers. > > Should we go in this direction? If we can get agreement with the tools community (Martin said he will poll them) I believe this is the best way to go. It is a path that leads eventually to a much smaller and cleaner Fortran MPI implementation. I think we also need agreement from the MPI vendors with purely Fortran implementation for any of the MPI functions. -craig

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Rolf Rabenseifner 21 Mar '13

21 Mar '13

Thank you for the answers. Four comments: 1. About the linker names and intenal call sequence: >> I think that Bill's proposal kills the Fortran Profiling Interface. >> If the user defines a >> subroutine MPI_Test (....) >> do something >> call PMPI _Test() >> do something >> end subroutine MPI_Test Yes, this is true. Having these two implementation options A. Internal real MPI C calls with A1. standardized Fortran linker names, or with A2. non-standardized Fortran names, i.e. within a Fortran module. B. Standardized Fortran linker names. would kill the upper Fortran example as soon as an implementation chooses A2. Bill, do you have an idea how to solve the A2 problem. Is it still possible to provide the standardized names when you somehow inline the wrappers? Then, we would have only A1 which allows that a pure PMPI Fortran user can freely choose to intercept the Fortran or the C interface. 2. About MPI_Send_desc: Yes, Bill, you are absolutely right. I thought that the MPI_Isend_desc internally calls MPI_Isend, but this would be an implementation that internally copies the non-contiguous Fortran application buffer into a contiguous scratch buffer and then calls MPI_Isend (the scratch buffer is released latest in MPI_Wait or MPI_Test). Optimized solutions would do flattening on the fly of the combination of the description of the Fortran buffer combined with the description in the datatype handle, and this double-flattening within the MPI_Isend_desc. Therefore special naming for option A must be defined. This is only for the MPI routine groups MPI_SEND and MPI_ISEND (see MPI-3.0, page 607 and ticket #364) 3. Routine groups: All what we do should be done per routine group, i.e., mpi.h tells by macros how it is done within the specific routine groups. 4. My apologies to Hubert. The list should have been > >> I need at least answers for > >> - mpich (Bill Long), > >> - OpenMPI (Jeff or Craig), - NEC MPI (Hubert), - other, if there are other independent mpi_f08 implementations, > >> - some tools people. Best regards Rolf ----- Original Message ----- > From: "Bill Long" <longb(a)cray.com> > To: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> > Cc: "Martin Schulz" <schulzm(a)llnl.gov>, "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> > Sent: Wednesday, March 20, 2013 9:40:08 PM > Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind > My impression is that Rolf's proposal was to offer two options: > > a) no bind(c) on the Fortran wrappers, no requirements on the Fortran > external names, but the wrappers are REQUIRED to call the C library > routines. > > b) specific external names are required for the Fortran-called > routines, > but they do not have to be wrappers. (Still a bit unclear to me in > this case where bind(c) might be required). > > On 3/20/13 2:58 PM, Jeff Squyres (jsquyres) wrote: > > Not only that, I think that there are some cases where tools *need* > > to intercept the Fortran call, not the C call. In OMPI, for example, > > we have a few corner cases where the Fortran MPI subroutine does > > *not* call the corresponding C MPI API function. Instead, it calls a > > back-end OMPI routine. > > > > True for implementations using option (b), but not for the (a) case. > > > One obvious case that jumps to mind is anything involving a > > callback. > > Looking at the callback examples on page 268, I have some questions: > > 1) The BIND(C) and an argument of type LOGICAL are likely > incompatible, > so either the BIND(C) or the LOGICAL needs to go. > > 2) The external names for the C and Fortran versions are different. I > assume this does not matter assuming that the function that is calling > these has only an address, and not a name, to call. But how does the > function know the calling signature for the callback routine with only > a > procedure pointer to work with? > > If I were writing a code with a callback like this, I would want to > assume the C prototype even of the code was in Fortran. I assume the > existence of the separate (and apparently flawed) Fortran interface is > for legacy code. But there should not be legacy mpi_f08 code. A bit > confusing. > > > > > If you have integer handles in C, they might be able to get away > > with not swizzling/de-swizzling the arguments to the callback to the > > proper language. But this is not always the case. > > > > > > > > On Mar 20, 2013, at 12:29 PM, Hubert Ritzdorf > > <Hubert.Ritzdorf(a)emea.nec.com> wrote: > > > >> Hi, > >> > >> I think that Bill's proposal kills the Fortran Profiling Interface. > >> > >> If the user defines a > >> > >> subroutine MPI_Test (....) > >> > >> do something > >> > >> call PMPI _Test() > >> > >> do something > >> > >> end subroutine MPI_Test > > I was under the impression that users would not directly call PMPI > routines. Those were for the tools people. Why would a normal user > want to do something like the above? > > >> > >> then this function will never be executed because the Fortrran > >> compiler may have already > >> inlined the MPI_Test subroutine in the original code or the module > >> version of MPI_Test is used. > > In the scope where this routine is called, the Fortran wrapper would > have to be inaccessible. That would be done by NOT having a USE of the > module in this scope. The external name of the module procedure would > be > mangled to include the name of the module, whereas a stand-alone > routine like this would have an external name like mpi_test_ . The > linker would see these as different routines. > > > Cheers, > Bill > > > >> > >> Hubert > >> ________________________________________ > >> From: mpi3-fortran-bounces(a)lists.mpi-forum.org > >> [mpi3-fortran-bounces(a)lists.mpi-forum.org] on behalf of Rolf > >> Rabenseifner [rabenseifner(a)hlrs.de] > >> Sent: Wednesday, March 20, 2013 6:09 PM > >> To: longb(a)cray.com; MPI-3 Fortran working group > >> Cc: Martin Schulz > >> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 > >> interface - issue with the LOGICAL kind > >> > >> This mail contains a proposal how we can solve the current > >> Fortran problems and possibly making the tools people also happy. > >> > >> I need at least answers for > >> - mpich (Bill Long), > >> - OpenMPI (Jeff or Craig), > >> - some tools people. > >> > >> Let me summarize what I've learnt so far for mpi_f08: > >> > >> - #364 about LOGICAL correction is a must. > >> - MPI-3.0 allows BIND(C) and non-BIND(C), with the > >> new restrictions in #364 > >> - BIND(C) needs TR29113, therefore any implementation now > >> should use non-BIND(C) for all non-TR29113 compilers. > >> - Most (or all) try to use the non-BIND(C) interface. > >> - Best way to do this are thin wrappers in Fortran that > >> internally call > >> - the oficial MPI-3.0 C binding > >> (here, the tools people have a problem because they > >> expect to wrap the Fortran interface and may therefore > >> double-wrap), or > >> - or an internal MPI-3.0 C routine > >> (which is half-perfect for the tools folk, because > >> they have to handle the implementation-dependant > >> Fortran names with underscores). > >> - And at least Bill wants to include the Fortran MPI > >> library routines itselves in the mpi_f08 module. > >> This is in conflict with the current MPI-3.0 > >> linker naming rules. > >> > >> Implications and proposal: > >> - We should rewrite the linker names section MPI-3.0 17.1.5 > >> with following goals: > >> - Allow additionally to define that the PMPI is provided > >> only on the MPI C interface and the Fortran MPI routines > >> internally call the C interface; > >> again, a macro definition in mpi.h tells whether this > >> is chosen. > >> Question: Would this cause a problem when linking a > >> C or Fortran user application with some own > >> MPI_Test etc. profiling routines that internally call > >> PMPI_Test? > >> - If Fortran profiling is done through the MPI C binding > >> then there are no rules about the Fortran linker names > >> needed, and therefore, one may put the thin Fortran > >> wrappers into a Fortran module. > >> - We can define BIND(C) for MPI routine groups as a > >> future option that is currently not used. > >> This would reduce the current effort for the tools people. > >> Is there anybody who wants to implement one of the Fortran > >> routine groups with BIND(C)? > >> > >> I would expect, that this is the best to solve the problem > >> that #364 prohibits BIND(C) for some routines and implementors > >> want to do the same method for all routines. > >> > >> Don't forget that BIND(C) and TR29113 is still important > >> for implementing these thin wrappers internally. > >> > >> Without using BIND(C) on the official mpi_f08 and mpi module > >> interface, some features of TR29113 are not used: > >> - the extension for interoperabple CHARACTER(LEN=*), > >> - the extension for OPTIONAL dummy arguments. > >> > >> Other TR29113 features are heavily used: > >> - The DIMENSION(..), TYPE(*) is needed for some internal > >> C-written MPI_Isend_desc. > >> - The ASYNCHRONOUS usage for communication buffers. > >> > >> Should we go in this direction? > >> > >> Best regards > >> Rolf > >> > >> > >> ----- Original Message ----- > >>> From: "Bill Long" <longb(a)cray.com> > >>> To: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> > >>> Cc: "Martin Schulz" <schulzm(a)llnl.gov>, "MPI-3 Fortran working > >>> group" <mpi3-fortran(a)lists.mpi-forum.org> > >>> Sent: Wednesday, March 20, 2013 4:01:52 PM > >>> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 > >>> interface - issue with the LOGICAL kind > >>> On 3/20/13 8:26 AM, Jeff Squyres (jsquyres) wrote: > >>>> On Mar 20, 2013, at 6:25 AM, Bill Long <longb(a)cray.com> wrote: > >>>> > >>>>> interface !--> MPI_Test > >>>>> > >>>>> ! int MPI_Test(MPI_Request *request, int *flag, MPI_Status > >>>>> *status); > >>>>> > >>>>> Function MPI_Test_C( request, flag, status) & > >>>>> BIND(C, name="MPI_Test") RESULT (res) > >>>>> import :: C_request, c_int, MPI_Status_C > >>>>> integer(C_request) :: request > >>>> > >>>> This is not correct. You cannot assume that MPI handles are > >>>> integers > >>>> (they're pointers in Open MPI). > >>> > >>> > >>> This is correct for mpich. OpenMPI would evidently have a > >>> different > >>> version. But the concept is still the same - make a correct > >>> interface > >>> to the C library routine, however it is defined by the > >>> implementation, > >>> and then call that from a thin Fortran wrapper that takes care of > >>> the > >>> issues related to LOGICAL, the use of default INTEGER, and > >>> OPTIONAL > >>> arguments. > >>> > >>> Cheers, > >>> Bill > >>> > >>> > >>>> > >>> > >>> -- > >>> Bill Long longb(a)cray.com > >>> Fortran Technical Support & voice: 651-605-9024 > >>> Bioinformatics Software Development fax: 651-605-9142 > >>> Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101 > >>> > >>> > >>> _______________________________________________ > >>> mpi3-fortran mailing list > >>> mpi3-fortran(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > >> > >> -- > >> Dr. Rolf Rabenseifner . . . . . . . . . .. email > >> rabenseifner(a)hlrs.de > >> High Performance Computing Center (HLRS) . phone > >> ++49(0)711/685-65530 > >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >> 685-65832 > >> Head of Dpmt Parallel Computing . . . > >> www.hlrs.de/people/rabenseifner > >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >> 1.307) > >> _______________________________________________ > >> mpi3-fortran mailing list > >> mpi3-fortran(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > >> > >> > >> Click > >> https://www.mailcontrol.com/sr/oUBxarVXlArGX2PQPOmvUrwgXREt4LZXdolWasoQmWDb… > >> to report this email as spam. > >> > >> _______________________________________________ > >> mpi3-fortran mailing list > >> mpi3-fortran(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > > > > > > -- > Bill Long longb(a)cray.com > Fortran Technical Support & voice: 651-605-9024 > Bioinformatics Software Development fax: 651-605-9142 > Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101 > > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Bill Long 20 Mar '13

20 Mar '13

My impression is that Rolf's proposal was to offer two options: a) no bind(c) on the Fortran wrappers, no requirements on the Fortran external names, but the wrappers are REQUIRED to call the C library routines. b) specific external names are required for the Fortran-called routines, but they do not have to be wrappers. (Still a bit unclear to me in this case where bind(c) might be required). On 3/20/13 2:58 PM, Jeff Squyres (jsquyres) wrote: > Not only that, I think that there are some cases where tools *need* to intercept the Fortran call, not the C call. In OMPI, for example, we have a few corner cases where the Fortran MPI subroutine does *not* call the corresponding C MPI API function. Instead, it calls a back-end OMPI routine. > True for implementations using option (b), but not for the (a) case. > One obvious case that jumps to mind is anything involving a callback. Looking at the callback examples on page 268, I have some questions: 1) The BIND(C) and an argument of type LOGICAL are likely incompatible, so either the BIND(C) or the LOGICAL needs to go. 2) The external names for the C and Fortran versions are different. I assume this does not matter assuming that the function that is calling these has only an address, and not a name, to call. But how does the function know the calling signature for the callback routine with only a procedure pointer to work with? If I were writing a code with a callback like this, I would want to assume the C prototype even of the code was in Fortran. I assume the existence of the separate (and apparently flawed) Fortran interface is for legacy code. But there should not be legacy mpi_f08 code. A bit confusing. > > If you have integer handles in C, they might be able to get away with not swizzling/de-swizzling the arguments to the callback to the proper language. But this is not always the case. > > > > On Mar 20, 2013, at 12:29 PM, Hubert Ritzdorf <Hubert.Ritzdorf(a)emea.nec.com> wrote: > >> Hi, >> >> I think that Bill's proposal kills the Fortran Profiling Interface. >> >> If the user defines a >> >> subroutine MPI_Test (....) >> >> do something >> >> call PMPI _Test() >> >> do something >> >> end subroutine MPI_Test I was under the impression that users would not directly call PMPI routines. Those were for the tools people. Why would a normal user want to do something like the above? >> >> then this function will never be executed because the Fortrran compiler may have already >> inlined the MPI_Test subroutine in the original code or the module version of MPI_Test is used. In the scope where this routine is called, the Fortran wrapper would have to be inaccessible. That would be done by NOT having a USE of the module in this scope. The external name of the module procedure would be mangled to include the name of the module, whereas a stand-alone routine like this would have an external name like mpi_test_ . The linker would see these as different routines. Cheers, Bill >> >> Hubert >> ________________________________________ >> From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mpi3-fortran-bounces(a)lists.mpi-forum.org] on behalf of Rolf Rabenseifner [rabenseifner(a)hlrs.de] >> Sent: Wednesday, March 20, 2013 6:09 PM >> To: longb(a)cray.com; MPI-3 Fortran working group >> Cc: Martin Schulz >> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind >> >> This mail contains a proposal how we can solve the current >> Fortran problems and possibly making the tools people also happy. >> >> I need at least answers for >> - mpich (Bill Long), >> - OpenMPI (Jeff or Craig), >> - some tools people. >> >> Let me summarize what I've learnt so far for mpi_f08: >> >> - #364 about LOGICAL correction is a must. >> - MPI-3.0 allows BIND(C) and non-BIND(C), with the >> new restrictions in #364 >> - BIND(C) needs TR29113, therefore any implementation now >> should use non-BIND(C) for all non-TR29113 compilers. >> - Most (or all) try to use the non-BIND(C) interface. >> - Best way to do this are thin wrappers in Fortran that >> internally call >> - the oficial MPI-3.0 C binding >> (here, the tools people have a problem because they >> expect to wrap the Fortran interface and may therefore >> double-wrap), or >> - or an internal MPI-3.0 C routine >> (which is half-perfect for the tools folk, because >> they have to handle the implementation-dependant >> Fortran names with underscores). >> - And at least Bill wants to include the Fortran MPI >> library routines itselves in the mpi_f08 module. >> This is in conflict with the current MPI-3.0 >> linker naming rules. >> >> Implications and proposal: >> - We should rewrite the linker names section MPI-3.0 17.1.5 >> with following goals: >> - Allow additionally to define that the PMPI is provided >> only on the MPI C interface and the Fortran MPI routines >> internally call the C interface; >> again, a macro definition in mpi.h tells whether this >> is chosen. >> Question: Would this cause a problem when linking a >> C or Fortran user application with some own >> MPI_Test etc. profiling routines that internally call >> PMPI_Test? >> - If Fortran profiling is done through the MPI C binding >> then there are no rules about the Fortran linker names >> needed, and therefore, one may put the thin Fortran >> wrappers into a Fortran module. >> - We can define BIND(C) for MPI routine groups as a >> future option that is currently not used. >> This would reduce the current effort for the tools people. >> Is there anybody who wants to implement one of the Fortran >> routine groups with BIND(C)? >> >> I would expect, that this is the best to solve the problem >> that #364 prohibits BIND(C) for some routines and implementors >> want to do the same method for all routines. >> >> Don't forget that BIND(C) and TR29113 is still important >> for implementing these thin wrappers internally. >> >> Without using BIND(C) on the official mpi_f08 and mpi module >> interface, some features of TR29113 are not used: >> - the extension for interoperabple CHARACTER(LEN=*), >> - the extension for OPTIONAL dummy arguments. >> >> Other TR29113 features are heavily used: >> - The DIMENSION(..), TYPE(*) is needed for some internal >> C-written MPI_Isend_desc. >> - The ASYNCHRONOUS usage for communication buffers. >> >> Should we go in this direction? >> >> Best regards >> Rolf >> >> >> ----- Original Message ----- >>> From: "Bill Long" <longb(a)cray.com> >>> To: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> >>> Cc: "Martin Schulz" <schulzm(a)llnl.gov>, "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> >>> Sent: Wednesday, March 20, 2013 4:01:52 PM >>> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind >>> On 3/20/13 8:26 AM, Jeff Squyres (jsquyres) wrote: >>>> On Mar 20, 2013, at 6:25 AM, Bill Long <longb(a)cray.com> wrote: >>>> >>>>> interface !--> MPI_Test >>>>> >>>>> ! int MPI_Test(MPI_Request *request, int *flag, MPI_Status >>>>> *status); >>>>> >>>>> Function MPI_Test_C( request, flag, status) & >>>>> BIND(C, name="MPI_Test") RESULT (res) >>>>> import :: C_request, c_int, MPI_Status_C >>>>> integer(C_request) :: request >>>> >>>> This is not correct. You cannot assume that MPI handles are integers >>>> (they're pointers in Open MPI). >>> >>> >>> This is correct for mpich. OpenMPI would evidently have a different >>> version. But the concept is still the same - make a correct interface >>> to the C library routine, however it is defined by the implementation, >>> and then call that from a thin Fortran wrapper that takes care of the >>> issues related to LOGICAL, the use of default INTEGER, and OPTIONAL >>> arguments. >>> >>> Cheers, >>> Bill >>> >>> >>>> >>> >>> -- >>> Bill Long longb(a)cray.com >>> Fortran Technical Support & voice: 651-605-9024 >>> Bioinformatics Software Development fax: 651-605-9142 >>> Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101 >>> >>> >>> _______________________________________________ >>> mpi3-fortran mailing list >>> mpi3-fortran(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >> _______________________________________________ >> mpi3-fortran mailing list >> mpi3-fortran(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran >> >> >> Click https://www.mailcontrol.com/sr/oUBxarVXlArGX2PQPOmvUrwgXREt4LZXdolWasoQmWDb… to report this email as spam. >> >> _______________________________________________ >> mpi3-fortran mailing list >> mpi3-fortran(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > > -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Jeff Squyres (jsquyres) 20 Mar '13

20 Mar '13

Not only that, I think that there are some cases where tools *need* to intercept the Fortran call, not the C call. In OMPI, for example, we have a few corner cases where the Fortran MPI subroutine does *not* call the corresponding C MPI API function. Instead, it calls a back-end OMPI routine. One obvious case that jumps to mind is anything involving a callback. If you have integer handles in C, they might be able to get away with not swizzling/de-swizzling the arguments to the callback to the proper language. But this is not always the case. On Mar 20, 2013, at 12:29 PM, Hubert Ritzdorf <Hubert.Ritzdorf(a)emea.nec.com> wrote: > Hi, > > I think that Bill's proposal kills the Fortran Profiling Interface. > > If the user defines a > > subroutine MPI_Test (....) > > do something > > call PMPI _Test() > > do something > > end subroutine MPI_Test > > then this function will never be executed because the Fortrran compiler may have already > inlined the MPI_Test subroutine in the original code or the module version of MPI_Test is used. > > Hubert > ________________________________________ > From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mpi3-fortran-bounces(a)lists.mpi-forum.org] on behalf of Rolf Rabenseifner [rabenseifner(a)hlrs.de] > Sent: Wednesday, March 20, 2013 6:09 PM > To: longb(a)cray.com; MPI-3 Fortran working group > Cc: Martin Schulz > Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind > > This mail contains a proposal how we can solve the current > Fortran problems and possibly making the tools people also happy. > > I need at least answers for > - mpich (Bill Long), > - OpenMPI (Jeff or Craig), > - some tools people. > > Let me summarize what I've learnt so far for mpi_f08: > > - #364 about LOGICAL correction is a must. > - MPI-3.0 allows BIND(C) and non-BIND(C), with the > new restrictions in #364 > - BIND(C) needs TR29113, therefore any implementation now > should use non-BIND(C) for all non-TR29113 compilers. > - Most (or all) try to use the non-BIND(C) interface. > - Best way to do this are thin wrappers in Fortran that > internally call > - the oficial MPI-3.0 C binding > (here, the tools people have a problem because they > expect to wrap the Fortran interface and may therefore > double-wrap), or > - or an internal MPI-3.0 C routine > (which is half-perfect for the tools folk, because > they have to handle the implementation-dependant > Fortran names with underscores). > - And at least Bill wants to include the Fortran MPI > library routines itselves in the mpi_f08 module. > This is in conflict with the current MPI-3.0 > linker naming rules. > > Implications and proposal: > - We should rewrite the linker names section MPI-3.0 17.1.5 > with following goals: > - Allow additionally to define that the PMPI is provided > only on the MPI C interface and the Fortran MPI routines > internally call the C interface; > again, a macro definition in mpi.h tells whether this > is chosen. > Question: Would this cause a problem when linking a > C or Fortran user application with some own > MPI_Test etc. profiling routines that internally call > PMPI_Test? > - If Fortran profiling is done through the MPI C binding > then there are no rules about the Fortran linker names > needed, and therefore, one may put the thin Fortran > wrappers into a Fortran module. > - We can define BIND(C) for MPI routine groups as a > future option that is currently not used. > This would reduce the current effort for the tools people. > Is there anybody who wants to implement one of the Fortran > routine groups with BIND(C)? > > I would expect, that this is the best to solve the problem > that #364 prohibits BIND(C) for some routines and implementors > want to do the same method for all routines. > > Don't forget that BIND(C) and TR29113 is still important > for implementing these thin wrappers internally. > > Without using BIND(C) on the official mpi_f08 and mpi module > interface, some features of TR29113 are not used: > - the extension for interoperabple CHARACTER(LEN=*), > - the extension for OPTIONAL dummy arguments. > > Other TR29113 features are heavily used: > - The DIMENSION(..), TYPE(*) is needed for some internal > C-written MPI_Isend_desc. > - The ASYNCHRONOUS usage for communication buffers. > > Should we go in this direction? > > Best regards > Rolf > > > ----- Original Message ----- >> From: "Bill Long" <longb(a)cray.com> >> To: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> >> Cc: "Martin Schulz" <schulzm(a)llnl.gov>, "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> >> Sent: Wednesday, March 20, 2013 4:01:52 PM >> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind >> On 3/20/13 8:26 AM, Jeff Squyres (jsquyres) wrote: >>> On Mar 20, 2013, at 6:25 AM, Bill Long <longb(a)cray.com> wrote: >>> >>>> interface !--> MPI_Test >>>> >>>> ! int MPI_Test(MPI_Request *request, int *flag, MPI_Status >>>> *status); >>>> >>>> Function MPI_Test_C( request, flag, status) & >>>> BIND(C, name="MPI_Test") RESULT (res) >>>> import :: C_request, c_int, MPI_Status_C >>>> integer(C_request) :: request >>> >>> This is not correct. You cannot assume that MPI handles are integers >>> (they're pointers in Open MPI). >> >> >> This is correct for mpich. OpenMPI would evidently have a different >> version. But the concept is still the same - make a correct interface >> to the C library routine, however it is defined by the implementation, >> and then call that from a thin Fortran wrapper that takes care of the >> issues related to LOGICAL, the use of default INTEGER, and OPTIONAL >> arguments. >> >> Cheers, >> Bill >> >> >>> >> >> -- >> Bill Long longb(a)cray.com >> Fortran Technical Support & voice: 651-605-9024 >> Bioinformatics Software Development fax: 651-605-9142 >> Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101 >> >> >> _______________________________________________ >> mpi3-fortran mailing list >> mpi3-fortran(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > > > Click https://www.mailcontrol.com/sr/oUBxarVXlArGX2PQPOmvUrwgXREt4LZXdolWasoQmWDb… to report this email as spam. > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Hubert Ritzdorf 20 Mar '13

20 Mar '13

Hi, I think that Bill's proposal kills the Fortran Profiling Interface. If the user defines a subroutine MPI_Test (....) do something call PMPI _Test() do something end subroutine MPI_Test then this function will never be executed because the Fortrran compiler may have already inlined the MPI_Test subroutine in the original code or the module version of MPI_Test is used. Hubert ________________________________________ From: mpi3-fortran-bounces(a)lists.mpi-forum.org [mpi3-fortran-bounces(a)lists.mpi-forum.org] on behalf of Rolf Rabenseifner [rabenseifner(a)hlrs.de] Sent: Wednesday, March 20, 2013 6:09 PM To: longb(a)cray.com; MPI-3 Fortran working group Cc: Martin Schulz Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind This mail contains a proposal how we can solve the current Fortran problems and possibly making the tools people also happy. I need at least answers for - mpich (Bill Long), - OpenMPI (Jeff or Craig), - some tools people. Let me summarize what I've learnt so far for mpi_f08: - #364 about LOGICAL correction is a must. - MPI-3.0 allows BIND(C) and non-BIND(C), with the new restrictions in #364 - BIND(C) needs TR29113, therefore any implementation now should use non-BIND(C) for all non-TR29113 compilers. - Most (or all) try to use the non-BIND(C) interface. - Best way to do this are thin wrappers in Fortran that internally call - the oficial MPI-3.0 C binding (here, the tools people have a problem because they expect to wrap the Fortran interface and may therefore double-wrap), or - or an internal MPI-3.0 C routine (which is half-perfect for the tools folk, because they have to handle the implementation-dependant Fortran names with underscores). - And at least Bill wants to include the Fortran MPI library routines itselves in the mpi_f08 module. This is in conflict with the current MPI-3.0 linker naming rules. Implications and proposal: - We should rewrite the linker names section MPI-3.0 17.1.5 with following goals: - Allow additionally to define that the PMPI is provided only on the MPI C interface and the Fortran MPI routines internally call the C interface; again, a macro definition in mpi.h tells whether this is chosen. Question: Would this cause a problem when linking a C or Fortran user application with some own MPI_Test etc. profiling routines that internally call PMPI_Test? - If Fortran profiling is done through the MPI C binding then there are no rules about the Fortran linker names needed, and therefore, one may put the thin Fortran wrappers into a Fortran module. - We can define BIND(C) for MPI routine groups as a future option that is currently not used. This would reduce the current effort for the tools people. Is there anybody who wants to implement one of the Fortran routine groups with BIND(C)? I would expect, that this is the best to solve the problem that #364 prohibits BIND(C) for some routines and implementors want to do the same method for all routines. Don't forget that BIND(C) and TR29113 is still important for implementing these thin wrappers internally. Without using BIND(C) on the official mpi_f08 and mpi module interface, some features of TR29113 are not used: - the extension for interoperabple CHARACTER(LEN=*), - the extension for OPTIONAL dummy arguments. Other TR29113 features are heavily used: - The DIMENSION(..), TYPE(*) is needed for some internal C-written MPI_Isend_desc. - The ASYNCHRONOUS usage for communication buffers. Should we go in this direction? Best regards Rolf ----- Original Message ----- > From: "Bill Long" <longb(a)cray.com> > To: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> > Cc: "Martin Schulz" <schulzm(a)llnl.gov>, "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> > Sent: Wednesday, March 20, 2013 4:01:52 PM > Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind > On 3/20/13 8:26 AM, Jeff Squyres (jsquyres) wrote: > > On Mar 20, 2013, at 6:25 AM, Bill Long <longb(a)cray.com> wrote: > > > >> interface !--> MPI_Test > >> > >> ! int MPI_Test(MPI_Request *request, int *flag, MPI_Status > >> *status); > >> > >> Function MPI_Test_C( request, flag, status) & > >> BIND(C, name="MPI_Test") RESULT (res) > >> import :: C_request, c_int, MPI_Status_C > >> integer(C_request) :: request > > > > This is not correct. You cannot assume that MPI handles are integers > > (they're pointers in Open MPI). > > > This is correct for mpich. OpenMPI would evidently have a different > version. But the concept is still the same - make a correct interface > to the C library routine, however it is defined by the implementation, > and then call that from a thin Fortran wrapper that takes care of the > issues related to LOGICAL, the use of default INTEGER, and OPTIONAL > arguments. > > Cheers, > Bill > > > > > > -- > Bill Long longb(a)cray.com > Fortran Technical Support & voice: 651-605-9024 > Bioinformatics Software Development fax: 651-605-9142 > Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101 > > > _______________________________________________ > mpi3-fortran mailing list > mpi3-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) _______________________________________________ mpi3-fortran mailing list mpi3-fortran(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran Click https://www.mailcontrol.com/sr/oUBxarVXlArGX2PQPOmvUrwgXREt4LZXdolWasoQmWDb… to report this email as spam.

1 0

Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind
by Bill Long 20 Mar '13

20 Mar '13

Hi Rolf, This looks like good progress toward a solution. There is one missing piece that I think is needed. If the tools target only the C functions, then the names of the versions of the C functions that take a choice/buf argument that is a pointer to a descriptor need to be standardized. For the moment I'm using MPI_Send_cdesc as the companion of MPI_Send. Besides the spelling of the function name, only the specification for the buf argument is changed - all the other arguments and the return type are the same as for MPI_Send. Since this is a simple replacement pattern, we would need only a paragraph to describe the naming rule, and not a complete list of the new prototypes. The actual form of the name modification is certainly open for discussion. I picked appending "_cdesc" because the new argument declaration is CFI_cdesc_t, so it seemed like it would be easy to remember. Because of the way these descriptors work, the routines that implement operations like MPI_Send that use a descriptor have to be written in C anyway. If the interfaces is otherwise the same as for the traditional MPI_Send then the tools packages would have a much easier time (no INTEGER or LOGICAL Fortran issues). For the case of thin Fortran wrappers without BIND(C) and tools looking at the C functions only, these names would be required. For other implementation options, it is not as clear if it is necessary to require these C function names. However, if they were required universally, then C programmers would have the option of calling these functions directly, which would seem like a nice facility for them, particularly where performance is an issue. Cheers, Bill On 3/20/13 12:09 PM, Rolf Rabenseifner wrote: > This mail contains a proposal how we can solve the current > Fortran problems and possibly making the tools people also happy. > > I need at least answers for > - mpich (Bill Long), > - OpenMPI (Jeff or Craig), > - some tools people. > > Let me summarize what I've learnt so far for mpi_f08: > > - #364 about LOGICAL correction is a must. > - MPI-3.0 allows BIND(C) and non-BIND(C), with the > new restrictions in #364 > - BIND(C) needs TR29113, therefore any implementation now > should use non-BIND(C) for all non-TR29113 compilers. > - Most (or all) try to use the non-BIND(C) interface. > - Best way to do this are thin wrappers in Fortran that > internally call > - the oficial MPI-3.0 C binding > (here, the tools people have a problem because they > expect to wrap the Fortran interface and may therefore > double-wrap), or > - or an internal MPI-3.0 C routine > (which is half-perfect for the tools folk, because > they have to handle the implementation-dependant > Fortran names with underscores). > - And at least Bill wants to include the Fortran MPI > library routines itselves in the mpi_f08 module. > This is in conflict with the current MPI-3.0 > linker naming rules. > > Implications and proposal: > - We should rewrite the linker names section MPI-3.0 17.1.5 > with following goals: > - Allow additionally to define that the PMPI is provided > only on the MPI C interface and the Fortran MPI routines > internally call the C interface; > again, a macro definition in mpi.h tells whether this > is chosen. > Question: Would this cause a problem when linking a > C or Fortran user application with some own > MPI_Test etc. profiling routines that internally call > PMPI_Test? > - If Fortran profiling is done through the MPI C binding > then there are no rules about the Fortran linker names > needed, and therefore, one may put the thin Fortran > wrappers into a Fortran module. > - We can define BIND(C) for MPI routine groups as a > future option that is currently not used. > This would reduce the current effort for the tools people. > Is there anybody who wants to implement one of the Fortran > routine groups with BIND(C)? > > I would expect, that this is the best to solve the problem > that #364 prohibits BIND(C) for some routines and implementors > want to do the same method for all routines. > > Don't forget that BIND(C) and TR29113 is still important > for implementing these thin wrappers internally. > > Without using BIND(C) on the official mpi_f08 and mpi module > interface, some features of TR29113 are not used: > - the extension for interoperabple CHARACTER(LEN=*), > - the extension for OPTIONAL dummy arguments. > > Other TR29113 features are heavily used: > - The DIMENSION(..), TYPE(*) is needed for some internal > C-written MPI_Isend_desc. > - The ASYNCHRONOUS usage for communication buffers. > > Should we go in this direction? > > Best regards > Rolf > > > ----- Original Message ----- >> From: "Bill Long" <longb(a)cray.com> >> To: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> >> Cc: "Martin Schulz" <schulzm(a)llnl.gov>, "MPI-3 Fortran working group" <mpi3-fortran(a)lists.mpi-forum.org> >> Sent: Wednesday, March 20, 2013 4:01:52 PM >> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind >> On 3/20/13 8:26 AM, Jeff Squyres (jsquyres) wrote: >>> On Mar 20, 2013, at 6:25 AM, Bill Long <longb(a)cray.com> wrote: >>> >>>> interface !--> MPI_Test >>>> >>>> ! int MPI_Test(MPI_Request *request, int *flag, MPI_Status >>>> *status); >>>> >>>> Function MPI_Test_C( request, flag, status) & >>>> BIND(C, name="MPI_Test") RESULT (res) >>>> import :: C_request, c_int, MPI_Status_C >>>> integer(C_request) :: request >>> >>> This is not correct. You cannot assume that MPI handles are integers >>> (they're pointers in Open MPI). >> >> >> This is correct for mpich. OpenMPI would evidently have a different >> version. But the concept is still the same - make a correct interface >> to the C library routine, however it is defined by the implementation, >> and then call that from a thin Fortran wrapper that takes care of the >> issues related to LOGICAL, the use of default INTEGER, and OPTIONAL >> arguments. >> >> Cheers, >> Bill >> >> >>> >> >> -- >> Bill Long longb(a)cray.com >> Fortran Technical Support & voice: 651-605-9024 >> Bioinformatics Software Development fax: 651-605-9142 >> Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101 >> >> >> _______________________________________________ >> mpi3-fortran mailing list >> mpi3-fortran(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran > -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101

1 0