December 2015 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPIWG Fortran] F08 and pair types?
by Rolf Rabenseifner 19 Dec '15

19 Dec '15

Dear Jeff and Hubert, I strongly recommend that you both check my answer from Dec. 11, 2015 to this topic (see below). It stays in the Fortran world, i.e., it fits exactly to all reasons listed by Hubert. Best regards Rolf ----- Forwarded Message ----- From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> To: "MPI-WG Fortran working group" <mpiwg-fortran(a)lists.mpi-forum.org> Sent: Friday, December 11, 2015 11:38:00 AM Subject: Re: [MPIWG Fortran] F08 and pair types? A few comments - shortcut: F08 and pair types is a good idea and helps to get MPI-4.0 more consistent. 1. After the fiasco with the MPI-3.0 BIND(C) back-end and a completely rewritten Fortran back-end section without BIND(C), I would strongly recommend to stay within Fortran without C. One possible reason: if somebody wants to add something to MAX/MINLOC with a datatype that is not directly C compatible (like LOGICAL), then we would have an inconsistent Interface. This applies to the type of the numerical data itself, i.e., the following application code would fit to the MPI datatype handle MPI_DOUBLE_PRECISION_INT_TYPE TYPE MY_DOUBLE_PRECISION_INT_TYPE SEQUENCE DOUBLE PRECISION :: VAL INTEGER :: LOC END TYPE and we should ***never*** do TYPE MY_DOUBLE_PRECISION_INT_TYPE > SEQUENCE > REAL(kind=C_DOUBLE) :: VAL > INTEGER :: LOC > END TYPE and it also applies to the whole stucture, i.e., we should ***never*** do TYPE, BIND(C) :: MY_DOUBLE_PRECISION_INT_TYPE DOUBLE PRECISION :: VAL INTEGER :: LOC END TYPE nor TYPE, BIND(C) :: MY_DOUBLE_PRECISION_INT_TYPE REAL(kind=C_DOUBLE) :: VAL INTEGER :: LOC END TYPE 2. The names for Fortran and C should be clearly different. In Fortran we should define the following datatype handles: - new MPI_DOUBLE_PRECISION_INT_TYPE (29 character :-) - new MPI_REAL_INT and - existing MPI_2INTEGER (but with this new meaning [i.e. SEQUENCE derived type with two INTEGERs], which is memory-wise identical to the current definition of an array of two INTEGERs) 3. It is up to the MPI library to understand the alignment rules for these types. Implication: Because we will define this through MPI_TYPE_CREATE_STRUCT as on MPI-3.1 page 180 lines 33-42, we must define MPI_TYPE_CREATE_STRUCT correctly. This means, that the attomatic alignment epsilon on page 105 for Fortran types must be according to the rules for Fortran derived types with the SEQUENCE attribute. Currently it is not defined, which is bad for old MPI-1.1 application when BIND(C) was not yet invented. The text of MPI-3.1 Section 17.1.15 Fortran Derived Types must also reflect this now well-defined definition of the Fortran alignment epsilon. This change is a change from undefined to defined, i.e., existing correct MPI applications must not have a problem with this change. Some MPI libraries do not yet calculate correct Fortran alignments (in this sense), because they apply C alignments to Fortran types. These MPI libraries must be corrected. 4. If somebody uses MPI_FLOAT_INT together with MAX/MIN_LOC in Fortran, then this is already allowed based on MPI-3.1 page 659 lines 1-9! A Fortran programmer may program the buffer for MPI_FLOAT_INT with TYPE, BIND(C) :: my_float_int REAL(kind=C_FLOAT) :: VAL INTEGER(kind=C_INT) :: LOC END TYPE 5. All said is of course valid for all three mpi_f08 module, mpi module, and mpif.h I hope that my comments are not inconflicht with previous discussions. Best regards Rolf ----- Original Message ----- > From: "Hubert Ritzdorf" <Hubert.Ritzdorf(a)EMEA.NEC.COM> > To: "MPI-WG Fortran working group" <mpiwg-fortran(a)lists.mpi-forum.org> > Sent: Friday, December 18, 2015 8:34:36 PM > Subject: Re: [MPIWG Fortran] F08 and pair types? > Jeff Squyres wrote: > >>Hmm. Ok, that's the opposite advice we got for MPI-3.0 (which was now admittedly >>several years ago). >> >>Soo... which way should it go these days? >> >>- use native Fortran datatypes >>- use C datatypes (via BIND(C)) > > > I think that MPI Fortran Interface should support native Fortran datatypes in > pair types ! > > (*) We (NEC) don't know any relevant Fortran application which doesn't use > native Fortran datatypes. > If you switch for C Datatypes in MINLOC/MAXLOC, all application programmers must > solve > the portability problems (possibly loosing accuracy in reduce operations). > > (*) Nearly all Fortran compilers support compile time options such as "-r8" or > "-size real64" > which would cause problems with the C datatypes. > And this options are used by application programmers ! > > (*) C datatypes would be inconsistent with functions such as > MPI_Type_create_f90_real. > > (*) Supporting Fortran subarrays without supporting native Fortran datatypes > doesn't make sense for me. > > Hubert > > > > _______________________________________________ > mpiwg-fortran mailing list > mpiwg-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Hubert Ritzdorf 18 Dec '15

18 Dec '15

Jeff Squyres wrote: >Hmm. Ok, that's the opposite advice we got for MPI-3.0 (which was now admittedly several years ago). > >Soo... which way should it go these days? > >- use native Fortran datatypes >- use C datatypes (via BIND(C)) I think that MPI Fortran Interface should support native Fortran datatypes in pair types ! (*) We (NEC) don't know any relevant Fortran application which doesn't use native Fortran datatypes. If you switch for C Datatypes in MINLOC/MAXLOC, all application programmers must solve the portability problems (possibly loosing accuracy in reduce operations). (*) Nearly all Fortran compilers support compile time options such as "-r8" or "-size real64" which would cause problems with the C datatypes. And this options are used by application programmers ! (*) C datatypes would be inconsistent with functions such as MPI_Type_create_f90_real. (*) Supporting Fortran subarrays without supporting native Fortran datatypes doesn't make sense for me. Hubert

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Bill Long 11 Dec '15

11 Dec '15

On Dec 9, 2015, at 7:45 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > On Dec 8, 2015, at 6:31 PM, Bill Long <longb(a)cray.com> wrote: >> >> As I understand the high-level situation, this is all for MPI_REDUCE and friends, and MPI_2INTEGER and MPI_MINLOC or MPI_MAXLOC are all just magic constants that the MPI_REDUCE routine internally decodes to determine what to do. The actual data, the in and out arguments of MPI_REDUCE, are choice arguments (void *). > >> Current codes have the actual argument IN declared as INTEGER :: IN (2, K) where K is the number of things in the list to be reduced. IN(1,*) is the “VAL" value, and IN(2,*) is the “LOC” value. Since the corresponding dummy argument is (void *) you can pass any actual type, since only the memory layout matters. > > Correct. > >> If the above is correct, then the following declarations would have the same memory layouts for the actual arguments: >> >> INTEGER :: IN(2, K) >> >> and >> >> TYPE MPI_2INTEGER_TYPE >> SEQUENCE >> INTEGER :: VAL >> INTEGER :: LOC >> END TYPE >> >> TYPE(MPI_2INTEGER_TYPE) :: IN(K) >> >> This seems like sugar coating for the 2INTEGER case. > > Yes, exactly. > > Is there a way to do it without SEQUENCE (since that is frowned upon)? There are two ways to define a type that has enforced memory layout. One is SEQUENCE types which (for default numeric and logical components) are laid out as a sequence of numeric memory storage units (2 for double precision and 1 for real, integer, logical). The other is to make the type a BIND(C) type, in which case the C compiler’s rules determine the layout. However, knowing the layout is only important for the legacy interfaces. I would not expect any new feature to be back-ported to those old interfaces. After all, I assume the intention is to deprecate and eventually delete them. Why waste ink putting more lipstick on the old pig? The F08 interfaces include enough information sent to the callee that it can figure out the memory layout. > >> But, the following would likely not be equivalent: >> >> DOUBLE PRECISION :: IN(2, K) >> >> and >> >> TYPE MPI_DOUBLE_INT_TYPE >> SEQUENCE >> DOUBLE PRECISION :: VAL >> INTEGER :: LOC >> END TYPE >> >> TYPE(MPI_DOUBLE_INT_TYPE) :: IN(K) > > Right -- I'm not trying to make those equivalent. > > To be clear: > > - the only one where we need this equivalence MPI_2INTEGER. > - the intent is to deprecate MPI_2DOUBLE_PRECISION and MPI_2REAL > --> I assume that these are historical artifacts that were created before derived types existed > - and then create 2 new types: MPI_DOUBLE_PRECISION_INTEGER and MPI_REAL_INTEGER > --> these will be analogous to their C counterparts (but using Fortran types, not C types) > >>> 2. I wasn't looking to emulate the C types in Fortran -- I think we've had this discussion before. The intent is to have Fortran-native types (that would be natural for Fortran programmers). >> >> I think BIND(C) is “natural” for Fortran programmers these days. This stuff has been around for over a decade. The use of “DOUBLE PRECISION” is far less natural these days. Essentially everyone has switched to REAL(KIND= ) instead. That stuff is 25 years old. > > Hmm. Ok, that's the opposite advice we got for MPI-3.0 (which was now admittedly several years ago). > > Soo... which way should it go these days? > > - use native Fortran datatypes > - use C datatypes (via BIND(C)) > > I.e., when users want to use MINLOC / MAXLOC in fortran, will they be using this across DOUBLE PRECISION values, or Type(C_DOUBLE)? > The difference between DOUBLE PRECISION and REAL(C_DOUBLE) is that the first can change depending on compilation options like “-i8” which, with normal sizes of today would be 8 bytes (without -i8) or 16 bytes (with -i8). Unless there is an intent to support quad floats on the C side, I’d avoid the DOUBLE PRECISION option. Using the C types makes things unambiguously correct when the arguments make it to the C library code. Cheers, Bill > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-fortran mailing list > mpiwg-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN 55101

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Rolf Rabenseifner 11 Dec '15

11 Dec '15

A few comments - shortcut: F08 and pair types is a good idea and helps to get MPI-4.0 more consistent. 1. After the fiasco with the MPI-3.0 BIND(C) back-end and a completely rewritten Fortran back-end section without BIND(C), I would strongly recommend to stay within Fortran without C. One possible reason: if somebody wants to add something to MAX/MINLOC with a datatype that is not directly C compatible (like LOGICAL), then we would have an inconsistent Interface. This applies to the type of the numerical data itself, i.e., the following application code would fit to the MPI datatype handle MPI_DOUBLE_PRECISION_INT_TYPE TYPE MY_DOUBLE_PRECISION_INT_TYPE SEQUENCE DOUBLE PRECISION :: VAL INTEGER :: LOC END TYPE and we should ***never*** do TYPE MY_DOUBLE_PRECISION_INT_TYPE > SEQUENCE > REAL(kind=C_DOUBLE) :: VAL > INTEGER :: LOC > END TYPE and it also applies to the whole stucture, i.e., we should ***never*** do TYPE, BIND(C) :: MY_DOUBLE_PRECISION_INT_TYPE DOUBLE PRECISION :: VAL INTEGER :: LOC END TYPE nor TYPE, BIND(C) :: MY_DOUBLE_PRECISION_INT_TYPE REAL(kind=C_DOUBLE) :: VAL INTEGER :: LOC END TYPE 2. The names for Fortran and C should be clearly different. In Fortran we should define the following datatype handles: - new MPI_DOUBLE_PRECISION_INT_TYPE (29 character :-) - new MPI_REAL_INT and - existing MPI_2INTEGER (but with this new meaning [i.e. SEQUENCE derived type with two INTEGERs], which is memory-wise identical to the current definition of an array of two INTEGERs) 3. It is up to the MPI library to understand the alignment rules for these types. Implication: Because we will define this through MPI_TYPE_CREATE_STRUCT as on MPI-3.1 page 180 lines 33-42, we must define MPI_TYPE_CREATE_STRUCT correctly. This means, that the attomatic alignment epsilon on page 105 for Fortran types must be according to the rules for Fortran derived types with the SEQUENCE attribute. Currently it is not defined, which is bad for old MPI-1.1 application when BIND(C) was not yet invented. The text of MPI-3.1 Section 17.1.15 Fortran Derived Types must also reflect this now well-defined definition of the Fortran alignment epsilon. This change is a change from undefined to defined, i.e., existing correct MPI applications must not have a problem with this change. Some MPI libraries do not yet calculate correct Fortran alignments (in this sense), because they apply C alignments to Fortran types. These MPI libraries must be corrected. 4. If somebody uses MPI_FLOAT_INT together with MAX/MIN_LOC in Fortran, then this is already allowed based on MPI-3.1 page 659 lines 1-9! A Fortran programmer may program the buffer for MPI_FLOAT_INT with TYPE, BIND(C) :: my_float_int REAL(kind=C_FLOAT) :: VAL INTEGER(kind=C_INT) :: LOC END TYPE 5. All said is of course valid for all three mpi_f08 module, mpi module, and mpif.h I hope that my comments are not inconflicht with previous discussions. Best regards Rolf ----- Original Message ----- > From: "Craig Rasmussen" <rasmus(a)cas.uoregon.edu> > To: "MPI-WG Fortran working group" <mpiwg-fortran(a)lists.mpi-forum.org> > Sent: Friday, December 11, 2015 2:24:05 AM > Subject: Re: [MPIWG Fortran] F08 and pair types? > Jeff Squyres wrote: > >>Hmm. Ok, that's the opposite advice we got for MPI-3.0 (which was now admittedly >>several years ago). >> >>Soo... which way should it go these days? >> >>- use native Fortran datatypes >>- use C datatypes (via BIND(C)) > > I don't think the advice has changed since MPI-3.0. I believe the preference is > for the application programmer to see native Fortran datatypes, not, for > example, REAL(C_DOUBLE). However, BIND(C) kinds can be used in the MPI_F08 > module to declare the new datatypes, e.g., > > TYPE MPI_DOUBLE_INT_TYPE > SEQUENCE > REAL(kind=C_DOUBLE) :: VAL > INTEGER :: LOC > END TYPE > > as the application programmer will only use TYPE(MPI_DOUBLE_INT_TYPE) in a type > declaration and not know that it is defined in terms of native C types. > > I also don't think it matters that SEQUENCE is used to define > MPI_DOUBLE_INT_TYPE even if it is no longer considered to be so pretty. > However, I'm still confused by how this is used in C. Jeff S. in responding to > Bill L. said: > >>> How do you handle MPI_SHORT_INT? A short immediately followed in memory by an >>> int will cause the int to have bad memory alignment. >> >> Right. MPI knows this; MPI_SHORT_INT will include padding between the SHORT and >> INT if it needs to. >> > > How does a C programmer get the value returned in MPI_SHORT_INT? By accessing a > member of a C struct? Google suggests MPI_SHORT_INT is a struct {short, int} as > well as a pair of short integers followed by a 32 bit integer. Since these data > types really do contain the proper amount of padding (despite what I infer from > reading the MPI 3.1 standard), I think Bill's suggestion of declaring the > Fortran types to be BIND(C) equivalent to a C struct is the right thing to do. > Note that the Fortran application programmer won't ever see the BIND(C) > attribute. > > Final note: I was searching the WRONG Fortran standard document when I couldn't > find SEQUENCE and therefore said it looked like it was no longer in the > standard. My bad... :-( > > -craig > > > -craig > > On Wed, Dec 9, 2015 at 5:45 AM, Jeff Squyres (jsquyres) < jsquyres(a)cisco.com > > wrote: > > > On Dec 8, 2015, at 6:31 PM, Bill Long < longb(a)cray.com > wrote: >> >> As I understand the high-level situation, this is all for MPI_REDUCE and >> friends, and MPI_2INTEGER and MPI_MINLOC or MPI_MAXLOC are all just magic >> constants that the MPI_REDUCE routine internally decodes to determine what to >> do. The actual data, the in and out arguments of MPI_REDUCE, are choice >> arguments (void *). > >> Current codes have the actual argument IN declared as INTEGER :: IN (2, K) where >> K is the number of things in the list to be reduced. IN(1,*) is the “VAL" >> value, and IN(2,*) is the “LOC” value. Since the corresponding dummy argument >> is (void *) you can pass any actual type, since only the memory layout matters. > > Correct. > >> If the above is correct, then the following declarations would have the same >> memory layouts for the actual arguments: >> >> INTEGER :: IN(2, K) >> >> and >> >> TYPE MPI_2INTEGER_TYPE >> SEQUENCE >> INTEGER :: VAL >> INTEGER :: LOC >> END TYPE >> >> TYPE(MPI_2INTEGER_TYPE) :: IN(K) >> >> This seems like sugar coating for the 2INTEGER case. > > Yes, exactly. > > Is there a way to do it without SEQUENCE (since that is frowned upon)? > >> But, the following would likely not be equivalent: >> >> DOUBLE PRECISION :: IN(2, K) >> >> and >> >> TYPE MPI_DOUBLE_INT_TYPE >> SEQUENCE >> DOUBLE PRECISION :: VAL >> INTEGER :: LOC >> END TYPE >> >> TYPE(MPI_DOUBLE_INT_TYPE) :: IN(K) > > Right -- I'm not trying to make those equivalent. > > To be clear: > > - the only one where we need this equivalence MPI_2INTEGER. > - the intent is to deprecate MPI_2DOUBLE_PRECISION and MPI_2REAL > --> I assume that these are historical artifacts that were created before > derived types existed > - and then create 2 new types: MPI_DOUBLE_PRECISION_INTEGER and MPI_REAL_INTEGER > --> these will be analogous to their C counterparts (but using Fortran types, > not C types) > >>> 2. I wasn't looking to emulate the C types in Fortran -- I think we've had this >>> discussion before. The intent is to have Fortran-native types (that would be >>> natural for Fortran programmers). >> >> I think BIND(C) is “natural” for Fortran programmers these days. This stuff has >> been around for over a decade. The use of “DOUBLE PRECISION” is far less >> natural these days. Essentially everyone has switched to REAL(KIND= ) instead. >> That stuff is 25 years old. > > Hmm. Ok, that's the opposite advice we got for MPI-3.0 (which was now admittedly > several years ago). > > Soo... which way should it go these days? > > - use native Fortran datatypes > - use C datatypes (via BIND(C)) > > I.e., when users want to use MINLOC / MAXLOC in fortran, will they be using this > across DOUBLE PRECISION values, or Type(C_DOUBLE)? > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-fortran mailing list > mpiwg-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran > > > _______________________________________________ > mpiwg-fortran mailing list > mpiwg-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Craig Rasmusen 10 Dec '15

10 Dec '15

Jeff Squyres wrote: >Hmm. Ok, that's the opposite advice we got for MPI-3.0 (which was now admittedly several years ago). > >Soo... which way should it go these days? > >- use native Fortran datatypes >- use C datatypes (via BIND(C)) I don't think the advice has changed since MPI-3.0. I believe the preference is for the application programmer to see native Fortran datatypes, not, for example, REAL(C_DOUBLE). However, BIND(C) kinds can be used in the MPI_F08 module to declare the new datatypes, e.g., TYPE MPI_DOUBLE_INT_TYPE SEQUENCE REAL(kind=C_DOUBLE) :: VAL INTEGER :: LOC END TYPE as the application programmer will only use TYPE(MPI_DOUBLE_INT_TYPE) in a type declaration and not know that it is defined in terms of native C types. I also don't think it matters that SEQUENCE is used to define MPI_DOUBLE_INT_TYPE even if it is no longer considered to be so pretty. However, I'm still confused by how this is used in C. Jeff S. in responding to Bill L. said: >> How do you handle MPI_SHORT_INT? A short immediately followed in memory by an int will cause the int to have bad memory alignment. > > Right. MPI knows this; MPI_SHORT_INT will include padding between the SHORT and INT if it needs to. > How does a C programmer get the value returned in MPI_SHORT_INT? By accessing a member of a C struct? Google suggests MPI_SHORT_INT is a struct {short, int} as well as a pair of short integers followed by a 32 bit integer. Since these data types really do contain the proper amount of padding (despite what I infer from reading the MPI 3.1 standard), I think Bill's suggestion of declaring the Fortran types to be BIND(C) equivalent to a C struct is the right thing to do. Note that the Fortran application programmer won't ever see the BIND(C) attribute. Final note: I was searching the WRONG Fortran standard document when I couldn't find SEQUENCE and therefore said it looked like it was no longer in the standard. My bad... :-( -craig -craig On Wed, Dec 9, 2015 at 5:45 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > On Dec 8, 2015, at 6:31 PM, Bill Long <longb(a)cray.com> wrote: > > > > As I understand the high-level situation, this is all for MPI_REDUCE and > friends, and MPI_2INTEGER and MPI_MINLOC or MPI_MAXLOC are all just magic > constants that the MPI_REDUCE routine internally decodes to determine what > to do. The actual data, the in and out arguments of MPI_REDUCE, are choice > arguments (void *). > > > Current codes have the actual argument IN declared as INTEGER :: IN (2, > K) where K is the number of things in the list to be reduced. IN(1,*) is > the “VAL" value, and IN(2,*) is the “LOC” value. Since the corresponding > dummy argument is (void *) you can pass any actual type, since only the > memory layout matters. > > Correct. > > > If the above is correct, then the following declarations would have the > same memory layouts for the actual arguments: > > > > INTEGER :: IN(2, K) > > > > and > > > > TYPE MPI_2INTEGER_TYPE > > SEQUENCE > > INTEGER :: VAL > > INTEGER :: LOC > > END TYPE > > > > TYPE(MPI_2INTEGER_TYPE) :: IN(K) > > > > This seems like sugar coating for the 2INTEGER case. > > Yes, exactly. > > Is there a way to do it without SEQUENCE (since that is frowned upon)? > > > But, the following would likely not be equivalent: > > > > DOUBLE PRECISION :: IN(2, K) > > > > and > > > > TYPE MPI_DOUBLE_INT_TYPE > > SEQUENCE > > DOUBLE PRECISION :: VAL > > INTEGER :: LOC > > END TYPE > > > > TYPE(MPI_DOUBLE_INT_TYPE) :: IN(K) > > Right -- I'm not trying to make those equivalent. > > To be clear: > > - the only one where we need this equivalence MPI_2INTEGER. > - the intent is to deprecate MPI_2DOUBLE_PRECISION and MPI_2REAL > --> I assume that these are historical artifacts that were created > before derived types existed > - and then create 2 new types: MPI_DOUBLE_PRECISION_INTEGER and > MPI_REAL_INTEGER > --> these will be analogous to their C counterparts (but using Fortran > types, not C types) > > >> 2. I wasn't looking to emulate the C types in Fortran -- I think we've > had this discussion before. The intent is to have Fortran-native types > (that would be natural for Fortran programmers). > > > > I think BIND(C) is “natural” for Fortran programmers these days. This > stuff has been around for over a decade. The use of “DOUBLE PRECISION” is > far less natural these days. Essentially everyone has switched to > REAL(KIND= ) instead. That stuff is 25 years old. > > Hmm. Ok, that's the opposite advice we got for MPI-3.0 (which was now > admittedly several years ago). > > Soo... which way should it go these days? > > - use native Fortran datatypes > - use C datatypes (via BIND(C)) > > I.e., when users want to use MINLOC / MAXLOC in fortran, will they be > using this across DOUBLE PRECISION values, or Type(C_DOUBLE)? > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-fortran mailing list > mpiwg-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran >

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Rolf Rabenseifner 10 Dec '15

10 Dec '15

> 3. Deprecate the 2 old MPI datatypes that we don't want to encourage use of anymore: > > ----- > MPI_2REAL > MPI_2DOUBLE_PRECISION I would prefer to use the same wording as for mpif.h, i.e.: The use of MPI_2REAL and MPI_2DOUBLE_PRECISION is strongly discouraged and may be deprecated in a future Version of MPI. See MPI-3.1 page 611 lines 20-21. Rolf ----- Original Message ----- > From: "longb" <longb(a)cray.com> > To: "MPI-WG Fortran working group" <mpiwg-fortran(a)lists.mpi-forum.org> > Sent: Tuesday, December 8, 2015 10:14:04 PM > Subject: Re: [MPIWG Fortran] F08 and pair types? > On Dec 8, 2015, at 2:13 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > >> We talked about this in detail at the SJC meeting this morning. >> >> 1. In MPI 3.1 section 5.9.4 p180, the following types are defined for fortran: >> >> ----- >> MPI_2REAL >> MPI_2DOUBLE_PRECISION >> MPI_2INTEGER >> ----- >> >> The following types are defined for C: >> >> ----- >> MPI_FLOAT_INT >> MPI_DOUBLE_INT >> MPI_LONG_INT >> MPI_2INT >> MPI_SHORT_INT >> MPI_LONG_DOUBLE_INT >> —— > > > I assume the C versions are structs with two members of the obvious types. One > could mimic these in Fortran: > > use,intrinsic :: iso_c_binding > > type,bind(c) :: mpi_float_int ! the bind(c) ensures the C compiler’s memory > layout > real(c_float) :: val > integer(c_int) :: loc > end type > > and facilitate calling the C routines directly. > > When you say REAL , DOUBLE PRECISION, and INTEGER, do you mean the “default” > ones. I.e. the ones with a bit-size that can depend on compiler options? > That complicates things, combinatorially if the compile specifies -r8, -i4, > for example. > > >> >> 2. What Jeff Hammond asks is quite reasonable; it seems like we should use the >> Fortran derived types for MINLOC and MAXLOC functionality. The old Fortran >> types (2REAL, 2DOUBLE_PRECISION, 2INTEGER) exist because there were no derived >> types back in the early 90's. >> >> I think we can make three proposals (NOTE: we need to cross reference these >> proposals with the RMA and collective working groups, but I want to get the >> Fortran stuff correct first): >> >> 2a. Change the definition of MPI_INTEGER from >> >> ----- >> INTEGER FOO(2) >> ----- >> >> to (please correct my fortran syntax; you all know I have some of it wrong >> below...) >> >> ------ >> TYPE :: MPI_2INTEGER_TYPE, SEQUENCE >> INTEGER VAL >> INTEGER LOC >> END TYPE >> —— > > > This syntax would be > > TYPE :: MPI_2INTEGER_TYPE > SEQUENCE > NTEGER VAL > INTEGER LOC > END TYPE > > > However, apart from using default types, most Fortran gurus would be revolted at > the use of the archaic SEQUENCE specification. This is something that should > already have been on the MPI deprecated list. > > > >> >> Craig assures me that these two Fortran types are exactly the same in memory >> representation. Hence, even though we're technically changing the back-end >> representation of MPI_2INTEGER, we're changing it to an exactly equivalent >> definition. Hence, old codes will not break -- we're just updating to use >> newer Fortran syntax. >> >> Also, create two new types: >> >> ----- >> TYPE :: MPI_REAL_INTEGER_TYPE, SEQUENCE >> REAL VAL >> INTEGER LOC >> END TYPE >> >> TYPE :: MPI_DOUBLE_PRECISION_INTEGER_TYPE, SEQUENCE >> DOUBLE PRECISION VAL >> INTEGER LOC >> END TYPE >> ----- >> >> And MPI datatypes corresponding to these two new types: MPI_REAL_INTEGER, >> MPI_DOUBLE_PRECISION_INTEGER >> >> Note: with this one old type (2INTEGER) and the 2 new types (REAL_INTEGER and >> DOUBLE_PRECISION_INTEGER) correspond to the C types MPI_2INT, MPI_FLOAT_INT, >> MPI_DOUBLE_INT, respectively. >> >> 2b. Create three additional new Fortran types to make parity with the remaining >> C types (MPI_LONG_INT, MPI_SHORT_INT, MPI_LONG_DOUBLE_INT) -- the names below >> are completely up for negotiation, and may even be longer than 31 characters: >> >> ----- >> TYPE :: MPI_LONGINTEGER_INTEGER_TYPE, SEQUENCE >> INTEGER(KIND=whatever) VAL >> INTEGER LOC >> END TYPE >> >> TYPE :: MPI_SHORTINTEGER_INTEGER_TYPE, SEQUENCE >> INTEGER(KIND=whatever) VAL >> INTEGER LOC >> END TYPE >> >> TYPE :: MPI_QUAD_PRECISION_INTEGER_TYPE, SEQUENCE >> DOUBLE PRECISION(KIND=whatever) VAL > > REAL(KIND=whatever) VAL > > The single, double, and quad versions would be identical except for the values > for “whatever”. > > >> INTEGER LOC >> END TYPE >> ----- >> >> These 3 new Fortran types would then make Fortran support the same types as C. > > I would recommend use of the bind(c) types instead, as illustrated above, if you > want to duplicate the C types. > >> >> 3. Deprecate the 2 old MPI datatypes that we don't want to encourage use of >> anymore: >> >> ----- >> MPI_2REAL >> MPI_2DOUBLE_PRECISION > > Seems like a good idea. > > Cheers, > Bill > >> ----- >> >> Thoughts? >> >> >> >>> On Dec 8, 2015, at 10:31 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: >>> >>> Sorry, but I was too fast with my negative answer. >>> Yes, we have both arguments in the reduction >>> and therefore it would be simple to add a new >>> datatype to the list of allowed datatypes for >>> MIN/MAXLOC in Fortran. >>> >>> Deprecation is too expensive and there is no need. >>> >>> Rolf >>> >>> >>> ----- Original Message ----- >>>> From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >>>> To: "MPI-WG Fortran working group" <mpiwg-fortran(a)lists.mpi-forum.org> >>>> Sent: Tuesday, December 8, 2015 7:22:45 PM >>>> Subject: Re: [MPIWG Fortran] F08 and pair types? >>> >>>> I'm sorry, but I expect that deprecation of a language binding >>>> is not possible for a feature that is still valid in other languages (here C). >>>> >>>> This means, that MINLOC and MAXLOC are as they are defined >>>> and there is no reason to deprecate them. >>>> They are not wrong and nobody want to pay the maintenance costs >>>> for modifying existing applications that use MIN/MAXLOC. >>>> >>>> The only way to do it better, is to add a new feature, e.g., >>>> MINLOC_F08 and MAXLOC_F08 which is valid only for the new >>>> mpi_f08 module. >>>> >>>> Rolf >>>> >>>> ----- Original Message ----- >>>>> From: "Craig Rasmussen" <rasmus(a)cas.uoregon.edu> >>>>> To: "MPI-WG Fortran working group" <mpiwg-fortran(a)lists.mpi-forum.org> >>>>> Sent: Tuesday, December 8, 2015 6:30:37 PM >>>>> Subject: Re: [MPIWG Fortran] F08 and pair types? >>>> >>>>> The old stuff could be deprecated because all existing compilers that I know of >>>>> support user-defined types. I assume that "deprecated" means that it still >>>>> exists in libraries for legacy apps but is no longer really defined in the >>>>> standard. >>>>> >>>>> -craig >>>>> >>>>> On Tue, Dec 8, 2015 at 9:19 AM, Craig Rasmusen < rasmus(a)cas.uoregon.edu > wrote: >>>>> >>>>> >>>>> >>>>> Fortran is a modern language? Wut... >>>>> >>>>> Actually seems like a simple and good change. For example, could define: >>>>> >>>>> MPI_REAL_INT and MPI_INTEGER_INT >>>>> >>>>> However, I'm not sure what would be in the structs (user-defined types). I'll >>>>> have to ask Squyres what the text following >>>>> >>>>> "The datatype MPI_FLOAT_INT is as if defined by the following sequence of >>>>> instructions" >>>>> >>>>> means. >>>>> >>>>> -craig >>>>> >>>>> On Mon, Dec 7, 2015 at 4:20 PM, Jeff Hammond < jeff.science(a)gmail.com > wrote: >>>>> >>>>> >>>>> >>>>> I have a reasonable understanding of why the legacy Fortran bindings use 2REAL, >>>>> 2DOUBLE_PRECISION and 2INTEGER with {MAX,MIN}LOC reductions. >>>>> >>>>> Doesn't modern Fortran provide a way to create struct-like things along the >>>>> lines of C, such that more appropriate pair types could be used for these? >>>>> >>>>> The relevant text is section 5.9.4 of MPI 3.1. >>>>> >>>>> Jeff >>>>> >>>>> -- >>>>> Jeff Hammond >>>>> jeff.science(a)gmail.com >>>>> http://jeffhammond.github.io/ >>>>> >>>>> _______________________________________________ >>>>> mpiwg-fortran mailing list >>>>> mpiwg-fortran(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpiwg-fortran mailing list >>>>> mpiwg-fortran(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran >>>> >>>> -- >>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >>>> _______________________________________________ >>>> mpiwg-fortran mailing list >>>> mpiwg-fortran(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >>> _______________________________________________ >>> mpiwg-fortran mailing list >>> mpiwg-fortran(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran >> >> >> -- >> Jeff Squyres >> jsquyres(a)cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> _______________________________________________ >> mpiwg-fortran mailing list >> mpiwg-fortran(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran > > Bill Long > longb(a)cray.com > Fortran Technical Support & voice: > 651-605-9024 > Bioinformatics Software Development fax: 651-605-9142 > Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN 55101 > > > _______________________________________________ > mpiwg-fortran mailing list > mpiwg-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Bill Long 09 Dec '15

09 Dec '15

On Dec 9, 2015, at 7:45 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: >> >> But, the following would likely not be equivalent: >> >> DOUBLE PRECISION :: IN(2, K) >> >> and >> >> TYPE MPI_DOUBLE_INT_TYPE >> SEQUENCE >> DOUBLE PRECISION :: VAL >> INTEGER :: LOC >> END TYPE >> >> TYPE(MPI_DOUBLE_INT_TYPE) :: IN(K) > > Right -- I'm not trying to make those equivalent. > > To be clear: > > - the only one where we need this equivalence MPI_2INTEGER. > - the intent is to deprecate MPI_2DOUBLE_PRECISION and MPI_2REAL > --> I assume that these are historical artifacts that were created before derived types existed > - and then create 2 new types: MPI_DOUBLE_PRECISION_INTEGER and MPI_REAL_INTEGER > --> these will be analogous to their C counterparts (but using Fortran types, not C types) There would be compatibility ramifications. For the legacy Fortran interfaces, the called MPI_REDUCE function would be interpreting the memory incorrectly for the DOUBLE case for sure, and maybe the REAL case as well, since it is currently expecting the “LOC” values to be coming in as floating point values. The DOUBLE case has the added problem of memory alignment differences. Perhaps the new types should be permitted ONLY for the F08 interface. For the F08 interfaces, the arguments are coming in as TYPE(*),DIMENSION(..). The assumed-rank argument is currently rank 2, with 2*K total elements. The new incoming argument will be rank 1, with K elements, each of which is larger than with the old version. The MPI_REDUCE code could be modified in this case, based on the type, element_size, and rank parts of the descriptor, to figure out whether the in/out arguments are the new or old convention and branch to code that does the right thing. But it is still the case that this will cause recoding work in the MPI library routines. Cheers, Bill Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN 55101

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Jeff Squyres (jsquyres) 09 Dec '15

09 Dec '15

On Dec 8, 2015, at 6:31 PM, Bill Long <longb(a)cray.com> wrote: > > As I understand the high-level situation, this is all for MPI_REDUCE and friends, and MPI_2INTEGER and MPI_MINLOC or MPI_MAXLOC are all just magic constants that the MPI_REDUCE routine internally decodes to determine what to do. The actual data, the in and out arguments of MPI_REDUCE, are choice arguments (void *). > Current codes have the actual argument IN declared as INTEGER :: IN (2, K) where K is the number of things in the list to be reduced. IN(1,*) is the “VAL" value, and IN(2,*) is the “LOC” value. Since the corresponding dummy argument is (void *) you can pass any actual type, since only the memory layout matters. Correct. > If the above is correct, then the following declarations would have the same memory layouts for the actual arguments: > > INTEGER :: IN(2, K) > > and > > TYPE MPI_2INTEGER_TYPE > SEQUENCE > INTEGER :: VAL > INTEGER :: LOC > END TYPE > > TYPE(MPI_2INTEGER_TYPE) :: IN(K) > > This seems like sugar coating for the 2INTEGER case. Yes, exactly. Is there a way to do it without SEQUENCE (since that is frowned upon)? > But, the following would likely not be equivalent: > > DOUBLE PRECISION :: IN(2, K) > > and > > TYPE MPI_DOUBLE_INT_TYPE > SEQUENCE > DOUBLE PRECISION :: VAL > INTEGER :: LOC > END TYPE > > TYPE(MPI_DOUBLE_INT_TYPE) :: IN(K) Right -- I'm not trying to make those equivalent. To be clear: - the only one where we need this equivalence MPI_2INTEGER. - the intent is to deprecate MPI_2DOUBLE_PRECISION and MPI_2REAL --> I assume that these are historical artifacts that were created before derived types existed - and then create 2 new types: MPI_DOUBLE_PRECISION_INTEGER and MPI_REAL_INTEGER --> these will be analogous to their C counterparts (but using Fortran types, not C types) >> 2. I wasn't looking to emulate the C types in Fortran -- I think we've had this discussion before. The intent is to have Fortran-native types (that would be natural for Fortran programmers). > > I think BIND(C) is “natural” for Fortran programmers these days. This stuff has been around for over a decade. The use of “DOUBLE PRECISION” is far less natural these days. Essentially everyone has switched to REAL(KIND= ) instead. That stuff is 25 years old. Hmm. Ok, that's the opposite advice we got for MPI-3.0 (which was now admittedly several years ago). Soo... which way should it go these days? - use native Fortran datatypes - use C datatypes (via BIND(C)) I.e., when users want to use MINLOC / MAXLOC in fortran, will they be using this across DOUBLE PRECISION values, or Type(C_DOUBLE)? -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Bill Long 08 Dec '15

08 Dec '15

On Dec 8, 2015, at 7:02 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > On Dec 8, 2015, at 2:44 PM, Bill Long <longb(a)cray.com> wrote: >> >> >>> and goes on to define an MPI Datatype that is a C float followed immediately in memory by a C int. So the MPI standard actually defines a memory representation rather than a C struct. >> >> How do you handle MPI_SHORT_INT? A short immediately followed in memory by an int will cause the int to have bad memory alignment. > > Right. MPI knows this; MPI_SHORT_INT will include padding between the SHORT and INT if it needs to. > >> Was it the case that the starting point for the rules was the SEQUENCE type in Fortran (which might have been the only option at the time) and then force C to conform? Today it is easier to get Fortran to mimic what is natural for C. > > There are two issues here: > > 1. The "good" MPI datatype name of MPI_2INTEGER is already being used; it currently means "INTEGER foo(2)". We want to update it to mean a derived type that contains 2 INTEGERs and still be able to use the MPI datatype name "MPI_2INTEGER" without breaking existing codes. As I understand the high-level situation, this is all for MPI_REDUCE and friends, and MPI_2INTEGER and MPI_MINLOC or MPI_MAXLOC are all just magic constants that the MPI_REDUCE routine internally decodes to determine what to do. The actual data, the in and out arguments of MPI_REDUCE, are choice arguments (void *). Current codes have the actual argument IN declared as INTEGER :: IN (2, K) where K is the number of things in the list to be reduced. IN(1,*) is the “VAL" value, and IN(2,*) is the “LOC” value. Since the corresponding dummy argument is (void *) you can pass any actual type, since only the memory layout matters. > > I hear you that SEQUENCE is considered bad programming/style. Is having the equivalence between INTEGER foo(2) and a derived type with 2 INTEGERs do-able without using SEQUENCE? If the above is correct, then the following declarations would have the same memory layouts for the actual arguments: INTEGER :: IN(2, K) and TYPE MPI_2INTEGER_TYPE SEQUENCE INTEGER :: VAL INTEGER :: LOC END TYPE TYPE(MPI_2INTEGER_TYPE) :: IN(K) This seems like sugar coating for the 2INTEGER case. But, the following would likely not be equivalent: DOUBLE PRECISION :: IN(2, K) and TYPE MPI_DOUBLE_INT_TYPE SEQUENCE DOUBLE PRECISION :: VAL INTEGER :: LOC END TYPE TYPE(MPI_DOUBLE_INT_TYPE) :: IN(K) since the IN(1:2, n) section of the current IN array will have 2 doubles, or 16 bytes in most cases. One element of the new IN would have only 12 bytes. If K is > 1, there is potential for trouble. Certainly no guarantee from the Fortran standard. The standard does require that INTEGER and REAL (both default) as part of a SEQUENCE type are the same size in memory, so the MPI_REAL_INT_TYPE version would seem to be OK. > > 2. I wasn't looking to emulate the C types in Fortran -- I think we've had this discussion before. The intent is to have Fortran-native types (that would be natural for Fortran programmers). I think BIND(C) is “natural” for Fortran programmers these days. This stuff has been around for over a decade. The use of “DOUBLE PRECISION” is far less natural these days. Essentially everyone has switched to REAL(KIND= ) instead. That stuff is 25 years old. Cheers, Bill > I might not have been clear when I said "parity with the C types" -- what I meant was having "Fortran-natural equivalents of the 6 C MPI_Datatypes". > > Make sense? > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-fortran mailing list > mpiwg-fortran(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN 55101

1 0

Re: [MPIWG Fortran] F08 and pair types?
by Jeff Squyres (jsquyres) 08 Dec '15

08 Dec '15

On Dec 8, 2015, at 2:44 PM, Bill Long <longb(a)cray.com> wrote: > > >> and goes on to define an MPI Datatype that is a C float followed immediately in memory by a C int. So the MPI standard actually defines a memory representation rather than a C struct. > > How do you handle MPI_SHORT_INT? A short immediately followed in memory by an int will cause the int to have bad memory alignment. Right. MPI knows this; MPI_SHORT_INT will include padding between the SHORT and INT if it needs to. > Was it the case that the starting point for the rules was the SEQUENCE type in Fortran (which might have been the only option at the time) and then force C to conform? Today it is easier to get Fortran to mimic what is natural for C. There are two issues here: 1. The "good" MPI datatype name of MPI_2INTEGER is already being used; it currently means "INTEGER foo(2)". We want to update it to mean a derived type that contains 2 INTEGERs and still be able to use the MPI datatype name "MPI_2INTEGER" without breaking existing codes. I hear you that SEQUENCE is considered bad programming/style. Is having the equivalence between INTEGER foo(2) and a derived type with 2 INTEGERs do-able without using SEQUENCE? 2. I wasn't looking to emulate the C types in Fortran -- I think we've had this discussion before. The intent is to have Fortran-native types (that would be natural for Fortran programmers). I might not have been clear when I said "parity with the C types" -- what I meant was having "Fortran-natural equivalents of the 6 C MPI_Datatypes". Make sense? -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0