On Wed, Apr 03, 2013 at 09:19:59AM -0500, Bob Cernohous wrote:
I've been out sick and haven't followed this conversation too closely, but I've gotten a support note that seems somewhat related -- at least they're asking about external32. Any comments from the experts before I try to dig in? ROMIO supports external32? A quick search shows external32 code is common, not ad_bg specific.
external32 is indeed a part of the standard. for 15 years ROMIO had no external32 support, and no one seemed to care too much. Mostly, the role of external32 -- a portable file format -- is better played by pnetcdf and hdf5, which in addition to being portable are also self-describing. Intel contributed external32 support to ROMIO, but Dave and I are pretty sure it's not going to work when type-on-platform is different from type-in-external32 See the end of https://trac.mpich.org/projects/mpich/ticket/1754
The MPI standard defines a portable data representation for MPI I/O, see e.g. [1]. On BGQ, attempting to write data via MPI I/O with the data representation set to 'external32', MPI does not report any errors but the data written is not in the correct format. Furthermore trying to read the data back with the same data representation fails and does not reproduce the original data.
Right. I think the Intel-contributed external32 work, warts and all, went into MPICH 3.0 sorry we did not return an error in older versions. that's a bug.
The example Fortran code does the following: 1) write a few numbers (integer, float and double) to file via MPI I/O in native format 2) read the numbers back in native format 3) write the same numbers to file in external32 format 4) output MPI errors/success when using MPI I/O 5) read the numbers back from file
1+2 work flawlessly 3+4 show no errors and instead indicate MPI SUCCESS 5 fails to reproduce correct numbers
Once you re-sync with MPICH, this use case will work -- integer, float, and double have the same size types in native and external32 format.
The data written in 1) appears as big endian data as expected. The data written in 3) seems to be little endian (for integer and float) and strangely byte-swapped for doubles.
[1] http://www.mpi-forum.org/docs/mpi21-report/node285.htm
Bob Cernohous: (T/L 553) 507-253-6093
[email protected] IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829
Chaos reigns within. Reflect, repent, and reboot. Order shall return.
From: Dave Goodell <[email protected]> To: Larry Baker <[email protected]>, Cc: [email protected] Date: 04/02/2013 04:48 PM Subject: Re: [mpich-devel] binding a new MPI type with fortran Sent by: [email protected]
On Apr 2, 2013, at 4:34 PM CDT, Larry Baker <[email protected]> wrote:
Jeff referred me to the MPI 3.0 spec ( http://mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf) for data sizes. But, even it is contradictory. For example, in section 3.2.2, it says an MPI_INTEGER is a Fortran (default KIND) INTEGER. Fortran does not specify the size of a (default KIND) INTEGER. However, Table 13.2 says it is 4 bytes. It could just as easily be 8 bytes in a 64-bit execution environment. In that case, Fortran requires that the size of all the default KIND numeric types in Table 13.2 would double in size. But, not necessarily pointers; pointers in Fortran do not have a specified size (storage unit).
Table 13.2 lists sizes for a specific kind of encoding of MPI datatypes known as "external32". You should not use it as a reference for the actual size of the MPI datatype, which must instead always match the actual language type in the corresponding language binding.
The MPI 3.0 spec says if you send an MPI_INTEGER, you have to receive an MPI_INTEGER. I didn't exhaustively search the MPI 3.0 spec, but you might also have to make sure the default KIND INTEGER is the same for the sender and receiver. This might be beyond the scope of MPI, and I do not know how exhaustive the run-time tests can be to detect violations of the rules. (A message would have to carry type metadata for the payload data in addition to the payload data itself.) Section 17 had lots of information about Fortran and C data sizes, including automatically conversions. I didn't read enough to know if the spec covers the case of exchanging default KIND INTEGERs between two executables with different sizes. Maybe it is true that MPI mandates something like lexical scoping rules, such that, for example, all members of an MPI application must assign the same semantics to their fundamental data types -- the default KIND INTEGER, etc., in the case of Fortran.
Most users should ignore the rules regarding conversions on heterogeneous systems. They don't make a lot of sense even if you're an MPI expert, and:
1) almost no MPI implementations properly support this behavior
2) almost no user codes are written correctly to deal with it, even if #1 were true
-Dave
-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA