November 2018 - mpi-comments - mailman3.cels.anl.gov

Re: [Mpi-comments] Request: Canonical MPI_Pack_external with different data representations
by Christoph Niethammer 12 Nov '18

12 Nov '18

Hello, I would like to add to the discussion that, Open MPI currently uses external32 for the mentioned MPI_pack_external routine, but as far as I see, an extension to other formats should be straight forward via its converter mechanism. Maybe someone else can comment on the other MPI implementations: MPICH, ... Best Christoph Niethammer ----- Mensaje original ----- De: "mpi-comments" <mpi-comments(a)lists.mpi-forum.org> Para: "mpi-comments" <mpi-comments(a)lists.mpi-forum.org> CC: "Mor, Omri" <omrimor2(a)illinois.edu> Enviados: Viernes, 9 de Noviembre 2018 19:23:18 Asunto: Re: [Mpi-comments] Request: Canonical MPI_Pack_external with different data representations > After more than 20 years, how about making use of the "future extensibility" > for the next standard to introduce more datarep arguments (like e.g. "native”)? It could also be useful to define an “external64” data representation, to support LP64 systems better. Why is MPI_AINT 8 bytes in “external32”? I suppose that it was designed to support the IL32P64 (LLP64) data model, used in Windows. Unrelated, it may also be useful to define an MPI_SIZE_T datatype that can be used with size_t variables. Though I guess MPI_Aint and MPI_AINT can be used for that purpose as well. > I also wonder how to determine the exact size of a pack buffer. > " > A call to MPI_PACK_SIZE(incount, datatype, comm, size) returns in size an upper bound > on the increment in position that is effected by a call to MPI_PACK(inbuf, incount, datatype, > outbuf, outcount, position, comm).[...] > > Rationale. The call returns an upper bound, rather than an exact bound, since the > exact amount of space needed to pack the message may depend on the context (e.g., > first message packed in a packing unit may take more space). (End of rationale.) > "(MPI3.1, p. 135) > There seems not to exist e.g. a routine to add up the sizes (INOUT size instead of position) > analogous to the Pack procedure. Is there a function to determine the extra size for the > first message (see Rationale). > How should the exact buffer size (and not only an upper bound) be determinated before packing? For the canonical MPI_PACK/MPI_UNPACK, there’s MPI_PACK_EXTERNAL_SIZE, which returns the exact size of the pack buffer. For the non-canonical MPI_PACK/MPI_UNPACK, you can only get an upper bound on the pack size, as the function doesn’t know if it’s the first message in a series of pack/unpack calls or not. MPI_PACK may, depending on the implementation, only need to store the header for the first message, while the others can be stored directly; however, MPI_PACK_SIZE can only assume that it’s the first message and thus would need space for the header as well. Omri Mor _______________________________________________ mpi-comments mailing list mpi-comments(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpi-comments

1 0

Re: [Mpi-comments] Request: Canonical MPI_Pack_external with different data representations
by Mor, Omri 09 Nov '18

09 Nov '18

> After more than 20 years, how about making use of the "future extensibility" > for the next standard to introduce more datarep arguments (like e.g. "native”)? It could also be useful to define an “external64” data representation, to support LP64 systems better. Why is MPI_AINT 8 bytes in “external32”? I suppose that it was designed to support the IL32P64 (LLP64) data model, used in Windows. Unrelated, it may also be useful to define an MPI_SIZE_T datatype that can be used with size_t variables. Though I guess MPI_Aint and MPI_AINT can be used for that purpose as well. > I also wonder how to determine the exact size of a pack buffer. > " > A call to MPI_PACK_SIZE(incount, datatype, comm, size) returns in size an upper bound > on the increment in position that is effected by a call to MPI_PACK(inbuf, incount, datatype, > outbuf, outcount, position, comm).[...] > > Rationale. The call returns an upper bound, rather than an exact bound, since the > exact amount of space needed to pack the message may depend on the context (e.g., > first message packed in a packing unit may take more space). (End of rationale.) > "(MPI3.1, p. 135) > There seems not to exist e.g. a routine to add up the sizes (INOUT size instead of position) > analogous to the Pack procedure. Is there a function to determine the extra size for the > first message (see Rationale). > How should the exact buffer size (and not only an upper bound) be determinated before packing? For the canonical MPI_PACK/MPI_UNPACK, there’s MPI_PACK_EXTERNAL_SIZE, which returns the exact size of the pack buffer. For the non-canonical MPI_PACK/MPI_UNPACK, you can only get an upper bound on the pack size, as the function doesn’t know if it’s the first message in a series of pack/unpack calls or not. MPI_PACK may, depending on the implementation, only need to store the header for the first message, while the others can be stored directly; however, MPI_PACK_SIZE can only assume that it’s the first message and thus would need space for the header as well. Omri Mor

1 0

Request: Canonical MPI_Pack_external with different data representations
by Martin Bernreuther 09 Nov '18

09 Nov '18

Hello, since MPI-2.0 there's a "Canonical MPI_PACK and MPI_UNPACK" (p. 78, section 4.16) (also see e.g. https://www.mpi-forum.org/docs/mpi-2.1/mpi21-report-bw/node83.htm#Node83) which didn't change much over time: " These functions read/write data to/from the buffer in the "external32" data format specified in Section 13.5.2, and calculate the size needed for packing. Their first arguments specify the data format, for future extensibility, but currently the only valid value of the datarep argument is "external32." "(taken from MPI-3.1, p. 138, section 4.3) After more than 20 years, how about making use of the "future extensibility" for the next standard to introduce more datarep arguments (like e.g. "native")? With MPI_PACK for I/O there's a " Advice to users. The type MPI_PACKED is treated as bytes and is not converted. The user should be aware that MPI_PACK has the option of placing a header in the beginning of the pack buﬀer. (End of advice to users.) "(MPI-31, p. 539) A "header" might be undesirable for other programs to read and this seems not to be an issue using MPI_Pack_external with MPI_BYTE, since there is no header: " Rationale. MPI_PACK_EXTERNAL specifies that there is no header on the message and further specifies the exact format of the data. Since MPI_PACK may (and is allowed to) use a header, the datatype MPI_PACKED cannot be used for data packed with MPI_PACK_EXTERNAL. (End of rationale.) "(MPI-31, p. 138) Thus the canonical version seems to have an advantage for MPI-I/O here. However if a user wants to write e.g. "little endian" on an actual x86 architecture, this seems to be excluded by the actual standard. Thank you, Martin Bernreuther P.S.: I also wonder how to determine the exact size of a pack buffer. " A call to MPI_PACK_SIZE(incount, datatype, comm, size) returns in size an upper bound on the increment in position that is effected by a call to MPI_PACK(inbuf, incount, datatype, outbuf, outcount, position, comm).[...] Rationale. The call returns an upper bound, rather than an exact bound, since the exact amount of space needed to pack the message may depend on the context (e.g., first message packed in a packing unit may take more space). (End of rationale.) "(MPI3.1, p. 135) There seems not to exist e.g. a routine to add up the sizes (INOUT size instead of position) analogous to the Pack procedure. Is there a function to determine the extra size for the first message (see Rationale). How should the exact buffer size (and not only an upper bound) be determinated before packing? -- ________________________________________________________________________ Dr.-Ing. Martin Bernreuther University of Stuttgart High Performance Computing Center (HLRS) Nobelstrasse 19 (Office: 1.307) 70569 Stuttgart, Germany Phone: (++49-(0)711) 685-64542 Fax: (++49-(0)711) 685-54542, -87279, -65832 E-Mail: bernreuther(a)hlrs.de URL: http://www.hlrs.de/people/bernreuther/

1 0