October 2019 - mpiwg-large-counts - mailman3.cels.anl.gov

Re: [Mpiwg-large-counts] Pythonizing instructions
by HOLMES Daniel 19 Oct '19

19 Oct '19

Hi Jeff, Thanks for doing this. I’ll take a look on my Monday. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 19 Oct 2019, at 21:26, Jeff Squyres (jsquyres) via mpiwg-large-counts <mpiwg-large-counts(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: As discussed in our last meeting, I've written a first take at instructions for Chapter Authors for how to write Pythonized functions. These instructions are not yet intended for a wide audience -- they are intended solely for the working group. Let's iterate on them here in the WG and get them to a good enough state before sharing them widely: https://github.com/mpiwg-large-count/large-count-issues/wiki/pythonizing Puri/Tony/Dan: please try to use these instructions to Pythonize a few bindings. The PR for the Pythonization is https://github.com/mpiwg-large-count/mpi-standard/pull/2. All even-numbered chapters are done, and a small number of odd-numbered chapters. Grep for `funcdef` in chapter *.tex files to find a chapter that hasn't been Pythonized yet (funcdef is the LaTeX macro for hard-coded LaTeX macros). Send your feedback here and/or we'll discuss on Wednesday. -- Jeff Squyres jsquyres(a)cisco.com<mailto:[email protected]> _______________________________________________ mpiwg-large-counts mailing list mpiwg-large-counts(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts

1 0

Pythonizing instructions
by Jeff Squyres (jsquyres) 19 Oct '19

19 Oct '19

As discussed in our last meeting, I've written a first take at instructions for Chapter Authors for how to write Pythonized functions. These instructions are not yet intended for a wide audience -- they are intended solely for the working group. Let's iterate on them here in the WG and get them to a good enough state before sharing them widely: https://github.com/mpiwg-large-count/large-count-issues/wiki/pythonizing Puri/Tony/Dan: please try to use these instructions to Pythonize a few bindings. The PR for the Pythonization is https://github.com/mpiwg-large-count/mpi-standard/pull/2. All even-numbered chapters are done, and a small number of odd-numbered chapters. Grep for `funcdef` in chapter *.tex files to find a chapter that hasn't been Pythonized yet (funcdef is the LaTeX macro for hard-coded LaTeX macros). Send your feedback here and/or we'll discuss on Wednesday. -- Jeff Squyres jsquyres(a)cisco.com

1 0

Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements
by schulzm＠in.tum.de 09 Oct '19

09 Oct '19

Hi Rolf, I can’t find an email from the 23rd - I have one from you on the 20th, 22nd, and then every day from the 24th to the 28th, but none on the 23rd :). Is this the same as below, or am I missing something? Let me take a look at this, but I’ll need a few days. Martin — Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de > On 9. Oct 2019, at 13:44, Rolf Rabenseifner via mpiwg-large-counts <mpiwg-large-counts(a)lists.mpi-forum.org> wrote: > > > ----- Forwarded Message ----- > From: Rolf Rabenseifner <rabenseifner(a)hlrs.de <mailto:[email protected]>> > To: HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]>> > Cc: Purushotham V. Bangalore <puri(a)uab.edu <mailto:[email protected]>>, Martin Ruefenacht <m.a.ruefenacht(a)gmail.com <mailto:[email protected]>>, Claudia Blaas-Schenner <claudia.blaas-schenner(a)tuwien.ac.at <mailto:[email protected]>>, Jeff Squyres <jsquyres(a)cisco.com <mailto:[email protected]>>, Anthony Skjellum <tony-skjellum(a)utc.edu <mailto:[email protected]>> > Sent: Tue, 08 Oct 2019 20:12:18 +0200 (CEST) > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements > > Hi all, > > (Dan and Puri, on Sep. 23, I sent also a copy of my email to > Martin. For me it Looks like that he had no complains) > > All the rest of answers are inline below. > > Best regards > Rolf > > > ----- Original Message ----- >> From: "HOLMES Daniel" <d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]>> >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de <mailto:[email protected]>> >> Cc: "Purushotham V. Bangalore" <puri(a)uab.edu <mailto:[email protected]>>, "Martin Ruefenacht" <m.a.ruefenacht(a)gmail.com <mailto:[email protected]>>, "Claudia Blaas-Schenner" >> <claudia.blaas-schenner(a)tuwien.ac.at <mailto:[email protected]>>, "Jeff Squyres" <jsquyres(a)cisco.com <mailto:[email protected]>>, "Anthony Skjellum" <tony-skjellum(a)utc.edu <mailto:[email protected]>> >> Sent: Monday, October 7, 2019 8:25:03 PM >> Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements > >> Hi Rolf, >> >> Apologies - I have no record of this email arriving before today. This probably >> means that I am not a member of te mpiwg-large-counts email list, which seems >> somewhat sub-optimal. >>> >>> To understand how big and large counts should be implemented in MPI-4, >>> it is important to understand the count, displacement and size model of in >>> MPI-3.1. >>> >>> As long as we do not have a common understanding of MPI-3.1, >>> we will have problems to define MPI-4. >> >> Agreed. >> >>> Therefore my clear question, do we agree on the rules above? >> >> In short, no. >> In addition, these rules are incomplete. >> Also, there are breaches in MPI-3.1 for most of these rules. >> >> — >> >> Below this point in this email, I attempt to cover some of the disagreements >> with your rules and I include a missing rule. However, this is not the sum >> total of the knowledge/experience gained in/by the WG. Some of the other WG >> members may wish to disagree and I expect that some points will generate >> further discussion. >> >>> - an index into such an array, i.e., the number of an element. >>> Argument name / descriptions [Routine]: >>> -- array_of_displacements / displacement ..., in multiples of oldtype extent >>> (... integer) [MPI_TYPE_INDEXED] >>> -- sdispls / integer array (of length group size). Entry j specifies the >>> displacement ... [MPI_ALLTOALLV] >>> C-type in MPI-3.1: int >>> C-type in _l: MPI_Count >> >> The two examples you give are not the same. The array_of_displacements is a >> number, correct type MPI_Count (in C), as you state, because it is a “multiple >> of” <something>. > > It looks like, that for MPI_TYPE_INDEXED, you agreed. > >> However, each element in the sdispls array is a displacement >> relative to a memory location, sendbuf. >> The correct type is MPI_Aint (in C). >> Note that it is incorrect in MPI-3.1 for this parameter to be int[] (in C) >> because int is *not* a smaller version of MPI_Aint. A displacement must not be >> described as a number of bytes because of segmented address spaces. The phrase >> “displacement in bytes” is nonsense. > > > MPI-3.1 p171:7 clearly writes: sendbuf+sdispls[i]*extent(sendtype) > > This means if sendbuf would be declared as > type_X sendbuf[sendcount[i]]; > then sendbuf[sdispls[i]] would be exactly the same as we have in > MPI_TYPE_INDEXED: sdispls[i] is in index into an array of elements. > > To compare it again: > > - MPI_TYPE_INDEXED MPI-3.1 p89:9-10 and 15-16 > > IN array_of_displacements displacement for each block, > in multiples of oldtype extent (array of integer) > > int MPI_Type_indexed(..., const int array_of_displacements[], > > - MPI_ALLTOALLV MPI-31., p170:23-24 and p171:7 > > int MPI_Alltoallv(..., const int sdispls[], > > sendbuf+sdispls[i]*extent(sendtype) > > > Second reason: When defining MPI_NEIGHBOR_ALLTOALL|V|W, we > clearly revisited MPI_ALLTOALL|V|W and decided to correct wrong types. > For MPI(_NEIGHBOR)_ALLTOALL and MPI(_NEIGHBOR)_ALLTOALLV, we decided > that all is correct, whereas vor MPI(_NEIGHBOR)_ALLTOALLW, we decided > that the correct type for the displs is MPI_Aint (and not int). > > Therefore, I cannot see any difference, and the MPI Forum also did not > see this difference when they standardized MPI_NEIGHBOR_ALLTOALLV. > > >>> - number of bytes >>> Argument name / descriptions [Routine]: >>> -- size / size of window in bytes (non-negative integer) [MPI_WIN_CREATE] >>> -- outsize / output buffer size, in bytes (integer) [MPI_PACK_EXTERNAL] >>> C-type in MPI-3.1: MPI_Aint >>> C-type in _l: MPI_Aint >>> (Wrong) C-types in MPI-3.1: int [MPI_PACK, MPI_TYPE_SIZE] >>> C-type (corrected) in _l: MPI_Aint [MPI_PACK, MPI_TYPE_SIZE] >> >> A number of <something> is a number, not a displacement. The correct “large” >> type is MPI_Count (in C). There are places in MPI-3.1 where this is, correctly, >> int or MPI_Count but there are other places in MPI-3.1 where it is, >> incorrectly, MPI_Aint. > > I never said that that a number is a displacement. > MPI_Aint is clearly used for > - addresses relative to a buffer begin, > - absolute addresses (returned by MPI_GET_ADDRESS) > - number of bytes. > > MPI-3.1 says > 2.5.6 Absolute Addresses and Relative Address Displacements > Some MPI procedures use address arguments that represent an absolute address in the calling > program, or relative displacement arguments that represent differences of two absolute > addresses. The datatype of such arguments is MPI_Aint in C and INTEGER (KIND= > MPI_ADDRESS_KIND) in Fortran. > > Relative or absolute addresses ==> MPI_Aint. > It is not written MPI_Aint ==> rel. or abs. addresses. > > Examples for number of bytes: > > MPI_TYPE_CREATE_HVECTOR, result of MPI_AINT_DIFF, MPI_TYPE_GET_EXTENT, > MPI_WIN_CREATE, MPI_WIN_ALLOCATE, MPI_WIN_ALLOCATE_SHARED, > MPI_WIN_SHARED_QUERY, MPI_WIN_ATTACH. > > This is a feature, not a bug. > > Example for byte-displacements: > > MPI_TYPE_CREATE_HINDEXED, MPI_TYPE_CREATE_HINDEXED_BLOCK, > MPI_TYPE_CREATE_STRUCT, result of MPI_AINT_DIFF. > > A real exception is MPI_PUT, ... with MPI_Aint target_disp, > and target_addr = window_base + target_disp x disp_unit. > > >>> - smaller number in bytes >>> Some argument names: e.g., disp_unit >>> Description: local unit size for displacements, in bytes (positive integer) >>> C-type in MPI-3.1: int >>> C-type in _l: still int? or MPI_Count? >> >> Why must disp_unit be smaller, and is it really a number of bytes? The premise >> for disp_unit is that a window might consist of a sequence of locations in >> memory that are used to store values of a particular type (typically >> represented by an MPI datatype). The disp_unit gives an indication of the >> extent of the datatype, a quantity by which all offset values for the window >> will be scaled, in order that address arithmetic adding (offset*disp_unit) to >> base_address produces a new address that is one of the sequence of locations in >> memory used to store values, i.e. the location of the beginning of one of the >> datatypes. This indicates to me that the correct type for disp_unit is MPI_Aint >> (in C). Note that, if true, this means that using int (as is done in MPI-3.1) >> is incorrect, even prior to the large count changes because int is *not* a >> smaller version of MPI_Aint. > > Yes. disp_unit is an extent of a type, which is a number of bytes, which > should have been MPI_Aint (as in MPI_TYPE_GET_EXTENT). > > Therefore > C-type in _l: should be MPI_Aint. > > >> A missing rule concerns "length arguments” (see section 2.5.2 in MPI-3.1 for a >> definition). >> The length of the array_of_requests for the MPI_Waitany function is given by a >> parameter called count, which is of type int (in C). This is a different class >> of type than any discussed so far. The WG has flip-flopped with regards to >> whether this class of type should be enlarged or not. Our current position is >> (I believe) to leave this class of types alone, i.e. they will remain int (in >> C). > > I have to think about. > > Best regards > Rolf > > >> >> Cheers, >> Dan. >> — >> Dr Daniel Holmes PhD >> Architect (HPC Research) >> d.holmes(a)epcc.ed.ac.uk <mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>> >> Phone: +44 (0) 131 651 3465 >> Mobile: +44 (0) 7940 524 088 >> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT >> — >> The University of Edinburgh is a charitable body, registered in Scotland, with >> registration number SC005336. >> — >> >> On 7 Oct 2019, at 18:09, Rolf Rabenseifner >> <rabenseifner(a)hlrs.de <mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>>> wrote: >> >> Dear all, >> >> I've never seen any answer to my email. >> Maybe there isn't any answer, maybe it never reached me. >> >> Please give me an advice. >> >> Best regards >> Rolf >> >> ----- Forwarded Message ----- >> From: Rolf Rabenseifner via mpiwg-large-counts >> <mpiwg-large-counts(a)lists.mpi-forum.org <mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>>> >> To: >> mpiwg-large-counts(a)lists.mpi-forum.org <mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>> >> Cc: Rolf Rabenseifner <rabenseifner(a)hlrs.de <mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>>> >> Sent: Mon, 23 Sep 2019 17:15:09 +0200 (CEST) >> Subject: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, >> and byte and nonbyte displacements >> >> Dear all, >> >> To understand how big and large counts should be implemented in MPI-4, >> it is important to understand the count, displacement and size model of in >> MPI-3.1. >> >> As long as we do not have a common understanding of MPI-3.1, >> we will have problems to define MPI-4. >> >> Therefore here my understanding of MPI-3. We have >> >> - number of array elements with elements of a given type (typically represented >> by an MPI datatype handle). >> Usual argument name: count >> Usual description: number of elements in ... buffer (non-negative integer) >> C-type in MPI-3.1: int >> C-type in _l: MPI_Count >> >> - an index into such an array, i.e., the number of an element. >> Argument name / descriptions [Routine]: >> -- array_of_displacements / displacement ..., in multiples of oldtype extent >> (... integer) [MPI_TYPE_INDEXED] >> -- sdispls / integer array (of length group size). Entry j specifies the >> displacement ... [MPI_ALLTOALLV] >> C-type in MPI-3.1: int >> C-type in _l: MPI_Count >> >> - number of bytes >> Argument name / descriptions [Routine]: >> -- size / size of window in bytes (non-negative integer) [MPI_WIN_CREATE] >> -- outsize / output buffer size, in bytes (integer) [MPI_PACK_EXTERNAL] >> C-type in MPI-3.1: MPI_Aint >> C-type in _l: MPI_Aint >> (Wrong) C-types in MPI-3.1: int [MPI_PACK, MPI_TYPE_SIZE] >> C-type (corrected) in _l: MPI_Aint [MPI_PACK, MPI_TYPE_SIZE] >> >> - smaller number in bytes >> Some argument names: e.g., disp_unit >> Description: local unit size for displacements, in bytes (positive integer) >> C-type in MPI-3.1: int >> C-type in _l: still int? or MPI_Count? >> >> - Position or relative byte displacement within an array of bytes. >> Such values can be calculated as any sum and product of in, long, long long, >> and MPI_Aint as long as MPI_Aint value contains a pure integer size value, >> i.e., an (integer) difference of two absolute addresses within one sequential >> storage, see MPI-3.1 page 115 line 31, or a MPI datatype extent, retrieved, >> e.g., with MPI_TYPE_GET_EXTENT. >> Argument names / description: >> -- position / current position in buffer, in bytes (integer) [MPI_PACK_EXTERNAL] >> -- array_of_displacements / byte displacement of each block (array of integer) >> [MPI_TYPE_CREATE_STRUCT] >> C-type in MPI-3.1: MPI_Aint >> C-type in _l: MPI_Aint >> (Wrong) C-types in MPI-3.1: int [MPI_PACK, MPI_ALLTOALLW] >> C-type (corrected) in _l: MPI_Aint [MPI_PACK, MPI_ALLTOALLW] >> >> - Absolute address values for byte displacements. >> These values are also valid for all byte displacements in datatype routines >> and in MPI_NEIGHBOR_ALLTOALLW, provided that they are used in combination >> with buffer=MPI_BOTTOM. >> They cannot be used in MPI_ALLTOALLW. >> With "C-type (corrected) in _l: MPI_Aint [MPI_ALLTOALLW]", >> they are also usable with MPI_ALLTOALLW. >> >> >> I already looked at the Large/Big Count pdf and saw that in the datatype chapter >> these rules were broken, for example for the ...PACK/UNPACK... routines. >> >> >> Therefore my clear question, do we agree on the rules above? >> >> >> Already detected bugs in Version from Sep. 13, 2019: >> >> - page 127: >> No idea why you changed the name from MPI_GET_ELEMENTS to MPI_TYPE_GET_ELEMENTS. >> Should be reverted. >> >> - MPI_PACK: >> outsize and position should be handled identical to those in MPI_PACK_EXTERNAL, >> i.e., >> both are MPI_Aint... >> >> - MPI_PACK_SIZE: >> size should be handled identical to that in MPI_PACK_EXTERNAL_SIZE, i.e., >> MPI_Aint... >> >> - MPI_UNPACK: >> insize and position should be handled identical to those in MPI_UNPACK_EXTERNAL, >> i.e., >> both are MPI_Aint... >> >> - MPI_Type_contiguous: the large count _l version is missing >> >> - MPI_Type_create_darray >> -- array_of_distribs must be INTEGER, because it holds enumeration values >> and nothing else. >> -- array_of_dargs requires significant explanation because it can >> hold an enumeration (INTEGER) and also large count values which >> can cause not understandable compiler error reports >> in Fortran, because using MPI_COUNT_KIND array_of_gsizes values >> together with an INTEGER enumeration constant, >> here MPI_DISTRIBUTE_DFLT_DARG, would cause a compiler message >> like "no matching interface found". >> >> Two possible text/interface solutions: >> - If using the mpi_f08 module and MPI_DISTRIBUTE_DFLT_DARG together >> with large Count Version of this procedure, i.e., >> INTEGER(KIND=MPI_COUNT_KIND) array_of_gsizes and array_of_dargs >> arguments, then one should use >> INT(MPI_DISTRIBUTE_DFLT_DARG, MPI_COUNT_KIND) >> instead of >> MPI_DISTRIBUTE_DFLT_DARG. >> - overloading with two versions (long,normal) and (long,long) >> but i would recommend the first solution because it does >> not require additional MPI library implementation overhead. >> >> >> Before I can continue with reviewing, the principles above must be >> cleared/discussed/agreed/... >> >> Best regards >> Rolf >> >> >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email >> rabenseifner(a)hlrs.de <mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>> . >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . >> Head of Dpmt Parallel Computing . . . >> www.hlrs.de/people/rabenseifner <http://www.hlrs.de/people/rabenseifner><http://www.hlrs.de/people/rabenseifner <http://www.hlrs.de/people/rabenseifner>> . >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . >> _______________________________________________ >> mpiwg-large-counts mailing list >> mpiwg-large-counts(a)lists.mpi-forum.org <mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>> >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts <https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts> >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de <mailto:[email protected]> . >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner <http://www.hlrs.de/people/rabenseifner> . >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de <mailto:[email protected]> . > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner <http://www.hlrs.de/people/rabenseifner> . > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de <mailto:[email protected]> . > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts

1 0

Re: [Mpiwg-large-counts] BigCount working group
by Rolf Rabenseifner 09 Oct '19

09 Oct '19

I just sent my detailed reply to Dan's reply (and including Dan's text) to this mailing list. Therefore, I hope that it goes also somehow to the archive. Jeff, I expect you know also very well the principles of using MPI_Aint since MPI-1, continuously used until now. Best regards Rolf ----- Jeff Squyres (jsquyres) via mpiwg-large-counts <mpiwg-large-counts(a)lists.mpi-forum.org> wrote: > Through Rolf's post (https://lists.mpi-forum.org/pipermail/mpiwg-large-counts/2019-September/000…) the current WG was made aware of this mailing list. :-) > > We're going to start using this mailing list for working group kinds of things. > > I posted the notes from our meeting today on the WG GitHub repo wiki: > > https://github.com/mpiwg-large-count/large-count-issues/wiki > > -- > Jeff Squyres > jsquyres(a)cisco.com > > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Fwd: Re: Large Count - the principles for counts, sizes, and byte and nonbyte displacements
by Rolf Rabenseifner 09 Oct '19

09 Oct '19

----- Forwarded Message ----- From: Rolf Rabenseifner <rabenseifner(a)hlrs.de> To: HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> Cc: Purushotham V. Bangalore <puri(a)uab.edu>, Martin Ruefenacht <m.a.ruefenacht(a)gmail.com>, Claudia Blaas-Schenner <claudia.blaas-schenner(a)tuwien.ac.at>, Jeff Squyres <jsquyres(a)cisco.com>, Anthony Skjellum <tony-skjellum(a)utc.edu> Sent: Tue, 08 Oct 2019 20:12:18 +0200 (CEST) Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements Hi all, (Dan and Puri, on Sep. 23, I sent also a copy of my email to Martin. For me it Looks like that he had no complains) All the rest of answers are inline below. Best regards Rolf ----- Original Message ----- > From: "HOLMES Daniel" <d.holmes(a)epcc.ed.ac.uk> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "Purushotham V. Bangalore" <puri(a)uab.edu>, "Martin Ruefenacht" <m.a.ruefenacht(a)gmail.com>, "Claudia Blaas-Schenner" > <claudia.blaas-schenner(a)tuwien.ac.at>, "Jeff Squyres" <jsquyres(a)cisco.com>, "Anthony Skjellum" <tony-skjellum(a)utc.edu> > Sent: Monday, October 7, 2019 8:25:03 PM > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements > Hi Rolf, > > Apologies - I have no record of this email arriving before today. This probably > means that I am not a member of te mpiwg-large-counts email list, which seems > somewhat sub-optimal. >> >> To understand how big and large counts should be implemented in MPI-4, >> it is important to understand the count, displacement and size model of in >> MPI-3.1. >> >> As long as we do not have a common understanding of MPI-3.1, >> we will have problems to define MPI-4. > > Agreed. > >> Therefore my clear question, do we agree on the rules above? > > In short, no. > In addition, these rules are incomplete. > Also, there are breaches in MPI-3.1 for most of these rules. > > — > > Below this point in this email, I attempt to cover some of the disagreements > with your rules and I include a missing rule. However, this is not the sum > total of the knowledge/experience gained in/by the WG. Some of the other WG > members may wish to disagree and I expect that some points will generate > further discussion. > >> - an index into such an array, i.e., the number of an element. >> Argument name / descriptions [Routine]: >> -- array_of_displacements / displacement ..., in multiples of oldtype extent >> (... integer) [MPI_TYPE_INDEXED] >> -- sdispls / integer array (of length group size). Entry j specifies the >> displacement ... [MPI_ALLTOALLV] >> C-type in MPI-3.1: int >> C-type in _l: MPI_Count > > The two examples you give are not the same. The array_of_displacements is a > number, correct type MPI_Count (in C), as you state, because it is a “multiple > of” <something>. It looks like, that for MPI_TYPE_INDEXED, you agreed. > However, each element in the sdispls array is a displacement > relative to a memory location, sendbuf. > The correct type is MPI_Aint (in C). > Note that it is incorrect in MPI-3.1 for this parameter to be int[] (in C) > because int is *not* a smaller version of MPI_Aint. A displacement must not be > described as a number of bytes because of segmented address spaces. The phrase > “displacement in bytes” is nonsense. MPI-3.1 p171:7 clearly writes: sendbuf+sdispls[i]*extent(sendtype) This means if sendbuf would be declared as type_X sendbuf[sendcount[i]]; then sendbuf[sdispls[i]] would be exactly the same as we have in MPI_TYPE_INDEXED: sdispls[i] is in index into an array of elements. To compare it again: - MPI_TYPE_INDEXED MPI-3.1 p89:9-10 and 15-16 IN array_of_displacements displacement for each block, in multiples of oldtype extent (array of integer) int MPI_Type_indexed(..., const int array_of_displacements[], - MPI_ALLTOALLV MPI-31., p170:23-24 and p171:7 int MPI_Alltoallv(..., const int sdispls[], sendbuf+sdispls[i]*extent(sendtype) Second reason: When defining MPI_NEIGHBOR_ALLTOALL|V|W, we clearly revisited MPI_ALLTOALL|V|W and decided to correct wrong types. For MPI(_NEIGHBOR)_ALLTOALL and MPI(_NEIGHBOR)_ALLTOALLV, we decided that all is correct, whereas vor MPI(_NEIGHBOR)_ALLTOALLW, we decided that the correct type for the displs is MPI_Aint (and not int). Therefore, I cannot see any difference, and the MPI Forum also did not see this difference when they standardized MPI_NEIGHBOR_ALLTOALLV. >> - number of bytes >> Argument name / descriptions [Routine]: >> -- size / size of window in bytes (non-negative integer) [MPI_WIN_CREATE] >> -- outsize / output buffer size, in bytes (integer) [MPI_PACK_EXTERNAL] >> C-type in MPI-3.1: MPI_Aint >> C-type in _l: MPI_Aint >> (Wrong) C-types in MPI-3.1: int [MPI_PACK, MPI_TYPE_SIZE] >> C-type (corrected) in _l: MPI_Aint [MPI_PACK, MPI_TYPE_SIZE] > > A number of <something> is a number, not a displacement. The correct “large” > type is MPI_Count (in C). There are places in MPI-3.1 where this is, correctly, > int or MPI_Count but there are other places in MPI-3.1 where it is, > incorrectly, MPI_Aint. I never said that that a number is a displacement. MPI_Aint is clearly used for - addresses relative to a buffer begin, - absolute addresses (returned by MPI_GET_ADDRESS) - number of bytes. MPI-3.1 says 2.5.6 Absolute Addresses and Relative Address Displacements Some MPI procedures use address arguments that represent an absolute address in the calling program, or relative displacement arguments that represent differences of two absolute addresses. The datatype of such arguments is MPI_Aint in C and INTEGER (KIND= MPI_ADDRESS_KIND) in Fortran. Relative or absolute addresses ==> MPI_Aint. It is not written MPI_Aint ==> rel. or abs. addresses. Examples for number of bytes: MPI_TYPE_CREATE_HVECTOR, result of MPI_AINT_DIFF, MPI_TYPE_GET_EXTENT, MPI_WIN_CREATE, MPI_WIN_ALLOCATE, MPI_WIN_ALLOCATE_SHARED, MPI_WIN_SHARED_QUERY, MPI_WIN_ATTACH. This is a feature, not a bug. Example for byte-displacements: MPI_TYPE_CREATE_HINDEXED, MPI_TYPE_CREATE_HINDEXED_BLOCK, MPI_TYPE_CREATE_STRUCT, result of MPI_AINT_DIFF. A real exception is MPI_PUT, ... with MPI_Aint target_disp, and target_addr = window_base + target_disp x disp_unit. >> - smaller number in bytes >> Some argument names: e.g., disp_unit >> Description: local unit size for displacements, in bytes (positive integer) >> C-type in MPI-3.1: int >> C-type in _l: still int? or MPI_Count? > > Why must disp_unit be smaller, and is it really a number of bytes? The premise > for disp_unit is that a window might consist of a sequence of locations in > memory that are used to store values of a particular type (typically > represented by an MPI datatype). The disp_unit gives an indication of the > extent of the datatype, a quantity by which all offset values for the window > will be scaled, in order that address arithmetic adding (offset*disp_unit) to > base_address produces a new address that is one of the sequence of locations in > memory used to store values, i.e. the location of the beginning of one of the > datatypes. This indicates to me that the correct type for disp_unit is MPI_Aint > (in C). Note that, if true, this means that using int (as is done in MPI-3.1) > is incorrect, even prior to the large count changes because int is *not* a > smaller version of MPI_Aint. Yes. disp_unit is an extent of a type, which is a number of bytes, which should have been MPI_Aint (as in MPI_TYPE_GET_EXTENT). Therefore C-type in _l: should be MPI_Aint. > A missing rule concerns "length arguments” (see section 2.5.2 in MPI-3.1 for a > definition). > The length of the array_of_requests for the MPI_Waitany function is given by a > parameter called count, which is of type int (in C). This is a different class > of type than any discussed so far. The WG has flip-flopped with regards to > whether this class of type should be enlarged or not. Our current position is > (I believe) to leave this class of types alone, i.e. they will remain int (in > C). I have to think about. Best regards Rolf > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Architect (HPC Research) > d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> > Phone: +44 (0) 131 651 3465 > Mobile: +44 (0) 7940 524 088 > Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT > — > The University of Edinburgh is a charitable body, registered in Scotland, with > registration number SC005336. > — > > On 7 Oct 2019, at 18:09, Rolf Rabenseifner > <rabenseifner(a)hlrs.de<mailto:[email protected]>> wrote: > > Dear all, > > I've never seen any answer to my email. > Maybe there isn't any answer, maybe it never reached me. > > Please give me an advice. > > Best regards > Rolf > > ----- Forwarded Message ----- > From: Rolf Rabenseifner via mpiwg-large-counts > <mpiwg-large-counts(a)lists.mpi-forum.org<mailto:[email protected]>> > To: > mpiwg-large-counts(a)lists.mpi-forum.org<mailto:[email protected]> > Cc: Rolf Rabenseifner <rabenseifner(a)hlrs.de<mailto:[email protected]>> > Sent: Mon, 23 Sep 2019 17:15:09 +0200 (CEST) > Subject: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, > and byte and nonbyte displacements > > Dear all, > > To understand how big and large counts should be implemented in MPI-4, > it is important to understand the count, displacement and size model of in > MPI-3.1. > > As long as we do not have a common understanding of MPI-3.1, > we will have problems to define MPI-4. > > Therefore here my understanding of MPI-3. We have > > - number of array elements with elements of a given type (typically represented > by an MPI datatype handle). > Usual argument name: count > Usual description: number of elements in ... buffer (non-negative integer) > C-type in MPI-3.1: int > C-type in _l: MPI_Count > > - an index into such an array, i.e., the number of an element. > Argument name / descriptions [Routine]: > -- array_of_displacements / displacement ..., in multiples of oldtype extent > (... integer) [MPI_TYPE_INDEXED] > -- sdispls / integer array (of length group size). Entry j specifies the > displacement ... [MPI_ALLTOALLV] > C-type in MPI-3.1: int > C-type in _l: MPI_Count > > - number of bytes > Argument name / descriptions [Routine]: > -- size / size of window in bytes (non-negative integer) [MPI_WIN_CREATE] > -- outsize / output buffer size, in bytes (integer) [MPI_PACK_EXTERNAL] > C-type in MPI-3.1: MPI_Aint > C-type in _l: MPI_Aint > (Wrong) C-types in MPI-3.1: int [MPI_PACK, MPI_TYPE_SIZE] > C-type (corrected) in _l: MPI_Aint [MPI_PACK, MPI_TYPE_SIZE] > > - smaller number in bytes > Some argument names: e.g., disp_unit > Description: local unit size for displacements, in bytes (positive integer) > C-type in MPI-3.1: int > C-type in _l: still int? or MPI_Count? > > - Position or relative byte displacement within an array of bytes. > Such values can be calculated as any sum and product of in, long, long long, > and MPI_Aint as long as MPI_Aint value contains a pure integer size value, > i.e., an (integer) difference of two absolute addresses within one sequential > storage, see MPI-3.1 page 115 line 31, or a MPI datatype extent, retrieved, > e.g., with MPI_TYPE_GET_EXTENT. > Argument names / description: > -- position / current position in buffer, in bytes (integer) [MPI_PACK_EXTERNAL] > -- array_of_displacements / byte displacement of each block (array of integer) > [MPI_TYPE_CREATE_STRUCT] > C-type in MPI-3.1: MPI_Aint > C-type in _l: MPI_Aint > (Wrong) C-types in MPI-3.1: int [MPI_PACK, MPI_ALLTOALLW] > C-type (corrected) in _l: MPI_Aint [MPI_PACK, MPI_ALLTOALLW] > > - Absolute address values for byte displacements. > These values are also valid for all byte displacements in datatype routines > and in MPI_NEIGHBOR_ALLTOALLW, provided that they are used in combination > with buffer=MPI_BOTTOM. > They cannot be used in MPI_ALLTOALLW. > With "C-type (corrected) in _l: MPI_Aint [MPI_ALLTOALLW]", > they are also usable with MPI_ALLTOALLW. > > > I already looked at the Large/Big Count pdf and saw that in the datatype chapter > these rules were broken, for example for the ...PACK/UNPACK... routines. > > > Therefore my clear question, do we agree on the rules above? > > > Already detected bugs in Version from Sep. 13, 2019: > > - page 127: > No idea why you changed the name from MPI_GET_ELEMENTS to MPI_TYPE_GET_ELEMENTS. > Should be reverted. > > - MPI_PACK: > outsize and position should be handled identical to those in MPI_PACK_EXTERNAL, > i.e., > both are MPI_Aint... > > - MPI_PACK_SIZE: > size should be handled identical to that in MPI_PACK_EXTERNAL_SIZE, i.e., > MPI_Aint... > > - MPI_UNPACK: > insize and position should be handled identical to those in MPI_UNPACK_EXTERNAL, > i.e., > both are MPI_Aint... > > - MPI_Type_contiguous: the large count _l version is missing > > - MPI_Type_create_darray > -- array_of_distribs must be INTEGER, because it holds enumeration values > and nothing else. > -- array_of_dargs requires significant explanation because it can > hold an enumeration (INTEGER) and also large count values which > can cause not understandable compiler error reports > in Fortran, because using MPI_COUNT_KIND array_of_gsizes values > together with an INTEGER enumeration constant, > here MPI_DISTRIBUTE_DFLT_DARG, would cause a compiler message > like "no matching interface found". > > Two possible text/interface solutions: > - If using the mpi_f08 module and MPI_DISTRIBUTE_DFLT_DARG together > with large Count Version of this procedure, i.e., > INTEGER(KIND=MPI_COUNT_KIND) array_of_gsizes and array_of_dargs > arguments, then one should use > INT(MPI_DISTRIBUTE_DFLT_DARG, MPI_COUNT_KIND) > instead of > MPI_DISTRIBUTE_DFLT_DARG. > - overloading with two versions (long,normal) and (long,long) > but i would recommend the first solution because it does > not require additional MPI library implementation overhead. > > > Before I can continue with reviewing, the principles above must be > cleared/discussed/agreed/... > > Best regards > Rolf > > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email > rabenseifner(a)hlrs.de<mailto:[email protected]> . > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > Head of Dpmt Parallel Computing . . . > www.hlrs.de/people/rabenseifner<http://www.hlrs.de/people/rabenseifner> . > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org<mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

BigCount working group
by Jeff Squyres (jsquyres) 09 Oct '19

09 Oct '19

Through Rolf's post (https://lists.mpi-forum.org/pipermail/mpiwg-large-counts/2019-September/000…) the current WG was made aware of this mailing list. :-) We're going to start using this mailing list for working group kinds of things. I posted the notes from our meeting today on the WG GitHub repo wiki: https://github.com/mpiwg-large-count/large-count-issues/wiki -- Jeff Squyres jsquyres(a)cisco.com

1 0