Hi Rolf, Thanks. I’ve copied your response into the the issue to keep it all in one place (so we don’t have to search email to find it). Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) [email protected]<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 5 Nov 2019, at 17:58, Rolf Rabenseifner <[email protected]<mailto:[email protected]>> wrote: Sorry, I now detected that you wanted to point me also to ----- Jeff Hammond <[email protected]<mailto:[email protected]><mailto:[email protected]>> wrote: Rolf: Have you looked at https://github.com/mpiwg-large-count/large-count-issues/issues/6 ? where I found your text. Thank you for remembering me. About your choices: 1.change the name of MPI_Type_get_extext_x to MPI_Type_get_extext_l to fit in with all the other _l functions for large count. This is a bad idea because the MPI_Count variant is not (just) embiggening the MPI_Aint variant, it is providing a different role (target files, not absolute memory). Agreed, bad idea. 2.add a new MPI_Type_get_extext_l that uses MPI_Offset. This is superfluous because we already have the _x variant with MPI_Count. Agreed, bad idea, because there is no "int Count" Argument. 3.leave it well alone. This seems like the best approach. Partially agreed, because for the _x Versions in the derived datatype chapter, I would always use MPI_Count and not MPI_Offset, because one never know for which other purpose derived datatypes may be used in the future.. The concept is not restricted to Memory and I/O. I/O is only an example why we need to provide these _x Versions. Therefore I prefere: 4. leave it. It is the starting point for the other _X versions. Small diff between these two _x routines and the new ones is that for each new one, it does not have a visible Fortran _X, only the overloaded one, and each existing one must get the additional overloaded interface. As far as I understand Fortran, there will be then only one backend routine with two different interfaces, because the backend is the same _X routine for the explicit and for the overloaded MPI_Count routine. Best regards Rolf ----- Original Message ----- From: "mpiwg-large-counts" <[email protected]<mailto:[email protected]>> To: "HOLMES Daniel" <[email protected]<mailto:[email protected]>> Cc: "Rolf Rabenseifner" <[email protected]<mailto:[email protected]>>, "mpiwg-large-counts" <[email protected]<mailto:[email protected]>> Sent: Tuesday, November 5, 2019 6:10:25 PM Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements Hi Dan, our mailers do not want to work together. I cannot find your "lengthy comment on that issue" in this email. As you can see below, it is flattened. Originally , it was with huge indents, but without any ">". Text looked then líke d d r e s s e s In the telcon, we completely aggreed that the large count versions MPI...._l should keep all MPI_Aint and should add MPI_Aint where it was missing by being wrong (MPI_Alltoall_w and MPI_(Un)pack. We completely agreed that all the MPI_Aint discussion is done. MPI_Aint is an signed integer mis-used to also store absoulte adresses whatever there bits may mean. And we do not change this nor we introduce any new such strange type. And we agreed that there should be for the few derived datatype routines with MPI_Aint an additional set with MPI_Count (or MPI_Offset, which would be wierd) instead of MPI_Aint. These MPI_Count versions do not allow the use of absolute addresses. These MPI_Count versions are needed for the case that sizeof(MPI_Aint) is less than sizeof(MPI_Count), may de 4:8 or 8.12 or 8:16, depending on the memory size per MPI process and the filesystem size. I only proposed in my latest email, that exactly two of them already exist: MPI_Type_get_extent_x and MPI_Type_get_true_extent_x and we should keep them and take the postfix _x for this set of routines instead of throwing them away and reinvernting them again. For ... I would say, - MPI_TYPE_SIZE_X the _l Version should have MPI_Aint size because it is a byte size. - MPI_GET_ELEMENTS_X and MPI_STATUS_SET_ELEMENTS_X in my opinion, we should keep _X, and the _l version is then a duplicate of it. Best regards Rolf ----- Original Message ----- From: "HOLMES Daniel" <[email protected]<mailto:[email protected]>> To: "Rolf Rabenseifner" <[email protected]<mailto:[email protected]>> Cc: "mpiwg-large-counts" <[email protected]<mailto:[email protected]>> Sent: Tuesday, November 5, 2019 3:18:13 PM Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements Hi Rolf (et al), I wrote a lengthy comment on that issue to capture my current understanding of your “really wrong” assertion. Broadly, we agree - I just wanted to write down the reasoning and nuances of that outcome. Cheers, Dan. — Dr Daniel Holmes PhD Architect (HPC Research) [email protected]<mailto:[email protected]><mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 2 Nov 2019, at 09:13, Rolf Rabenseifner via mpiwg-large-counts <[email protected]<mailto:[email protected]><mailto:[email protected]>> wrote: After the Telcon it seems that this ticket is really wrong. Some or all of the routines may/should be kept. And it these routines arean essential part of the future large count concept. Thank you very much for pointing us to this ticket. Rolf ----- Jeff Hammond <[email protected]<mailto:[email protected]><mailto:[email protected]>> wrote: Rolf: Have you looked at https://github.com/mpiwg-large-count/large-count-issues/issues/6? Jeff On Fri, Nov 1, 2019 at 1:00 AM Rolf Rabenseifner <[email protected]> wrote: A small comment on the result of our telcon: - Postfix _l for int -> MPI_Count - Postfix _x for additionally MPI_Aint -> MPI_Count I.e., the additional routines in the derived datatype chapter. Two of them already exist MPI_Type_get_(true)extent_x In Fortran we will have then for the same routine two aliases: - the overload one without _x - and the explicit one with _x For both ones, the internal function name is the same, with _x. Best regards Rolf ----- Jeff Hammond <[email protected]> wrote: On Thu, Oct 31, 2019 at 7:48 AM Rolf Rabenseifner <[email protected]> wrote: Dear all, here my summary as input for our telcon today. In principle, it is a very simple question: with large Counts, do we - keep all MPI_Aint - or do we substitute MPI_Aint by MPI_Count? I haven't been involved as much lately but did we not use MPI_Count for count and element displacements in the large count proposal? We need to use MPI_Aint for offsets into memory because that is what this type is for. Jeff In principle, the MPI Forum answered this question already for MPI-3.0 in 2012 with a clear YES: int MPI_Type_get_extent(MPI_Datatype datatype, MPI_Aint *lb, MPI_Aint *extent) int MPI_Type_get_extent_x(MPI_Datatype datatype, MPI_Count *lb, MPI_Count *extent) About Jeff H. question: If we limit the API to not support MPI_Count means that an MPI implementation has not really such quality options when using I/O fileviews, because the API is restricted to MPI_Aint (which should be implemented based on the, e.g., 64bit memory system). About Jim's comment: Apologies, it's been a while since I looked at the I/O interfaces. If I/O only needs relative displacements that have normal integer semantics, then I don't see why MPI_Count would not work for this purpose. If you have an MPI_Aint that contains a relative displacement, it also has normal integer semantics and can be converted to an MPI_Count. Yes, but this automatically implies that the datatypes must also be able to handle MPI_Count. The only case we really need to look out for is when an integer type contains an absolute address. In those cases, the quantity in the variable cannot be treated as a normal integer and we need special routines to work with it. Yes, this happens when we extend MPI_Aint in the derived datatype routines to MPI_Count. But in principle, this is not a big Problem, as you all could see in the previous emails: - We must do for MPI_Count the same as we did for MPI_Aint, i.e., we'll have long versions of the routines MPI_Get_address, MPI_Aint_diff, MPI_Aint_add - And we must ensure that the type cast from MPI_Aint to MPI_Count works, which is a small new advice to implementors for MPI_Det_address. Therefore again my 4 questions: - Should the new large count routines be prepared for more than 10 or 20 Exabyte files where we need 64/65 or or 65/66 unsigned/signed integers for relative byte displacements or byte counts? If yes, then all MPI_Aint arguments must be substituted by MPI_Count. (In other words, do we want to be prepared for another 25 years of MPI? :-) As stated above, the MPI-Forum already decided 2012 with a YES. - Should we allow that these new routines are also used for memory description, where we typically need only the large MPI_Count "count" arguments? (or should we provide two different new routines for each routine that currently has int Count/... and MPI_Aint disp/... arguments) I expect, that nobody wants to have two different large versions of for example MPI_Type_create_struct. - Should we allow a mix of old and new routines, especially for memory-based usage, that old-style MPI_Get_address is used to retrieve an absolute address and then, e.g., new style MPI_Type_create_struct with MPI_Count blocklength and displacements is used? I expect that forbidding such a mix would be a problem for Software development. Often old-style modules must work together with new-style modules. - Do we want to require for this type cast of MPI_Aint addr into MPI_Count that it is allowed to do this cast with a normal assignment, rather than a special MPI function? I expect yes, because for must usage of MPI_Aint and MPI_Count, it is for relative displacements or byte counts, i.e. for normal integers and therefore automatic type cast between MPI_Aint and MPI_Count is a must. With yes to all four questions, the proposed solution above is the easiest way. Hope to see/hear you today in our telcon. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email [email protected]<mailto:[email protected]> . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner<http://www.hlrs.de/people/rabenseifner> . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .