October 2019 - mpiwg-large-counts - mailman3.cels.anl.gov

Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements
by Jim Dinan 29 Oct '19

29 Oct '19

If you do pointer arithmetic, the compiler will ensure that the result is correct. If you convert a pointer into an integer and then do the arithmetic, the compiler can't help you and the result is not portable. This is why MPI_Aint_add describes what it does in terms of pointer arithmetic. The confusing and frustrating thing about MPI_Aint is that it's one type for two very different purposes. Allowing direct +/- on MPI_Aint values that represent addresses is not portable and is a mistake that we tried to correct with MPI_Aint_add/diff (I am happy to strengthen should to must if needed). It's perfectly fine to do arithmetic on MPI_Aint values that are displacements. When you assign an MPI_Aint to an MPI_Count, there are two cases depending on what the bits in the MPI_Aint represent: absolute address and relative displacements. The case where you assign an address to a count doesn't make sense to me. Why would one do this and why should MPI support it? The case where you assign a displacement to a count seems fine, you would want sign extension to happen. ~Jim. On Tue, Oct 29, 2019 at 4:52 PM Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dear Jim, > > > (a3) Section 4.1.5 of MPI 3.1 states "To ensure portability, arithmetic > on > > absolute addresses should not be performed with the intrinsic operators > \-" > > and \+". > > The major problem is, that we decided "should" and not "maust" or "shall", > because there is such many existing MPI-1 ... MPI-3.0 code that must have > used + or - operators. > > The only objective, that is true from the beginning, that MPI addresses > must be > retrieved with MPI_Get_address. > > And the second also Major Problem is the new assigment of an MPI_Aint > value > into an MPI_Count variable with MPI_Count larger than MPI_Aint. > > Therefore, I would prefere, that we keep this "should" and design in long > term > MPI_Get_address in a way that in principle MPI_Aint_diff and _add > need not to do anythin else as the + or - operator. > > And this depends on the meaning of the unsigned addresses, i.e., > what is the sequence of addresses (i.e., is it really going from > 0 to FFFF...FFFF) and than mapping these addreses to the mathematical > sequence > of MPI_Aint which starts at -2**(n-1) and ends at 2**(n-1)-1. > > Thats all. For the moment, as far as the web and some emails told us, > we are fare away from this contiguous 64-bit address space (0 to > FFFF...FFFF). > > But we should be correctly prepared. > > Or in other words: > > (a2) Should be solved by MPI_Aint_add/diff. > In my opinion no, it must be solved by MPI_Get_addr > and MPI_Aint_add/diff can stay normal + or - operators. > > I should also mention, that of course all MPI routines that > accept MPI_BOOTOM must reverse the work of MPI_Get_address > to get back the real "unsigned" virtual addresses of the OS. > > The same what we already had if an implementation has chosen > to use the address of an MPI common block as base for MPI_BOTTOM. > Here, the MPI lib had the freedom to revert the mapping > within MPI_Get_addr or within all functions called with MPI_BOTTOM. > > Best regards > Rolf > > > > ----- Original Message ----- > > From: "Jim Dinan" <james.dinan(a)gmail.com> > > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > > Cc: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > > Sent: Tuesday, October 29, 2019 3:58:18 PM > > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for > counts, sizes, and byte and nonbyte displacements > > > Hi Rolf, > > > > (a1) seems to me like another artifact of storing an unsigned quantity > in a > > signed variable, i.e., the quantity in an MPI_Aint can be an unsigned > > address or a signed displacement. Since we don't have an unsigned type > for > > addresses, the user can't portably fix this above MPI. We will need to > add > > functions to deal with combinations of MPI_Aint and MPI_Counts. This is > > essentially why we needed MPI_Aint_add/diff. Or ... the golden (Au is > > gold) int ... MPI_Auint. > > > > (a2) Should be solved by MPI_Aint_add/diff. > > > > (a3) Section 4.1.5 of MPI 3.1 states "To ensure portability, arithmetic > on > > absolute addresses should not be performed with the intrinsic operators > \-" > > and \+". MPI_Aint_add was written carefully to indicate that the "base" > > argument is treated as an unsigned address and the "disp" argument is > > treated as a signed displacement. > > > > ~Jim. > > > > On Tue, Oct 29, 2019 at 5:19 AM Rolf Rabenseifner <rabenseifner(a)hlrs.de> > > wrote: > > > >> Dear Jim and all, > >> > >> I'm not sure whether I'm really able to understand your email. > >> > >> I take the MPI view: > >> > >> (1) An absolute address can stored in an MPI_Aint variable > >> with and only with MPI_Get_address or MPI_Aint_add. > >> > >> (2) A positive or negative number of bytes or a relative address > >> which is by definition the amount of bytes between two locations > >> in a MPI "sequential storage" (MPI-3.1 page 115) > >> can be assigned with any method to an MPI_Aint variable > >> as long as the original value fits into MPI_Aint. > >> In both languages automatic type cast (i.e., sign expansion) > >> is done. > >> > >> (3) If users misuse MPI_Aint for storing anything else into MPI_Aint > >> variable then this is out of scope of MPI. > >> If such values are used in a minus operation then it is > >> out of the scope of MPI whether this makes sense. > >> If the user is sure that the new value falls into category (2) > >> then all is fine as long as the user is correct. > >> > >> I expect that your => is not a "greater or equal than". > >> I expect that you noticed assignments. > >> > >> > intptr_t => MPI_Aint > >> "intptr_t: integer type capable of holding a pointer." > >> > >> > uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" > proposal?) > >> "uintptr_t: unsigned integer type capable of holding a pointer." > >> > >> may fall exactly exactly into (3) when used for pointers. > >> > >> > >> Especially on a 64 bit system the user may have in the future exactly > >> the problems (a), (a1), (a2) and (b) as described below. > >> But here, the user is responsible, to for example implement (a3), > >> whereas for MPI_Get_address, the implementors of the MPI library > >> are responsible and the MPI Forum may be responsible for giving > >> the correct advices. > >> > >> By the way, the golden MPI_Auint was never golden. > >> Such need was "resolved" by introducing MPI_Aint_diff and MPI_Aint_add > >> in MPI-3.1. > >> > >> > >> > ptrdiff_t => MPI_Aint > >> "std::ptrdiff_t is the signed integer type of the result of subtracting > >> two pointers." > >> > >> may perfectly fit to (2). > >> > >> All of the following falls into category (2): > >> > >> > size_t (sizeof) => MPI_Count, int > >> "sizeof( type ) (1) > >> sizeof expression (2) > >> Both versions are constant expressions of type std::size_t." > >> > >> > size_t (offsetof) => MPI_Aint, int > >> "Defined in header <cstddef> > >> #define offsetof(type, member) /*implementation-defined*/ > >> The macro offsetof expands to an integral constant expression > >> of type std::size_t, the value of which is the offset, in bytes, > >> from the beginning of an object of specified type to ist > >> specified member, including padding if any." > >> > >> Note that this offsetof has nothing to do with MPI_Offset. > >> > >> On a system with less than 2*31 byte and 4-byte int, it is guaranteed > >> that size_t => int works. > >> > >> On a system with less than 2*63 byte and 8-byte MPI_Aint, it is > guaranteed > >> that size_t => MPI_Aint works. > >> > >> Problem: size_t is unsigned, int and MPI_Aint are signed. > >> > >> MPI_Count should be defined in a way that on systems with more than > >> 2**63 Bytes of disc space, that MPI_Count can hold such values, > >> because > >> int .LE. {MPI_Aint, MPI_Offset} .LE. MPI_Count > >> > >> Therefore size_t => MPI_Count should always work. > >> > >> > ssize_t => Mostly for error handling. Out of scope for MPI? > >> "In short, ssize_t is the same as size_t, but is a signed type - > >> read ssize_t as “signed size_t”. ssize_t is able to represent > >> the number -1, which is returned by several system calls > >> and library functions as a way to indicate error. > >> For example, the read and write system calls: ... > >> ssize_t read(int fildes, void *buf, size_t nbyte); ..." > >> > >> ssize_t fits therefore better to MPI_Aint, because both > >> are signed types that can hold byte counts, but > >> the value -1 in a MPI_Aint variable stands for a > >> byte displacement of -1 bytes and not for an error code -1. > >> > >> > >> All use of (2) is in principle no problem. > >> ------------------------------------------ > >> > >> All the complex discussiuon of the last days is about (1): > >> > >> (1) An absolute address can stored in an MPI_Aint variable > >> with and only with MPI_Get_address or MPI_Aint_add. > >> > >> In MPI-1 to MPI-3.0 and still in MPI-3.1 (here as may be not portable), > >> we also allow > >> MPI_Aint variable := absolute address in MPI_Aint variable > >> + or - > >> a number of bytes (in any integer type). > >> > >> The result is then still in category (1). > >> > >> > >> For the difference of two absolute addresses, > >> MPI_Aint_diff can be used. The result is than MPI_Aint of category (2) > >> > >> In MPI-1 to MPI-3.0 and still in MPI-3.1 (here as may be not portable), > >> we also allow > >> MPI_Aint variable := absolute address in MPI_Aint variable > >> - absolute address in MPI_Aint variable. > >> > >> The result is then in category (2). > >> > >> > >> The problems we discuss the last days are about systems > >> that internally use unsigned addresses and the MPI library stores > >> these addresses into MPI_Aint variables and > >> > >> (a) a sequential storage can have virtual addresses that > >> are both in the area with highest bit =0 and other addresses > >> in the same sequential storage (i.e., same array or structure) > >> with highest bit =1. > >> > >> or > >> (b) some higher bits contain segment addresses. > >> > >> (b) is not a problem as long as a sequential storage resides > >> always within one Segment. > >> > >> Therefore, we only have to discuss (a). > >> > >> The two problems that we have is > >> (a1) that for the minus operations an integer overflow will > >> happen and must be ignored. > >> (a2) if such addresses are expanded to larger variables, > >> e.g., MPI_Count with more bits in MPI_Count than in MPI_Aint, > >> sign expansion will result in completely wring results. > >> > >> And here, the most simple trick is, > >> (a3) that MPI_Get_address really shall > >> map the contiguous unsigned range from 0 to 2**64-1 to the > >> signed (and also contiguous) range from -2**63 to 2**63-1 > >> by simple subtracting 2**63. > >> With this simple trick in MPI_Get_address, Problems > >> 8a1) and (a2) are resolved. > >> > >> It looks like that (a) and therefore (a1) and (a2) > >> may be far in the future. > >> But they may be less far in the future, if a system may > >> map the whole applications cluster address space > >> into virtual memory (not cache coherent, but accessible). > >> > >> > >> And all this is never or only partial written into the > >> MPI Standard, also all is (well) known by the MPI Forum, > >> with the following exceptions: > >> - (a2) is new. > >> - (a1) is solved in MPI-3.1 only for MPI_Aint_diff and > >> MPI_Aint_add, but not for the operators - and + > >> if a user will switch on integer overflow detection > >> in the future when we will have such large systems. > >> - (a3) is new and in principle solves the problem also > >> for + and - operators. > >> > >> At lease (a1)+(a2) should be added as rationale to MPI-4.0 > >> and (a3) as advice to implementors within the framework > >> of big count, because (a2) is newly coming with big count. > >> > >> I hope this helps a bit if you took the time to read > >> this long email. > >> > >> Best regards > >> Rolf > >> > >> > >> > >> ----- Original Message ----- > >> > From: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > >> > To: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > >> > Cc: "Jim Dinan" <james.dinan(a)gmail.com>, "James Dinan" < > >> james.dinan(a)intel.com> > >> > Sent: Monday, October 28, 2019 5:07:37 PM > >> > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for > >> counts, sizes, and byte and nonbyte displacements > >> > >> > Still not sure I see the issue. MPI's memory-related integers should > map > >> to > >> > types that serve the same function in C. If the base language is > broken > >> for > >> > segmented addressing, we won't be able to fix it in a library. Looking > >> at the > >> > mapping below, I don't see where we would have broken it: > >> > > >> > intptr_t => MPI_Aint > >> > uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" > proposal?) > >> > ptrdiff_t => MPI_Aint > >> > size_t (sizeof) => MPI_Count, int > >> > size_t (offsetof) => MPI_Aint, int > >> > ssize_t => Mostly for error handling. Out of scope for MPI? > >> > > >> > It sounds like there are some places where we used MPI_Aint in place > of > >> size_t > >> > for sizes. Not great, but MPI_Aint already needs to be at least as > large > >> as > >> > size_t, so this seems benign. > >> > > >> > ~Jim. > >> > > >> > On Fri, Oct 25, 2019 at 8:25 PM Dinan, James via mpiwg-large-counts < > [ > >> > mailto:[email protected] | > >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > >> > > >> > > >> > > >> > > >> > > >> > Jeff, thanks so much for opening up these old wounds. I’m not sure I > >> have enough > >> > context to contribute to the discussion. Where can I read up on the > >> issue with > >> > MPI_Aint? > >> > > >> > > >> > > >> > I’m glad to hear that C signed integers will finally have a > well-defined > >> > representation. > >> > > >> > > >> > > >> > ~Jim. > >> > > >> > > >> > > >> > > >> > From: Jeff Hammond < [ mailto:[email protected] | > >> jeff.science(a)gmail.com ] > >> > > > >> > Date: Thursday, October 24, 2019 at 7:03 PM > >> > To: "Jeff Squyres (jsquyres)" < [ mailto:[email protected] | > >> jsquyres(a)cisco.com > >> > ] > > >> > Cc: MPI BigCount Working Group < [ mailto: > >> mpiwg-large-counts(a)lists.mpi-forum.org > >> > | mpiwg-large-counts(a)lists.mpi-forum.org ] >, "Dinan, James" < [ > >> > mailto:[email protected] | james.dinan(a)intel.com ] > > >> > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for > >> counts, > >> > sizes, and byte and nonbyte displacements > >> > > >> > > >> > > >> > > >> > > >> > Jim (cc) suffered the most in MPI 3.0 days because of AINT_DIFF and > >> AINT_SUM, so > >> > maybe he wants to create this ticket. > >> > > >> > > >> > > >> > > >> > > >> > Jeff > >> > > >> > > >> > > >> > > >> > > >> > On Thu, Oct 24, 2019 at 2:41 PM Jeff Squyres (jsquyres) < [ > >> > mailto:[email protected] | jsquyres(a)cisco.com ] > wrote: > >> > > >> > > >> > > >> > > >> > > >> > Not opposed to ditching segmented addressing at all. We'd need a > ticket > >> for this > >> > ASAP, though. > >> > > >> > > >> > > >> > > >> > > >> > This whole conversation is predicated on: > >> > > >> > > >> > > >> > > >> > > >> > - MPI supposedly supports segmented addressing > >> > > >> > > >> > - MPI_Aint is not sufficient for modern segmented addressing (i.e., > >> representing > >> > an address that may not be in main RAM and is not mapped in to the > >> current > >> > process' linear address space) > >> > > >> > > >> > > >> > > >> > > >> > If we no longer care about segmented addressing, that makes a whole > >> bunch of > >> > BigCount stuff a LOT easier. E.g., MPI_Aint can basically be a > >> > non-segment-supporting address integer. AINT_DIFF and AINT_SUM can go > >> away, > >> > too. > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Oct 24, 2019, at 5:35 PM, Jeff Hammond via mpiwg-large-counts < [ > >> > mailto:[email protected] | > >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > >> > > >> > > >> > > >> > > >> > > >> > Rolf: > >> > > >> > > >> > > >> > Before anybody spends any time analyzing how we handle segmented > >> addressing, I > >> > want you to provide an example of a platform where this is relevant. > What > >> > system can you boot today that needs this and what MPI libraries have > >> expressed > >> > an interest in supporting it? > >> > > >> > > >> > > >> > > >> > > >> > For anyone who didn't hear, ISO C and C++ have finally committed to > >> > twos-complement integers ( [ > >> > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html > | > >> > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html > ] > >> , [ > >> > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm | > >> > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm ] ) because > >> modern > >> > programmers should not be limited by hardware designs from the 1960s. > We > >> should > >> > similarly not waste our time on obsolete features like segmentation. > >> > > >> > > >> > > >> > > >> > > >> > Jeff > >> > > >> > > >> > > >> > > >> > > >> > On Thu, Oct 24, 2019 at 10:13 AM Rolf Rabenseifner via > >> mpiwg-large-counts < [ > >> > mailto:[email protected] | > >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > >> > > >> > > >> > > >> > > >> >> I think that changes the conversation entirely, right? > >> > > >> > Not the first part, the state-of-current-MPI. > >> > > >> > It may change something for the future, or a new interface may be > needed. > >> > > >> > Please, can you describe how MPI_Get_address can work with the > >> > different variables from different memory segments. > >> > > >> > Or whether a completely new function or a set of functions is needed. > >> > > >> > If we can still express variables from all memory segments as > >> > input to MPI_Get_address, there may be still a way to flatten > >> > the result of some internal address-iquiry into a flattened > >> > signed integer with the same behavior as MPI_Aint today. > >> > > >> > If this is impossible, then new way of thinking and solution > >> > may be needed. > >> > > >> > I really want to see examples for all current stuff as you > >> > mentioned in your last email. > >> > > >> > Best regards > >> > Rolf > >> > > >> > ----- Original Message ----- > >> >> From: "Jeff Squyres" < [ mailto:[email protected] | > jsquyres(a)cisco.com > >> ] > > >> >> To: "Rolf Rabenseifner" < [ mailto:[email protected] | > >> rabenseifner(a)hlrs.de ] > >> >> > > >> >> Cc: "mpiwg-large-counts" < [ mailto: > >> mpiwg-large-counts(a)lists.mpi-forum.org | > >> >> mpiwg-large-counts(a)lists.mpi-forum.org ] > > >> >> Sent: Thursday, October 24, 2019 5:27:31 PM > >> >> Subject: Re: [Mpiwg-large-counts] Large Count - the principles for > >> counts, > >> >> sizes, and byte and nonbyte displacements > >> > > >> >> On Oct 24, 2019, at 11:15 AM, Rolf Rabenseifner > >> >> < [ mailto:[email protected] | rabenseifner(a)hlrs.de ] <mailto: [ > >> >> mailto:[email protected] | rabenseifner(a)hlrs.de ] >> wrote: > >> >> > >> >> For me, it looked like that there was some misunderstanding > >> >> of the concept that absolute and relative addresses > >> >> and number of bytes that can be stored in MPI_Aint. > >> >> > >> >> ...with the caveat that MPI_Aint -- as it is right now -- does not > >> support > >> >> modern segmented memory systems (i.e., where you need more than a > small > >> number > >> >> of bits to indicate the segment where the memory lives). > >> >> > >> >> I think that changes the conversation entirely, right? > >> >> > >> >> -- > >> >> Jeff Squyres > >> >> [ mailto:[email protected] | jsquyres(a)cisco.com ] <mailto: [ > >> >> mailto:[email protected] | jsquyres(a)cisco.com ] > > >> > > >> > -- > >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email [ mailto: > >> rabenseifner(a)hlrs.de | > >> > rabenseifner(a)hlrs.de ] . > >> > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > . > >> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > . > >> > Head of Dpmt Parallel Computing . . . [ > >> http://www.hlrs.de/people/rabenseifner | > >> > www.hlrs.de/people/rabenseifner ] . > >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > . > >> > _______________________________________________ > >> > mpiwg-large-counts mailing list > >> > [ mailto:[email protected] | > >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > >> > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > -- > >> > > >> > > >> > Jeff Hammond > >> > [ mailto:[email protected] | jeff.science(a)gmail.com ] > >> > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] > >> > > >> > > >> > _______________________________________________ > >> > mpiwg-large-counts mailing list > >> > [ mailto:[email protected] | > >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > >> > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > >> > > >> > > >> > > >> > > >> > > >> > > >> > -- > >> > Jeff Squyres > >> > [ mailto:[email protected] | jsquyres(a)cisco.com ] > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > -- > >> > > >> > > >> > Jeff Hammond > >> > [ mailto:[email protected] | jeff.science(a)gmail.com ] > >> > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] > >> > _______________________________________________ > >> > mpiwg-large-counts mailing list > >> > [ mailto:[email protected] | > >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > >> > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > >> > > >> > _______________________________________________ > >> > mpiwg-large-counts mailing list > >> > mpiwg-large-counts(a)lists.mpi-forum.org > >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts > >> > >> -- > >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . > >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . > >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . >

1 0

Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements
by Rolf Rabenseifner 29 Oct '19

29 Oct '19

Dear Jim, > (a3) Section 4.1.5 of MPI 3.1 states "To ensure portability, arithmetic on > absolute addresses should not be performed with the intrinsic operators \-" > and \+". The major problem is, that we decided "should" and not "maust" or "shall", because there is such many existing MPI-1 ... MPI-3.0 code that must have used + or - operators. The only objective, that is true from the beginning, that MPI addresses must be retrieved with MPI_Get_address. And the second also Major Problem is the new assigment of an MPI_Aint value into an MPI_Count variable with MPI_Count larger than MPI_Aint. Therefore, I would prefere, that we keep this "should" and design in long term MPI_Get_address in a way that in principle MPI_Aint_diff and _add need not to do anythin else as the + or - operator. And this depends on the meaning of the unsigned addresses, i.e., what is the sequence of addresses (i.e., is it really going from 0 to FFFF...FFFF) and than mapping these addreses to the mathematical sequence of MPI_Aint which starts at -2**(n-1) and ends at 2**(n-1)-1. Thats all. For the moment, as far as the web and some emails told us, we are fare away from this contiguous 64-bit address space (0 to FFFF...FFFF). But we should be correctly prepared. Or in other words: > (a2) Should be solved by MPI_Aint_add/diff. In my opinion no, it must be solved by MPI_Get_addr and MPI_Aint_add/diff can stay normal + or - operators. I should also mention, that of course all MPI routines that accept MPI_BOOTOM must reverse the work of MPI_Get_address to get back the real "unsigned" virtual addresses of the OS. The same what we already had if an implementation has chosen to use the address of an MPI common block as base for MPI_BOTTOM. Here, the MPI lib had the freedom to revert the mapping within MPI_Get_addr or within all functions called with MPI_BOTTOM. Best regards Rolf ----- Original Message ----- > From: "Jim Dinan" <james.dinan(a)gmail.com> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > Sent: Tuesday, October 29, 2019 3:58:18 PM > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements > Hi Rolf, > > (a1) seems to me like another artifact of storing an unsigned quantity in a > signed variable, i.e., the quantity in an MPI_Aint can be an unsigned > address or a signed displacement. Since we don't have an unsigned type for > addresses, the user can't portably fix this above MPI. We will need to add > functions to deal with combinations of MPI_Aint and MPI_Counts. This is > essentially why we needed MPI_Aint_add/diff. Or ... the golden (Au is > gold) int ... MPI_Auint. > > (a2) Should be solved by MPI_Aint_add/diff. > > (a3) Section 4.1.5 of MPI 3.1 states "To ensure portability, arithmetic on > absolute addresses should not be performed with the intrinsic operators \-" > and \+". MPI_Aint_add was written carefully to indicate that the "base" > argument is treated as an unsigned address and the "disp" argument is > treated as a signed displacement. > > ~Jim. > > On Tue, Oct 29, 2019 at 5:19 AM Rolf Rabenseifner <rabenseifner(a)hlrs.de> > wrote: > >> Dear Jim and all, >> >> I'm not sure whether I'm really able to understand your email. >> >> I take the MPI view: >> >> (1) An absolute address can stored in an MPI_Aint variable >> with and only with MPI_Get_address or MPI_Aint_add. >> >> (2) A positive or negative number of bytes or a relative address >> which is by definition the amount of bytes between two locations >> in a MPI "sequential storage" (MPI-3.1 page 115) >> can be assigned with any method to an MPI_Aint variable >> as long as the original value fits into MPI_Aint. >> In both languages automatic type cast (i.e., sign expansion) >> is done. >> >> (3) If users misuse MPI_Aint for storing anything else into MPI_Aint >> variable then this is out of scope of MPI. >> If such values are used in a minus operation then it is >> out of the scope of MPI whether this makes sense. >> If the user is sure that the new value falls into category (2) >> then all is fine as long as the user is correct. >> >> I expect that your => is not a "greater or equal than". >> I expect that you noticed assignments. >> >> > intptr_t => MPI_Aint >> "intptr_t: integer type capable of holding a pointer." >> >> > uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" proposal?) >> "uintptr_t: unsigned integer type capable of holding a pointer." >> >> may fall exactly exactly into (3) when used for pointers. >> >> >> Especially on a 64 bit system the user may have in the future exactly >> the problems (a), (a1), (a2) and (b) as described below. >> But here, the user is responsible, to for example implement (a3), >> whereas for MPI_Get_address, the implementors of the MPI library >> are responsible and the MPI Forum may be responsible for giving >> the correct advices. >> >> By the way, the golden MPI_Auint was never golden. >> Such need was "resolved" by introducing MPI_Aint_diff and MPI_Aint_add >> in MPI-3.1. >> >> >> > ptrdiff_t => MPI_Aint >> "std::ptrdiff_t is the signed integer type of the result of subtracting >> two pointers." >> >> may perfectly fit to (2). >> >> All of the following falls into category (2): >> >> > size_t (sizeof) => MPI_Count, int >> "sizeof( type ) (1) >> sizeof expression (2) >> Both versions are constant expressions of type std::size_t." >> >> > size_t (offsetof) => MPI_Aint, int >> "Defined in header <cstddef> >> #define offsetof(type, member) /*implementation-defined*/ >> The macro offsetof expands to an integral constant expression >> of type std::size_t, the value of which is the offset, in bytes, >> from the beginning of an object of specified type to ist >> specified member, including padding if any." >> >> Note that this offsetof has nothing to do with MPI_Offset. >> >> On a system with less than 2*31 byte and 4-byte int, it is guaranteed >> that size_t => int works. >> >> On a system with less than 2*63 byte and 8-byte MPI_Aint, it is guaranteed >> that size_t => MPI_Aint works. >> >> Problem: size_t is unsigned, int and MPI_Aint are signed. >> >> MPI_Count should be defined in a way that on systems with more than >> 2**63 Bytes of disc space, that MPI_Count can hold such values, >> because >> int .LE. {MPI_Aint, MPI_Offset} .LE. MPI_Count >> >> Therefore size_t => MPI_Count should always work. >> >> > ssize_t => Mostly for error handling. Out of scope for MPI? >> "In short, ssize_t is the same as size_t, but is a signed type - >> read ssize_t as “signed size_t”. ssize_t is able to represent >> the number -1, which is returned by several system calls >> and library functions as a way to indicate error. >> For example, the read and write system calls: ... >> ssize_t read(int fildes, void *buf, size_t nbyte); ..." >> >> ssize_t fits therefore better to MPI_Aint, because both >> are signed types that can hold byte counts, but >> the value -1 in a MPI_Aint variable stands for a >> byte displacement of -1 bytes and not for an error code -1. >> >> >> All use of (2) is in principle no problem. >> ------------------------------------------ >> >> All the complex discussiuon of the last days is about (1): >> >> (1) An absolute address can stored in an MPI_Aint variable >> with and only with MPI_Get_address or MPI_Aint_add. >> >> In MPI-1 to MPI-3.0 and still in MPI-3.1 (here as may be not portable), >> we also allow >> MPI_Aint variable := absolute address in MPI_Aint variable >> + or - >> a number of bytes (in any integer type). >> >> The result is then still in category (1). >> >> >> For the difference of two absolute addresses, >> MPI_Aint_diff can be used. The result is than MPI_Aint of category (2) >> >> In MPI-1 to MPI-3.0 and still in MPI-3.1 (here as may be not portable), >> we also allow >> MPI_Aint variable := absolute address in MPI_Aint variable >> - absolute address in MPI_Aint variable. >> >> The result is then in category (2). >> >> >> The problems we discuss the last days are about systems >> that internally use unsigned addresses and the MPI library stores >> these addresses into MPI_Aint variables and >> >> (a) a sequential storage can have virtual addresses that >> are both in the area with highest bit =0 and other addresses >> in the same sequential storage (i.e., same array or structure) >> with highest bit =1. >> >> or >> (b) some higher bits contain segment addresses. >> >> (b) is not a problem as long as a sequential storage resides >> always within one Segment. >> >> Therefore, we only have to discuss (a). >> >> The two problems that we have is >> (a1) that for the minus operations an integer overflow will >> happen and must be ignored. >> (a2) if such addresses are expanded to larger variables, >> e.g., MPI_Count with more bits in MPI_Count than in MPI_Aint, >> sign expansion will result in completely wring results. >> >> And here, the most simple trick is, >> (a3) that MPI_Get_address really shall >> map the contiguous unsigned range from 0 to 2**64-1 to the >> signed (and also contiguous) range from -2**63 to 2**63-1 >> by simple subtracting 2**63. >> With this simple trick in MPI_Get_address, Problems >> 8a1) and (a2) are resolved. >> >> It looks like that (a) and therefore (a1) and (a2) >> may be far in the future. >> But they may be less far in the future, if a system may >> map the whole applications cluster address space >> into virtual memory (not cache coherent, but accessible). >> >> >> And all this is never or only partial written into the >> MPI Standard, also all is (well) known by the MPI Forum, >> with the following exceptions: >> - (a2) is new. >> - (a1) is solved in MPI-3.1 only for MPI_Aint_diff and >> MPI_Aint_add, but not for the operators - and + >> if a user will switch on integer overflow detection >> in the future when we will have such large systems. >> - (a3) is new and in principle solves the problem also >> for + and - operators. >> >> At lease (a1)+(a2) should be added as rationale to MPI-4.0 >> and (a3) as advice to implementors within the framework >> of big count, because (a2) is newly coming with big count. >> >> I hope this helps a bit if you took the time to read >> this long email. >> >> Best regards >> Rolf >> >> >> >> ----- Original Message ----- >> > From: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> >> > To: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> >> > Cc: "Jim Dinan" <james.dinan(a)gmail.com>, "James Dinan" < >> james.dinan(a)intel.com> >> > Sent: Monday, October 28, 2019 5:07:37 PM >> > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for >> counts, sizes, and byte and nonbyte displacements >> >> > Still not sure I see the issue. MPI's memory-related integers should map >> to >> > types that serve the same function in C. If the base language is broken >> for >> > segmented addressing, we won't be able to fix it in a library. Looking >> at the >> > mapping below, I don't see where we would have broken it: >> > >> > intptr_t => MPI_Aint >> > uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" proposal?) >> > ptrdiff_t => MPI_Aint >> > size_t (sizeof) => MPI_Count, int >> > size_t (offsetof) => MPI_Aint, int >> > ssize_t => Mostly for error handling. Out of scope for MPI? >> > >> > It sounds like there are some places where we used MPI_Aint in place of >> size_t >> > for sizes. Not great, but MPI_Aint already needs to be at least as large >> as >> > size_t, so this seems benign. >> > >> > ~Jim. >> > >> > On Fri, Oct 25, 2019 at 8:25 PM Dinan, James via mpiwg-large-counts < [ >> > mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: >> > >> > >> > >> > >> > >> > Jeff, thanks so much for opening up these old wounds. I’m not sure I >> have enough >> > context to contribute to the discussion. Where can I read up on the >> issue with >> > MPI_Aint? >> > >> > >> > >> > I’m glad to hear that C signed integers will finally have a well-defined >> > representation. >> > >> > >> > >> > ~Jim. >> > >> > >> > >> > >> > From: Jeff Hammond < [ mailto:[email protected] | >> jeff.science(a)gmail.com ] >> > > >> > Date: Thursday, October 24, 2019 at 7:03 PM >> > To: "Jeff Squyres (jsquyres)" < [ mailto:[email protected] | >> jsquyres(a)cisco.com >> > ] > >> > Cc: MPI BigCount Working Group < [ mailto: >> mpiwg-large-counts(a)lists.mpi-forum.org >> > | mpiwg-large-counts(a)lists.mpi-forum.org ] >, "Dinan, James" < [ >> > mailto:[email protected] | james.dinan(a)intel.com ] > >> > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for >> counts, >> > sizes, and byte and nonbyte displacements >> > >> > >> > >> > >> > >> > Jim (cc) suffered the most in MPI 3.0 days because of AINT_DIFF and >> AINT_SUM, so >> > maybe he wants to create this ticket. >> > >> > >> > >> > >> > >> > Jeff >> > >> > >> > >> > >> > >> > On Thu, Oct 24, 2019 at 2:41 PM Jeff Squyres (jsquyres) < [ >> > mailto:[email protected] | jsquyres(a)cisco.com ] > wrote: >> > >> > >> > >> > >> > >> > Not opposed to ditching segmented addressing at all. We'd need a ticket >> for this >> > ASAP, though. >> > >> > >> > >> > >> > >> > This whole conversation is predicated on: >> > >> > >> > >> > >> > >> > - MPI supposedly supports segmented addressing >> > >> > >> > - MPI_Aint is not sufficient for modern segmented addressing (i.e., >> representing >> > an address that may not be in main RAM and is not mapped in to the >> current >> > process' linear address space) >> > >> > >> > >> > >> > >> > If we no longer care about segmented addressing, that makes a whole >> bunch of >> > BigCount stuff a LOT easier. E.g., MPI_Aint can basically be a >> > non-segment-supporting address integer. AINT_DIFF and AINT_SUM can go >> away, >> > too. >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Oct 24, 2019, at 5:35 PM, Jeff Hammond via mpiwg-large-counts < [ >> > mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: >> > >> > >> > >> > >> > >> > Rolf: >> > >> > >> > >> > Before anybody spends any time analyzing how we handle segmented >> addressing, I >> > want you to provide an example of a platform where this is relevant. What >> > system can you boot today that needs this and what MPI libraries have >> expressed >> > an interest in supporting it? >> > >> > >> > >> > >> > >> > For anyone who didn't hear, ISO C and C++ have finally committed to >> > twos-complement integers ( [ >> > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html | >> > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html ] >> , [ >> > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm | >> > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm ] ) because >> modern >> > programmers should not be limited by hardware designs from the 1960s. We >> should >> > similarly not waste our time on obsolete features like segmentation. >> > >> > >> > >> > >> > >> > Jeff >> > >> > >> > >> > >> > >> > On Thu, Oct 24, 2019 at 10:13 AM Rolf Rabenseifner via >> mpiwg-large-counts < [ >> > mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: >> > >> > >> > >> > >> >> I think that changes the conversation entirely, right? >> > >> > Not the first part, the state-of-current-MPI. >> > >> > It may change something for the future, or a new interface may be needed. >> > >> > Please, can you describe how MPI_Get_address can work with the >> > different variables from different memory segments. >> > >> > Or whether a completely new function or a set of functions is needed. >> > >> > If we can still express variables from all memory segments as >> > input to MPI_Get_address, there may be still a way to flatten >> > the result of some internal address-iquiry into a flattened >> > signed integer with the same behavior as MPI_Aint today. >> > >> > If this is impossible, then new way of thinking and solution >> > may be needed. >> > >> > I really want to see examples for all current stuff as you >> > mentioned in your last email. >> > >> > Best regards >> > Rolf >> > >> > ----- Original Message ----- >> >> From: "Jeff Squyres" < [ mailto:[email protected] | jsquyres(a)cisco.com >> ] > >> >> To: "Rolf Rabenseifner" < [ mailto:[email protected] | >> rabenseifner(a)hlrs.de ] >> >> > >> >> Cc: "mpiwg-large-counts" < [ mailto: >> mpiwg-large-counts(a)lists.mpi-forum.org | >> >> mpiwg-large-counts(a)lists.mpi-forum.org ] > >> >> Sent: Thursday, October 24, 2019 5:27:31 PM >> >> Subject: Re: [Mpiwg-large-counts] Large Count - the principles for >> counts, >> >> sizes, and byte and nonbyte displacements >> > >> >> On Oct 24, 2019, at 11:15 AM, Rolf Rabenseifner >> >> < [ mailto:[email protected] | rabenseifner(a)hlrs.de ] <mailto: [ >> >> mailto:[email protected] | rabenseifner(a)hlrs.de ] >> wrote: >> >> >> >> For me, it looked like that there was some misunderstanding >> >> of the concept that absolute and relative addresses >> >> and number of bytes that can be stored in MPI_Aint. >> >> >> >> ...with the caveat that MPI_Aint -- as it is right now -- does not >> support >> >> modern segmented memory systems (i.e., where you need more than a small >> number >> >> of bits to indicate the segment where the memory lives). >> >> >> >> I think that changes the conversation entirely, right? >> >> >> >> -- >> >> Jeff Squyres >> >> [ mailto:[email protected] | jsquyres(a)cisco.com ] <mailto: [ >> >> mailto:[email protected] | jsquyres(a)cisco.com ] > >> > >> > -- >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email [ mailto: >> rabenseifner(a)hlrs.de | >> > rabenseifner(a)hlrs.de ] . >> > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . >> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . >> > Head of Dpmt Parallel Computing . . . [ >> http://www.hlrs.de/people/rabenseifner | >> > www.hlrs.de/people/rabenseifner ] . >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . >> > _______________________________________________ >> > mpiwg-large-counts mailing list >> > [ mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] >> > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] >> > >> > >> > >> > >> > >> > >> > >> > >> > -- >> > >> > >> > Jeff Hammond >> > [ mailto:[email protected] | jeff.science(a)gmail.com ] >> > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] >> > >> > >> > _______________________________________________ >> > mpiwg-large-counts mailing list >> > [ mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] >> > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] >> > >> > >> > >> > >> > >> > >> > -- >> > Jeff Squyres >> > [ mailto:[email protected] | jsquyres(a)cisco.com ] >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > -- >> > >> > >> > Jeff Hammond >> > [ mailto:[email protected] | jeff.science(a)gmail.com ] >> > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] >> > _______________________________________________ >> > mpiwg-large-counts mailing list >> > [ mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] >> > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] >> > >> > _______________________________________________ >> > mpiwg-large-counts mailing list >> > mpiwg-large-counts(a)lists.mpi-forum.org >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements
by Jim Dinan 29 Oct '19

29 Oct '19

Hi Rolf, (a1) seems to me like another artifact of storing an unsigned quantity in a signed variable, i.e., the quantity in an MPI_Aint can be an unsigned address or a signed displacement. Since we don't have an unsigned type for addresses, the user can't portably fix this above MPI. We will need to add functions to deal with combinations of MPI_Aint and MPI_Counts. This is essentially why we needed MPI_Aint_add/diff. Or ... the golden (Au is gold) int ... MPI_Auint. (a2) Should be solved by MPI_Aint_add/diff. (a3) Section 4.1.5 of MPI 3.1 states "To ensure portability, arithmetic on absolute addresses should not be performed with the intrinsic operators \-" and \+". MPI_Aint_add was written carefully to indicate that the "base" argument is treated as an unsigned address and the "disp" argument is treated as a signed displacement. ~Jim. On Tue, Oct 29, 2019 at 5:19 AM Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dear Jim and all, > > I'm not sure whether I'm really able to understand your email. > > I take the MPI view: > > (1) An absolute address can stored in an MPI_Aint variable > with and only with MPI_Get_address or MPI_Aint_add. > > (2) A positive or negative number of bytes or a relative address > which is by definition the amount of bytes between two locations > in a MPI "sequential storage" (MPI-3.1 page 115) > can be assigned with any method to an MPI_Aint variable > as long as the original value fits into MPI_Aint. > In both languages automatic type cast (i.e., sign expansion) > is done. > > (3) If users misuse MPI_Aint for storing anything else into MPI_Aint > variable then this is out of scope of MPI. > If such values are used in a minus operation then it is > out of the scope of MPI whether this makes sense. > If the user is sure that the new value falls into category (2) > then all is fine as long as the user is correct. > > I expect that your => is not a "greater or equal than". > I expect that you noticed assignments. > > > intptr_t => MPI_Aint > "intptr_t: integer type capable of holding a pointer." > > > uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" proposal?) > "uintptr_t: unsigned integer type capable of holding a pointer." > > may fall exactly exactly into (3) when used for pointers. > > > Especially on a 64 bit system the user may have in the future exactly > the problems (a), (a1), (a2) and (b) as described below. > But here, the user is responsible, to for example implement (a3), > whereas for MPI_Get_address, the implementors of the MPI library > are responsible and the MPI Forum may be responsible for giving > the correct advices. > > By the way, the golden MPI_Auint was never golden. > Such need was "resolved" by introducing MPI_Aint_diff and MPI_Aint_add > in MPI-3.1. > > > > ptrdiff_t => MPI_Aint > "std::ptrdiff_t is the signed integer type of the result of subtracting > two pointers." > > may perfectly fit to (2). > > All of the following falls into category (2): > > > size_t (sizeof) => MPI_Count, int > "sizeof( type ) (1) > sizeof expression (2) > Both versions are constant expressions of type std::size_t." > > > size_t (offsetof) => MPI_Aint, int > "Defined in header <cstddef> > #define offsetof(type, member) /*implementation-defined*/ > The macro offsetof expands to an integral constant expression > of type std::size_t, the value of which is the offset, in bytes, > from the beginning of an object of specified type to ist > specified member, including padding if any." > > Note that this offsetof has nothing to do with MPI_Offset. > > On a system with less than 2*31 byte and 4-byte int, it is guaranteed > that size_t => int works. > > On a system with less than 2*63 byte and 8-byte MPI_Aint, it is guaranteed > that size_t => MPI_Aint works. > > Problem: size_t is unsigned, int and MPI_Aint are signed. > > MPI_Count should be defined in a way that on systems with more than > 2**63 Bytes of disc space, that MPI_Count can hold such values, > because > int .LE. {MPI_Aint, MPI_Offset} .LE. MPI_Count > > Therefore size_t => MPI_Count should always work. > > > ssize_t => Mostly for error handling. Out of scope for MPI? > "In short, ssize_t is the same as size_t, but is a signed type - > read ssize_t as “signed size_t”. ssize_t is able to represent > the number -1, which is returned by several system calls > and library functions as a way to indicate error. > For example, the read and write system calls: ... > ssize_t read(int fildes, void *buf, size_t nbyte); ..." > > ssize_t fits therefore better to MPI_Aint, because both > are signed types that can hold byte counts, but > the value -1 in a MPI_Aint variable stands for a > byte displacement of -1 bytes and not for an error code -1. > > > All use of (2) is in principle no problem. > ------------------------------------------ > > All the complex discussiuon of the last days is about (1): > > (1) An absolute address can stored in an MPI_Aint variable > with and only with MPI_Get_address or MPI_Aint_add. > > In MPI-1 to MPI-3.0 and still in MPI-3.1 (here as may be not portable), > we also allow > MPI_Aint variable := absolute address in MPI_Aint variable > + or - > a number of bytes (in any integer type). > > The result is then still in category (1). > > > For the difference of two absolute addresses, > MPI_Aint_diff can be used. The result is than MPI_Aint of category (2) > > In MPI-1 to MPI-3.0 and still in MPI-3.1 (here as may be not portable), > we also allow > MPI_Aint variable := absolute address in MPI_Aint variable > - absolute address in MPI_Aint variable. > > The result is then in category (2). > > > The problems we discuss the last days are about systems > that internally use unsigned addresses and the MPI library stores > these addresses into MPI_Aint variables and > > (a) a sequential storage can have virtual addresses that > are both in the area with highest bit =0 and other addresses > in the same sequential storage (i.e., same array or structure) > with highest bit =1. > > or > (b) some higher bits contain segment addresses. > > (b) is not a problem as long as a sequential storage resides > always within one Segment. > > Therefore, we only have to discuss (a). > > The two problems that we have is > (a1) that for the minus operations an integer overflow will > happen and must be ignored. > (a2) if such addresses are expanded to larger variables, > e.g., MPI_Count with more bits in MPI_Count than in MPI_Aint, > sign expansion will result in completely wring results. > > And here, the most simple trick is, > (a3) that MPI_Get_address really shall > map the contiguous unsigned range from 0 to 2**64-1 to the > signed (and also contiguous) range from -2**63 to 2**63-1 > by simple subtracting 2**63. > With this simple trick in MPI_Get_address, Problems > 8a1) and (a2) are resolved. > > It looks like that (a) and therefore (a1) and (a2) > may be far in the future. > But they may be less far in the future, if a system may > map the whole applications cluster address space > into virtual memory (not cache coherent, but accessible). > > > And all this is never or only partial written into the > MPI Standard, also all is (well) known by the MPI Forum, > with the following exceptions: > - (a2) is new. > - (a1) is solved in MPI-3.1 only for MPI_Aint_diff and > MPI_Aint_add, but not for the operators - and + > if a user will switch on integer overflow detection > in the future when we will have such large systems. > - (a3) is new and in principle solves the problem also > for + and - operators. > > At lease (a1)+(a2) should be added as rationale to MPI-4.0 > and (a3) as advice to implementors within the framework > of big count, because (a2) is newly coming with big count. > > I hope this helps a bit if you took the time to read > this long email. > > Best regards > Rolf > > > > ----- Original Message ----- > > From: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > > To: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > > Cc: "Jim Dinan" <james.dinan(a)gmail.com>, "James Dinan" < > james.dinan(a)intel.com> > > Sent: Monday, October 28, 2019 5:07:37 PM > > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for > counts, sizes, and byte and nonbyte displacements > > > Still not sure I see the issue. MPI's memory-related integers should map > to > > types that serve the same function in C. If the base language is broken > for > > segmented addressing, we won't be able to fix it in a library. Looking > at the > > mapping below, I don't see where we would have broken it: > > > > intptr_t => MPI_Aint > > uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" proposal?) > > ptrdiff_t => MPI_Aint > > size_t (sizeof) => MPI_Count, int > > size_t (offsetof) => MPI_Aint, int > > ssize_t => Mostly for error handling. Out of scope for MPI? > > > > It sounds like there are some places where we used MPI_Aint in place of > size_t > > for sizes. Not great, but MPI_Aint already needs to be at least as large > as > > size_t, so this seems benign. > > > > ~Jim. > > > > On Fri, Oct 25, 2019 at 8:25 PM Dinan, James via mpiwg-large-counts < [ > > mailto:[email protected] | > > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > > > > > > > > > > > > Jeff, thanks so much for opening up these old wounds. I’m not sure I > have enough > > context to contribute to the discussion. Where can I read up on the > issue with > > MPI_Aint? > > > > > > > > I’m glad to hear that C signed integers will finally have a well-defined > > representation. > > > > > > > > ~Jim. > > > > > > > > > > From: Jeff Hammond < [ mailto:[email protected] | > jeff.science(a)gmail.com ] > > > > > Date: Thursday, October 24, 2019 at 7:03 PM > > To: "Jeff Squyres (jsquyres)" < [ mailto:[email protected] | > jsquyres(a)cisco.com > > ] > > > Cc: MPI BigCount Working Group < [ mailto: > mpiwg-large-counts(a)lists.mpi-forum.org > > | mpiwg-large-counts(a)lists.mpi-forum.org ] >, "Dinan, James" < [ > > mailto:[email protected] | james.dinan(a)intel.com ] > > > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for > counts, > > sizes, and byte and nonbyte displacements > > > > > > > > > > > > Jim (cc) suffered the most in MPI 3.0 days because of AINT_DIFF and > AINT_SUM, so > > maybe he wants to create this ticket. > > > > > > > > > > > > Jeff > > > > > > > > > > > > On Thu, Oct 24, 2019 at 2:41 PM Jeff Squyres (jsquyres) < [ > > mailto:[email protected] | jsquyres(a)cisco.com ] > wrote: > > > > > > > > > > > > Not opposed to ditching segmented addressing at all. We'd need a ticket > for this > > ASAP, though. > > > > > > > > > > > > This whole conversation is predicated on: > > > > > > > > > > > > - MPI supposedly supports segmented addressing > > > > > > - MPI_Aint is not sufficient for modern segmented addressing (i.e., > representing > > an address that may not be in main RAM and is not mapped in to the > current > > process' linear address space) > > > > > > > > > > > > If we no longer care about segmented addressing, that makes a whole > bunch of > > BigCount stuff a LOT easier. E.g., MPI_Aint can basically be a > > non-segment-supporting address integer. AINT_DIFF and AINT_SUM can go > away, > > too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Oct 24, 2019, at 5:35 PM, Jeff Hammond via mpiwg-large-counts < [ > > mailto:[email protected] | > > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > > > > > > > > > > > > Rolf: > > > > > > > > Before anybody spends any time analyzing how we handle segmented > addressing, I > > want you to provide an example of a platform where this is relevant. What > > system can you boot today that needs this and what MPI libraries have > expressed > > an interest in supporting it? > > > > > > > > > > > > For anyone who didn't hear, ISO C and C++ have finally committed to > > twos-complement integers ( [ > > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html | > > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html ] > , [ > > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm | > > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm ] ) because > modern > > programmers should not be limited by hardware designs from the 1960s. We > should > > similarly not waste our time on obsolete features like segmentation. > > > > > > > > > > > > Jeff > > > > > > > > > > > > On Thu, Oct 24, 2019 at 10:13 AM Rolf Rabenseifner via > mpiwg-large-counts < [ > > mailto:[email protected] | > > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > > > > > > > > > >> I think that changes the conversation entirely, right? > > > > Not the first part, the state-of-current-MPI. > > > > It may change something for the future, or a new interface may be needed. > > > > Please, can you describe how MPI_Get_address can work with the > > different variables from different memory segments. > > > > Or whether a completely new function or a set of functions is needed. > > > > If we can still express variables from all memory segments as > > input to MPI_Get_address, there may be still a way to flatten > > the result of some internal address-iquiry into a flattened > > signed integer with the same behavior as MPI_Aint today. > > > > If this is impossible, then new way of thinking and solution > > may be needed. > > > > I really want to see examples for all current stuff as you > > mentioned in your last email. > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "Jeff Squyres" < [ mailto:[email protected] | jsquyres(a)cisco.com > ] > > >> To: "Rolf Rabenseifner" < [ mailto:[email protected] | > rabenseifner(a)hlrs.de ] > >> > > >> Cc: "mpiwg-large-counts" < [ mailto: > mpiwg-large-counts(a)lists.mpi-forum.org | > >> mpiwg-large-counts(a)lists.mpi-forum.org ] > > >> Sent: Thursday, October 24, 2019 5:27:31 PM > >> Subject: Re: [Mpiwg-large-counts] Large Count - the principles for > counts, > >> sizes, and byte and nonbyte displacements > > > >> On Oct 24, 2019, at 11:15 AM, Rolf Rabenseifner > >> < [ mailto:[email protected] | rabenseifner(a)hlrs.de ] <mailto: [ > >> mailto:[email protected] | rabenseifner(a)hlrs.de ] >> wrote: > >> > >> For me, it looked like that there was some misunderstanding > >> of the concept that absolute and relative addresses > >> and number of bytes that can be stored in MPI_Aint. > >> > >> ...with the caveat that MPI_Aint -- as it is right now -- does not > support > >> modern segmented memory systems (i.e., where you need more than a small > number > >> of bits to indicate the segment where the memory lives). > >> > >> I think that changes the conversation entirely, right? > >> > >> -- > >> Jeff Squyres > >> [ mailto:[email protected] | jsquyres(a)cisco.com ] <mailto: [ > >> mailto:[email protected] | jsquyres(a)cisco.com ] > > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email [ mailto: > rabenseifner(a)hlrs.de | > > rabenseifner(a)hlrs.de ] . > > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > > Head of Dpmt Parallel Computing . . . [ > http://www.hlrs.de/people/rabenseifner | > > www.hlrs.de/people/rabenseifner ] . > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > > _______________________________________________ > > mpiwg-large-counts mailing list > > [ mailto:[email protected] | > > mpiwg-large-counts(a)lists.mpi-forum.org ] > > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > > > > > > > > > > > > > > > > > > -- > > > > > > Jeff Hammond > > [ mailto:[email protected] | jeff.science(a)gmail.com ] > > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] > > > > > > _______________________________________________ > > mpiwg-large-counts mailing list > > [ mailto:[email protected] | > > mpiwg-large-counts(a)lists.mpi-forum.org ] > > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > > > > > > > > > > > > > > -- > > Jeff Squyres > > [ mailto:[email protected] | jsquyres(a)cisco.com ] > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Jeff Hammond > > [ mailto:[email protected] | jeff.science(a)gmail.com ] > > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] > > _______________________________________________ > > mpiwg-large-counts mailing list > > [ mailto:[email protected] | > > mpiwg-large-counts(a)lists.mpi-forum.org ] > > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > > > > _______________________________________________ > > mpiwg-large-counts mailing list > > mpiwg-large-counts(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . >

1 0

Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements
by Rolf Rabenseifner 29 Oct '19

29 Oct '19

Dear Jim and all, I'm not sure whether I'm really able to understand your email. I take the MPI view: (1) An absolute address can stored in an MPI_Aint variable with and only with MPI_Get_address or MPI_Aint_add. (2) A positive or negative number of bytes or a relative address which is by definition the amount of bytes between two locations in a MPI "sequential storage" (MPI-3.1 page 115) can be assigned with any method to an MPI_Aint variable as long as the original value fits into MPI_Aint. In both languages automatic type cast (i.e., sign expansion) is done. (3) If users misuse MPI_Aint for storing anything else into MPI_Aint variable then this is out of scope of MPI. If such values are used in a minus operation then it is out of the scope of MPI whether this makes sense. If the user is sure that the new value falls into category (2) then all is fine as long as the user is correct. I expect that your => is not a "greater or equal than". I expect that you noticed assignments. > intptr_t => MPI_Aint "intptr_t: integer type capable of holding a pointer." > uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" proposal?) "uintptr_t: unsigned integer type capable of holding a pointer." may fall exactly exactly into (3) when used for pointers. Especially on a 64 bit system the user may have in the future exactly the problems (a), (a1), (a2) and (b) as described below. But here, the user is responsible, to for example implement (a3), whereas for MPI_Get_address, the implementors of the MPI library are responsible and the MPI Forum may be responsible for giving the correct advices. By the way, the golden MPI_Auint was never golden. Such need was "resolved" by introducing MPI_Aint_diff and MPI_Aint_add in MPI-3.1. > ptrdiff_t => MPI_Aint "std::ptrdiff_t is the signed integer type of the result of subtracting two pointers." may perfectly fit to (2). All of the following falls into category (2): > size_t (sizeof) => MPI_Count, int "sizeof( type ) (1) sizeof expression (2) Both versions are constant expressions of type std::size_t." > size_t (offsetof) => MPI_Aint, int "Defined in header <cstddef> #define offsetof(type, member) /*implementation-defined*/ The macro offsetof expands to an integral constant expression of type std::size_t, the value of which is the offset, in bytes, from the beginning of an object of specified type to ist specified member, including padding if any." Note that this offsetof has nothing to do with MPI_Offset. On a system with less than 2*31 byte and 4-byte int, it is guaranteed that size_t => int works. On a system with less than 2*63 byte and 8-byte MPI_Aint, it is guaranteed that size_t => MPI_Aint works. Problem: size_t is unsigned, int and MPI_Aint are signed. MPI_Count should be defined in a way that on systems with more than 2**63 Bytes of disc space, that MPI_Count can hold such values, because int .LE. {MPI_Aint, MPI_Offset} .LE. MPI_Count Therefore size_t => MPI_Count should always work. > ssize_t => Mostly for error handling. Out of scope for MPI? "In short, ssize_t is the same as size_t, but is a signed type - read ssize_t as “signed size_t”. ssize_t is able to represent the number -1, which is returned by several system calls and library functions as a way to indicate error. For example, the read and write system calls: ... ssize_t read(int fildes, void *buf, size_t nbyte); ..." ssize_t fits therefore better to MPI_Aint, because both are signed types that can hold byte counts, but the value -1 in a MPI_Aint variable stands for a byte displacement of -1 bytes and not for an error code -1. All use of (2) is in principle no problem. ------------------------------------------ All the complex discussiuon of the last days is about (1): (1) An absolute address can stored in an MPI_Aint variable with and only with MPI_Get_address or MPI_Aint_add. In MPI-1 to MPI-3.0 and still in MPI-3.1 (here as may be not portable), we also allow MPI_Aint variable := absolute address in MPI_Aint variable + or - a number of bytes (in any integer type). The result is then still in category (1). For the difference of two absolute addresses, MPI_Aint_diff can be used. The result is than MPI_Aint of category (2) In MPI-1 to MPI-3.0 and still in MPI-3.1 (here as may be not portable), we also allow MPI_Aint variable := absolute address in MPI_Aint variable - absolute address in MPI_Aint variable. The result is then in category (2). The problems we discuss the last days are about systems that internally use unsigned addresses and the MPI library stores these addresses into MPI_Aint variables and (a) a sequential storage can have virtual addresses that are both in the area with highest bit =0 and other addresses in the same sequential storage (i.e., same array or structure) with highest bit =1. or (b) some higher bits contain segment addresses. (b) is not a problem as long as a sequential storage resides always within one Segment. Therefore, we only have to discuss (a). The two problems that we have is (a1) that for the minus operations an integer overflow will happen and must be ignored. (a2) if such addresses are expanded to larger variables, e.g., MPI_Count with more bits in MPI_Count than in MPI_Aint, sign expansion will result in completely wring results. And here, the most simple trick is, (a3) that MPI_Get_address really shall map the contiguous unsigned range from 0 to 2**64-1 to the signed (and also contiguous) range from -2**63 to 2**63-1 by simple subtracting 2**63. With this simple trick in MPI_Get_address, Problems 8a1) and (a2) are resolved. It looks like that (a) and therefore (a1) and (a2) may be far in the future. But they may be less far in the future, if a system may map the whole applications cluster address space into virtual memory (not cache coherent, but accessible). And all this is never or only partial written into the MPI Standard, also all is (well) known by the MPI Forum, with the following exceptions: - (a2) is new. - (a1) is solved in MPI-3.1 only for MPI_Aint_diff and MPI_Aint_add, but not for the operators - and + if a user will switch on integer overflow detection in the future when we will have such large systems. - (a3) is new and in principle solves the problem also for + and - operators. At lease (a1)+(a2) should be added as rationale to MPI-4.0 and (a3) as advice to implementors within the framework of big count, because (a2) is newly coming with big count. I hope this helps a bit if you took the time to read this long email. Best regards Rolf ----- Original Message ----- > From: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > To: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > Cc: "Jim Dinan" <james.dinan(a)gmail.com>, "James Dinan" <james.dinan(a)intel.com> > Sent: Monday, October 28, 2019 5:07:37 PM > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements > Still not sure I see the issue. MPI's memory-related integers should map to > types that serve the same function in C. If the base language is broken for > segmented addressing, we won't be able to fix it in a library. Looking at the > mapping below, I don't see where we would have broken it: > > intptr_t => MPI_Aint > uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" proposal?) > ptrdiff_t => MPI_Aint > size_t (sizeof) => MPI_Count, int > size_t (offsetof) => MPI_Aint, int > ssize_t => Mostly for error handling. Out of scope for MPI? > > It sounds like there are some places where we used MPI_Aint in place of size_t > for sizes. Not great, but MPI_Aint already needs to be at least as large as > size_t, so this seems benign. > > ~Jim. > > On Fri, Oct 25, 2019 at 8:25 PM Dinan, James via mpiwg-large-counts < [ > mailto:[email protected] | > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > > > > > > Jeff, thanks so much for opening up these old wounds. I’m not sure I have enough > context to contribute to the discussion. Where can I read up on the issue with > MPI_Aint? > > > > I’m glad to hear that C signed integers will finally have a well-defined > representation. > > > > ~Jim. > > > > > From: Jeff Hammond < [ mailto:[email protected] | jeff.science(a)gmail.com ] > > > Date: Thursday, October 24, 2019 at 7:03 PM > To: "Jeff Squyres (jsquyres)" < [ mailto:[email protected] | jsquyres(a)cisco.com > ] > > Cc: MPI BigCount Working Group < [ mailto:[email protected] > | mpiwg-large-counts(a)lists.mpi-forum.org ] >, "Dinan, James" < [ > mailto:[email protected] | james.dinan(a)intel.com ] > > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, > sizes, and byte and nonbyte displacements > > > > > > Jim (cc) suffered the most in MPI 3.0 days because of AINT_DIFF and AINT_SUM, so > maybe he wants to create this ticket. > > > > > > Jeff > > > > > > On Thu, Oct 24, 2019 at 2:41 PM Jeff Squyres (jsquyres) < [ > mailto:[email protected] | jsquyres(a)cisco.com ] > wrote: > > > > > > Not opposed to ditching segmented addressing at all. We'd need a ticket for this > ASAP, though. > > > > > > This whole conversation is predicated on: > > > > > > - MPI supposedly supports segmented addressing > > > - MPI_Aint is not sufficient for modern segmented addressing (i.e., representing > an address that may not be in main RAM and is not mapped in to the current > process' linear address space) > > > > > > If we no longer care about segmented addressing, that makes a whole bunch of > BigCount stuff a LOT easier. E.g., MPI_Aint can basically be a > non-segment-supporting address integer. AINT_DIFF and AINT_SUM can go away, > too. > > > > > > > > > > > > > > On Oct 24, 2019, at 5:35 PM, Jeff Hammond via mpiwg-large-counts < [ > mailto:[email protected] | > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > > > > > > Rolf: > > > > Before anybody spends any time analyzing how we handle segmented addressing, I > want you to provide an example of a platform where this is relevant. What > system can you boot today that needs this and what MPI libraries have expressed > an interest in supporting it? > > > > > > For anyone who didn't hear, ISO C and C++ have finally committed to > twos-complement integers ( [ > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html | > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html ] , [ > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm | > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm ] ) because modern > programmers should not be limited by hardware designs from the 1960s. We should > similarly not waste our time on obsolete features like segmentation. > > > > > > Jeff > > > > > > On Thu, Oct 24, 2019 at 10:13 AM Rolf Rabenseifner via mpiwg-large-counts < [ > mailto:[email protected] | > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: > > > > >> I think that changes the conversation entirely, right? > > Not the first part, the state-of-current-MPI. > > It may change something for the future, or a new interface may be needed. > > Please, can you describe how MPI_Get_address can work with the > different variables from different memory segments. > > Or whether a completely new function or a set of functions is needed. > > If we can still express variables from all memory segments as > input to MPI_Get_address, there may be still a way to flatten > the result of some internal address-iquiry into a flattened > signed integer with the same behavior as MPI_Aint today. > > If this is impossible, then new way of thinking and solution > may be needed. > > I really want to see examples for all current stuff as you > mentioned in your last email. > > Best regards > Rolf > > ----- Original Message ----- >> From: "Jeff Squyres" < [ mailto:[email protected] | jsquyres(a)cisco.com ] > >> To: "Rolf Rabenseifner" < [ mailto:[email protected] | rabenseifner(a)hlrs.de ] >> > >> Cc: "mpiwg-large-counts" < [ mailto:[email protected] | >> mpiwg-large-counts(a)lists.mpi-forum.org ] > >> Sent: Thursday, October 24, 2019 5:27:31 PM >> Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, >> sizes, and byte and nonbyte displacements > >> On Oct 24, 2019, at 11:15 AM, Rolf Rabenseifner >> < [ mailto:[email protected] | rabenseifner(a)hlrs.de ] <mailto: [ >> mailto:[email protected] | rabenseifner(a)hlrs.de ] >> wrote: >> >> For me, it looked like that there was some misunderstanding >> of the concept that absolute and relative addresses >> and number of bytes that can be stored in MPI_Aint. >> >> ...with the caveat that MPI_Aint -- as it is right now -- does not support >> modern segmented memory systems (i.e., where you need more than a small number >> of bits to indicate the segment where the memory lives). >> >> I think that changes the conversation entirely, right? >> >> -- >> Jeff Squyres >> [ mailto:[email protected] | jsquyres(a)cisco.com ] <mailto: [ >> mailto:[email protected] | jsquyres(a)cisco.com ] > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email [ mailto:[email protected] | > rabenseifner(a)hlrs.de ] . > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > Head of Dpmt Parallel Computing . . . [ http://www.hlrs.de/people/rabenseifner | > www.hlrs.de/people/rabenseifner ] . > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > _______________________________________________ > mpiwg-large-counts mailing list > [ mailto:[email protected] | > mpiwg-large-counts(a)lists.mpi-forum.org ] > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > > > > > > > > > -- > > > Jeff Hammond > [ mailto:[email protected] | jeff.science(a)gmail.com ] > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] > > > _______________________________________________ > mpiwg-large-counts mailing list > [ mailto:[email protected] | > mpiwg-large-counts(a)lists.mpi-forum.org ] > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > > > > > > > -- > Jeff Squyres > [ mailto:[email protected] | jsquyres(a)cisco.com ] > > > > > > > > > > > > -- > > > Jeff Hammond > [ mailto:[email protected] | jeff.science(a)gmail.com ] > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] > _______________________________________________ > mpiwg-large-counts mailing list > [ mailto:[email protected] | > mpiwg-large-counts(a)lists.mpi-forum.org ] > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] > > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements
by Jim Dinan 28 Oct '19

28 Oct '19

Still not sure I see the issue. MPI's memory-related integers should map to types that serve the same function in C. If the base language is broken for segmented addressing, we won't be able to fix it in a library. Looking at the mapping below, I don't see where we would have broken it: intptr_t => MPI_Aint uintptr_t => ??? (Anyone remember the MPI_Auint "golden Aint" proposal?) ptrdiff_t => MPI_Aint size_t (sizeof) => MPI_Count, int size_t (offsetof) => MPI_Aint, int ssize_t => Mostly for error handling. Out of scope for MPI? It sounds like there are some places where we used MPI_Aint in place of size_t for sizes. Not great, but MPI_Aint already needs to be at least as large as size_t, so this seems benign. ~Jim. On Fri, Oct 25, 2019 at 8:25 PM Dinan, James via mpiwg-large-counts < mpiwg-large-counts(a)lists.mpi-forum.org> wrote: > Jeff, thanks so much for opening up these old wounds. I’m not sure I have > enough context to contribute to the discussion. Where can I read up on the > issue with MPI_Aint? > > > > I’m glad to hear that C signed integers will finally have a well-defined > representation. > > > > ~Jim. > > > > *From: *Jeff Hammond <jeff.science(a)gmail.com> > *Date: *Thursday, October 24, 2019 at 7:03 PM > *To: *"Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> > *Cc: *MPI BigCount Working Group <mpiwg-large-counts(a)lists.mpi-forum.org>, > "Dinan, James" <james.dinan(a)intel.com> > *Subject: *Re: [Mpiwg-large-counts] Large Count - the principles for > counts, sizes, and byte and nonbyte displacements > > > > Jim (cc) suffered the most in MPI 3.0 days because of AINT_DIFF and > AINT_SUM, so maybe he wants to create this ticket. > > > > Jeff > > > > On Thu, Oct 24, 2019 at 2:41 PM Jeff Squyres (jsquyres) < > jsquyres(a)cisco.com> wrote: > > Not opposed to ditching segmented addressing at all. We'd need a ticket > for this ASAP, though. > > > > This whole conversation is predicated on: > > > > - MPI supposedly supports segmented addressing > > - MPI_Aint is not sufficient for modern segmented addressing (i.e., > representing an address that may not be in main RAM and is not mapped in to > the current process' linear address space) > > > > If we no longer care about segmented addressing, that makes a whole bunch > of BigCount stuff a LOT easier. E.g., MPI_Aint can basically be a > non-segment-supporting address integer. AINT_DIFF and AINT_SUM can go > away, too. > > > > > > > > On Oct 24, 2019, at 5:35 PM, Jeff Hammond via mpiwg-large-counts < > mpiwg-large-counts(a)lists.mpi-forum.org> wrote: > > > > Rolf: > > > Before anybody spends any time analyzing how we handle segmented > addressing, I want you to provide an example of a platform where this is > relevant. What system can you boot today that needs this and what MPI > libraries have expressed an interest in supporting it? > > > > For anyone who didn't hear, ISO C and C++ have finally committed to > twos-complement integers ( > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html, > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm) because modern > programmers should not be limited by hardware designs from the 1960s. We > should similarly not waste our time on obsolete features like segmentation. > > > > Jeff > > > > On Thu, Oct 24, 2019 at 10:13 AM Rolf Rabenseifner via mpiwg-large-counts < > mpiwg-large-counts(a)lists.mpi-forum.org> wrote: > > > I think that changes the conversation entirely, right? > > Not the first part, the state-of-current-MPI. > > It may change something for the future, or a new interface may be needed. > > Please, can you describe how MPI_Get_address can work with the > different variables from different memory segments. > > Or whether a completely new function or a set of functions is needed. > > If we can still express variables from all memory segments as > input to MPI_Get_address, there may be still a way to flatten > the result of some internal address-iquiry into a flattened > signed integer with the same behavior as MPI_Aint today. > > If this is impossible, then new way of thinking and solution > may be needed. > > I really want to see examples for all current stuff as you > mentioned in your last email. > > Best regards > Rolf > > ----- Original Message ----- > > From: "Jeff Squyres" <jsquyres(a)cisco.com> > > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > > Cc: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > > Sent: Thursday, October 24, 2019 5:27:31 PM > > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for > counts, sizes, and byte and nonbyte displacements > > > On Oct 24, 2019, at 11:15 AM, Rolf Rabenseifner > > <rabenseifner(a)hlrs.de<mailto:[email protected]>> wrote: > > > > For me, it looked like that there was some misunderstanding > > of the concept that absolute and relative addresses > > and number of bytes that can be stored in MPI_Aint. > > > > ...with the caveat that MPI_Aint -- as it is right now -- does not > support > > modern segmented memory systems (i.e., where you need more than a small > number > > of bits to indicate the segment where the memory lives). > > > > I think that changes the conversation entirely, right? > > > > -- > > Jeff Squyres > > jsquyres(a)cisco.com<mailto:[email protected]> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts > > > > > -- > > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts > > > > > -- > Jeff Squyres > jsquyres(a)cisco.com > > > > > > > -- > > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts >

1 0

Re: [Mpiwg-large-counts] Pythonizing + BigCount: virtual meeting on Nov 13?
by Wesley Bland 28 Oct '19

28 Oct '19

I don’t think anyone else has reserved it. I can put you on the schedule. > On Oct 26, 2019, at 9:15 AM, Jeff Squyres (jsquyres) via mpiwg-large-counts <mpiwg-large-counts(a)lists.mpi-forum.org> wrote: > > Martin Schulz -- > > Can you please confirm that the ExaCount WG can have the Nov 13 MPI Forum Virtual Meeting slot? > > Thanks. > > >> On Oct 23, 2019, at 10:11 AM, Jeff Squyres (jsquyres) via mpiwg-large-counts <mpiwg-large-counts(a)lists.mpi-forum.org> wrote: >> >> Martin S -- >> >> The LargeCount / BigCount / WhopperCount / GinormousCount / UseYouFavoriteWordCount WG would like to reserve the 2-hour virtual meeting on Wed Nov 13. This is the Wed before SC, and I'm sure it will be incredibly well attended! ;-) >> >> We'd like to cover two things: >> >> 1. The state of LaTeX Pythonizing >> (Recall: the Pythonization -- yes, that's totally a word; you must use it in a sentence with a colleague today -- of the MPI LaTeX is a separate effort than the Embiggening -- yes, that's also a word -- of "count" parameters. Specifically: we're doing a comparable-to-MPI-3.1 Pythonization of the MPI bindings in LaTeX without Embiggening or otherwise changing any parameters. Pythonization opens the door to multiple interesting things in the future, including Embiggening "count" parameters). >> >> 2. The state of BigCount / Embiggening >> There was a reading and much discussion at the last Forum meeting about Embiggening. We'd like to get some Forum feedback before the next Forum meeting in December. >> >> -- >> Jeff Squyres >> jsquyres(a)cisco.com >> >> _______________________________________________ >> mpiwg-large-counts mailing list >> mpiwg-large-counts(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts > > > -- > Jeff Squyres > jsquyres(a)cisco.com > > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts

1 0

Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements
by Rolf Rabenseifner 27 Oct '19

27 Oct '19

Dear Jeff H., comments inline. It is really a complex Topic. Thank you very much about your pointing to 57 bits Registers. I tried to understand by reading https://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details and the article around this topic. All the rest below. ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org>, "Jeff Squyres" <jsquyres(a)cisco.com>, "James Dinan" > <james.dinan(a)intel.com> > Sent: Friday, October 25, 2019 8:35:47 PM > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements > On Fri, Oct 25, 2019 at 5:20 AM Rolf Rabenseifner <rabenseifner(a)hlrs.de> > wrote: > >> Dear Jeff, >> >> > If we no longer care about segmented addressing, that makes a whole >> bunch of >> > BigCount stuff a LOT easier. E.g., MPI_Aint can basically be a >> > non-segment-supporting address integer. >> >> > AINT_DIFF and AINT_SUM can go away, too. >> >> Both statements are -- in my opinion -- incorrect. >> And the real problem is really ugly, see below. >> >> After we seem to agree that MPI_Aint is used as it is used, i.e., >> to currently store >> - absolute addresses (which means the bits of a 64-bit-unsigned address >> interpreted as a signed twos-complement 64 bit integer >> i.e., values between -2**63 and + 2**63-1 >> (and only here is the discussion about whether some higher bits may >> be used to address segments) >> - relative addresses between -2**63 and + 2**63-1 >> > > C and C++ do not allow one to do pointer arithmetic outside of a single > array so I'm not sure how one would generate relative addresses this large, > particularly on an x86_64 machine where the underlying memory addresses are > 48 or 57 bits. "C and C++ do not allow one to do pointer arithmetic outside of a single array" perfectly fits to the MPI Definition on "sequential storage" and the rule that a diff is only allowed within such sequential storage. About >> - relative addresses between -2**63 and + 2**63-1 I describe only what MPI allows. One should have in mind, that relative addresses also used as displacements in derived datatypes. Those can be larger than the address range of a real memory system, if a file is larger than this range AND the derived datatype is for example used to define a filetype, i.e., which portions of a globally accessed file are accessible for an individual MPI process. >> - byte counts between 0 and 2**63-1 >> > > All such uses should use functions with MPI_Count, not MPI_Aint, once those > functions are defined in MPI 4.0. Exactly for this proposal I detected a ***severe*** problem, described on the lines below with the two addresses addr1 and addr2 and using as example 8 bit MPI_Aint and 12 bit MPI_Count. >> And that for two absolute addresses within the same "sequential storage" >> (defined in MPI-3.1 Sect. 4.1.12 page 115 lines 17-19), it is allowed >> to use a minus operator (as Long as integer overflow detection is >> switched off) or MPI_Aint_diff. >> > > Again, pointer arithmetic has to be within a single array. It's pretty > hard to generate an overflow in this context. I'm pretty sure that you may be not right, at least in the future: MPI_Aint_diff was defined on he Background that at least in the future, the virtual addresses may span from lower to higher half of the virtual 64bit address space as shown in the right diagram in figure in https://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details and https://en.wikipedia.org/wiki/X86-64#/media/File:AMD64-canonical--64-bit.svg If an array goes from lower half to higher half then we have the problem described below with the two absolute addresses 127 and 129 in my example with only a 8 bits space. As long as in the virtual address space in user mode, only addresses in the lower half exist, we do not have the problem. When I understand correctly, we want to specify MPI-4.0 to be future-safe for the case that true 64-bit addressing will come in some future. (This may include systems that allow full direct addressibility of the whole cluster memory.) Are I'm right with this assumption or should MPI-4.0 only be usable for 56/57 bit addresses? >> In principle, the MPI standard is not fully consistent with that: >> >> MPI-3.1 page 102 lines 45-46 tell: >> "To ensure portability, arithmetic on MPI addresses >> must >> be performed using the MPI_AINT_ADD and MPI_AINT_DIFF functions." >> and >> > > Yes, it also says for portability, RMA needs to use MPI_Alloc_mem and not > malloc or similar arrays. That never matters in practice. Yes, I agree, nearly all applications may still use the minus Operator for MPI_Aint instead of MPI_Aint_diff. >> > > ... MPI-3.1 2.5.6 "Absolute >> > > Addresses and Relative Address Displacements" p16:39-43: >> > > >> > > "For retrieving absolute addresses or any calculation with absolute >> addresses, one >> > > should >> > > use the routines and functions provided in Section 4.1.5. Section >> > > 4.1.12 provides additional rules for the correct use of absolute >> addresses. For >> > > expressions with relative displacements or other usage without absolute >> > > addresses, intrinsic operators (e.g., +, -, *) can be used." >> >> And now about large counts, especially if we want to extent routines >> that currently use MPI_Aint to something larger, i.e., MPI_Aint_x or >> MPI_Count. >> > > MPI_Aint_x will not exist. MPI_Count is already a thing and is at least as > large as MPI_Aint and MPI_Offset. I only wanted to be general. Of course, I also expect that we should read below MPI_Aint_x (or MPI_Count) addr_x; always as MPI_Count) addr_x; >> Here, the major problem is the automatic cast within an assignment >> >> MPI_Aint addr; >> MPI_Aint_x (or MPI_Count) addr_x; >> >> MPI_Get_address(...., &addr); >> addr_x = addr; // ***this Statement is the problem**** >> >> > No. This statement is not a problem. MPI_Count is required to hold the > full range of MPI_Aint. You can assign MPI_Aint to MPI_Count without > truncation. You completely misunderstood my hint. The Problem is the sign Expansion and not a removal of high-order bits. In my example MPI_Count has 12 bits and MPI_Aint 8 bits and an array may span addresses from lower half to the higher half of the 8 bit address space. >> let's take my example fro a previous email (using 8 bit MPI_Aint) >> >> addr1 01111111 = 127 (signed int) = 127 (unsigned int) >> addr2 10000001 = -127 (signed int) = 129 (unsigned int) >> >> Internally the addreses are viewed by the hardware and OS as unsigned. >> MPI_Aint is interpreting the same bits as signed int. >> >> addr2-addr1 = 129 -127 = 2 (as unsigned int) >> but in a real application code with "-" operator: >> = -127 -127 = -254 >> --> signed int Overflow because 8 bit can express only -128 .. +127 >> --> detected or automatically corrected with +256 --> -254+256 = 2 >> >> And now with 12 bit MPI_Aint_x >> >> addr1_x := addr1 results in (by sign propagation) >> addr1_x = 000001111111 = 127 (signed int) = 127 (unsigned int) >> >> addr2_x := addr2 results in (by sign propagation) >> addr2_x = 111110000001 = -127 (signed int) = 129 (unsigned int) >> >> and then >> addr2_x - addr1_x = -127 - 127 = -254 >> which is a normal integer within 12bit, >> and therefore ***NO*** overflow correction!!!!!! >> >> And therefore a completely ***wrong*** result. >> >> Using two different types for absolute addresses seems to be a >> real problem in my opinion. >> >> >> And of course signed 64bit MPI_Aint does allow to specify only >> 2**63-1 bytes, which is about 8*1000**6 Bytes, >> which is only 8 Exabyte. >> >> On systems with less than 8 Exabyte per MPI process, this is not >> a problem for message passing, but it is a problem for I/O, >> > > That is why MPI_Count is potentially larger than MPI_Aint, because it also > has to hold MPI_Offset for IO purposes. As you could see, the problem arises if MPI_Count is really larger than MPI_Aint. >> and therefore for derived datatypes. >> And derived datayptes use MPI_Aint at several locations, >> and some of them with the possibility of providing absolute addresses. >> > > You are welcome to create a ticket for large-count datatype functions that > use MPI_Count if one does not already exist. I expect that the large Count Group wants to extend also all derived datatype routines to large counts to be able to Support large offsets in parallel I/O. And for this purpose, the large count working group must decide whether for example the long version of MPI_TYPE_CREATE_STRUCT will have not only MPI_Count (instead of int) array_of_blocklengths[], but also MPI_Count (instead of MPI_Aint) array_of_displacements[]. > Jeff >> >> A solution of this problem seems to be not trivial, or is there one? Yes, there is a solution: If a system provides a 64-bit unsigned integer address space which allows that an array or structure may go accross the middle F00000000..000 address, then MPI_Get_addressmust really map the contiguous **unsigned** address space from 0000...0000 until FFFF...FFFF to the also contiguous **signed** address space from F000...0000 until 7FFF...FFFF by simply subtracting F000...0000 from each unsigned address to achieve the corresponding **signed** address. This should be mentioned in advice to implementors. Rolf >> And always doing MPI_Aint with more than 8 bytes is also a no-option, >> based on the ABI discussion, and is also a waste of memory. >> >> >> Best regards >> Rolf >> >> >> ----- Original Message ----- >> > From: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> >> > To: "Jeff Squyres" <jsquyres(a)cisco.com> >> > Cc: "Jeff Hammond" <jeff.science(a)gmail.com>, "James Dinan" < >> james.dinan(a)intel.com>, "mpiwg-large-counts" >> > <mpiwg-large-counts(a)lists.mpi-forum.org> >> > Sent: Friday, October 25, 2019 1:02:35 AM >> > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for >> counts, sizes, and byte and nonbyte displacements >> >> > Jim (cc) suffered the most in MPI 3.0 days because of AINT_DIFF and >> AINT_SUM, so >> > maybe he wants to create this ticket. >> > >> > Jeff >> >> >> > On Thu, Oct 24, 2019 at 2:41 PM Jeff Squyres (jsquyres) < [ >> > mailto:[email protected] | jsquyres(a)cisco.com ] > wrote: >> > >> > >> > Not opposed to ditching segmented addressing at all. We'd need a ticket >> for this >> > ASAP, though. >> > >> > This whole conversation is predicated on: >> > >> > - MPI supposedly supports segmented addressing >> > - MPI_Aint is not sufficient for modern segmented addressing (i.e., >> representing >> > an address that may not be in main RAM and is not mapped in to the >> current >> > process' linear address space) >> > >> > If we no longer care about segmented addressing, that makes a whole >> bunch of >> > BigCount stuff a LOT easier. E.g., MPI_Aint can basically be a >> > non-segment-supporting address integer. AINT_DIFF and AINT_SUM can go >> away, >> > too. >> >> >> > On Oct 24, 2019, at 5:35 PM, Jeff Hammond via mpiwg-large-counts < [ >> > mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: >> > >> > Rolf: >> > >> > Before anybody spends any time analyzing how we handle segmented >> addressing, I >> > want you to provide an example of a platform where this is relevant. What >> > system can you boot today that needs this and what MPI libraries have >> expressed >> > an interest in supporting it? >> > >> > For anyone who didn't hear, ISO C and C++ have finally committed to >> > twos-complement integers ( [ >> > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html | >> > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r1.html ] >> , [ >> > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm | >> > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm ] ) because >> modern >> > programmers should not be limited by hardware designs from the 1960s. We >> should >> > similarly not waste our time on obsolete features like segmentation. >> > >> > Jeff >> > >> > On Thu, Oct 24, 2019 at 10:13 AM Rolf Rabenseifner via >> mpiwg-large-counts < [ >> > mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] > wrote: >> > >> > >> >> I think that changes the conversation entirely, right? >> > >> > Not the first part, the state-of-current-MPI. >> > >> > It may change something for the future, or a new interface may be needed. >> > >> > Please, can you describe how MPI_Get_address can work with the >> > different variables from different memory segments. >> > >> > Or whether a completely new function or a set of functions is needed. >> > >> > If we can still express variables from all memory segments as >> > input to MPI_Get_address, there may be still a way to flatten >> > the result of some internal address-iquiry into a flattened >> > signed integer with the same behavior as MPI_Aint today. >> > >> > If this is impossible, then new way of thinking and solution >> > may be needed. >> > >> > I really want to see examples for all current stuff as you >> > mentioned in your last email. >> > >> > Best regards >> > Rolf >> >> >> ----- Original Message ----- >> > From: "HOLMES Daniel" <d.holmes(a)epcc.ed.ac.uk> >> > To: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> >> > Cc: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff Squyres" < >> jsquyres(a)cisco.com> >> > Sent: Thursday, October 24, 2019 6:41:34 PM >> > Subject: Re: [Mpiwg-large-counts] Large Count - the principles for >> counts, sizes, and byte and nonbyte displacements >> >> > Hi Rolf & Jeff, >> > >> > I think this wiki article is instructive on this topic also: >> > https://en.wikipedia.org/wiki/X86_memory_segmentation >> > >> > This seems like a crazy memory addressing system to me personally, but >> it is a >> > (historic) example of a segmented addressing approach that MPI_Aint can >> > support. >> > >> > The “strange properties” for arithmetic are strange indeed, depending on >> what >> > the MPI_Aint stores and how. >> > >> > If MPI_Aint was 20 bits long and stores only the address, then it cannot >> be used >> > to determine uniquely which segment is being used or what the offset is >> within >> > that segment (there are 4096 possible answers). Does MPI need that more >> > detailed information? Probably - because segments were a way of >> implementing >> > memory protection, i.e. accessing a segment you did not have permission >> to >> > access led to a “segmentation fault” error. I do not know enough about >> these >> > old architectures to say whether an attempt to access the *same byte* >> using two >> > different segment:offset pairs that produce the *same* address could >> result in >> > different behaviour. That is, if I have access permissions for segment 3 >> but >> > not for segment 4, I can access {seg=3,offset=2^16-16} but can I access >> > {segment=4,offset=2^16-32}, which is the same byte? If not, then MPI >> needs to >> > store segment and offset inside MPI_Aint to be able to check and to set >> > registers correctly. >> > >> > If MPI_Aint is 32 bits long and stores the segment in the first 16 bits >> and the >> > offset in the last 16 bits, then the 20 bit address can be computed in a >> single >> > simple instruction and both segment and offset are immediately >> retrievable. >> > However, doing ordinary arithmetic with this bitwise representation is >> unwise >> > because it is a compound structure type. Let us subtract 1 from an >> MPI_Aint of >> > this layout which stores offset of 0 and some non-zero segment. We get >> offset >> > (2^16-1) in segment (s-1), which is not 1 byte before the previous >> MPI_Aint >> > because segments overlap. The same happens when adding and overflowing >> the >> > offset portion - it changes the segment in an incorrect way. Segment++ >> moves >> > the address forward only 16 bytes, not 2^16 bytes. >> > >> > The wrap-around from the end of the address space back to the beginning >> is also >> > a source of strange properties for arithmetic. >> > >> > One of the key statements from that wiki page is this: >> > >> > The root of the problem is that no appropriate address-arithmetic >> instructions >> > suitable for flat addressing of the entire memory range are >> available.[citation >> > needed] Flat addressing is possible by applying multiple instructions, >> which >> > however leads to slower programs. >> > >> > Cheers, >> > Dan. >> > — >> > Dr Daniel Holmes PhD >> > Architect (HPC Research) >> > d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> >> > Phone: +44 (0) 131 651 3465 >> > Mobile: +44 (0) 7940 524 088 >> > Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, >> EH8 9BT >> > — >> > The University of Edinburgh is a charitable body, registered in >> Scotland, with >> > registration number SC005336. >> > — >> >> >> > ----- Original Message ----- >> >> From: "Jeff Squyres" < [ mailto:[email protected] | jsquyres(a)cisco.com >> ] > >> >> To: "Rolf Rabenseifner" < [ mailto:[email protected] | >> rabenseifner(a)hlrs.de ] >> >> > >> >> Cc: "mpiwg-large-counts" < [ mailto: >> mpiwg-large-counts(a)lists.mpi-forum.org | >> >> mpiwg-large-counts(a)lists.mpi-forum.org ] > >> >> Sent: Thursday, October 24, 2019 5:27:31 PM >> >> Subject: Re: [Mpiwg-large-counts] Large Count - the principles for >> counts, >> >> sizes, and byte and nonbyte displacements >> > >> >> On Oct 24, 2019, at 11:15 AM, Rolf Rabenseifner >> >> < [ mailto:[email protected] | rabenseifner(a)hlrs.de ] <mailto: [ >> >> mailto:[email protected] | rabenseifner(a)hlrs.de ] >> wrote: >> >> >> >> For me, it looked like that there was some misunderstanding >> >> of the concept that absolute and relative addresses >> >> and number of bytes that can be stored in MPI_Aint. >> >> >> >> ...with the caveat that MPI_Aint -- as it is right now -- does not >> support >> >> modern segmented memory systems (i.e., where you need more than a small >> number >> >> of bits to indicate the segment where the memory lives). >> >> >> >> I think that changes the conversation entirely, right? >> >> >> >> -- >> >> Jeff Squyres >> >> [ mailto:[email protected] | jsquyres(a)cisco.com ] <mailto: [ >> >> mailto:[email protected] | jsquyres(a)cisco.com ] > >> > >> > -- >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email [ mailto: >> rabenseifner(a)hlrs.de | >> > rabenseifner(a)hlrs.de ] . >> > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . >> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . >> > Head of Dpmt Parallel Computing . . . [ >> http://www.hlrs.de/people/rabenseifner | >> > www.hlrs.de/people/rabenseifner ] . >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . >> > _______________________________________________ >> > mpiwg-large-counts mailing list >> > [ mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] >> > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] >> > >> > >> > -- >> > Jeff Hammond >> > [ mailto:[email protected] | jeff.science(a)gmail.com ] >> > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] >> > _______________________________________________ >> > mpiwg-large-counts mailing list >> > [ mailto:[email protected] | >> > mpiwg-large-counts(a)lists.mpi-forum.org ] >> > [ https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts | >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts ] >> > >> > >> > -- >> > Jeff Squyres >> > [ mailto:[email protected] | jsquyres(a)cisco.com ] >> > >> > >> > >> > -- >> > Jeff Hammond >> > [ mailto:[email protected] | jeff.science(a)gmail.com ] >> > [ http://jeffhammond.github.io/ | http://jeffhammond.github.io/ ] >> > >> > _______________________________________________ >> > mpiwg-large-counts mailing list >> > mpiwg-large-counts(a)lists.mpi-forum.org >> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Re: [Mpiwg-large-counts] Minor improvement to Pythonizing
by Jeff Squyres (jsquyres) 27 Oct '19

27 Oct '19

On Oct 27, 2019, at 1:38 PM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > > The index entries are well chosen according to relevance, > i.e., not always just all Locations. have to admit that I was always confused by the "MPI Declarations Index" -- I thought it was buggy because it was so incomplete. There is no description there of your stated intent. All it says in MPI-3.1 is: "This index refers to declarations needed in C, such as address kind integers, handles, etc. The underlined page numbers is the “main” reference (sometimes there are more than one when key concepts are discussed in multiple areas)." There's a lot of references for each type, but it is nowhere near complete. There's no indication that only *some* of the references to handles are listed. This led me to believe that it was erroneous. It sounds like there are three possibilities: 1. List all the pages where all the types are used. 2. List some of the pages where some of the types are used. But don't say why. 3. List some of the pages where some of the types are used, and give an explanation / rationale as to what exactly is shown, and why. I think we're in state 2. Either state 1 or 3 would be significantly better. I took a stab at state 1, but I don't have strong feelings about it. Specifically: it sounds like you'd prefer state 3. If you have specific rationale for why specific selections were made, can you amend the introduction text to the MPI Declarations Index? I'm guessing this can be an errata, or possibly even a ticket 0...? -- Jeff Squyres jsquyres(a)cisco.com

1 0

Re: [Mpiwg-large-counts] Minor improvement to Pythonizing
by Rolf Rabenseifner 27 Oct '19

27 Oct '19

Dear Jeff, > I made a minor improvement to the Pythonizing script this morning: it now > automatically adds a \cdeclindex{} entry to each LIS rendering for each "MPI_*" > C type that is found in a given MPI procedure. This seems to be the contrary: The index entries are well chosen according to relevance, i.e., not always just all Locations. And exactly 1-3 defining locations are underlined. - MPI_Comm, MPI_Datatype: all generation and handling routines, but never the routines, where such a handle is just an Input argument, i.e., just used. For some others, all locations may be there: - MPI_File ? - MPI_Op ? - MPI_Request ??? Doing this when implementing the index, was many hours of work. I had not the time to check all. But having all MPI_Comm or MPI_Datatype in the index makes it unusable! Best regards Rolf ----- Original Message ----- > From: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > To: "mpiwg-large-counts" <mpiwg-large-counts(a)lists.mpi-forum.org> > Cc: "Jeff Squyres" <jsquyres(a)cisco.com> > Sent: Saturday, October 26, 2019 5:39:22 PM > Subject: [Mpiwg-large-counts] Minor improvement to Pythonizing > I made a minor improvement to the Pythonizing script this morning: it now > automatically adds a \cdeclindex{} entry to each LIS rendering for each "MPI_*" > C type that is found in a given MPI procedure. > > Specifically, there are a *lot* of places in the old .tex that manually added a > \cdeclindex{} entry for some or all of the MPI handles that are used in a given > binding. The old LaTeX was not consistent, though: > > - Sometimes only some of the handle types were \cdeclindex{}'ed > - Sometimes none of the handle types were \cdeclindex{}'ed > - Sometimes all of the handle types were \cdeclindex{}'ed > - Sometimes non-handle types were \cdeclindex{}'ed (e.g., MPI_Aint) > > The Pythonizing improvement this morning always lists a \cdeclindex{} for every > C type that is passed that begins with "MPI_" -- to include MPI_Aint, > MPI_Count, MPI_Status, MPI_Comm, etc. (because I saw some of these in the > existing hard-coded LaTeX). > > The end result is that the handle type index is significantly more detailed / > accurate than it was in MPI-3.1. > > Hopefully this is not controversial. > > -- > Jeff Squyres > jsquyres(a)cisco.com > > _______________________________________________ > mpiwg-large-counts mailing list > mpiwg-large-counts(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Minor improvement to Pythonizing
by Jeff Squyres (jsquyres) 26 Oct '19

26 Oct '19

I made a minor improvement to the Pythonizing script this morning: it now automatically adds a \cdeclindex{} entry to each LIS rendering for each "MPI_*" C type that is found in a given MPI procedure. Specifically, there are a *lot* of places in the old .tex that manually added a \cdeclindex{} entry for some or all of the MPI handles that are used in a given binding. The old LaTeX was not consistent, though: - Sometimes only some of the handle types were \cdeclindex{}'ed - Sometimes none of the handle types were \cdeclindex{}'ed - Sometimes all of the handle types were \cdeclindex{}'ed - Sometimes non-handle types were \cdeclindex{}'ed (e.g., MPI_Aint) The Pythonizing improvement this morning always lists a \cdeclindex{} for every C type that is passed that begins with "MPI_" -- to include MPI_Aint, MPI_Count, MPI_Status, MPI_Comm, etc. (because I saw some of these in the existing hard-coded LaTeX). The end result is that the handle type index is significantly more detailed / accurate than it was in MPI-3.1. Hopefully this is not controversial. -- Jeff Squyres jsquyres(a)cisco.com

1 0