August 2012 - mpiwg-rma - mailman3.cels.anl.gov

[Mpi3-rma] GPI (high performance pgas)
by maik peterson 30 Aug '12

30 Aug '12

has anyone ever tried to couple GPI (european pgas implementation) with mpi in the sense of rma ? we tried it standalone and it was blowing away mpi in most of the test cases. but when it comes to porting it would be nice to have mpi still available in the background and just focusing on perf. critical parts with pgas.

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Jim Dinan 28 Aug '12

28 Aug '12

On 8/28/12 2:33 PM, Dries Kimpe wrote: >> Breaking backwards compatibility is worse than whatever portability >> issues you can come up with, especially with something as fundamental >> as MPI_Aint. > > Good point. > > Question: do you know of any instances where MPI_Aint is used to hold a > negative value? In e.g. MPI_Type_create_hindexed, the MPI_Aint displacement arguments can represent both positive and negative displacements. ~Jim.

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Dries Kimpe 28 Aug '12

28 Aug '12

* Jeff Hammond <jhammond(a)alcf.anl.gov> [2012-08-28 14:02:09]: > Breaking backwards compatibility is worse than whatever portability > issues you can come up with, especially with something as fundamental > as MPI_Aint. Good point. Question: do you know of any instances where MPI_Aint is used to hold a negative value? Or, even better, if the existing standard specifies, for all use cases of MPI_Aint, that the value should be non-negative then there should not be any issue with backwards compatibility when changing MPI_Aint to unsigned? The 2.2 standard states that MPI_Aint "must have the same width and encode address values in the same manner such that address values in one language may be passed directly to another language without conversion." (pg 15) It also says that MPI_Aint is used to represent an 'absolute' address (even though MPI_BOTTOM indicates the start of the address range). On pg 18, it says that 'MPI_Aint is defined to be an integer of the size needed to hold /any valid address/ on the target architecture.' Also, note that MPI_Aint is used by MPI_GET_ADDRESS. Meaning the problem is not limited to RMA. Dries

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Jim Dinan 28 Aug '12

28 Aug '12

Hi All, Here is the suggested change to address this issue. In the paragraph on page 425, line 1, add the bracketed sentence: In the case of a window created with MPI_WIN_CREATE_DYNAMIC, the target_disp for all RMA functions is the address at the target; i.e., the effective window_base is MPI_BOTTOM and the disp_unit is one. [Thus, for dynamic windows, the target_disp argument to all RMA functions in interpreted as unsigned.] Users should use MPI_GET_ADDRESS at the target process to determine the address of a target memory location and communicate this address to the origin process. ~Jim. On 8/28/12 9:46 AM, Jim Dinan wrote: > Hi All, > > Currently, target displacements in put/get/acc/etc are defined to be > non-negative integers of type, MPI_Aint. MPI_Aint is defined to be a > signed integer. However, displacements associated with dynamic windows > are addresses (unsigned) and can range into the space of negative integers. > > It seems like the least impact change would be to add a special case > that the displacement argument will be interpreted as unsigned for > dynamic windows. Changing the interface to take an 'unsigned MPI_Aint' > would be the most clear (and preserve the existing and intended new > semantics), but would break backward compatibility. > > Interested to hear your thoughts. > > ~Jim.

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Jeff Hammond 28 Aug '12

28 Aug '12

Breaking backwards compatibility is worse than whatever portability issues you can come up with, especially with something as fundamental as MPI_Aint. Jeff On Tue, Aug 28, 2012 at 1:45 PM, Dries Kimpe <dkimpe(a)mcs.anl.gov> wrote: > > * Jeff Hammond <jhammond(a)alcf.anl.gov> [2012-08-28 13:22:47]: > >> I imagine that HPC-oriented operating systems do not provide virtual >> addresses in the problematic range anyways. Maybe VM randomization >> causes this but that's not common in HPC. > >> In any case, MPI RMA requires MPI_Alloc_mem for portable performance. >> MPI_Alloc_mem should be capable of returning only pointers in the >> lower half of the address space, if by no other means than using mmap >> intelligently. > >> I believe that this is only a problem on 32-bit systems where losing 2 >> of the 4 GB is a problem. On a 64-bit system, losing half of the 18 >> exabytes in the address space is unlikely to be a problem, given that >> an exabyte of DRAM would require something like a white dwarf to >> power, meaning that it is likely that the OS can find a way to VM map >> the physical memory present into the lower half of the address range. > > On 32 bit, it is very likely that addresses from the stack variables will > have the highest bit set. This is even more true for 32 bit programs > running under 64 bit kernels. > > In linux, on 64 bit X86_64, this will _currently_ not be a problem. All > user memory will be between 0000000000000000 - 00007fffffffffff. > > Also, remember that these are virtual addresses. If MPI_Alloc_mem is > calling malloc/new (which can call mmap for large allocations), addresses > will be from the high range, which can have the sign bit set. > > But all this is besides the point. The address map/choices are not fixed, > and can change at any time (and do differ between operating systems), > and MPI cannot change how the C library/runtime returns memory from > malloc. > > (And then we're ignoring tools like valgrind/memory profilers/...) > > Ignoring the sign issue will very likely cause portability issues. > > Dries > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Dries Kimpe 28 Aug '12

28 Aug '12

* Jeff Hammond <jhammond(a)alcf.anl.gov> [2012-08-28 13:22:47]: > I imagine that HPC-oriented operating systems do not provide virtual > addresses in the problematic range anyways. Maybe VM randomization > causes this but that's not common in HPC. > In any case, MPI RMA requires MPI_Alloc_mem for portable performance. > MPI_Alloc_mem should be capable of returning only pointers in the > lower half of the address space, if by no other means than using mmap > intelligently. > I believe that this is only a problem on 32-bit systems where losing 2 > of the 4 GB is a problem. On a 64-bit system, losing half of the 18 > exabytes in the address space is unlikely to be a problem, given that > an exabyte of DRAM would require something like a white dwarf to > power, meaning that it is likely that the OS can find a way to VM map > the physical memory present into the lower half of the address range. On 32 bit, it is very likely that addresses from the stack variables will have the highest bit set. This is even more true for 32 bit programs running under 64 bit kernels. In linux, on 64 bit X86_64, this will _currently_ not be a problem. All user memory will be between 0000000000000000 - 00007fffffffffff. Also, remember that these are virtual addresses. If MPI_Alloc_mem is calling malloc/new (which can call mmap for large allocations), addresses will be from the high range, which can have the sign bit set. But all this is besides the point. The address map/choices are not fixed, and can change at any time (and do differ between operating systems), and MPI cannot change how the C library/runtime returns memory from malloc. (And then we're ignoring tools like valgrind/memory profilers/...) Ignoring the sign issue will very likely cause portability issues. Dries

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Jeff Hammond 28 Aug '12

28 Aug '12

I imagine that HPC-oriented operating systems do not provide virtual addresses in the problematic range anyways. Maybe VM randomization causes this but that's not common in HPC. In any case, MPI RMA requires MPI_Alloc_mem for portable performance. MPI_Alloc_mem should be capable of returning only pointers in the lower half of the address space, if by no other means than using mmap intelligently. I believe that this is only a problem on 32-bit systems where losing 2 of the 4 GB is a problem. On a 64-bit system, losing half of the 18 exabytes in the address space is unlikely to be a problem, given that an exabyte of DRAM would require something like a white dwarf to power, meaning that it is likely that the OS can find a way to VM map the physical memory present into the lower half of the address range. Jeff On Tue, Aug 28, 2012 at 12:31 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > This would mean that the user can't use half of the address space for > dynamic windows. Not really ideal. > > ~Jim. > > > On 8/28/12 10:05 AM, Jeff Hammond wrote: >> >> Is it possible for implementations to prevent the user from attaching >> memory to a dynamic window that would cause this overflow? For >> example, MPI_Win_attach could fail if the base+size indicated that >> addresses could be used that were greater than the capacity of >> MPI_Aint? That seems like a way better option than breaking backwards >> compatibility or attaching all sorts of caveats to the use of MPI_Aint >> that a user is never going to remember. >> >> Jeff >> >> On Tue, Aug 28, 2012 at 10:00 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >>> >>> Sorry, this wasn't too clear. A more detailed explanation of the problem >>> is: >>> >>> Passing a dynamic window displacement to an RMA operation currently >>> requires >>> the user to put an address (which is an unsigned quantity) into an >>> MPI_Aint >>> (which is signed). With the current spec, passing a negative >>> displacement >>> to an RMA operation is an error. When interpreted as signed integers, >>> dynamic window displacements (i.e. addresses) can be negative because the >>> unsigned to signed conversion effectively overflows. >>> >>> ~Jim. >>> >>> >>> On 8/28/12 9:49 AM, Jeff Hammond wrote: >>>> >>>> >>>> I do not understand this: >>>> >>>> "displacements associated with dynamic windows are addresses >>>> (unsigned) and can range into the space of negative integers" >>>> >>>> How can they "range into the space of negative integers" if they are >>>> unsigned? >>>> >>>> Jeff >>>> >>>> On Tue, Aug 28, 2012 at 9:46 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >>>>> >>>>> >>>>> Hi All, >>>>> >>>>> Currently, target displacements in put/get/acc/etc are defined to be >>>>> non-negative integers of type, MPI_Aint. MPI_Aint is defined to be a >>>>> signed >>>>> integer. However, displacements associated with dynamic windows are >>>>> addresses (unsigned) and can range into the space of negative integers. >>>>> >>>>> It seems like the least impact change would be to add a special case >>>>> that >>>>> the displacement argument will be interpreted as unsigned for dynamic >>>>> windows. Changing the interface to take an 'unsigned MPI_Aint' would >>>>> be >>>>> the >>>>> most clear (and preserve the existing and intended new semantics), but >>>>> would >>>>> break backward compatibility. >>>>> >>>>> Interested to hear your thoughts. >>>>> >>>>> ~Jim. >>> >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> >> >> > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Jim Dinan 28 Aug '12

28 Aug '12

This would mean that the user can't use half of the address space for dynamic windows. Not really ideal. ~Jim. On 8/28/12 10:05 AM, Jeff Hammond wrote: > Is it possible for implementations to prevent the user from attaching > memory to a dynamic window that would cause this overflow? For > example, MPI_Win_attach could fail if the base+size indicated that > addresses could be used that were greater than the capacity of > MPI_Aint? That seems like a way better option than breaking backwards > compatibility or attaching all sorts of caveats to the use of MPI_Aint > that a user is never going to remember. > > Jeff > > On Tue, Aug 28, 2012 at 10:00 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >> Sorry, this wasn't too clear. A more detailed explanation of the problem >> is: >> >> Passing a dynamic window displacement to an RMA operation currently requires >> the user to put an address (which is an unsigned quantity) into an MPI_Aint >> (which is signed). With the current spec, passing a negative displacement >> to an RMA operation is an error. When interpreted as signed integers, >> dynamic window displacements (i.e. addresses) can be negative because the >> unsigned to signed conversion effectively overflows. >> >> ~Jim. >> >> >> On 8/28/12 9:49 AM, Jeff Hammond wrote: >>> >>> I do not understand this: >>> >>> "displacements associated with dynamic windows are addresses >>> (unsigned) and can range into the space of negative integers" >>> >>> How can they "range into the space of negative integers" if they are >>> unsigned? >>> >>> Jeff >>> >>> On Tue, Aug 28, 2012 at 9:46 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >>>> >>>> Hi All, >>>> >>>> Currently, target displacements in put/get/acc/etc are defined to be >>>> non-negative integers of type, MPI_Aint. MPI_Aint is defined to be a >>>> signed >>>> integer. However, displacements associated with dynamic windows are >>>> addresses (unsigned) and can range into the space of negative integers. >>>> >>>> It seems like the least impact change would be to add a special case that >>>> the displacement argument will be interpreted as unsigned for dynamic >>>> windows. Changing the interface to take an 'unsigned MPI_Aint' would be >>>> the >>>> most clear (and preserve the existing and intended new semantics), but >>>> would >>>> break backward compatibility. >>>> >>>> Interested to hear your thoughts. >>>> >>>> ~Jim. >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > >

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Jeff Hammond 28 Aug '12

28 Aug '12

Is it possible for implementations to prevent the user from attaching memory to a dynamic window that would cause this overflow? For example, MPI_Win_attach could fail if the base+size indicated that addresses could be used that were greater than the capacity of MPI_Aint? That seems like a way better option than breaking backwards compatibility or attaching all sorts of caveats to the use of MPI_Aint that a user is never going to remember. Jeff On Tue, Aug 28, 2012 at 10:00 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > Sorry, this wasn't too clear. A more detailed explanation of the problem > is: > > Passing a dynamic window displacement to an RMA operation currently requires > the user to put an address (which is an unsigned quantity) into an MPI_Aint > (which is signed). With the current spec, passing a negative displacement > to an RMA operation is an error. When interpreted as signed integers, > dynamic window displacements (i.e. addresses) can be negative because the > unsigned to signed conversion effectively overflows. > > ~Jim. > > > On 8/28/12 9:49 AM, Jeff Hammond wrote: >> >> I do not understand this: >> >> "displacements associated with dynamic windows are addresses >> (unsigned) and can range into the space of negative integers" >> >> How can they "range into the space of negative integers" if they are >> unsigned? >> >> Jeff >> >> On Tue, Aug 28, 2012 at 9:46 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >>> >>> Hi All, >>> >>> Currently, target displacements in put/get/acc/etc are defined to be >>> non-negative integers of type, MPI_Aint. MPI_Aint is defined to be a >>> signed >>> integer. However, displacements associated with dynamic windows are >>> addresses (unsigned) and can range into the space of negative integers. >>> >>> It seems like the least impact change would be to add a special case that >>> the displacement argument will be interpreted as unsigned for dynamic >>> windows. Changing the interface to take an 'unsigned MPI_Aint' would be >>> the >>> most clear (and preserve the existing and intended new semantics), but >>> would >>> break backward compatibility. >>> >>> Interested to hear your thoughts. >>> >>> ~Jim. > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-rma] Target displacement sign issue
by Jim Dinan 28 Aug '12

28 Aug '12

Sorry, this wasn't too clear. A more detailed explanation of the problem is: Passing a dynamic window displacement to an RMA operation currently requires the user to put an address (which is an unsigned quantity) into an MPI_Aint (which is signed). With the current spec, passing a negative displacement to an RMA operation is an error. When interpreted as signed integers, dynamic window displacements (i.e. addresses) can be negative because the unsigned to signed conversion effectively overflows. ~Jim. On 8/28/12 9:49 AM, Jeff Hammond wrote: > I do not understand this: > > "displacements associated with dynamic windows are addresses > (unsigned) and can range into the space of negative integers" > > How can they "range into the space of negative integers" if they are unsigned? > > Jeff > > On Tue, Aug 28, 2012 at 9:46 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: >> Hi All, >> >> Currently, target displacements in put/get/acc/etc are defined to be >> non-negative integers of type, MPI_Aint. MPI_Aint is defined to be a signed >> integer. However, displacements associated with dynamic windows are >> addresses (unsigned) and can range into the space of negative integers. >> >> It seems like the least impact change would be to add a special case that >> the displacement argument will be interpreted as unsigned for dynamic >> windows. Changing the interface to take an 'unsigned MPI_Aint' would be the >> most clear (and preserve the existing and intended new semantics), but would >> break backward compatibility. >> >> Interested to hear your thoughts. >> >> ~Jim.

1 0