July 2012 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?
by William Gropp 26 Jul '12

26 Jul '12

Thanks for the support :) I too vote for #2. I note that applications do not need to use the disp_unit in accessing data under this model, but will need it if they are acting to replace PUT etc (as they might if the code is designed to work with and without shared memory). Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Jul 25, 2012, at 2:16 PM, Torsten Hoefler wrote: > Hi Jim, > > Bill's argument convinced me, I think we should have the displ_unit in > the query function because this would support future systems and comes > at essentially no cost. If the unit is the same on all processes, it can > be stored. This can be determined in O(log(P)) or O(1) time with an info > argument. Compression is trivial. > >> We should try to reach a decision on this issue before the end of the >> week. I think it would be helpful to hear some other opinions. So far, >> we have three options to choose from: >> >> 1) Require that the displacement unit be the same at all processes for >> MPI_Win_allocate_shared(). > This would never allow us to run across heterogeneous systems. I don't > know any heterogeneous system that supports the unified model but GPUs > are coming :-). I agree to Jim that coherency would be tricky, so this > hardware is unlikely. We could also always lift this restriction in > future MPI versions, so I'm not opposed. > >> 2) Allow different displacement units at each process in shared memory >> windows and allow processes to determine the displacement unit at remote >> processes via MPI_Win_shared_query(). This is in addition to the MPI-2 >> self-query already possible with MPI_Win_get_attr(MPI_WIN_DISP_UNIT). > Yes, this is what we have at this point. > >> 3) Allow different displacement units at each process in shared memory >> windows and provide only the MPI-2 self-query already possible with >> MPI_Win_get_attr(MPI_WIN_DISP_UNIT). Manual communication would be >> needed to exchange disp_unit information. > This would require a wording change. > >> A (very) rough summary of the discussion on this issue: >> >> * Given that non-uniform disp_unit values are currently allowed, >> processes may need to use disp_unit in address calculations. > They may check if all processes' disp_unit is the same, but that's > another complication in the interface. > >> * Requiring programmers to use the disp_unit at each process in address >> calculations diminishes the usefulness of the shared memory programming >> model. It's also not clear if there is a use case to support the >> non-uniform disp_unit model. > Agreed. > > I vote for #2. > > Best, > Torsten > > -- > ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] MPI-3.1 RMA planning
by Pavan Balaji 26 Jul '12

26 Jul '12

On 07/26/2012 05:05 AM, Torsten Hoefler wrote: > Why would an MPI implementation ever want to return SEPARATE if UNIFIED > would be supported? And why would a user want this? An MPI implementation might want to put the public copy in a special memory region, such as a symmetric address space or shared memory region. In this case, the public copy is different from the private copy. For the user's perspective, SEPARATE might be sufficient if (s)he is only planning to do PUT/GET operations, for example, in which case UNIFIED and SEPARATE might not have much of a semantic difference. >> I realize that the above recommendation is bad in many ways including: >> (1) we are changing the semantics of "asserts" here, and (2) this >> might force local PUT/GET operations to always go through the network >> as if they are issued by a remote process because of cache coherence >> issues and thus might lose performance. But this is just to get some >> discussion started. > I'm not sure about this ... but I think this needs some face-to-face time. I agree. I only have the problem, but no solution currently. Some group discussion (either telecon or face-to-face) will be good. > I would like to add some info arguments to the list of things to > consider. I don't have the full list at this point, but we have the > "same_size" argument for create and allocate. However, we have no > "same_displ_unit", which goes by the same rationale. We could also add > some info arguments to dynamic windows to mitigate some of the > implementation issues (allow optimized implementations on RMA systems). Sounds good. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3.1 RMA planning
by Torsten Hoefler 26 Jul '12

26 Jul '12

Pavan, > Now that the MPI-3.0 changes are almost done, I wanted to start > discussing what pieces we want to add into MPI-3.1. Here are a few > things I'd like to have considered: > > 1. We currently have an assert for MPI_MODE_NOSTORE. But there's no > assert for MPI_MODE_NOLOAD. > > Use case: This is an optimization for the separate window case. > Logically, when an epoch starts, the public copy of the window is > copied into the private copy to allow for load/store accesses. When > the epoch ends, the private copy is copied back into the public copy. > The MPI_MODE_NOSTORE assert allows the MPI implementation to get rid > of the copy at the close of the epoch. If both NOLOAD and NOSTORE > asserts are given, the copy at the start of the epoch can be avoided > as well. Makes sense. > 2. We currently allow the MPI implementation to create a window in > either the unified or separate mode. However, there is no way for the > user to say that SEPARATE is sufficient (like with threading modes). > For example, even on cache coherent systems, if the user calls > MPI_WIN_CREATE() the MPI implementation might choose to provide the > SEPARATE memory model and create a different public copy of the window > in symmetrically allocated memory or on a shared memory region. But > this will disable some capabilities for the user (e.g., simultaneous > load/store and PUT/GET accesses to different parts of the same window). > > My recommendation: define that UNIFIED > SEPARATE (which we already > kind of do, but don't explicitly state it), and allow the user to give > a "required" value as an info argument. The "provided" value of the > memory model can already be queried through the attribute. As with > threading modes, the MPI implementation is allowed to return a > "provided" value which is less than, greater than, or the same as the > "required" value. > > While the current use case might only be for WIN_CREATE, I'd also > recommend that we add it to all window creation routines for symmetry. > Also, WIN_ALLOCATE or WIN_ALLOCATE_SHARED might not run into such > restrictions, but the user might still prefer WIN_CREATE if (s)he > wants to expose already allocated memory that the application is > already working on. Why would an MPI implementation ever want to return SEPARATE if UNIFIED would be supported? And why would a user want this? > 3. In our semantics and correctness section, for the SEPARATE memory > model, we state that a PUT/ACCUMULATE must not access a target window > once a local update or PUT/GET to an overlapping window has started. > I'd recommend that we change this to "... to the same or an > overlapping window has started", since it obviously applies to the > same window as well, and not just overlapping windows. That's close to a ticket 0 :). > 4. Currently, in the SEPARATE memory model, we do not allow > PUT/ACCUMULATE to a target window once a local update or PUT/GET to > that window has started. This makes sense for local load/store, but > might not for local PUT/GET since those could directly target the > public copy. However, when a process opens an epoch on its local > window, the MPI implementation has to assume that it will perform > load/store operations as well. The user can provide MPI_MODE_NOLOAD > and MPI_MODE_NOSTORE assertions, but these are just hints and the MPI > implementation is allowed to ignore them. Thus, from the user's > perspective, even if (s)he gives the NOLOAD/NOSTORE guarantees, such > accesses are still not allowed. > > Possible (very lame) recommendation: Make NOLOAD/NOSTORE assertions > mandatory for the MPI implementation to honor. Thus, when the > application gives these assertions, it is no longer restricted from > simultaneously performing PUT/GET operations locally and remotely to > disjoint locations of the window. > > I realize that the above recommendation is bad in many ways including: > (1) we are changing the semantics of "asserts" here, and (2) this > might force local PUT/GET operations to always go through the network > as if they are issued by a remote process because of cache coherence > issues and thus might lose performance. But this is just to get some > discussion started. I'm not sure about this ... but I think this needs some face-to-face time. I would like to add some info arguments to the list of things to consider. I don't have the full list at this point, but we have the "same_size" argument for create and allocate. However, we have no "same_displ_unit", which goes by the same rationale. We could also add some info arguments to dynamic windows to mitigate some of the implementation issues (allow optimized implementations on RMA systems). All the Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

[Mpi3-rma] MPI-3.1 RMA planning
by Pavan Balaji 25 Jul '12

25 Jul '12

Folks, Now that the MPI-3.0 changes are almost done, I wanted to start discussing what pieces we want to add into MPI-3.1. Here are a few things I'd like to have considered: 1. We currently have an assert for MPI_MODE_NOSTORE. But there's no assert for MPI_MODE_NOLOAD. Use case: This is an optimization for the separate window case. Logically, when an epoch starts, the public copy of the window is copied into the private copy to allow for load/store accesses. When the epoch ends, the private copy is copied back into the public copy. The MPI_MODE_NOSTORE assert allows the MPI implementation to get rid of the copy at the close of the epoch. If both NOLOAD and NOSTORE asserts are given, the copy at the start of the epoch can be avoided as well. 2. We currently allow the MPI implementation to create a window in either the unified or separate mode. However, there is no way for the user to say that SEPARATE is sufficient (like with threading modes). For example, even on cache coherent systems, if the user calls MPI_WIN_CREATE() the MPI implementation might choose to provide the SEPARATE memory model and create a different public copy of the window in symmetrically allocated memory or on a shared memory region. But this will disable some capabilities for the user (e.g., simultaneous load/store and PUT/GET accesses to different parts of the same window). My recommendation: define that UNIFIED > SEPARATE (which we already kind of do, but don't explicitly state it), and allow the user to give a "required" value as an info argument. The "provided" value of the memory model can already be queried through the attribute. As with threading modes, the MPI implementation is allowed to return a "provided" value which is less than, greater than, or the same as the "required" value. While the current use case might only be for WIN_CREATE, I'd also recommend that we add it to all window creation routines for symmetry. Also, WIN_ALLOCATE or WIN_ALLOCATE_SHARED might not run into such restrictions, but the user might still prefer WIN_CREATE if (s)he wants to expose already allocated memory that the application is already working on. 3. In our semantics and correctness section, for the SEPARATE memory model, we state that a PUT/ACCUMULATE must not access a target window once a local update or PUT/GET to an overlapping window has started. I'd recommend that we change this to "... to the same or an overlapping window has started", since it obviously applies to the same window as well, and not just overlapping windows. 4. Currently, in the SEPARATE memory model, we do not allow PUT/ACCUMULATE to a target window once a local update or PUT/GET to that window has started. This makes sense for local load/store, but might not for local PUT/GET since those could directly target the public copy. However, when a process opens an epoch on its local window, the MPI implementation has to assume that it will perform load/store operations as well. The user can provide MPI_MODE_NOLOAD and MPI_MODE_NOSTORE assertions, but these are just hints and the MPI implementation is allowed to ignore them. Thus, from the user's perspective, even if (s)he gives the NOLOAD/NOSTORE guarantees, such accesses are still not allowed. Possible (very lame) recommendation: Make NOLOAD/NOSTORE assertions mandatory for the MPI implementation to honor. Thus, when the application gives these assertions, it is no longer restricted from simultaneously performing PUT/GET operations locally and remotely to disjoint locations of the window. I realize that the above recommendation is bad in many ways including: (1) we are changing the semantics of "asserts" here, and (2) this might force local PUT/GET operations to always go through the network as if they are issued by a remote process because of cache coherence issues and thus might lose performance. But this is just to get some discussion started. Look forward to your comments. Regards, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?
by Rolf Rabenseifner 25 Jul '12

25 Jul '12

I would like to see an additional advice to users that tell that the usual way to use disp_unit as extent of the same datatype on all processes of a window, i.e., to have the identical value on all processes on homogeneous systems. Without such ann advice, users tend to "misuse" such features. One may add a rationale that non-identical values may need O(p) memory to store the disp_uni information on each process, i.e., on all processes together O(p**2) memory. Such a hint can be added to the existing advice: MPI-2.2 p337:1-5: Advice to users. Common choices for disp_unit are 1 (no scaling), and (in C syntax) sizeof(type), for a window that consists of an array of elements of type type. The later choice will allow one to use array indices in RMA calls, and have those scaled correctly to byte displacements, even in a heterogeneous environment. (End of advice to users.) I asked earlier for such an advice, but it was lost in the discussion of the final meaning of the interface. If the RMA roup can formulate such an advice, still enough general, I expect this would be helpful. This is of course true for all routines using disp_unit. Rolf ----- Original Message ----- > From: "Torsten Hoefler" <htor(a)illinois.edu> > To: "MPI 3.0 Remote Memory Access working group" <mpi3-rma(a)lists.mpi-forum.org> > Sent: Wednesday, July 25, 2012 9:16:50 PM > Subject: Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query? > Hi Jim, > > Bill's argument convinced me, I think we should have the displ_unit in > the query function because this would support future systems and comes > at essentially no cost. If the unit is the same on all processes, it > can > be stored. This can be determined in O(log(P)) or O(1) time with an > info > argument. Compression is trivial. > > > We should try to reach a decision on this issue before the end of > > the > > week. I think it would be helpful to hear some other opinions. So > > far, > > we have three options to choose from: > > > > 1) Require that the displacement unit be the same at all processes > > for > > MPI_Win_allocate_shared(). > This would never allow us to run across heterogeneous systems. I don't > know any heterogeneous system that supports the unified model but GPUs > are coming :-). I agree to Jim that coherency would be tricky, so this > hardware is unlikely. We could also always lift this restriction in > future MPI versions, so I'm not opposed. > > > 2) Allow different displacement units at each process in shared > > memory > > windows and allow processes to determine the displacement unit at > > remote > > processes via MPI_Win_shared_query(). This is in addition to the > > MPI-2 > > self-query already possible with > > MPI_Win_get_attr(MPI_WIN_DISP_UNIT). > Yes, this is what we have at this point. > > > 3) Allow different displacement units at each process in shared > > memory > > windows and provide only the MPI-2 self-query already possible with > > MPI_Win_get_attr(MPI_WIN_DISP_UNIT). Manual communication would be > > needed to exchange disp_unit information. > This would require a wording change. > > > A (very) rough summary of the discussion on this issue: > > > > * Given that non-uniform disp_unit values are currently allowed, > > processes may need to use disp_unit in address calculations. > They may check if all processes' disp_unit is the same, but that's > another complication in the interface. > > > * Requiring programmers to use the disp_unit at each process in > > address > > calculations diminishes the usefulness of the shared memory > > programming > > model. It's also not clear if there is a use case to support the > > non-uniform disp_unit model. > Agreed. > > I vote for #2. > > Best, > Torsten > > -- > ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?
by Torsten Hoefler 25 Jul '12

25 Jul '12

Hi Jim, Bill's argument convinced me, I think we should have the displ_unit in the query function because this would support future systems and comes at essentially no cost. If the unit is the same on all processes, it can be stored. This can be determined in O(log(P)) or O(1) time with an info argument. Compression is trivial. > We should try to reach a decision on this issue before the end of the > week. I think it would be helpful to hear some other opinions. So far, > we have three options to choose from: > > 1) Require that the displacement unit be the same at all processes for > MPI_Win_allocate_shared(). This would never allow us to run across heterogeneous systems. I don't know any heterogeneous system that supports the unified model but GPUs are coming :-). I agree to Jim that coherency would be tricky, so this hardware is unlikely. We could also always lift this restriction in future MPI versions, so I'm not opposed. > 2) Allow different displacement units at each process in shared memory > windows and allow processes to determine the displacement unit at remote > processes via MPI_Win_shared_query(). This is in addition to the MPI-2 > self-query already possible with MPI_Win_get_attr(MPI_WIN_DISP_UNIT). Yes, this is what we have at this point. > 3) Allow different displacement units at each process in shared memory > windows and provide only the MPI-2 self-query already possible with > MPI_Win_get_attr(MPI_WIN_DISP_UNIT). Manual communication would be > needed to exchange disp_unit information. This would require a wording change. > A (very) rough summary of the discussion on this issue: > > * Given that non-uniform disp_unit values are currently allowed, > processes may need to use disp_unit in address calculations. They may check if all processes' disp_unit is the same, but that's another complication in the interface. > * Requiring programmers to use the disp_unit at each process in address > calculations diminishes the usefulness of the shared memory programming > model. It's also not clear if there is a use case to support the > non-uniform disp_unit model. Agreed. I vote for #2. Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?
by Jim Dinan 25 Jul '12

25 Jul '12

Hi All, We should try to reach a decision on this issue before the end of the week. I think it would be helpful to hear some other opinions. So far, we have three options to choose from: 1) Require that the displacement unit be the same at all processes for MPI_Win_allocate_shared(). 2) Allow different displacement units at each process in shared memory windows and allow processes to determine the displacement unit at remote processes via MPI_Win_shared_query(). This is in addition to the MPI-2 self-query already possible with MPI_Win_get_attr(MPI_WIN_DISP_UNIT). 3) Allow different displacement units at each process in shared memory windows and provide only the MPI-2 self-query already possible with MPI_Win_get_attr(MPI_WIN_DISP_UNIT). Manual communication would be needed to exchange disp_unit information. A (very) rough summary of the discussion on this issue: * Given that non-uniform disp_unit values are currently allowed, processes may need to use disp_unit in address calculations. * Requiring programmers to use the disp_unit at each process in address calculations diminishes the usefulness of the shared memory programming model. It's also not clear if there is a use case to support the non-uniform disp_unit model. Thanks, ~Jim.

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?
by Jim Dinan 24 Jul '12

24 Jul '12

You can already query this in MPI-2 using MPI_Win_get_attr(win, MPI_WIN_DISP_UNIT, &disp_unit, &flag). ~Jim. On 07/24/2012 10:12 PM, Rajeev Thakur wrote: > Even the Win_query_shared function returns a process's own disp_unit, which would need to be communicated to other processes. > > Rajeev > > On Jul 24, 2012, at 7:32 PM, William Gropp wrote: > >> This does require some two-sided communication, which seems unfortunate for a one-sided interface. >> >> Bill >> >> William Gropp >> Director, Parallel Computing Institute >> Deputy Director for Research >> Institute for Advanced Computing Applications and Technologies >> Paul and Cynthia Saylor Professor of Computer Science >> University of Illinois Urbana-Champaign >> >> >> >> On Jul 24, 2012, at 2:10 PM, Dave Goodell wrote: >> >>> I think Jim's point (correct me if I'm wrong, Jim) is that there is a third option, which is to allow processes to query their own disp_unit with the MPI_WIN_DISP_UNIT attr and then communicate that information manually. >>> >>> -Dave >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?
by Rajeev Thakur 24 Jul '12

24 Jul '12

Even the Win_query_shared function returns a process's own disp_unit, which would need to be communicated to other processes. Rajeev On Jul 24, 2012, at 7:32 PM, William Gropp wrote: > This does require some two-sided communication, which seems unfortunate for a one-sided interface. > > Bill > > William Gropp > Director, Parallel Computing Institute > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > On Jul 24, 2012, at 2:10 PM, Dave Goodell wrote: > >> I think Jim's point (correct me if I'm wrong, Jim) is that there is a third option, which is to allow processes to query their own disp_unit with the MPI_WIN_DISP_UNIT attr and then communicate that information manually. >> >> -Dave > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?
by William Gropp 24 Jul '12

24 Jul '12

This does require some two-sided communication, which seems unfortunate for a one-sided interface. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Jul 24, 2012, at 2:10 PM, Dave Goodell wrote: > I think Jim's point (correct me if I'm wrong, Jim) is that there is a third option, which is to allow processes to query their own disp_unit with the MPI_WIN_DISP_UNIT attr and then communicate that information manually. > > -Dave

1 0