March 2011 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 09:22:49PM -0500, Rajeev Thakur wrote: > All can be used, but none are needed, right? We might need to talk about synch or flush. Some shared memory systems are loosely coupled and need membars to guarantee correct ordering. > Currently, in the unified model you can do: > > lock > put(A) > unlock > barrier barrier > x = A > > With shared memory, you could do > > A=10 > barrier barrier > x = A > > Right? Not really. The barrier and the x=A might be re-ordered. This is also the biggest issue with our definition of the unified model, some shared memory systems might actually not be able to support it :-(. We had a long discussion in Stuttgart about this. > If the above is true, it might be a bit confusing to users that in the > RMA chapter you use put, get, and synchronization functions for some > things and can directly read/write memory for some other things. Well, the standard is not a user documentation :-). But yes, having this special kind of window requires some effort to understand it. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 15 Mar '11

15 Mar '11

All can be used, but none are needed, right? Currently, in the unified model you can do: lock put(A) unlock barrier barrier x = A With shared memory, you could do A=10 barrier barrier x = A Right? If the above is true, it might be a bit confusing to users that in the RMA chapter you use put, get, and synchronization functions for some things and can directly read/write memory for some other things. Rajeev On Mar 15, 2011, at 9:17 PM, Torsten Hoefler wrote: > On Tue, Mar 15, 2011 at 09:03:40PM -0500, Rajeev Thakur wrote: >> Since processes sharing memory can/will access it directly using >> memory references (not put/get), and this proposal is for the unified >> memory model, we need to say whether any of the synchronization >> functions in the RMA chapter are needed at all for such usage. > Well, put/get to local windows are also allowed, so this would not be an > exception. It's even nicely defined because the chapter defines the > behavior for overlapping windows, and a shared window could be treated > semantically like multiple overlapping windows. I am not sure if all are > needed but I would say that all can be used. > > Yes, load/store accesses would probably be the most common use (plus > maybe the new (portable) atomics!). > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 06:44:49PM -0700, Bronis R. de Supinski wrote: > > Torsten: > > I suggested a joint meeting because that will make it > clear on the schedule. I was not thinking about adding > administrative overhead. I would like to be present. > I will likely miss the Tuesday morning sessions due to > a long telecon that I need to attend. Ok. This decision is up to the WG chairs. > I think some sort of "best effort" wording will be needed. > It is possible that processes are on the same node but > that no shared memory is available. In that case distinct > physical memory regions could be returned. Yes. > I am not quite following the group comments. Are you saying > that the different windows are returned to different shared > memory domains? Yes > I thought the window would be available to all processes but > physically shared memory is used when possible. Well, this would be an option too -- but I think it's not catching the gist of the idea. > How is the group returned? How do you query it? MPI_Win_get_group :-). Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 09:03:40PM -0500, Rajeev Thakur wrote: > Since processes sharing memory can/will access it directly using > memory references (not put/get), and this proposal is for the unified > memory model, we need to say whether any of the synchronization > functions in the RMA chapter are needed at all for such usage. Well, put/get to local windows are also allowed, so this would not be an exception. It's even nicely defined because the chapter defines the behavior for overlapping windows, and a shared window could be treated semantically like multiple overlapping windows. I am not sure if all are needed but I would say that all can be used. Yes, load/store accesses would probably be the most common use (plus maybe the new (portable) atomics!). Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Rajeev Thakur 15 Mar '11

15 Mar '11

Since processes sharing memory can/will access it directly using memory references (not put/get), and this proposal is for the unified memory model, we need to say whether any of the synchronization functions in the RMA chapter are needed at all for such usage. Rajeev On Mar 15, 2011, at 3:25 PM, Torsten Hoefler wrote: > Hello all, > > I apologize for the cross-post but this is of interest to two groups. I > have an alternative proposal for the shared memory support. I believe > that it would naturally integrate into the One Sided chapter as a > special type of window. This would save some additional function calls > and seems to make semantical definitions tighter and the whole standard > more orthogonal. Thanks to Brian, Bill, Marc, and Ron for preliminary > discussions of the idea. > > Here is the proposed new call: > > MPI_Win_allocate_shared(size, info, comm, *baseptr, *win); > > This would create a special window in the "unified" model which is > referencing the same physical memory on all processes in a shared memory > domain. The group of processes that share memory can be queried with > MPI_Win_get_group() and a new MPI_WIN_FLAVOR_SHARED would be added as > window type. > > The whole concept is even mightier because one could now support NCC > shared memory (I remember than Hubert mentioned that in one of the > discussions) if we would allow the "separate" window type for it. > However, this seems to introduce all kinds of complex interactions > (e.g., does everybody logically have it's own public copy of the *same* > memory). I'm not proposing this at this point but it might be worth > a discussion! > > This would make the new proposed calls "MPI_COMM_SHM_FREE(shm)" and > "MPI_SHM_FENCE" superfluous and would enable all synchronizations and > portable (!!) atomic operations that we defined in the RMA chapter > immediately. This would also remove the need to have a special > MPI_COMM_SHM as required in the hybrid proposal. > > Any comments are welcome. I can talk about this at the next meeting > (most likely in the RMA group on Tuesday). > > All the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Bronis R. de Supinski 15 Mar '11

15 Mar '11

Torsten: I suggested a joint meeting because that will make it clear on the schedule. I was not thinking about adding administrative overhead. I would like to be present. I will likely miss the Tuesday morning sessions due to a long telecon that I need to attend. I think some sort of "best effort" wording will be needed. It is possible that processes are on the same node but that no shared memory is available. In that case distinct physical memory regions could be returned. I am not quite following the group comments. Are you saying that the different windows are returned to different shared memory domains? I thought the window would be available to all processes but physically shared memory is used when possible. How is the group returned? How do you query it? Bronis On Tue, 15 Mar 2011, Torsten Hoefler wrote: > Bronis, > >> What happens if the processes do not have shared physical >> memory? > If the communicator had P processes, then the new window will have P > windows (or groups) with one process in each. > >> Is it just that physical shared memory is used >> when possible? What if multple nodes are included in comm? > Yes, the call is similar to a comm split where all nodes that share a > common memory have the same color and all nodes that don't have > different colors. It will create multiple distinct windows. The user can > query the group to see which processes share a window. > >> Will each node use shared memory? That seems fairly >> reasonable. > Yes, it's basically the same semantics as Ron's original proposal where > a new communicator MPI_COMM_SHM (?) was defined. This functionality will > be assimilated into one call and group query (which is more logical > because one doesn't necessarily need a full communicator). A new use for > groups :-). > >> Overall, I think this proposal is good. I do think a number of details >> have to be worked out before it is ready but I agree that discussing >> it at a Forum meeting is the way to go. > Yes, there will be some discussion, but it could be worth it. > >> Perhaps we could request a joint RMA/Hybrid WG session to discuss it? >> I think it is a mistake to cover it just within one group or the >> other. > Sure, that would be best. We could also just pick one or the other and > make sure all key people from the both working groups are present ;-). > Seems like less administrative overhead. > > Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 >

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

Bronis, > What happens if the processes do not have shared physical > memory? If the communicator had P processes, then the new window will have P windows (or groups) with one process in each. > Is it just that physical shared memory is used > when possible? What if multple nodes are included in comm? Yes, the call is similar to a comm split where all nodes that share a common memory have the same color and all nodes that don't have different colors. It will create multiple distinct windows. The user can query the group to see which processes share a window. > Will each node use shared memory? That seems fairly > reasonable. Yes, it's basically the same semantics as Ron's original proposal where a new communicator MPI_COMM_SHM (?) was defined. This functionality will be assimilated into one call and group query (which is more logical because one doesn't necessarily need a full communicator). A new use for groups :-). > Overall, I think this proposal is good. I do think a number of details > have to be worked out before it is ready but I agree that discussing > it at a Forum meeting is the way to go. Yes, there will be some discussion, but it could be worth it. > Perhaps we could request a joint RMA/Hybrid WG session to discuss it? > I think it is a mistake to cover it just within one group or the > other. Sure, that would be best. We could also just pick one or the other and make sure all key people from the both working groups are present ;-). Seems like less administrative overhead. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-hybridpm] [Mpi3-rma] Alternative Proposal for Shared Memory Support
by Bronis R. de Supinski 15 Mar '11

15 Mar '11

Torsten: What happens if the processes do not have shared physical memory? Is it just that physical shared memory is used when possible? What if multple nodes are included in comm? Will each node use shared memory? That seems fairly reasonable. Overall, I think this proposal is good. I do think a number of details have to be worked out before it is ready but I agree that discussing it at a Forum meeting is the way to go. Perhaps we could request a joint RMA/Hybrid WG session to discuss it? I think it is a mistake to cover it just within one group or the other. Bronis On Tue, 15 Mar 2011, Torsten Hoefler wrote: > Hello all, > > I apologize for the cross-post but this is of interest to two groups. I > have an alternative proposal for the shared memory support. I believe > that it would naturally integrate into the One Sided chapter as a > special type of window. This would save some additional function calls > and seems to make semantical definitions tighter and the whole standard > more orthogonal. Thanks to Brian, Bill, Marc, and Ron for preliminary > discussions of the idea. > > Here is the proposed new call: > > MPI_Win_allocate_shared(size, info, comm, *baseptr, *win); > > This would create a special window in the "unified" model which is > referencing the same physical memory on all processes in a shared memory > domain. The group of processes that share memory can be queried with > MPI_Win_get_group() and a new MPI_WIN_FLAVOR_SHARED would be added as > window type. > > The whole concept is even mightier because one could now support NCC > shared memory (I remember than Hubert mentioned that in one of the > discussions) if we would allow the "separate" window type for it. > However, this seems to introduce all kinds of complex interactions > (e.g., does everybody logically have it's own public copy of the *same* > memory). I'm not proposing this at this point but it might be worth > a discussion! > > This would make the new proposed calls "MPI_COMM_SHM_FREE(shm)" and > "MPI_SHM_FENCE" superfluous and would enable all synchronizations and > portable (!!) atomic operations that we defined in the RMA chapter > immediately. This would also remove the need to have a special > MPI_COMM_SHM as required in the hybrid proposal. > > Any comments are welcome. I can talk about this at the next meeting > (most likely in the RMA group on Tuesday). > > All the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

[Mpi3-hybridpm] Alternative Proposal for Shared Memory Support
by Torsten Hoefler 15 Mar '11

15 Mar '11

Hello all, I apologize for the cross-post but this is of interest to two groups. I have an alternative proposal for the shared memory support. I believe that it would naturally integrate into the One Sided chapter as a special type of window. This would save some additional function calls and seems to make semantical definitions tighter and the whole standard more orthogonal. Thanks to Brian, Bill, Marc, and Ron for preliminary discussions of the idea. Here is the proposed new call: MPI_Win_allocate_shared(size, info, comm, *baseptr, *win); This would create a special window in the "unified" model which is referencing the same physical memory on all processes in a shared memory domain. The group of processes that share memory can be queried with MPI_Win_get_group() and a new MPI_WIN_FLAVOR_SHARED would be added as window type. The whole concept is even mightier because one could now support NCC shared memory (I remember than Hubert mentioned that in one of the discussions) if we would allow the "separate" window type for it. However, this seems to introduce all kinds of complex interactions (e.g., does everybody logically have it's own public copy of the *same* memory). I'm not proposing this at this point but it might be worth a discussion! This would make the new proposed calls "MPI_COMM_SHM_FREE(shm)" and "MPI_SHM_FENCE" superfluous and would enable all synchronizations and portable (!!) atomic operations that we defined in the RMA chapter immediately. This would also remove the need to have a special MPI_COMM_SHM as required in the hybrid proposal. Any comments are welcome. I can talk about this at the next meeting (most likely in the RMA group on Tuesday). All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-hybridpm] [Fwd: Re: [Fwd: Re: Helper threads proposal]]
by Pavan Balaji 15 Mar '11

15 Mar '11

The previous attachment didn't go through. Resending. On 03/14/2011 09:51 PM, Pavan Balaji wrote: > > Thanks Howard. I'm cc'ing the Hybrid list as well. > > -- Pavan > > On 03/14/2011 01:34 PM, Howard Pritchard wrote: >> HI Pavan, >> >> Here's what one of cray's compiler guys says about the proposal. >> Based on he's feedback, I've no objections, except probably >> the example errors he notes should be fixed. >> >> Howard >> > > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0