February 2011 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by Hubert Ritzdorf 25 Feb '11

25 Feb '11

I think we should extend the current view of the MPI_Shm handle. It shouldn't represent only a single shared memory region. It should be possible to handle several shared memory regions under a single MPI_shm handle. In this case, an attach function could look like MPI_Comm_shm_attach (MPI_Comm comm, MPI_shm shm_in, MPI_shm shmem_out, MPI_Info info) The Communicator "comm" would be any communicator created for example by an MPI_Comm_create_shmem () function call. "shm_in" might be any shared memory handle, located on the process call MPI_Comm_shm_attach. The function would create a new handle "shmem_out" containing the shared memory regions of all "shm_in" handles of comm. The communicator "comm" is associated to "shmem_out". The "shm_out" handle would represent N shared memory regions. An inquiry function MPI_Comm_shmem_num_regions could return the number of shared memory regions. An inquiry function MPI_Comm_shmem_region_info() could return a triple (baseptr, size and key) of the i-th shared memory region or a subset of shared memory regions. The unique key is required to identify the shared memory regions by the MPI processes. Hubert

1 0

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by William Gropp 22 Feb '11

22 Feb '11

I think that we should think more expandably - there are likely to be a number of different constraints that we need to satisfy, particularly 5 or 10 years from now. So I'd like to avoid either predefined names (like MPI_COMM_SHM) or a routine to create a communicator based on a single constraint. One simple extension would be MPI_Comm_create_shm( MPI_Comm incomm, MPI_Info properties, MPI_Comm *outcomm ) with the desired properties (e.g., samenode=true, samesocket=true, samel3cache=true, sametile=true, ...) in MPI_Info. Bill On Feb 22, 2011, at 2:24 PM, James Dinan wrote: > Hi Ron, > > On 2/17/11 12:34 PM, Ron Brightwell wrote: >> On 02/17/2011 09:18 AM, James Dinan wrote: >>> I tend to agree with this - MPI_COMM_SHM seems like it's standing >>> in for >>> more sophisticated functionality to help the programmer build >>> communicators based on system topology. There was similar feedback >>> in >>> the forum as well (although the discussion digressed into "what if >>> we >>> have DSM and how should MPI deal with that" which also raises >>> interesting questions). >>> >>> On future and even current systems the on-node topology is >>> becoming more >>> complex. We could, for example, want to map shared memory on a >>> per-socket basis rather than across all sockets on the node to avoid >>> sending data over the intra-node interconnect. >> >> This is somewhat where the proposal started. We tossed around the >> idea >> of providing MPI_COMM_SOCKET, MPI_COMM_NODE, MPI_COMM_CACHE, etc. >> Given >> that you can't assume any kind of affinity - processes may move >> freely >> between sockets - the complexity of figuring out a portable way to >> expose hierarchy was significant. Our options seemed to be to say >> nothing about how the application discovers which processes can share >> memory or only say as little as possible, since anything else is >> just a >> potential optimization. People weren't happy with the former since it >> impacted portability. > > Should we consider a function like MPI_Comm_split_shm(MPI_Comm parent, > MPI_Comm *comm_shm) since a constant MPI_COMM_SHM will not be able to > capture dynamic processes joining/leaving the computation? > > ~Jim. > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by James Dinan 22 Feb '11

22 Feb '11

Hi Ron, On 2/17/11 12:34 PM, Ron Brightwell wrote: > On 02/17/2011 09:18 AM, James Dinan wrote: >> I tend to agree with this - MPI_COMM_SHM seems like it's standing in for >> more sophisticated functionality to help the programmer build >> communicators based on system topology. There was similar feedback in >> the forum as well (although the discussion digressed into "what if we >> have DSM and how should MPI deal with that" which also raises >> interesting questions). >> >> On future and even current systems the on-node topology is becoming more >> complex. We could, for example, want to map shared memory on a >> per-socket basis rather than across all sockets on the node to avoid >> sending data over the intra-node interconnect. > > This is somewhat where the proposal started. We tossed around the idea > of providing MPI_COMM_SOCKET, MPI_COMM_NODE, MPI_COMM_CACHE, etc. Given > that you can't assume any kind of affinity - processes may move freely > between sockets - the complexity of figuring out a portable way to > expose hierarchy was significant. Our options seemed to be to say > nothing about how the application discovers which processes can share > memory or only say as little as possible, since anything else is just a > potential optimization. People weren't happy with the former since it > impacted portability. Should we consider a function like MPI_Comm_split_shm(MPI_Comm parent, MPI_Comm *comm_shm) since a constant MPI_COMM_SHM will not be able to capture dynamic processes joining/leaving the computation? ~Jim.

1 0

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by Hubert Ritzdorf 21 Feb '11

21 Feb '11

It would be possible to create an attach function corresponding to the MPI_Intercomm_create interface. For example: MPI_Comm_shm_attach (MI_Comm local_comm, int local_leader, MPI_Shm local_shm, MPI_Comm bridge_comm, int remote_leader, int tag, MPI_Comm *comm_out, MPI_Shm *shm_out) The function would attach (*) the shared memory region of the remote communicator, (*) create an (intra or inter) communicator containing the remote communicator and local communicator, and (*) would return a new shmem descriptor "shm_out" to the shared memory of the remote communicator. In this case, you could use MPI_Comm_shm_free still for freeing the shared memory. It's already collective and MPI can internally count the number of attachments. You would need a new constant MPI_SHM_NULL (null handle for MPI_Shm) which is a valid input for input argument "local_shmem" of MPI_Comm_shm_attach. In addition, there should be a function MPI_Shm_comm (MPI_Shm shm, MPI_Comm *comm) which returns the communicator to a MPI_shm handle and a function MPI_Shm_baseptr (MPI_shm shm, void *baseptr) which returns the baseptr to a MPI_shm handle. These functions would be required by external libraries and to get the "baseptr" of handle "shm_out" above. Hubert PS: The last argument of MPI_Comm_shm_alloc in the draft (version 1.0, Page 19, Line 17) must be MPI_shm *shm (instead of MPI_shm shm). >-----Original Message----- >From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org [mailto:mpi3-hybridpm- >bounces(a)lists.mpi-forum.org] On Behalf Of James Dinan >Sent: Friday, February 18, 2011 12:14 AM >To: mpi3-hybridpm(a)lists.mpi-forum.org >Subject: Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces > >Right, there is a mismatch between MPI handles and shared memory >handles. IPC shared memory relies on having a portable handle >(filename, etc) in order to support asynchronous attach/detach to/from >shared allocations. In order to fully support this, we would also need >to have a portable handle. There was an earlier suggestion of letting >the programmer supply a handle. We didn't seem to like it, but that >would get around the problem of needing to share and MPI handle. > >Another alternative would be to make attaching to a shmem region >collective. This is tricky too since it would be collective across a >new communicator that includes the added processes. We would also need >a collective detach. > > ~Jim. > >On 2/17/11 2:27 PM, Jeff Squyres wrote: >> In fact, chapter 2 explicitly says that handles only have meaning in the >local process. >> >> >> On Feb 17, 2011, at 3:19 PM, Hubert Ritzdorf wrote: >> >>>> >>>>> >>>>> MPI_Comm_shm_attach() is a little tricky - in the current proposal I >>>>> don't think we've defined the shm handle to be portable. Can we make >it >>>>> valid to send this to another node-local process in order for them to >>>>> call attach? >>>>> >>>> >>>> Is there an example in MPI where we allow sending an MPI handle to >>>> object between ranks? This seems like a bad idea to me. >>>> >>> >>> No, there isn't. >>> >>> Hubert >>> >>> _______________________________________________ >>> Mpi3-hybridpm mailing list >>> Mpi3-hybridpm(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> >> > >_______________________________________________ >Mpi3-hybridpm mailing list >Mpi3-hybridpm(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by James Dinan 17 Feb '11

17 Feb '11

Right, there is a mismatch between MPI handles and shared memory handles. IPC shared memory relies on having a portable handle (filename, etc) in order to support asynchronous attach/detach to/from shared allocations. In order to fully support this, we would also need to have a portable handle. There was an earlier suggestion of letting the programmer supply a handle. We didn't seem to like it, but that would get around the problem of needing to share and MPI handle. Another alternative would be to make attaching to a shmem region collective. This is tricky too since it would be collective across a new communicator that includes the added processes. We would also need a collective detach. ~Jim. On 2/17/11 2:27 PM, Jeff Squyres wrote: > In fact, chapter 2 explicitly says that handles only have meaning in the local process. > > > On Feb 17, 2011, at 3:19 PM, Hubert Ritzdorf wrote: > >>> >>>> >>>> MPI_Comm_shm_attach() is a little tricky - in the current proposal I >>>> don't think we've defined the shm handle to be portable. Can we make it >>>> valid to send this to another node-local process in order for them to >>>> call attach? >>>> >>> >>> Is there an example in MPI where we allow sending an MPI handle to >>> object between ranks? This seems like a bad idea to me. >>> >> >> No, there isn't. >> >> Hubert >> >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > >

1 0

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by Jeff Squyres 17 Feb '11

17 Feb '11

In fact, chapter 2 explicitly says that handles only have meaning in the local process. On Feb 17, 2011, at 3:19 PM, Hubert Ritzdorf wrote: >> >>> >>> MPI_Comm_shm_attach() is a little tricky - in the current proposal I >>> don't think we've defined the shm handle to be portable. Can we make it >>> valid to send this to another node-local process in order for them to >>> call attach? >>> >> >> Is there an example in MPI where we allow sending an MPI handle to >> object between ranks? This seems like a bad idea to me. >> > > No, there isn't. > > Hubert > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by Hubert Ritzdorf 17 Feb '11

17 Feb '11

> >> >> MPI_Comm_shm_attach() is a little tricky - in the current proposal I >> don't think we've defined the shm handle to be portable. Can we make it >> valid to send this to another node-local process in order for them to >> call attach? >> > >Is there an example in MPI where we allow sending an MPI handle to >object between ranks? This seems like a bad idea to me. > No, there isn't. Hubert

1 0

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by Ron Brightwell 17 Feb '11

17 Feb '11

On 02/17/2011 09:18 AM, James Dinan wrote: > Hi Hubert, > > I tend to agree with this - MPI_COMM_SHM seems like it's standing in for > more sophisticated functionality to help the programmer build > communicators based on system topology. There was similar feedback in > the forum as well (although the discussion digressed into "what if we > have DSM and how should MPI deal with that" which also raises > interesting questions). > > On future and even current systems the on-node topology is becoming more > complex. We could, for example, want to map shared memory on a > per-socket basis rather than across all sockets on the node to avoid > sending data over the intra-node interconnect. This is somewhat where the proposal started. We tossed around the idea of providing MPI_COMM_SOCKET, MPI_COMM_NODE, MPI_COMM_CACHE, etc. Given that you can't assume any kind of affinity - processes may move freely between sockets - the complexity of figuring out a portable way to expose hierarchy was significant. Our options seemed to be to say nothing about how the application discovers which processes can share memory or only say as little as possible, since anything else is just a potential optimization. People weren't happy with the former since it impacted portability. > > MPI_Comm_shm_attach() is a little tricky - in the current proposal I > don't think we've defined the shm handle to be portable. Can we make it > valid to send this to another node-local process in order for them to > call attach? > Is there an example in MPI where we allow sending an MPI handle to object between ranks? This seems like a bad idea to me. -Ron

1 0

Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces
by James Dinan 17 Feb '11

17 Feb '11

Hi Hubert, I tend to agree with this - MPI_COMM_SHM seems like it's standing in for more sophisticated functionality to help the programmer build communicators based on system topology. There was similar feedback in the forum as well (although the discussion digressed into "what if we have DSM and how should MPI deal with that" which also raises interesting questions). On future and even current systems the on-node topology is becoming more complex. We could, for example, want to map shared memory on a per-socket basis rather than across all sockets on the node to avoid sending data over the intra-node interconnect. MPI_Comm_shm_attach() is a little tricky - in the current proposal I don't think we've defined the shm handle to be portable. Can we make it valid to send this to another node-local process in order for them to call attach? Best, ~Jim. On 2/17/11 3:58 AM, Hubert Ritzdorf wrote: > Yes, this is the problem. I'm working also in a European project for > Climate modeling which use coupled applications/models for simulation. > They may start with a single process, which dynamically spawns new parallel applications/models depending on the simulation. Coupled application exchange 3d volume data and could put common chemistry data into shared memory. Thus, it makes sense to place processes of different applications/models on the same node (or on another node on systems which provide shared memory over several nodes). > In order to create the proposed MPI_COMM_SHM for MPI_COMM_WORLD in MPI_Init(), the mpi implementer would probably use also MPI_Comm_split() with an appropriate color. Thus, we should provide the corresponding functionality to the user and he/she can create the communicator. > > For dynamic applications, an additional MPI_Comm_shm_attach () function would be useful in order to attach to already existing shared memory regions. > > Hubert > > > >> -----Original Message----- >> From: Snir, Marc [mailto:[email protected]] >> Sent: Thursday, February 17, 2011 4:21 AM >> To: Hubert Ritzdorf >> Cc: mpi3-hybridpm(a)lists.mpi-forum.org; Hubert Ritzdorf >> Subject: Re: [Mpi3-hybridpm] SHMEM Interface in Draft External Interfaces >> >> Could explain the problem with dynamic process management? The only >> potential problem would be if dynamic process management is used to add >> new processes on existing nodes; if the allocation is in full nodes, I see >> no problem. >> >> Marc Snir >> 4323 Siebel Center, 201 N Goodwin, IL 61801 >> Tel (217) 244 6568 >> Web http://www.cs.uiuc.edu/homes/snir >> >> On 2/16/11 11:32 AM, "Hubert Ritzdorf" >> <hritzdorf(a)hpce.nec.com<mailto:[email protected]>> wrote: >> >> Hi, >> >> I have 2 notes to chapter MPI and Shared Memory in External Interface >> draft "ei-2-v1.0.pdf": >> >> (*) I am missing a function MPI_Shmem_flush(shm) >> which flushes the shared memory on not cache-coherent systems >> (corresponding to shmem functions shmem_udcflush() or >> shmem_udcflush_line()). >> >> (*) The concept of >> the intracommunicator MPI_COMM_SHM of all processes the local process >> can share >> doesn't work for dynamic process management. I would propose >> >> (*) to define a special constant MPI_COLOR_SHM and >> (*) to create a communicator of all processes the local process >> can share >> in communicator comm. with function >> >> MPI_Comm_split (comm., MPI_COLOR_SHM, key, comm._shm) >> >> Best regards >> >> Hubert Ritzdorf >> NEC Germany > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] MPI_HELPER_TEAM_CREATE description of INFO
by Bronis R. de Supinski 17 Feb '11

17 Feb '11

Doug: I think Pavan will be changing to something more along the lines of "static" and "dynamic", which seemed to sit better with the other forum members. In that case, I think the initial description is in terms of what the implementation does (assign work to the threads statically or dynamically). I agree that we should also state what the implications are from the application perspective (LEAVE is synchronizing across threads so all threads must call it for any to complete it versus LEAVE has thread-local completion semantics -- the differences are significant enough that I think two different LEAVE functions are appropriate), perhaps in an advice to users... Bronis On Thu, 17 Feb 2011, Douglas Miller wrote: > Hi Pavan, > > I'm still looking for a description of the INFO keys "balanced" and "singular" that I like. I guess what seems odd to me about the description in v1.0 is that "balanced" is described in terms of what the implementation does, while "singular" is in terms of what the application does. Wouldn't it be less confusing if both spoke to how the application writer can determine if their code fits a particular model? The way I imagine it, the application writer will have a block of code (in mind or in source files) that does MPI calls and they think it is a candidate for parallelism using a helper team. They would then compare their code to the requirements for "balanced" and "singular" and set them as appropriate. > > I guess we would then have "advice to implementers" that helps guide the implementer on how they should handle the INFO. > > would you agree? I can make an attempt to come up with some text for "balanced" from the application developer's point of view. > > thanks, > _______________________________________________ > Douglas Miller BlueGene Messaging Development > IBM Corp., Rochester, MN USA Bldg 030-2 A410 > dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM >

1 0