September 2011 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] Fix for win_allocate_shared
by Torsten Hoefler 23 Sep '11

23 Sep '11

On Fri, 2011-09-23 at 12:56 -0500, Pavan Balaji wrote: > On 09/23/2011 12:47 PM, Barrett, Brian W wrote: > > It might not be able to. If non-coherent load/store is byte access > > (or word access), it can provide a separate public and private window > > in separate and meet the semantics. If it provides cacheline access > > to non-coherent space, it will not be able to provide shared memory > > across uncached space; such is the cost of standardization. > > Do we know any non-cache-coherent system that provides byte access (note > that word access is not sufficient)? > > What I'm arguing is that we are not giving any additional flexibility > for non-cache-coherent systems with this new "clarified semantics". In > fact, we are making it more difficult for future MPI versions to > actually define a semantics by saying that "multiple remote stores to > the same window have to work" -- we'll be stuck with this if we say that > in the standard now. I see your point but would like to postpone to the telecon. > Instead, if we leave it undefined, in MPI-3.1 (or 4.0), we might be able > to separate out local load/store and remote load/store operations and > say that multiple remote load/store operations is undefined (or > erroneous or whatever). > > So the options I'm proposing are: > > 1. Either define remote load/stores in SEPARATE. I think we can just define that concurrent accesses are undefined. > (or) > > 2. Leave it as undefined. > > Attempting a half-definition is making it worse than leaving it undefined. It's not a half-definition. It's very clear in that it's not supporting some architectures (that may not expose shared memory windows then). All the Best, Torsten

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Torsten Hoefler 23 Sep '11

23 Sep '11

On Fri, Sep 23, 2011 at 12:40:43PM -0500, Pavan Balaji wrote: > > On 09/23/2011 12:39 PM, Torsten Hoefler wrote: >> It clarifies semantics for separate (the first part) and the advice to >> implementers makes the semantic restriction explicit. > > What I mean is -- how will an MPI implementation on non-cache-coherent > system be able to expose shared memory with this restriction? Our decision was: Either expose non-cacheable memory or not at all. Well, the decision is not final, it's very easy to add a rule that says that the outcome of conflicting remote/local load/store accesses is undefined. Torsten PS: Sorry for my sluggish response, we're all stuck on an island with internet issues ... ;-) -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Barrett, Brian W 23 Sep '11

23 Sep '11

I disagree. It provides a rational target for future architectures and provides the ability of all cache coherent systems to support shared memory, something I'm not convinced the unified model does. Brian ----- Original Message ----- From: Pavan Balaji [mailto:[email protected]] Sent: Friday, September 23, 2011 11:56 AM To: mpi3-rma(a)lists.mpi-forum.org <mpi3-rma(a)lists.mpi-forum.org> Cc: Barrett, Brian W; Torsten Hoefler <htor(a)illinois.edu>; mpi3-hybridpm(a)lists.mpi-forum.org <mpi3-hybridpm(a)lists.mpi-forum.org> Subject: Re: [Mpi3-rma] Fix for win_allocate_shared On 09/23/2011 12:47 PM, Barrett, Brian W wrote: > It might not be able to. If non-coherent load/store is byte access > (or word access), it can provide a separate public and private window > in separate and meet the semantics. If it provides cacheline access > to non-coherent space, it will not be able to provide shared memory > across uncached space; such is the cost of standardization. Do we know any non-cache-coherent system that provides byte access (note that word access is not sufficient)? What I'm arguing is that we are not giving any additional flexibility for non-cache-coherent systems with this new "clarified semantics". In fact, we are making it more difficult for future MPI versions to actually define a semantics by saying that "multiple remote stores to the same window have to work" -- we'll be stuck with this if we say that in the standard now. Instead, if we leave it undefined, in MPI-3.1 (or 4.0), we might be able to separate out local load/store and remote load/store operations and say that multiple remote load/store operations is undefined (or erroneous or whatever). So the options I'm proposing are: 1. Either define remote load/stores in SEPARATE. (or) 2. Leave it as undefined. Attempting a half-definition is making it worse than leaving it undefined. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Pavan Balaji 23 Sep '11

23 Sep '11

On 09/23/2011 12:47 PM, Barrett, Brian W wrote: > It might not be able to. If non-coherent load/store is byte access > (or word access), it can provide a separate public and private window > in separate and meet the semantics. If it provides cacheline access > to non-coherent space, it will not be able to provide shared memory > across uncached space; such is the cost of standardization. Do we know any non-cache-coherent system that provides byte access (note that word access is not sufficient)? What I'm arguing is that we are not giving any additional flexibility for non-cache-coherent systems with this new "clarified semantics". In fact, we are making it more difficult for future MPI versions to actually define a semantics by saying that "multiple remote stores to the same window have to work" -- we'll be stuck with this if we say that in the standard now. Instead, if we leave it undefined, in MPI-3.1 (or 4.0), we might be able to separate out local load/store and remote load/store operations and say that multiple remote load/store operations is undefined (or erroneous or whatever). So the options I'm proposing are: 1. Either define remote load/stores in SEPARATE. (or) 2. Leave it as undefined. Attempting a half-definition is making it worse than leaving it undefined. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Barrett, Brian W 23 Sep '11

23 Sep '11

It might not be able to. If non-coherent load/store is byte access (or word access), it can provide a separate public and private window in separate and meet the semantics. If it provides cacheline access to non-coherent space, it will not be able to provide shared memory across uncached space; such is the cost of standardization. Brian ----- Original Message ----- From: Pavan Balaji [mailto:[email protected]] Sent: Friday, September 23, 2011 11:40 AM To: Torsten Hoefler <htor(a)illinois.edu> Cc: mpi3-rma(a)lists.mpi-forum.org <mpi3-rma(a)lists.mpi-forum.org>; mpi3-hybridpm(a)lists.mpi-forum.org <mpi3-hybridpm(a)lists.mpi-forum.org> Subject: Re: [Mpi3-rma] Fix for win_allocate_shared On 09/23/2011 12:39 PM, Torsten Hoefler wrote: > It clarifies semantics for separate (the first part) and the advice to > implementers makes the semantic restriction explicit. What I mean is -- how will an MPI implementation on non-cache-coherent system be able to expose shared memory with this restriction? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Pavan Balaji 23 Sep '11

23 Sep '11

On 09/23/2011 12:39 PM, Torsten Hoefler wrote: > It clarifies semantics for separate (the first part) and the advice to > implementers makes the semantic restriction explicit. What I mean is -- how will an MPI implementation on non-cache-coherent system be able to expose shared memory with this restriction? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Torsten Hoefler 23 Sep '11

23 Sep '11

On Fri, 2011-09-23 at 12:35 -0500, Pavan Balaji wrote: > On 09/23/2011 12:32 PM, Torsten Hoefler wrote: > > On Fri, 2011-09-23 at 12:25 -0500, Pavan Balaji wrote: > >> On 09/23/2011 02:39 AM, Torsten Hoefler wrote: > >>> "The semantic rules for shared memory windows in the MPI_RMA_SEPARATE > >>> model allow concurrent load/store accesses from two distinct processes > >>> to the same target window that is remote to both processes. An > >>> implementation cannot expose shared memory windows on hardware that does > >>> not support this access mode." > >> > >> This seems to suggest that it won't work for non-cache-coherent systems. > >> Didn't we decide against that? > > Brian probably wants to comment on this :-). > > > > Here's my recap: The issue was that many architectures may not be able > > to support UNIFIED. So we don't want to limit the SEPARATE model too > > much in order to still make it useful. If we later, based on our > > experience, find that we need this limitation to support existing > > architectures, then we can add the restriction. However, if we add it > > now, it may limiting the programming model unnecessarily. > > I'm not sure how this is better than my original suggestion of leaving > this as "undefined" for SEPARATE. That way, it is possible to define it > in the later MPI standards. But Jim and you didn't want to do that. It clarifies semantics for separate (the first part) and the advice to implementers makes the semantic restriction explicit. Best, Torsten

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Pavan Balaji 23 Sep '11

23 Sep '11

On 09/23/2011 12:32 PM, Torsten Hoefler wrote: > On Fri, 2011-09-23 at 12:25 -0500, Pavan Balaji wrote: >> On 09/23/2011 02:39 AM, Torsten Hoefler wrote: >>> "The semantic rules for shared memory windows in the MPI_RMA_SEPARATE >>> model allow concurrent load/store accesses from two distinct processes >>> to the same target window that is remote to both processes. An >>> implementation cannot expose shared memory windows on hardware that does >>> not support this access mode." >> >> This seems to suggest that it won't work for non-cache-coherent systems. >> Didn't we decide against that? > Brian probably wants to comment on this :-). > > Here's my recap: The issue was that many architectures may not be able > to support UNIFIED. So we don't want to limit the SEPARATE model too > much in order to still make it useful. If we later, based on our > experience, find that we need this limitation to support existing > architectures, then we can add the restriction. However, if we add it > now, it may limiting the programming model unnecessarily. I'm not sure how this is better than my original suggestion of leaving this as "undefined" for SEPARATE. That way, it is possible to define it in the later MPI standards. But Jim and you didn't want to do that. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Torsten Hoefler 23 Sep '11

23 Sep '11

On Fri, 2011-09-23 at 12:25 -0500, Pavan Balaji wrote: > On 09/23/2011 02:39 AM, Torsten Hoefler wrote: > > "The semantic rules for shared memory windows in the MPI_RMA_SEPARATE > > model allow concurrent load/store accesses from two distinct processes > > to the same target window that is remote to both processes. An > > implementation cannot expose shared memory windows on hardware that does > > not support this access mode." > > This seems to suggest that it won't work for non-cache-coherent systems. > Didn't we decide against that? Brian probably wants to comment on this :-). Here's my recap: The issue was that many architectures may not be able to support UNIFIED. So we don't want to limit the SEPARATE model too much in order to still make it useful. If we later, based on our experience, find that we need this limitation to support existing architectures, then we can add the restriction. However, if we add it now, it may limiting the programming model unnecessarily. All the Best, Torsten

1 0

Re: [Mpi3-rma] Fix for win_allocate_shared
by Pavan Balaji 23 Sep '11

23 Sep '11

On 09/23/2011 02:39 AM, Torsten Hoefler wrote: > "The semantic rules for shared memory windows in the MPI_RMA_SEPARATE > model allow concurrent load/store accesses from two distinct processes > to the same target window that is remote to both processes. An > implementation cannot expose shared memory windows on hardware that does > not support this access mode." This seems to suggest that it won't work for non-cache-coherent systems. Didn't we decide against that? -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0