- mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Single RMA synchronization for several window handles
by Rolf Rabenseifner 08 Aug '14

08 Aug '14

Dynamic windows do not provide the easy-to-use window base mechanism, i.e., the origin must know the absolute address on the target process. Same for target_disp_unit. Dynamic Windows also require that all have the same communicator (because same win handle). My proposal would allow full service for all RMA operations (and not only MPI_BOTTOM Quality), full service for Win_create or Win_allocate or Win_allocate_shared and together with combined synchronization. Is this worth to consider? Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Friday, August 8, 2014 5:06:14 PM > Subject: Re: [mpiwg-rma] Single RMA synchronization for several window handles > > Why not simply use MPI_Win_create_dynamic, attach the separate memory > regions, and use that MPI_Win? Adding an option for a collective > memory allocator would make this almost equivalent, and wouldn’t add > additional overhead beyond what is already requried. > > Bill > > William Gropp > Director, Parallel Computing Institute Thomas M. Siebel Chair in > Computer Science > University of Illinois Urbana-Champaign > > On Aug 8, 2014, at 9:18 AM, Rolf Rabenseifner < rabenseifner(a)hlrs.de > > wrote: > > > Jim, > > your topic "Reducing Synchronization Overhead Through Bundled > Communication" may get also help if we would be able to > combine several window handles to one superset window handle. > > If you have several windows for different buffers, but > only one synchronization pattern, e.g. MPI_Win_fince > then currently you must call MPI_Win_fence seperately > for each window handle. > > I would propose: > > MPI_Win_combine (/*IN*/ int count, > /*IN*/ MPI_Win *win, > /*IN*/ MPI_Comm comm, > /*OUT*/ MPI_Win *win_combined) > > The process group of comm must contain the process groups of all win. > The resulting window handle win_combined can be used only > in RMA synchronization calls and other helper routines, > but not for dynamic window allocation nor for any > RMA communication routine. > Collective synchronization routines must be called by all processes > of comm. > The semantics of an RMA synchronization call using win_combined > is defined as if the calls were seperately issued for > each window handle of the array win. If group handles > are part of the argument list of the synchronization call > then the appropriate subset is used for each window handle in win. > > What do you think about this idea for MPI-4.0? > > Best regards > Rolf > > ----- Original Message ----- > > > From: "Jim Dinan" < james.dinan(a)gmail.com > > To: "MPI WG Remote Memory Access working group" < > mpiwg-rma(a)lists.mpi-forum.org > > Sent: Thursday, August 7, 2014 4:08:32 PM > Subject: [mpiwg-rma] RMA Notification > > > > Hi All, > > > I have added a new proposal for an RMA notification extension: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/439 > > > I would like to bring this forward for the RMA WG to consider as an > MPI-4 extension. > > > Cheers, > ~Jim. > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Single RMA synchronization for several window handles
by William Gropp 08 Aug '14

08 Aug '14

Why not simply use MPI_Win_create_dynamic, attach the separate memory regions, and use that MPI_Win? Adding an option for a collective memory allocator would make this almost equivalent, and wouldn’t add additional overhead beyond what is already requried. Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Aug 8, 2014, at 9:18 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Jim, > > your topic "Reducing Synchronization Overhead Through Bundled > Communication" may get also help if we would be able to > combine several window handles to one superset window handle. > > If you have several windows for different buffers, but > only one synchronization pattern, e.g. MPI_Win_fince > then currently you must call MPI_Win_fence seperately > for each window handle. > > I would propose: > > MPI_Win_combine (/*IN*/ int count, > /*IN*/ MPI_Win *win, > /*IN*/ MPI_Comm comm, > /*OUT*/ MPI_Win *win_combined) > > The process group of comm must contain the process groups of all win. > The resulting window handle win_combined can be used only > in RMA synchronization calls and other helper routines, > but not for dynamic window allocation nor for any > RMA communication routine. > Collective synchronization routines must be called by all processes > of comm. > The semantics of an RMA synchronization call using win_combined > is defined as if the calls were seperately issued for > each window handle of the array win. If group handles > are part of the argument list of the synchronization call > then the appropriate subset is used for each window handle in win. > > What do you think about this idea for MPI-4.0? > > Best regards > Rolf > > ----- Original Message ----- >> From: "Jim Dinan" <james.dinan(a)gmail.com> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Thursday, August 7, 2014 4:08:32 PM >> Subject: [mpiwg-rma] RMA Notification >> >> >> >> Hi All, >> >> >> I have added a new proposal for an RMA notification extension: >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/439 >> >> >> I would like to bring this forward for the RMA WG to consider as an >> MPI-4 extension. >> >> >> Cheers, >> ~Jim. >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Single RMA synchronization for several window handles
by Rolf Rabenseifner 08 Aug '14

08 Aug '14

Jim, your topic "Reducing Synchronization Overhead Through Bundled Communication" may get also help if we would be able to combine several window handles to one superset window handle. If you have several windows for different buffers, but only one synchronization pattern, e.g. MPI_Win_fince then currently you must call MPI_Win_fence seperately for each window handle. I would propose: MPI_Win_combine (/*IN*/ int count, /*IN*/ MPI_Win *win, /*IN*/ MPI_Comm comm, /*OUT*/ MPI_Win *win_combined) The process group of comm must contain the process groups of all win. The resulting window handle win_combined can be used only in RMA synchronization calls and other helper routines, but not for dynamic window allocation nor for any RMA communication routine. Collective synchronization routines must be called by all processes of comm. The semantics of an RMA synchronization call using win_combined is defined as if the calls were seperately issued for each window handle of the array win. If group handles are part of the argument list of the synchronization call then the appropriate subset is used for each window handle in win. What do you think about this idea for MPI-4.0? Best regards Rolf ----- Original Message ----- > From: "Jim Dinan" <james.dinan(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, August 7, 2014 4:08:32 PM > Subject: [mpiwg-rma] RMA Notification > > > > Hi All, > > > I have added a new proposal for an RMA notification extension: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/439 > > > I would like to bring this forward for the RMA WG to consider as an > MPI-4 extension. > > > Cheers, > ~Jim. > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

RMA Notification
by Jim Dinan 07 Aug '14

07 Aug '14

Hi All, I have added a new proposal for an RMA notification extension: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/439 I would like to bring this forward for the RMA WG to consider as an MPI-4 extension. Cheers, ~Jim.

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by maik peterson 07 Aug '14

07 Aug '14

i really wonder how a team can work on rma technologies without understanding the basics of todays computer systems ? 2014-08-05 20:01 GMT+02:00 Dave Goodell (dgoodell) <dgoodell(a)cisco.com>: > On Aug 5, 2014, at 12:37 PM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> > wrote: > > > Dave, > > > > thank you for this helpful answer. > > > > My question was related to > > > >> The sentence p436:45-46 > >> "The order in which data is written is not > >> specified unless further synchronization is used." > > > > Your citation helps, because it tells the usual expectation: > >> 1. Each CPU will always perceive its own memory accesses as occurring > >> in program order. > > But in the MPI Standard nothing should be expected as usual. > > Right, since the MPI Standard is at least one more level removed from the > architecture. The behavior of an MPI program is also dependent on the > behavior of the C or Fortran programming language implementation, which > isn't addressed at all by the McKenney model. > > I think Bronis made an important, valid point earlier: > > Simply put, shared memory programming without compiler > > assistance is not something ordinary programmers should do. > > > > The text p436:43-48 may be modified into > > Advice to users. > > If accesses in the RMA unified model are not synchronized (with > > locks or flushes, see Section 11.5.3), load and store operations might > observe changes > > to the memory while they are in progress. The order in which data is > written is not > > specified unless further synchronization is used. This might lead to > inconsistent views > > on memory and programs that assume that a transfer is complete by only > checking > > parts of the message are erroneous. > > NEW: The only consistent view is that each process will always > > perceive its own memory accesses as occurring in program order. > > (End of advice to users.) > > > > But one can argue that this is already expected by everyone. > > In my opinion, it is better to expect on MPI shared Memory > > windows only semantics that is written black on White > > in the MPI Standard. > > I'll defer to the more active members of the RMA working group on this > proposed modification. In general, I'm really not fond of the entire MPI > RMA shared memory feature. I think it is something that should have been > left to a library outside of MPI, and that it's impossible to clearly > specify shared memory programming behavior in the MPI Forum's vernacular. > > -Dave > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] attendance at next Forum mtg
by Dave Goodell (dgoodell) 06 Aug '14

06 Aug '14

On Aug 6, 2014, at 11:29 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > Who from the RMA working group plans to attend the next Forum meeting in Japan? I'll be there, though I can't claim to have been particularly active in RMA lately. -Dave

1 0

Re: [mpiwg-rma] attendance at next Forum mtg
by Jeff Hammond 06 Aug '14

06 Aug '14

Jim will be there... On Wed, Aug 6, 2014 at 9:29 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > Who from the RMA working group plans to attend the next Forum meeting in Japan? > > Rajeev > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

attendance at next Forum mtg
by Rajeev Thakur 06 Aug '14

06 Aug '14

Who from the RMA working group plans to attend the next Forum meeting in Japan? Rajeev

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Dave Goodell (dgoodell) 05 Aug '14

05 Aug '14

On Aug 5, 2014, at 12:37 PM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dave, > > thank you for this helpful answer. > > My question was related to > >> The sentence p436:45-46 >> "The order in which data is written is not >> specified unless further synchronization is used." > > Your citation helps, because it tells the usual expectation: >> 1. Each CPU will always perceive its own memory accesses as occurring >> in program order. > But in the MPI Standard nothing should be expected as usual. Right, since the MPI Standard is at least one more level removed from the architecture. The behavior of an MPI program is also dependent on the behavior of the C or Fortran programming language implementation, which isn't addressed at all by the McKenney model. I think Bronis made an important, valid point earlier: > Simply put, shared memory programming without compiler > assistance is not something ordinary programmers should do. > The text p436:43-48 may be modified into > Advice to users. > If accesses in the RMA unified model are not synchronized (with > locks or flushes, see Section 11.5.3), load and store operations might observe changes > to the memory while they are in progress. The order in which data is written is not > specified unless further synchronization is used. This might lead to inconsistent views > on memory and programs that assume that a transfer is complete by only checking > parts of the message are erroneous. > NEW: The only consistent view is that each process will always > perceive its own memory accesses as occurring in program order. > (End of advice to users.) > > But one can argue that this is already expected by everyone. > In my opinion, it is better to expect on MPI shared Memory > windows only semantics that is written black on White > in the MPI Standard. I'll defer to the more active members of the RMA working group on this proposed modification. In general, I'm really not fond of the entire MPI RMA shared memory feature. I think it is something that should have been left to a library outside of MPI, and that it's impossible to clearly specify shared memory programming behavior in the MPI Forum's vernacular. -Dave

1 0

Re: [mpiwg-rma] Short question on the ccNUMA memory reality
by Rolf Rabenseifner 05 Aug '14

05 Aug '14

Dave, thank you for this helpful answer. My question was related to > The sentence p436:45-46 > "The order in which data is written is not > specified unless further synchronization is used." Your citation helps, because it tells the usual expectation: > 1. Each CPU will always perceive its own memory accesses as occurring > in program order. But in the MPI Standard nothing should be expected as usual. The text p436:43-48 may be modified into Advice to users. If accesses in the RMA unified model are not synchronized (with locks or flushes, see Section 11.5.3), load and store operations might observe changes to the memory while they are in progress. The order in which data is written is not specified unless further synchronization is used. This might lead to inconsistent views on memory and programs that assume that a transfer is complete by only checking parts of the message are erroneous. NEW: The only consistent view is that each process will always perceive its own memory accesses as occurring in program order. (End of advice to users.) But one can argue that this is already expected by everyone. In my opinion, it is better to expect on MPI shared Memory windows only semantics that is written black on White in the MPI Standard. Best regards Rolf ----- Original Message ----- > From: "Dave Goodell (dgoodell)" <dgoodell(a)cisco.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Cc: "Bill Long" <longb(a)cray.com> > Sent: Tuesday, August 5, 2014 6:10:01 PM > Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory reality > > On Aug 5, 2014, at 2:53 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > > > > On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner > > <rabenseifner(a)hlrs.de> wrote: > > > >> 1. Question (sequential consistency on one location): > >> ------------ > >> > >> Do I understand correctly that in the following patter > >> on a shared Memory or a ccNUM shared memory > >> > >> rank 0 rank 1 > >> print x > >> x=val_1 print x > >> x=val_2 print x > >> print x > >> > >> the print statements can print only in the following > >> sequence > >> - some times the previous value > >> - some times val_1 > >> - and after some time val_2 and it then stays to print val_2 > >> > >> and that it can never be that a sequence with val_2 before val_1 > >> can be produced, i.e., > >> old_val > >> val_2 > >> val_1 > >> val_2 > >> is impossible. > > > > When there are dependencies between statements, the compiler and > > architecture don’t reorder them. > > Rolf, for a bit of clarification on this, I use Paul McKenney's > theoretical "Ordering-Hostile Architecture" [1] as my guideline for > what an arbitrary architecture might choose to do. ***This is > separate from any language/compiler (non-)guarantees.*** > > [1] > http://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook-e1.pdf > > """ > C.6.1 Ordering-Hostile Architecture > A number of ordering-hostile computer systems have been produced over > the decades, but the nature of the hostility has always been > extremely subtle, and understanding it has required detailed > knowledge of the specific hardware. Rather than picking on a > specific hardware vendor, and as a presumably attractive alternative > to dragging the reader through detailed technical specifications, > let us instead de- > sign a mythical but maximally memory-ordering-hostile computer > architecture.4 > This hardware must obey the following ordering con- straints [McK05a, > McK05b]: > > 1. Each CPU will always perceive its own memory accesses as occurring > in program order. > > 2. CPUs will reorder a given operation with a store only if the two > operations are referencing different locations. > > 3. All of a given CPU’s loads preceding a read memory barrier > (smp_rmb()) will be perceived by all CPUs to precede any loads > following that read memory barrier. > > 4. All of a given CPU’s stores preceding a write mem- ory barrier > (smp_wmb()) will be perceived by all CPUs to precede any stores > following that write memory barrier. > > 5. All of a given CPU’s accesses (loads and stores) preceding a full > memory barrier (smp_mb()) will be perceived by all CPUs to precede > any accesses following that memory barrier. > """ > > Point 2 is particularly relevant here. > > >> Also other values are impossible, e.g., some bit or byte-mix > >> from val_1 and val_2. > > > > That is possible, unless you use some form of atomicity. > > Right, though such "weird" patterns are most likely to occur due to > some compiler behavior, not at the architectural level, assuming > that "x" is word-sized or smaller and naturally aligned. > > -Dave > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0