August 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] ticket 456
by Rolf Rabenseifner 31 Aug '14

31 Aug '14

Dear Jeff, answers/comments are below inlined: ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Sunday, August 31, 2014 7:42:51 AM > Subject: Re: [mpiwg-rma] ticket 456 > > On Sat, Aug 30, 2014 at 7:49 AM, Rolf Rabenseifner > <rabenseifner(a)hlrs.de> wrote: > > Jeff, > > > > great! > > > > Several comments: > > > > ________ > > About your > > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > > > line 48-50 (barrier + 2nd fence) are not needed. > > That's why the Barrier is commented out :-) > > The extra Fence is there because I like to be pedantic and do not > assume that it is always visible to the programmer that a prior Fence > call has been made. If the example is used for testing whether an MPI implementation fulfills #456, then it's synchronization-calls should be as minimal as possible. > > > fence on line 42 may be needed to guarantee that any initialization > > is finished. > > Line 46 completes the initialization epoch. What I wanted to say: after line 42, the window has some value which is visible for all processes and is probably not 42. > > fence on line 52 seems to be also not needed: > > - not according to #456 > > - not if substituting the store by a MPI_Put. > > It would be only needed, if the load on line 51 would be a MPI_Get. > > This is why I hate Win_fence. It was poorly designed. It has no > beginning or end. It would have been so much better to have > Fence_begin, Fence_end and Fence_middle (or whatever it could have > been called - it has the same notion of Win_sync of ending and > beginning an epoch). I understand, but it is as it is. > > ________ > > About your > > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > > > Line 17: I would recommend MPI_Abort > > Lines 48, 50, 58, 59 are not needed. > > Result: > > > > 46 if (rank==0) { > > 49 *shptr = 42; > > 52 MPI_Win_post(MPI_GROUP_ONE, 0, shwin); > > 53 MPI_Win_wait(shwin); > > 55 } else if (rank==1) { > > 56 int lint; > > 61 MPI_Win_start(MPI_GROUP_ZERO, 0, shwin); > > 62 lint = *rptr; > > 63 MPI_Win_complete(shwin); > > You may be right. I never use PSCW and it was hard enough to come up > with that example as it was. In https://fs.hlrs.de/projects/par/par_prog_ws/practical/MPI.zip in subdirectory mpi/course/C/1sided/ in the file halo_1sided_store_win_alloc_shared_pscw.c you can find a PSCW example that did not worked with Cray's MPI because it removed internally the needed synchronization with Post and Start because there were no RMA calls, or because any synchonization is delayed to the late execution of the RMA calls. > > This example would illustrate the write-read-rule of #456 > > (i.e. pattern with variable A) > > A=val_1 > > Sync-to-P1 --> Sync-from-P0 > > load(A) > > with > > Sync-to-P1 --> Sync-from-P0 > > being > > 2. MPI_Win_post --> MPI_Win_start > > > > This example would also work when substituting line 62 > > by MPI_Get. > > > > All Patterns in #456 are based on corresponding > > Patterns with MPI_Get and MPI_Put. > > > > If this example does not work with an existing MPI libary, > > then because it optimizes the synchronization away > > because there are no RMA calls. > > I explicitly verified that MPICH has the memory barriers required to > make this example work. I leave it as an exercise to the reader to > examine other implementations :-) If you are doing it as above, i.e., only one PSCW and store before post and load after start, then you may get some surprise. I'm on travel without access to "my" Cray. > > _________ > > About your > > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > > > Perfect. > > This example illustrates again the write-read-rule of #456 > > (i.e. pattern with variable A) > > with > > Sync-to-P1 --> Sync-from-P0 > > being > > 4. MPI_Win_sync > > Any-process-sync- > > -from-P0-to-P1 --> Any-process-sync- > > -from-P0-to-P1 4) > > MPI_Win_sync > > > > _________ > > About your > > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > > > Perfect. > > This example illustrates again the write-read-rule of #456 > > (i.e. pattern with variable A), > > here the lock/unlock pattern with exclusive lock in P1. > > > > _________ > > About your > > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > > > I would expect that it works, > > but it is not based on a pattern that currently exists in #456. > > > > This is a bug of #456. > > > > The current wording > > > > Patterns with lock/unlock synchronization: > > > > Within passive target communication, two locks L1 and L2 > > may be scheduled L1 before L2 or L2 before L1. In the > > following patterns, the arrow means that the lock in P0 > > was scheduled before the lock in P1. > > > > is not good enough. It should read: > > > > Within passive target communication, two locks L1 and L2 > > may be scheduled "L1 released before L2 granted" or > > "L2 released before L1 granted" in the case > > of two locks with at least one exclusive, or in the case of any > > lock with an > > additional synchronization (e.g., with point-to-point or > > collective communication) in between. In the > > following patterns, the arrow means that the lock in P0 > > was released before the lock in P1 was granted, independent > > of the method used to achieve this schedule. > > > > In the patters itself, I'll remove the word "shared" and > > "exclusive". > > > > A nice examlple would also be > > > > Process P0 Process P1 > > A=0 B=0 > > MPI_Win_fence MPI_Win_fence > > MPI_Win_lock MPI_Win_lock > > exclusive exclusive > > A=val_1 B=val_2 > > Bnew=load(B) Anew=load(A) > > MPI_Win_unlock MPI_Win_unlock > > I have not started on examples that mix sync modes but I can try to > do that next week. The fence was only to be sure that the initialization is visible on both processes. > > The two rules write-read and read-write together > > guarantee that either > > - in P0 Bnew=0 and in P1 Anew=val_1 > > or > > - in P0 Bnew=val_2 and in P1 Anew=0 > > > > But this is a combination of several basic rules in #456. > > > > _________ > > In General: > > > > #456 tries to define these 3*4 + 3 = 15 patterns, > > which should be enough as far as I oversee, > > plus two examples, 11.13 to show the translation of > > the win_sync write-read-pattern and read-write-pattern > > in real software, and 11.14 the pairing of memory barriers > > without further synchronization between the processes. > > > > Your test examples should be very helpful to test, > > whether an MPI library fulfills these patterns > > and to illustrate the compressed wording in #456. > > Yeah, the tests are necessary but not sufficient to confirm > implementation correctness. So far I have only run them on x86 > laptop, which is probably the easiest case for which an > implementation can be correct. Yes, but helpful. I started the discussion after I detected that my PSCW test-program does not work with mpich and Cray's MPI. Later on I got the information that Hubert detected this inconsistence of the MPI standard already two years ago. Best regards Rolf > > Best, > > Jeff > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "Jeff Hammond" <jeff.science(a)gmail.com> > >> To: "MPI Forum" <mpiwg-rma(a)lists.mpi-forum.org> > >> Sent: Friday, August 29, 2014 8:25:40 PM > >> Subject: [mpiwg-rma] ticket 456 > >> > >> Rolf, > >> > >> I find you pseudocode confusion, hence I am creating examples in C > >> that I can compile and run. I will try to come up with a case for > >> everyone of the examples in your ticket. See > >> https://github.com/jeffhammond/HPCInfo/tree/master/mpi/rma/shared-memory-wi…. > >> > >> Best, > >> > >> Jeff > >> > >> > >> -- > >> Jeff Hammond > >> jeff.science(a)gmail.com > >> http://jeffhammond.github.io/ > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] ticket 456
by Jeff Hammond 31 Aug '14

31 Aug '14

On Sat, Aug 30, 2014 at 7:49 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Jeff, > > great! > > Several comments: > > ________ > About your > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > line 48-50 (barrier + 2nd fence) are not needed. That's why the Barrier is commented out :-) The extra Fence is there because I like to be pedantic and do not assume that it is always visible to the programmer that a prior Fence call has been made. > fence on line 42 may be needed to guarantee that any initialization > is finished. Line 46 completes the initialization epoch. > fence on line 52 seems to be also not needed: > - not according to #456 > - not if substituting the store by a MPI_Put. > It would be only needed, if the load on line 51 would be a MPI_Get. This is why I hate Win_fence. It was poorly designed. It has no beginning or end. It would have been so much better to have Fence_begin, Fence_end and Fence_middle (or whatever it could have been called - it has the same notion of Win_sync of ending and beginning an epoch). > ________ > About your > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > Line 17: I would recommend MPI_Abort > Lines 48, 50, 58, 59 are not needed. > Result: > > 46 if (rank==0) { > 49 *shptr = 42; > 52 MPI_Win_post(MPI_GROUP_ONE, 0, shwin); > 53 MPI_Win_wait(shwin); > 55 } else if (rank==1) { > 56 int lint; > 61 MPI_Win_start(MPI_GROUP_ZERO, 0, shwin); > 62 lint = *rptr; > 63 MPI_Win_complete(shwin); You may be right. I never use PSCW and it was hard enough to come up with that example as it was. > This example would illustrate the write-read-rule of #456 > (i.e. pattern with variable A) > A=val_1 > Sync-to-P1 --> Sync-from-P0 > load(A) > with > Sync-to-P1 --> Sync-from-P0 > being > 2. MPI_Win_post --> MPI_Win_start > > This example would also work when substituting line 62 > by MPI_Get. > > All Patterns in #456 are based on corresponding > Patterns with MPI_Get and MPI_Put. > > If this example does not work with an existing MPI libary, > then because it optimizes the synchronization away > because there are no RMA calls. I explicitly verified that MPICH has the memory barriers required to make this example work. I leave it as an exercise to the reader to examine other implementations :-) > _________ > About your > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > Perfect. > This example illustrates again the write-read-rule of #456 > (i.e. pattern with variable A) > with > Sync-to-P1 --> Sync-from-P0 > being > 4. MPI_Win_sync > Any-process-sync- > -from-P0-to-P1 --> Any-process-sync- > -from-P0-to-P1 4) > MPI_Win_sync > > _________ > About your > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > Perfect. > This example illustrates again the write-read-rule of #456 > (i.e. pattern with variable A), > here the lock/unlock pattern with exclusive lock in P1. > > _________ > About your > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… > > I would expect that it works, > but it is not based on a pattern that currently exists in #456. > > This is a bug of #456. > > The current wording > > Patterns with lock/unlock synchronization: > > Within passive target communication, two locks L1 and L2 > may be scheduled L1 before L2 or L2 before L1. In the > following patterns, the arrow means that the lock in P0 > was scheduled before the lock in P1. > > is not good enough. It should read: > > Within passive target communication, two locks L1 and L2 > may be scheduled "L1 released before L2 granted" or > "L2 released before L1 granted" in the case > of two locks with at least one exclusive, or in the case of any lock with an > additional synchronization (e.g., with point-to-point or > collective communication) in between. In the > following patterns, the arrow means that the lock in P0 > was released before the lock in P1 was granted, independent > of the method used to achieve this schedule. > > In the patters itself, I'll remove the word "shared" and "exclusive". > > A nice examlple would also be > > Process P0 Process P1 > A=0 B=0 > MPI_Win_fence MPI_Win_fence > MPI_Win_lock MPI_Win_lock > exclusive exclusive > A=val_1 B=val_2 > Bnew=load(B) Anew=load(A) > MPI_Win_unlock MPI_Win_unlock I have not started on examples that mix sync modes but I can try to do that next week. > The two rules write-read and read-write together > guarantee that either > - in P0 Bnew=0 and in P1 Anew=val_1 > or > - in P0 Bnew=val_2 and in P1 Anew=0 > > But this is a combination of several basic rules in #456. > > _________ > In General: > > #456 tries to define these 3*4 + 3 = 15 patterns, > which should be enough as far as I oversee, > plus two examples, 11.13 to show the translation of > the win_sync write-read-pattern and read-write-pattern > in real software, and 11.14 the pairing of memory barriers > without further synchronization between the processes. > > Your test examples should be very helpful to test, > whether an MPI library fulfills these patterns > and to illustrate the compressed wording in #456. Yeah, the tests are necessary but not sufficient to confirm implementation correctness. So far I have only run them on x86 laptop, which is probably the easiest case for which an implementation can be correct. Best, Jeff > Best regards > Rolf > > ----- Original Message ----- >> From: "Jeff Hammond" <jeff.science(a)gmail.com> >> To: "MPI Forum" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Friday, August 29, 2014 8:25:40 PM >> Subject: [mpiwg-rma] ticket 456 >> >> Rolf, >> >> I find you pseudocode confusion, hence I am creating examples in C >> that I can compile and run. I will try to come up with a case for >> everyone of the examples in your ticket. See >> https://github.com/jeffhammond/HPCInfo/tree/master/mpi/rma/shared-memory-wi…. >> >> Best, >> >> Jeff >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] ticket 456
by Rolf Rabenseifner 30 Aug '14

30 Aug '14

Jeff, great! Several comments: ________ About your https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… line 48-50 (barrier + 2nd fence) are not needed. fence on line 42 may be needed to guarantee that any initialization is finished. fence on line 52 seems to be also not needed: - not according to #456 - not if substituting the store by a MPI_Put. It would be only needed, if the load on line 51 would be a MPI_Get. ________ About your https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… Line 17: I would recommend MPI_Abort Lines 48, 50, 58, 59 are not needed. Result: 46 if (rank==0) { 49 *shptr = 42; 52 MPI_Win_post(MPI_GROUP_ONE, 0, shwin); 53 MPI_Win_wait(shwin); 55 } else if (rank==1) { 56 int lint; 61 MPI_Win_start(MPI_GROUP_ZERO, 0, shwin); 62 lint = *rptr; 63 MPI_Win_complete(shwin); This example would illustrate the write-read-rule of #456 (i.e. pattern with variable A) A=val_1 Sync-to-P1 --> Sync-from-P0 load(A) with Sync-to-P1 --> Sync-from-P0 being 2. MPI_Win_post --> MPI_Win_start This example would also work when substituting line 62 by MPI_Get. All Patterns in #456 are based on corresponding Patterns with MPI_Get and MPI_Put. If this example does not work with an existing MPI libary, then because it optimizes the synchronization away because there are no RMA calls. _________ About your https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… Perfect. This example illustrates again the write-read-rule of #456 (i.e. pattern with variable A) with Sync-to-P1 --> Sync-from-P0 being 4. MPI_Win_sync Any-process-sync- -from-P0-to-P1 --> Any-process-sync- -from-P0-to-P1 4) MPI_Win_sync _________ About your https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… Perfect. This example illustrates again the write-read-rule of #456 (i.e. pattern with variable A), here the lock/unlock pattern with exclusive lock in P1. _________ About your https://github.com/jeffhammond/HPCInfo/blob/master/mpi/rma/shared-memory-wi… I would expect that it works, but it is not based on a pattern that currently exists in #456. This is a bug of #456. The current wording Patterns with lock/unlock synchronization: Within passive target communication, two locks L1 and L2 may be scheduled L1 before L2 or L2 before L1. In the following patterns, the arrow means that the lock in P0 was scheduled before the lock in P1. is not good enough. It should read: Within passive target communication, two locks L1 and L2 may be scheduled "L1 released before L2 granted" or "L2 released before L1 granted" in the case of two locks with at least one exclusive, or in the case of any lock with an additional synchronization (e.g., with point-to-point or collective communication) in between. In the following patterns, the arrow means that the lock in P0 was released before the lock in P1 was granted, independent of the method used to achieve this schedule. In the patters itself, I'll remove the word "shared" and "exclusive". A nice examlple would also be Process P0 Process P1 A=0 B=0 MPI_Win_fence MPI_Win_fence MPI_Win_lock MPI_Win_lock exclusive exclusive A=val_1 B=val_2 Bnew=load(B) Anew=load(A) MPI_Win_unlock MPI_Win_unlock The two rules write-read and read-write together guarantee that either - in P0 Bnew=0 and in P1 Anew=val_1 or - in P0 Bnew=val_2 and in P1 Anew=0 But this is a combination of several basic rules in #456. _________ In General: #456 tries to define these 3*4 + 3 = 15 patterns, which should be enough as far as I oversee, plus two examples, 11.13 to show the translation of the win_sync write-read-pattern and read-write-pattern in real software, and 11.14 the pairing of memory barriers without further synchronization between the processes. Your test examples should be very helpful to test, whether an MPI library fulfills these patterns and to illustrate the compressed wording in #456. Best regards Rolf ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "MPI Forum" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Friday, August 29, 2014 8:25:40 PM > Subject: [mpiwg-rma] ticket 456 > > Rolf, > > I find you pseudocode confusion, hence I am creating examples in C > that I can compile and run. I will try to come up with a case for > everyone of the examples in your ticket. See > https://github.com/jeffhammond/HPCInfo/tree/master/mpi/rma/shared-memory-wi…. > > Best, > > Jeff > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

ticket 456
by Jeff Hammond 29 Aug '14

29 Aug '14

Rolf, I find you pseudocode confusion, hence I am creating examples in C that I can compile and run. I will try to come up with a case for everyone of the examples in your ticket. See https://github.com/jeffhammond/HPCInfo/tree/master/mpi/rma/shared-memory-wi…. Best, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Fwd: [MPI Forum] #456: MPI 3.0 Errata: MPI RMA synchronization for MPI shared memory
by Rolf Rabenseifner 28 Aug '14

28 Aug '14

Dear members of the RMA WG, after first partial reviews by Jeff Hammond, Jed Brown and Dave Goodell, I could fill the TO-DO areas and hopefully finalize #456. https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456 Just in time before the Sep.1 deadline. It is based on the result of the last telecon and the general rules shared memory barriers. The way of describing is based on the goal to define the outcome and not the implementation of the MPI library. The solution touches only shared memory windows. I hope that the RMA WG can discuss it before or in Kobe. In my opinion, MPI-3.1 should not be finalized before this problem is resolved. It is an open topic on the RMA agenda since 2 years, since the request for clarification from 11-Sep-2012 about "MPI_Win_allocate_shared and synchronization functions". Best regards Rolf ----- Original Message ----- > From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, August 14, 2014 1:00:55 AM > Subject: [mpiwg-rma] Fwd: [MPI Forum] #456: MPI 3.0 Errata: MPI RMA synchronization for MPI shared memory > > Dear Bill and whole RMA working Group, > > I tried to bring all my knowledge together to solve the > "shared memory synchronization problem". > > I know, that you Bill also wanted to solve this problem. > Please take my ticket > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456 > as a basis or stone quarry or ... . > > For me, it is important to really find all locations that must > be touched to solve the Problem and to have a consistent solution. > > I hope that I found all of them and my solution is consistent. > > The solution is free of technical implementation internals, > i.e., the local memory barriers are only mentioned in a rationale. > > I hope you all enjoy the proposal and it is a basis for > further improvements. > > Best bregards > Rolf > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Next RMA telecon - Re: doodle poll for RMA working group telecon
by William Gropp 18 Aug '14

18 Aug '14

Rolf, Thanks for putting together #456; it helps to have this all in one place. The first thing that we need to do is decide between options A and B, noting that the increased synchronization required of (A) applies only to shared memory windows (and this still does not require a barrier, though that’s likely to be the easiest implementation). My preference is to keep the discussion of remote load/store separate from MPI RMA operations and to provide text specific for this case. I don’t believe that there will be enough members of the RMA group at the next meeting to come to a consensus on this, and the issues are subtle, so I don’t think it will be possible to get this resolved before or at the next meeting. Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Aug 15, 2014, at 3:06 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Dear Bill and Rajeev, > > the Sep. 1 deadline is in 17 days. > I know that the original plan from the last telecon > 2 weeks ago was that you, Bill make an initial proposal. > I didn't felt that I would be able to do such a proposal. > > After all the helpful discussion by email, > I felt that I now have enough information. > I hope that it was okay for you that I tried to put all > what I knew together into the proposal #456. > > To get further progress on this problem, I expect we > should have another telecon in the first half of next > week (already less than 14 days before the deadline). > > What is your opinion? > > I'm thankful that I was teached about the background, > especially by Dave, Pavan, Bronis, Bill Long, and you, Bill. > > Therefore I hope that my proposal is technically > a correct solution (but still incomplete for the two > FLUSH operations). > I hope that you or somebody else can fill this gap. > > Best regards > Rolf > > ----- Original Message ----- >> From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Monday, July 28, 2014 4:26:05 PM >> Subject: Re: [mpiwg-rma] doodle poll for RMA working group telecon >> >> A reminder of this telecon on Wednesday. Please read ticket 437 and >> the other tickets mentioned in it. >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437 >> >> Rajeev >> >> On Jul 19, 2014, at 5:11 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> >> wrote: >> >>> OK, let's have the telecon on Wed, July 30, 9.00am Central Time >>> (7am Pacific, 4pm German time). >>> >>> We can use this telecon number: 1-567-314-8345, passcode: >>> 262144 >>> >>> The agenda is ticket 437 and other tickets mentioned therein. >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437 >>> >>> Rajeev >>> >>> >>> On Jul 18, 2014, at 2:26 PM, William Gropp <wgropp(a)illinois.edu> >>> wrote: >>> >>>> Starting with the first available time, 9 CDT on July 30, is fine >>>> with me. >>>> Bill >>>> >>>> William Gropp >>>> Director, Parallel Computing Institute >>>> Thomas M. Siebel Chair in Computer Science >>>> University of Illinois Urbana-Champaign >>>> >>>> >>>> >>>> >>>> >>>> On Jul 18, 2014, at 10:45 AM, Rolf Rabenseifner wrote: >>>> >>>>> Dear all, >>>>> >>>>> Rajeev, thank you for starting the doodle. >>>>> I expect that you are still on vacation. >>>>> Therefore a short summary before the weekend starts (in Europe): >>>>> >>>>> As I can see at http://doodle.com/hx3p4cbwfcyszam6 >>>>> there are only three slots available where all can come together: >>>>> >>>>> Wednesday July 30, >>>>> - 09:00-10:00 USA Central time >>>>> - 16:00-17:00 Berlin timezone >>>>> >>>>> Wednesday July 30, >>>>> - 12:00-13:00 USA Central time >>>>> - 19:00-20:00 Berlin timezone >>>>> (this slot is also available, because my evening-date was >>>>> removed) >>>>> >>>>> and Thursday July 31, >>>>> - 12:00-13:00 USA Central time >>>>> - 19:00-20:00 Berlin timezone >>>>> >>>>>> We need to discuss a few tickets over telecon. Let us start with >>>>>> a >>>>>> telecon for ticket 437 that Rolf has created: >>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437. It is >>>>>> to >>>>>> clarify whether remote accesses to shared memory are to be >>>>>> considered similar to RMA operations or local loads/stores. A >>>>>> few >>>>>> related tickets are linked there. >>>>> >>>>> I would recommend that we start with the first slot and >>>>> we will see, whether we need to continue with the other slots. >>>>> Otherwise those other slots could be used for other RMA tickets. >>>>> >>>>> Okay? >>>>> >>>>> Participants on the doodle were: >>>>> - Rajeev Thakur >>>>> - Pavan Balaji >>>>> - Jeff Hammond >>>>> - Rolf Rabenseifner >>>>> - Bill Gropp >>>>> - Jim Dinan >>>>> - Hubert Ritzdorf >>>>> >>>>> I hope that other members of the RMA WG can also participate, >>>>> especially if you will go to the Japanese MPI forum meeting. >>>>> >>>>> Best regards >>>>> Rolf >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> >>>>>> To: "MPI WG Remote Memory Access working group" >>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>> Sent: Monday, July 14, 2014 8:39:12 PM >>>>>> Subject: [mpiwg-rma] doodle poll for RMA working group telecon >>>>>> >>>>>> We need to discuss a few tickets over telecon. Let us start with >>>>>> a >>>>>> telecon for ticket 437 that Rolf has created: >>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437. It is >>>>>> to >>>>>> clarify whether remote accesses to shared memory are to be >>>>>> considered similar to RMA operations or local loads/stores. A >>>>>> few >>>>>> related tickets are linked there. >>>>>> >>>>>> Please indicate your availability in the weeks of July 21 and 28 >>>>>> in >>>>>> the doodle poll at http://doodle.com/hx3p4cbwfcyszam6. I have >>>>>> listed >>>>>> only a few slots on each day to accommodate European and Pacific >>>>>> time zones. >>>>>> >>>>>> Thanks, >>>>>> Rajeev >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>> >>>>> -- >>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>> rabenseifner(a)hlrs.de >>>>> High Performance Computing Center (HLRS) . phone >>>>> ++49(0)711/685-65530 >>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>> 685-65832 >>>>> Head of Dpmt Parallel Computing . . . >>>>> www.hlrs.de/people/rabenseifner >>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>>> 1.307) >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Next RMA telecon - Re: doodle poll for RMA working group telecon
by Rolf Rabenseifner 15 Aug '14

15 Aug '14

Dear Bill and Rajeev, the Sep. 1 deadline is in 17 days. I know that the original plan from the last telecon 2 weeks ago was that you, Bill make an initial proposal. I didn't felt that I would be able to do such a proposal. After all the helpful discussion by email, I felt that I now have enough information. I hope that it was okay for you that I tried to put all what I knew together into the proposal #456. To get further progress on this problem, I expect we should have another telecon in the first half of next week (already less than 14 days before the deadline). What is your opinion? I'm thankful that I was teached about the background, especially by Dave, Pavan, Bronis, Bill Long, and you, Bill. Therefore I hope that my proposal is technically a correct solution (but still incomplete for the two FLUSH operations). I hope that you or somebody else can fill this gap. Best regards Rolf ----- Original Message ----- > From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Monday, July 28, 2014 4:26:05 PM > Subject: Re: [mpiwg-rma] doodle poll for RMA working group telecon > > A reminder of this telecon on Wednesday. Please read ticket 437 and > the other tickets mentioned in it. > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437 > > Rajeev > > On Jul 19, 2014, at 5:11 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> > wrote: > > > OK, let's have the telecon on Wed, July 30, 9.00am Central Time > > (7am Pacific, 4pm German time). > > > > We can use this telecon number: 1-567-314-8345, passcode: > > 262144 > > > > The agenda is ticket 437 and other tickets mentioned therein. > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437 > > > > Rajeev > > > > > > On Jul 18, 2014, at 2:26 PM, William Gropp <wgropp(a)illinois.edu> > > wrote: > > > >> Starting with the first available time, 9 CDT on July 30, is fine > >> with me. > >> Bill > >> > >> William Gropp > >> Director, Parallel Computing Institute > >> Thomas M. Siebel Chair in Computer Science > >> University of Illinois Urbana-Champaign > >> > >> > >> > >> > >> > >> On Jul 18, 2014, at 10:45 AM, Rolf Rabenseifner wrote: > >> > >>> Dear all, > >>> > >>> Rajeev, thank you for starting the doodle. > >>> I expect that you are still on vacation. > >>> Therefore a short summary before the weekend starts (in Europe): > >>> > >>> As I can see at http://doodle.com/hx3p4cbwfcyszam6 > >>> there are only three slots available where all can come together: > >>> > >>> Wednesday July 30, > >>> - 09:00-10:00 USA Central time > >>> - 16:00-17:00 Berlin timezone > >>> > >>> Wednesday July 30, > >>> - 12:00-13:00 USA Central time > >>> - 19:00-20:00 Berlin timezone > >>> (this slot is also available, because my evening-date was > >>> removed) > >>> > >>> and Thursday July 31, > >>> - 12:00-13:00 USA Central time > >>> - 19:00-20:00 Berlin timezone > >>> > >>>> We need to discuss a few tickets over telecon. Let us start with > >>>> a > >>>> telecon for ticket 437 that Rolf has created: > >>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437. It is > >>>> to > >>>> clarify whether remote accesses to shared memory are to be > >>>> considered similar to RMA operations or local loads/stores. A > >>>> few > >>>> related tickets are linked there. > >>> > >>> I would recommend that we start with the first slot and > >>> we will see, whether we need to continue with the other slots. > >>> Otherwise those other slots could be used for other RMA tickets. > >>> > >>> Okay? > >>> > >>> Participants on the doodle were: > >>> - Rajeev Thakur > >>> - Pavan Balaji > >>> - Jeff Hammond > >>> - Rolf Rabenseifner > >>> - Bill Gropp > >>> - Jim Dinan > >>> - Hubert Ritzdorf > >>> > >>> I hope that other members of the RMA WG can also participate, > >>> especially if you will go to the Japanese MPI forum meeting. > >>> > >>> Best regards > >>> Rolf > >>> > >>> > >>> ----- Original Message ----- > >>>> From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> > >>>> To: "MPI WG Remote Memory Access working group" > >>>> <mpiwg-rma(a)lists.mpi-forum.org> > >>>> Sent: Monday, July 14, 2014 8:39:12 PM > >>>> Subject: [mpiwg-rma] doodle poll for RMA working group telecon > >>>> > >>>> We need to discuss a few tickets over telecon. Let us start with > >>>> a > >>>> telecon for ticket 437 that Rolf has created: > >>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/437. It is > >>>> to > >>>> clarify whether remote accesses to shared memory are to be > >>>> considered similar to RMA operations or local loads/stores. A > >>>> few > >>>> related tickets are linked there. > >>>> > >>>> Please indicate your availability in the weeks of July 21 and 28 > >>>> in > >>>> the doodle poll at http://doodle.com/hx3p4cbwfcyszam6. I have > >>>> listed > >>>> only a few slots on each day to accommodate European and Pacific > >>>> time zones. > >>>> > >>>> Thanks, > >>>> Rajeev > >>>> > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>> > >>> > >>> -- > >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > >>> rabenseifner(a)hlrs.de > >>> High Performance Computing Center (HLRS) . phone > >>> ++49(0)711/685-65530 > >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >>> 685-65832 > >>> Head of Dpmt Parallel Computing . . . > >>> www.hlrs.de/people/rabenseifner > >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >>> 1.307) > >>> _______________________________________________ > >>> mpiwg-rma mailing list > >>> mpiwg-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Fwd: [MPI Forum] #456: MPI 3.0 Errata: MPI RMA synchronization for MPI shared memory
by Rolf Rabenseifner 13 Aug '14

13 Aug '14

Dear Bill and whole RMA working Group, I tried to bring all my knowledge together to solve the "shared memory synchronization problem". I know, that you Bill also wanted to solve this problem. Please take my ticket https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456 as a basis or stone quarry or ... . For me, it is important to really find all locations that must be touched to solve the Problem and to have a consistent solution. I hope that I found all of them and my solution is consistent. The solution is free of technical implementation internals, i.e., the local memory barriers are only mentioned in a rationale. I hope you all enjoy the proposal and it is a basis for further improvements. Best bregards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Single RMA synchronization for several window handles
by Rolf Rabenseifner 11 Aug '14

11 Aug '14

If a programmer uses several windows then he/she may have these multi-sync problems. If he/she uses only one window, than there is no problem. With my idea, I wanted to show a solution, which may be simple for the case that the Communicator is the same on all windows. But essentially more important (because MPI-2.0 errata) is the #437 discussion and ist solution. Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Monday, August 11, 2014 7:48:43 PM > Subject: Re: [mpiwg-rma] Single RMA synchronization for several window handles > > > I’ve written 7-pt stencil code with RMA and it does not require this > many RMA synchronizations. > > > Bill > > > > > > > > William Gropp > Director, Parallel Computing Institute Thomas M. Siebel Chair in > Computer Science > > > University of Illinois Urbana-Champaign > > > > > > > > On Aug 11, 2014, at 10:47 AM, Rolf Rabenseifner < > rabenseifner(a)hlrs.de > wrote: > > > It is not function call overhead, it is my 7 point-stencil-example > executing 6 times a synchronization pattern (e.g. all what is needed > for MPI_Win_fence) or only one time this synchronization pattern. > And as far as I see, the multiple synchronizations > cannot be discarded by the MPI library based on some > intelligent optimization. > > More than this, one RMA epoch needs typically two synchroniizations, > i.e., the question is whether 12 or two MPI_Win_fence are needed; > i.e., difference is 10, i.e., my proposal would allow > that 10 times the MPI_Win_fence latency is removed. > > Rolf > > ----- Original Message ----- > > > From: "Jeff Hammond" < jeff.science(a)gmail.com > > To: "MPI WG Remote Memory Access working group" < > mpiwg-rma(a)lists.mpi-forum.org > > Sent: Monday, August 11, 2014 4:09:19 PM > Subject: Re: [mpiwg-rma] Single RMA synchronization for several > window handles > > If function call overhead matters, MPI is probably the wrong model. > > You'll have to prove to me that this (function call overhead) is > significant compared to the cost of synchronization using empirical > data on some system. > > Jeff > > Sent from my iPhone > > > > On Aug 11, 2014, at 1:49 AM, Rolf Rabenseifner > < rabenseifner(a)hlrs.de > wrote: > > Jim and all, > > It is not syntactic sugar. > It is a latency optimizing enhancement: > > If you are doing neighbor communication with 1-sided > communication to 6 neighbors - based on 6 windows that > are all defined in MPI_COMM_WORLD, then currently > you need to call 6 times the synchronization calls > which implies at maximum a 6 times larger latency, > e.g., 6 times MPI_Win_fence instead of one call. > > If you are using MPI on shared memory windows, > then the same latency problem exists, but here, > the trick with dynamic windows is impossible > because shared memory windows must be allocated > with MPI_Win_allocate_shared. > > Of course, it is more complicated to define > MPI_Win_combine on a superset of communicators used > for all the combined windows. > > As a first effort, it would be already helpful > to define it for the same communicator. > To allow enhancements in a future MPI version, > I still would recommend to have the comm argument > as part of the argument list. > > Rolf > > ----- Original Message ----- > > > From: "Jim Dinan" < james.dinan(a)gmail.com > > To: "MPI WG Remote Memory Access working group" > < mpiwg-rma(a)lists.mpi-forum.org > > Sent: Sunday, August 10, 2014 5:48:02 PM > Subject: Re: [mpiwg-rma] Single RMA synchronization for several > window handles > > > > Hi Rolf, > > > I had initially proposed this in the context of passive target > RMA. > Active target RMA already notifies the receiver when data has > arrived, but there is no efficient way to get such a notification > in > passive target RMA. I think what you are proposing here would be > syntactic sugar on top of the existing interface --- could I > implement this by fencing every window to determine that all > transfers are completed? > > > I agree with the comments regarding dynamic windows. The merged > window would contain discontiguous buffers; thus, it would lose > the > ability to do offset-based addressing and would need to use > absolute > (BOTTOM-based) addressing. > > > ~Jim. > > > > On Fri, Aug 8, 2014 at 10:18 AM, Rolf Rabenseifner < > rabenseifner(a)hlrs.de > wrote: > > > Jim, > > your topic "Reducing Synchronization Overhead Through Bundled > Communication" may get also help if we would be able to > combine several window handles to one superset window handle. > > If you have several windows for different buffers, but > only one synchronization pattern, e.g. MPI_Win_fince > then currently you must call MPI_Win_fence seperately > for each window handle. > > I would propose: > > MPI_Win_combine (/*IN*/ int count, > /*IN*/ MPI_Win *win, > /*IN*/ MPI_Comm comm, > /*OUT*/ MPI_Win *win_combined) > > The process group of comm must contain the process groups of all > win. > The resulting window handle win_combined can be used only > in RMA synchronization calls and other helper routines, > but not for dynamic window allocation nor for any > RMA communication routine. > Collective synchronization routines must be called by all > processes > of comm. > The semantics of an RMA synchronization call using win_combined > is defined as if the calls were seperately issued for > each window handle of the array win. If group handles > are part of the argument list of the synchronization call > then the appropriate subset is used for each window handle in win. > > What do you think about this idea for MPI-4.0? > > Best regards > Rolf > > ----- Original Message ----- > > > From: "Jim Dinan" < james.dinan(a)gmail.com > > To: "MPI WG Remote Memory Access working group" < > mpiwg-rma(a)lists.mpi-forum.org > > Sent: Thursday, August 7, 2014 4:08:32 PM > Subject: [mpiwg-rma] RMA Notification > > > > Hi All, > > > I have added a new proposal for an RMA notification extension: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/439 > > > I would like to bring this forward for the RMA WG to consider as > an > MPI-4 extension. > > > Cheers, > ~Jim. > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email > rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone > ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > 685-65832 > Head of Dpmt Parallel Computing . . . > www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email > rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone > ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > 685-65832 > Head of Dpmt Parallel Computing . . . > www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] Single RMA synchronization for several window handles
by Rajeev Thakur 11 Aug '14

11 Aug '14

But it is not always possible to pass an assert. For example, if just one process does a put to another, no other process can use the assert MPI_MODE_NOPRECEDE and all must synchronize. Rajeev On Aug 11, 2014, at 3:50 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Then the right assertions are the solution, not a new function. > > Jeff > > On Mon, Aug 11, 2014 at 1:49 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> Unless the right asserts are passed, a process in a fence has to wait until all other processes have reached their fence. That is because, in the absence of asserts, the process doesn't know if it is the target of RMA operations from another process. >> >> Rajeev >> >> On Aug 11, 2014, at 3:22 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >>> Does the MPI standard require fence to do a barrier? If not, then my >>> argument still stands. The implementation only has to sync once per >>> remote target across N fences and the rest are nothing more than >>> function call overhead. >>> >>> If we want to optimize for functional call overhead, let's consider >>> MPI_Win_fencev, which makes way more sense than creating >>> franken-windows with this combiner function. >>> >>> Jeff >>> >>> On Mon, Aug 11, 2014 at 7:29 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>> It's more than function call overhead. Fence usually includes a barrier, so >>>> this is one barrier versus several. >>>> >>>> >>>> On Mon, Aug 11, 2014 at 10:09 AM, Jeff Hammond <jeff.science(a)gmail.com> >>>> wrote: >>>>> >>>>> If function call overhead matters, MPI is probably the wrong model. >>>>> >>>>> You'll have to prove to me that this (function call overhead) is >>>>> significant compared to the cost of synchronization using empirical data on >>>>> some system. >>>>> >>>>> Jeff >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Aug 11, 2014, at 1:49 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> >>>>>> wrote: >>>>>> >>>>>> Jim and all, >>>>>> >>>>>> It is not syntactic sugar. >>>>>> It is a latency optimizing enhancement: >>>>>> >>>>>> If you are doing neighbor communication with 1-sided >>>>>> communication to 6 neighbors - based on 6 windows that >>>>>> are all defined in MPI_COMM_WORLD, then currently >>>>>> you need to call 6 times the synchronization calls >>>>>> which implies at maximum a 6 times larger latency, >>>>>> e.g., 6 times MPI_Win_fence instead of one call. >>>>>> >>>>>> If you are using MPI on shared memory windows, >>>>>> then the same latency problem exists, but here, >>>>>> the trick with dynamic windows is impossible >>>>>> because shared memory windows must be allocated >>>>>> with MPI_Win_allocate_shared. >>>>>> >>>>>> Of course, it is more complicated to define >>>>>> MPI_Win_combine on a superset of communicators used >>>>>> for all the combined windows. >>>>>> >>>>>> As a first effort, it would be already helpful >>>>>> to define it for the same communicator. >>>>>> To allow enhancements in a future MPI version, >>>>>> I still would recommend to have the comm argument >>>>>> as part of the argument list. >>>>>> >>>>>> Rolf >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Jim Dinan" <james.dinan(a)gmail.com> >>>>>>> To: "MPI WG Remote Memory Access working group" >>>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>>> Sent: Sunday, August 10, 2014 5:48:02 PM >>>>>>> Subject: Re: [mpiwg-rma] Single RMA synchronization for several window >>>>>>> handles >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Rolf, >>>>>>> >>>>>>> >>>>>>> I had initially proposed this in the context of passive target RMA. >>>>>>> Active target RMA already notifies the receiver when data has >>>>>>> arrived, but there is no efficient way to get such a notification in >>>>>>> passive target RMA. I think what you are proposing here would be >>>>>>> syntactic sugar on top of the existing interface --- could I >>>>>>> implement this by fencing every window to determine that all >>>>>>> transfers are completed? >>>>>>> >>>>>>> >>>>>>> I agree with the comments regarding dynamic windows. The merged >>>>>>> window would contain discontiguous buffers; thus, it would lose the >>>>>>> ability to do offset-based addressing and would need to use absolute >>>>>>> (BOTTOM-based) addressing. >>>>>>> >>>>>>> >>>>>>> ~Jim. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Aug 8, 2014 at 10:18 AM, Rolf Rabenseifner < >>>>>>> rabenseifner(a)hlrs.de > wrote: >>>>>>> >>>>>>> >>>>>>> Jim, >>>>>>> >>>>>>> your topic "Reducing Synchronization Overhead Through Bundled >>>>>>> Communication" may get also help if we would be able to >>>>>>> combine several window handles to one superset window handle. >>>>>>> >>>>>>> If you have several windows for different buffers, but >>>>>>> only one synchronization pattern, e.g. MPI_Win_fince >>>>>>> then currently you must call MPI_Win_fence seperately >>>>>>> for each window handle. >>>>>>> >>>>>>> I would propose: >>>>>>> >>>>>>> MPI_Win_combine (/*IN*/ int count, >>>>>>> /*IN*/ MPI_Win *win, >>>>>>> /*IN*/ MPI_Comm comm, >>>>>>> /*OUT*/ MPI_Win *win_combined) >>>>>>> >>>>>>> The process group of comm must contain the process groups of all win. >>>>>>> The resulting window handle win_combined can be used only >>>>>>> in RMA synchronization calls and other helper routines, >>>>>>> but not for dynamic window allocation nor for any >>>>>>> RMA communication routine. >>>>>>> Collective synchronization routines must be called by all processes >>>>>>> of comm. >>>>>>> The semantics of an RMA synchronization call using win_combined >>>>>>> is defined as if the calls were seperately issued for >>>>>>> each window handle of the array win. If group handles >>>>>>> are part of the argument list of the synchronization call >>>>>>> then the appropriate subset is used for each window handle in win. >>>>>>> >>>>>>> What do you think about this idea for MPI-4.0? >>>>>>> >>>>>>> Best regards >>>>>>> Rolf >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Jim Dinan" < james.dinan(a)gmail.com > >>>>>>>> To: "MPI WG Remote Memory Access working group" < >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>>>>>>> Sent: Thursday, August 7, 2014 4:08:32 PM >>>>>>>> Subject: [mpiwg-rma] RMA Notification >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> >>>>>>>> I have added a new proposal for an RMA notification extension: >>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/439 >>>>>>>> >>>>>>>> >>>>>>>> I would like to bring this forward for the RMA WG to consider as an >>>>>>>> MPI-4 extension. >>>>>>>> >>>>>>>> >>>>>>>> Cheers, >>>>>>>> ~Jim. >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> -- >>>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >>>>>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >>>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >>>>>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >>>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> -- >>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >>>>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >>>>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0