February 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Inconsistency of MPI_WIN_FENCE semantic
by Zhao, Xin 25 Feb '14

25 Feb '14

Hi Pavan, Is Get posted on rank 0 or rank 1? Also Flush is only used for passive target. Xin ________________________________________ From: mpiwg-rma [mpiwg-rma-bounces(a)lists.mpi-forum.org] on behalf of Balaji, Pavan [balaji(a)anl.gov] Sent: Tuesday, February 25, 2014 10:22 PM To: MPI WG Remote Memory Access working group Subject: Re: [mpiwg-rma] Inconsistency of MPI_WIN_FENCE semantic Your example is too simplistic. Here’s another example — Fence if (rank == 0) { PUT(to rank 1); Flush } Fence Get Fence Without barrier semantics, the above program will lead to incorrect results if GET is expecting to see the value deposited by the PUT. Are you saying this is an incorrect program? I’m not sure I agree with that. The standard states that FENCE does not have barrier semantics when NOPRECEDE is given. I think the intention here was to say that it does have barrier semantics when that assertion is not given. — Pavan On Feb 25, 2014, at 10:10 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > The second sentence refers to case where there are a sequence of fences > > Fence > Fence > Put > Fence > > The second fence above need not be a barrier particularly if it had a NOPRECEDE. > > > Note that in a fence-put-fence you are allowed to do a barrier in the first fence, but you are not required to. You can do a deferred mode where the first fence is a no-op, all RMA ops are queued, and everything is done in the second fence and determining that all processes have reached there. > > Rajeev > > > > On Feb 25, 2014, at 9:48 PM, "Balaji, Pavan" <balaji(a)anl.gov> > wrote: > >> >> Besides, the second sentence says, if MPI_MODE_NOPRECEDE is given, it’ll not act as a barrier. >> >> What’s the necessity for that if WIN_FENCE does not act as a barrier even when MPI_MODE_NOPRECEDE is not given. >> >> — Pavan >> >> On Feb 25, 2014, at 9:38 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >>> >>> In practice, it’ll need to have barrier semantics. Otherwise, PUT will need to be a two-sided operation to ensure that it’s not issued before the other process calls MPI_WIN_FENCE. >>> >>> — Pavan >>> >>> On Feb 25, 2014, at 9:21 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>> >>>>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >>>> >>>> It only says "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". NOT "This requires MPI_WIN_FENCE that starts an epoch to act as an barrier." >>>> >>>> Why does the fence have to act as a barrier. The handshake could be done when first RMA operation is called after the fence. >>>> >>>> Rajeev >>>> >>>> >>>> On Feb 25, 2014, at 8:41 PM, "Zhao, Xin" <xinzhao3(a)illinois.edu> wrote: >>>> >>>>> Hi all, >>>>> >>>>> There is an inconsistency of MPI_WIN_FENCE semantic in MPI 3.0 Standard that makes me confused: >>>>> >>>>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >>>>> >>>>> (2) However, (1) contradict with the word at end of P441: "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier". >>>>> >>>>> Should the word of (1) add: "when MPI_MODE_NOPRECEDE is not given"? >>>>> >>>>> Thanks, >>>>> >>>>> Xin >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Inconsistency of MPI_WIN_FENCE semantic
by Balaji, Pavan 25 Feb '14

25 Feb '14

Your example is too simplistic. Here’s another example — Fence if (rank == 0) { PUT(to rank 1); Flush } Fence Get Fence Without barrier semantics, the above program will lead to incorrect results if GET is expecting to see the value deposited by the PUT. Are you saying this is an incorrect program? I’m not sure I agree with that. The standard states that FENCE does not have barrier semantics when NOPRECEDE is given. I think the intention here was to say that it does have barrier semantics when that assertion is not given. — Pavan On Feb 25, 2014, at 10:10 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > The second sentence refers to case where there are a sequence of fences > > Fence > Fence > Put > Fence > > The second fence above need not be a barrier particularly if it had a NOPRECEDE. > > > Note that in a fence-put-fence you are allowed to do a barrier in the first fence, but you are not required to. You can do a deferred mode where the first fence is a no-op, all RMA ops are queued, and everything is done in the second fence and determining that all processes have reached there. > > Rajeev > > > > On Feb 25, 2014, at 9:48 PM, "Balaji, Pavan" <balaji(a)anl.gov> > wrote: > >> >> Besides, the second sentence says, if MPI_MODE_NOPRECEDE is given, it’ll not act as a barrier. >> >> What’s the necessity for that if WIN_FENCE does not act as a barrier even when MPI_MODE_NOPRECEDE is not given. >> >> — Pavan >> >> On Feb 25, 2014, at 9:38 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >>> >>> In practice, it’ll need to have barrier semantics. Otherwise, PUT will need to be a two-sided operation to ensure that it’s not issued before the other process calls MPI_WIN_FENCE. >>> >>> — Pavan >>> >>> On Feb 25, 2014, at 9:21 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>> >>>>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >>>> >>>> It only says "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". NOT "This requires MPI_WIN_FENCE that starts an epoch to act as an barrier." >>>> >>>> Why does the fence have to act as a barrier. The handshake could be done when first RMA operation is called after the fence. >>>> >>>> Rajeev >>>> >>>> >>>> On Feb 25, 2014, at 8:41 PM, "Zhao, Xin" <xinzhao3(a)illinois.edu> wrote: >>>> >>>>> Hi all, >>>>> >>>>> There is an inconsistency of MPI_WIN_FENCE semantic in MPI 3.0 Standard that makes me confused: >>>>> >>>>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >>>>> >>>>> (2) However, (1) contradict with the word at end of P441: "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier". >>>>> >>>>> Should the word of (1) add: "when MPI_MODE_NOPRECEDE is not given"? >>>>> >>>>> Thanks, >>>>> >>>>> Xin >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Inconsistency of MPI_WIN_FENCE semantic
by Rajeev Thakur 25 Feb '14

25 Feb '14

The second sentence refers to case where there are a sequence of fences Fence Fence Put Fence The second fence above need not be a barrier particularly if it had a NOPRECEDE. Note that in a fence-put-fence you are allowed to do a barrier in the first fence, but you are not required to. You can do a deferred mode where the first fence is a no-op, all RMA ops are queued, and everything is done in the second fence and determining that all processes have reached there. Rajeev On Feb 25, 2014, at 9:48 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > Besides, the second sentence says, if MPI_MODE_NOPRECEDE is given, it’ll not act as a barrier. > > What’s the necessity for that if WIN_FENCE does not act as a barrier even when MPI_MODE_NOPRECEDE is not given. > > — Pavan > > On Feb 25, 2014, at 9:38 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >> >> In practice, it’ll need to have barrier semantics. Otherwise, PUT will need to be a two-sided operation to ensure that it’s not issued before the other process calls MPI_WIN_FENCE. >> >> — Pavan >> >> On Feb 25, 2014, at 9:21 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> >>>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >>> >>> It only says "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". NOT "This requires MPI_WIN_FENCE that starts an epoch to act as an barrier." >>> >>> Why does the fence have to act as a barrier. The handshake could be done when first RMA operation is called after the fence. >>> >>> Rajeev >>> >>> >>> On Feb 25, 2014, at 8:41 PM, "Zhao, Xin" <xinzhao3(a)illinois.edu> wrote: >>> >>>> Hi all, >>>> >>>> There is an inconsistency of MPI_WIN_FENCE semantic in MPI 3.0 Standard that makes me confused: >>>> >>>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >>>> >>>> (2) However, (1) contradict with the word at end of P441: "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier". >>>> >>>> Should the word of (1) add: "when MPI_MODE_NOPRECEDE is not given"? >>>> >>>> Thanks, >>>> >>>> Xin >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Inconsistency of MPI_WIN_FENCE semantic
by Balaji, Pavan 25 Feb '14

25 Feb '14

Besides, the second sentence says, if MPI_MODE_NOPRECEDE is given, it’ll not act as a barrier. What’s the necessity for that if WIN_FENCE does not act as a barrier even when MPI_MODE_NOPRECEDE is not given. — Pavan On Feb 25, 2014, at 9:38 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > In practice, it’ll need to have barrier semantics. Otherwise, PUT will need to be a two-sided operation to ensure that it’s not issued before the other process calls MPI_WIN_FENCE. > > — Pavan > > On Feb 25, 2014, at 9:21 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > >>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >> >> It only says "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". NOT "This requires MPI_WIN_FENCE that starts an epoch to act as an barrier." >> >> Why does the fence have to act as a barrier. The handshake could be done when first RMA operation is called after the fence. >> >> Rajeev >> >> >> On Feb 25, 2014, at 8:41 PM, "Zhao, Xin" <xinzhao3(a)illinois.edu> wrote: >> >>> Hi all, >>> >>> There is an inconsistency of MPI_WIN_FENCE semantic in MPI 3.0 Standard that makes me confused: >>> >>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >>> >>> (2) However, (1) contradict with the word at end of P441: "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier". >>> >>> Should the word of (1) add: "when MPI_MODE_NOPRECEDE is not given"? >>> >>> Thanks, >>> >>> Xin >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Inconsistency of MPI_WIN_FENCE semantic
by Balaji, Pavan 25 Feb '14

25 Feb '14

In practice, it’ll need to have barrier semantics. Otherwise, PUT will need to be a two-sided operation to ensure that it’s not issued before the other process calls MPI_WIN_FENCE. — Pavan On Feb 25, 2014, at 9:21 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. > > It only says "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". NOT "This requires MPI_WIN_FENCE that starts an epoch to act as an barrier." > > Why does the fence have to act as a barrier. The handshake could be done when first RMA operation is called after the fence. > > Rajeev > > > On Feb 25, 2014, at 8:41 PM, "Zhao, Xin" <xinzhao3(a)illinois.edu> wrote: > >> Hi all, >> >> There is an inconsistency of MPI_WIN_FENCE semantic in MPI 3.0 Standard that makes me confused: >> >> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. >> >> (2) However, (1) contradict with the word at end of P441: "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier". >> >> Should the word of (1) add: "when MPI_MODE_NOPRECEDE is not given"? >> >> Thanks, >> >> Xin >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Inconsistency of MPI_WIN_FENCE semantic
by Rajeev Thakur 25 Feb '14

25 Feb '14

> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. It only says "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". NOT "This requires MPI_WIN_FENCE that starts an epoch to act as an barrier." Why does the fence have to act as a barrier. The handshake could be done when first RMA operation is called after the fence. Rajeev On Feb 25, 2014, at 8:41 PM, "Zhao, Xin" <xinzhao3(a)illinois.edu> wrote: > Hi all, > > There is an inconsistency of MPI_WIN_FENCE semantic in MPI 3.0 Standard that makes me confused: > > (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. > > (2) However, (1) contradict with the word at end of P441: "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier". > > Should the word of (1) add: "when MPI_MODE_NOPRECEDE is not given"? > > Thanks, > > Xin > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Inconsistency of MPI_WIN_FENCE semantic
by Zhao, Xin 25 Feb '14

25 Feb '14

Hi all, There is an inconsistency of MPI_WIN_FENCE semantic in MPI 3.0 Standard that makes me confused: (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier. (2) However, (1) contradict with the word at end of P441: "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier". Should the word of (1) add: "when MPI_MODE_NOPRECEDE is not given"? Thanks, Xin

1 0

Problems with RMA synchronization in combination with load/store shared memory accesses
by Rolf Rabenseifner 21 Feb '14

21 Feb '14

Dear member of the RMA group and especially the mpich developers, I have real problems with the new shared memory in MPI-3.0, i.e., the load/stores together with the RMA synchronization causes wrong execution results. The attached 1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip contains - 1sided/halo_1sided_put_win_alloc.c The basis that works. It uses MPI_Put and MPI_Win_fence for duplex left/right halo communication. - 1sided/halo_1sided_store_win_alloc_shared.c This is the same, but a shared memory window is used and the MPU_Put is substituted by storing the data in the neighbors window. Same MPI_Win_fence with same assertions. This does not work, although I'm sure that my assertions are correct. Known possibilities: - I'm wrong and was not able to understand the assertions on MPI-3.0 p452:8-19. - I'm wrong because it is invalid to use the MPI_Win_fence together with the shared memory windows. - mpich has a bug. (The first two possibilities are the reason, why I use this Forum email list) - 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c This is a work-around-for Cray that works on our Cray and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED. It also runs on another mpich installation. - 1sided/halo_1sided_store_win_alloc_shared_pscw.c Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait and it does not work for any assertions. Same possibilities as above. - 1sided/halo_1sided_store_win_alloc_shared_query.c - 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c Same as halo_1sided_store_win_alloc_shared.c but non-contigues windows are used. Same problems as above. - 1sided/halo_1sided_store_win_alloc_shared_othersync.c This version uses the synchronization according to #413 and it is tested and works on two platforms. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal Readings
by Rolf Rabenseifner 20 Feb '14

20 Feb '14

I added Rajeevs proposals/changes to the ticket https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/413#comment:6 ----- Original Message ----- > From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, February 20, 2014 7:22:28 PM > Subject: Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal Readings > > Jeff, > > The compromise would be, that such examples now are the basis > for the 4.0-future to check whether the wording in 4.0 > will be as correct and complete as we all wish it should be. > > Do both is the answer. I hope the RMA WG has the power to > optimize the binding text 4.0. For the moment, I'm happy enough > when such examples and advices makes the standard more > readable. > > Please, can you put your effort into the changes that > are needed that the binding text for all synchronization > in the unified memory will be 100% clear in 4.0. > > I hope the example and the text in the proposed advice > in #413 (both not binding text!) is now clear enough. > > The only open topic seems to be the location of the advice. > Maybe that Bill or someone else has an opinion? > > Best regards > Rolf > > ----- Original Message ----- > > From: "Jeff Hammond" <jeff.science(a)gmail.com> > > To: "MPI WG Remote Memory Access working group" > > <mpiwg-rma(a)lists.mpi-forum.org> > > Sent: Thursday, February 20, 2014 7:08:32 PM > > Subject: Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal Readings > > > > I guess what's implicit in some of my objection to this proposal is > > that it is a bandaid. If we want to make the RMA chapter truly > > user > > friendly, we ought to go end to end and address all of the places > > where it's confusing as heck and propose that for 4.0. > > > > Sorry for pulling the "let's be holistic" card but I just don't > > accept > > that Rolf has found the only place where we need to clarify things > > about RMA. I'm not even sure it's the most important one. The > > same_op_no_op stuff with Accumulate still doesn't make sense / seem > > right to me. > > > > Jeff > > > > On Thu, Feb 20, 2014 at 11:23 AM, Jim Dinan <james.dinan(a)gmail.com> > > wrote: > > > Jeff, > > > > > > What is meant here is "MPI processes" and "threads sharing an MPI > > > process." > > > AFAIK, the MPI standard uses the term "threads" as shorthand for > > > "threads > > > sharing an MPI process." If you think this isn't a reasonable > > > reading of > > > the proposed text, please help us to make it clearer. This text > > > is > > > meant to > > > illustrate a broad range of non-RMA synchronizations, including > > > the > > > case > > > where shared memory window communication occurs within the same > > > MPI > > > process > > > -- i.e. between threads sharing the same MPI process -- and > > > synchronization > > > is accomplished with e.g. a shared variable. > > > > > > I suggest that we try to avoid an existential debate. Several WG > > > members > > > who are knowledgeable about the shared memory window text have > > > voiced > > > support for this proposal -- taking into account concerns raised > > > about > > > non-normative text -- and I think this is sufficient for the WG > > > to > > > seriously > > > consider the proposal. > > > > > > ~Jim. > > > > > > > > > > > > On Thu, Feb 20, 2014 at 11:43 AM, Jeff Hammond > > > <jeff.science(a)gmail.com> > > > wrote: > > >> > > >> Is the use of processes here MPI processes or OS processes? As > > >> MPI > > >> processes can be threads, I consider it to be redundant to say > > >> "processes and threads" if the former is taken to be the MPI > > >> type. > > >> > > >> I propose the following alternative: > > >> > > >> Advice to users: Shared memory programming is hard. Please take > > >> all > > >> the usual precautions when programming shared memory using the > > >> MPI > > >> constructs that enable it, just as you would any other API that > > >> exposes this concept. > > >> > > >> Jeff > > >> > > >> On Thu, Feb 20, 2014 at 10:25 AM, Jim Dinan > > >> <james.dinan(a)gmail.com> wrote: > > >> > Thinking about this more, we should also mention threads. > > >> > Here's an > > >> > updated > > >> > draft. I propose that we incorporate this change with Rolf's > > >> > example > > >> > (the > > >> > X.YY should reference Rolf's example). The following text > > >> > would > > >> > be > > >> > added to > > >> > page 451, line 3: > > >> > > > >> > Advice to users: MPI_WIN_SYNC can be used to order store > > >> > operations and > > >> > make > > >> > store updates to the window visible to other processes and > > >> > threads. Use > > >> > of > > >> > this routine is necessary to ensure portable behavior when > > >> > point-to-point, > > >> > collective, or shared memory synchronization is used in place > > >> > of > > >> > an RMA > > >> > synchronization routine. MPI_WIN_SYNC should be called by the > > >> > writer > > >> > before > > >> > the non-RMA synchronization operation and called by the reader > > >> > after the > > >> > non-RMA synchronization, as shown in Example X.YY. > > >> > > > >> > ~Jim. > > >> > > > >> > > > >> > On Thu, Feb 20, 2014 at 6:43 AM, Jim Dinan > > >> > <james.dinan(a)gmail.com> > > >> > wrote: > > >> >> > > >> >> Hi Rolf, > > >> >> > > >> >> I agree -- we have had several discussions in the RMA WG > > >> >> about > > >> >> this > > >> >> ambiguity, but I don't think we have a proposal for a > > >> >> clarification. I > > >> >> think the consensus was that MPI 3.0 is technically correct, > > >> >> albeit it > > >> >> hard > > >> >> to understand. Can we add an MPI_WIN_SYNC advice to users to > > >> >> your > > >> >> proposal? > > >> >> > > >> >> Advice to users: MPI_WIN_SYNC can be used to order store > > >> >> operations and > > >> >> make store updates to the window visible to other processes. > > >> >> Use of > > >> >> this > > >> >> routine is necessary to ensure portable behavior when > > >> >> point-to-point, > > >> >> collective, or shared memory synchronization is used in place > > >> >> of an RMA > > >> >> synchronization routine. MPI_WIN_SYNC should be called by > > >> >> the > > >> >> writer > > >> >> before > > >> >> the non-RMA synchronization operation and by the reader after > > >> >> the > > >> >> non-RMA > > >> >> synchronization, as shown in Example X.YY. > > >> >> > > >> >> ~Jim. > > >> >> > > >> >> > > >> >> On Thu, Feb 20, 2014 at 4:55 AM, Rolf Rabenseifner > > >> >> <rabenseifner(a)hlrs.de> > > >> >> wrote: > > >> >>> > > >> >>> Jeff, > > >> >>> > > >> >>> the problem is that we are in the unified model. > > >> >>> > > >> >>> I expect that nobody would expect that > > >> >>> > > >> >>> "the purposes of synchronizing the private and public > > >> >>> window" > > >> >>> > > >> >>> (from your cited text) is needed if > > >> >>> > > >> >>> "public and private copies are identical", > > >> >>> > > >> >>> see MPI-3.0 p436:37-40, which say > > >> >>> > > >> >>> "In the RMA unified model, public and private copies are > > >> >>> identical and > > >> >>> updates via put > > >> >>> or accumulate calls are eventually observed by load > > >> >>> operations > > >> >>> without > > >> >>> additional RMA > > >> >>> calls. A store access to a window is eventually visible to > > >> >>> remote get > > >> >>> or > > >> >>> accumulate calls > > >> >>> without additional RMA calls." > > >> >>> > > >> >>> MPI-3.0 p456:3ff say > > >> >>> > > >> >>> "In the MPI_WIN_UNIFIED memory model, the rules are much > > >> >>> simpler > > >> >>> because > > >> >>> the public > > >> >>> and private windows are the same. ..." > > >> >>> > > >> >>> and especially p456:34-36 > > >> >>> > > >> >>> "This permits updates to memory with > > >> >>> store operations without requiring an RMA epoch." > > >> >>> > > >> >>> I read all this text and thought that I do not need any > > >> >>> additional > > >> >>> synchronization besides the (empty) pt-to-pt Messages. > > >> >>> The members of the RMA working group convinced me > > >> >>> that the MPI_WIN_SYNC is needed to guarantee that a locally > > >> >>> visible X=13 may not be remote visible without the > > >> >>> MPI_WIN_SYNC > > >> >>> although the MPI-3.0 text clearly says > > >> >>> "in the RMA unified model, public and private copies are > > >> >>> identical". > > >> >>> > > >> >>> Currently, there is no example in this section showing the > > >> >>> behavior > > >> >>> in the unified model with only using load/store, i.e. > > >> >>> without > > >> >>> any > > >> >>> RMA call. All existing examples use some PUT or GET. > > >> >>> > > >> >>> I tried to fill this gap to prevent any mis-interpretation > > >> >>> of p436:37-40 and p456:3-p457:3. > > >> >>> > > >> >>> Best regards > > >> >>> Rolf > > >> >>> > > >> >>> > > >> >>> ----- Original Message ----- > > >> >>> > From: "Jeff Hammond" <jeff.science(a)gmail.com> > > >> >>> > To: "MPI WG Remote Memory Access working group" > > >> >>> > <mpiwg-rma(a)lists.mpi-forum.org> > > >> >>> > Cc: "Jeff Squyres" <jsquyres(a)cisco.com> > > >> >>> > Sent: Wednesday, February 19, 2014 7:19:14 PM > > >> >>> > Subject: Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal Readings > > >> >>> > > > >> >>> > Other than interactions that are unique to Fortran, I do > > >> >>> > not > > >> >>> > understand what is unclear about the following text from > > >> >>> > MPI-3: > > >> >>> > > > >> >>> > "For the purposes of synchronizing the private and public > > >> >>> > window, > > >> >>> > MPI_WIN_SYNC has the effect of ending and reopening an > > >> >>> > access and > > >> >>> > exposure epoch on the window." > > >> >>> > > > >> >>> > Thus, the valid usage is prescribed by its effective > > >> >>> > equivalence to > > >> >>> > "MPI_WIN_UNLOCK; MPI_WIN_LOCK;". I apologize if the WG > > >> >>> > has > > >> >>> > been > > >> >>> > sloppy in how we've discussed MPI_WIN_SYNC, but I do not > > >> >>> > feel the > > >> >>> > standard is ambiguous. > > >> >>> > > > >> >>> > Now, if you are arguing that Fortran is a special problem > > >> >>> > for > > >> >>> > MPI_WIN_SYNC, then I will gladly support your argument > > >> >>> > that > > >> >>> > Fortran > > >> >>> > is > > >> >>> > a special problem for lots of things :-) > > >> >>> > > > >> >>> > Jeff > > >> >>> > > > >> >>> > On Wed, Feb 19, 2014 at 11:44 AM, Rolf Rabenseifner > > >> >>> > <rabenseifner(a)hlrs.de> wrote: > > >> >>> > > Jim, > > >> >>> > > > > >> >>> > > Yes Jim, you are fully right and I updated ticket 413 > > >> >>> > > according to > > >> >>> > > your corrections. > > >> >>> > > Thank you for your carefully reading and your > > >> >>> > > corrections. > > >> >>> > > > > >> >>> > > The reason for this ticket is very simple: > > >> >>> > > Nothing about the use of MPI_Win_sync for the use-case > > >> >>> > > in this example is really explained by MPI-3.0. > > >> >>> > > I expect, that for MPI-4.0, the rules for RMA > > >> >>> > > synchronization > > >> >>> > > for shared memory windows must be revisited. > > >> >>> > > But this would be another ticket. > > >> >>> > > > > >> >>> > > Best regards > > >> >>> > > Rolf > > >> >>> > > > > >> >>> > > ----- Original Message ----- > > >> >>> > >> From: "Jim Dinan" <james.dinan(a)gmail.com> > > >> >>> > >> To: "MPI WG Remote Memory Access working group" > > >> >>> > >> <mpiwg-rma(a)lists.mpi-forum.org> > > >> >>> > >> Cc: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff > > >> >>> > >> Squyres" > > >> >>> > >> <jsquyres(a)cisco.com> > > >> >>> > >> Sent: Monday, February 17, 2014 11:30:42 PM > > >> >>> > >> Subject: Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal > > >> >>> > >> Readings > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> Rolf, > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> I think this ticket needs to be reviewed by the RMA WG > > >> >>> > >> before > > >> >>> > >> moving > > >> >>> > >> it forward. I would suggest updating the text to > > >> >>> > >> incorporate the > > >> >>> > >> following changes: > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> Example 11.13 demonstrates the proper synchronization > > >> >>> > >> in > > >> >>> > >> the > > >> >>> > >> unified > > >> >>> > >> memory model when a data transfer is implemented with > > >> >>> > >> load and > > >> >>> > >> store > > >> >>> > >> (instead of MPI_PUT or MPI_GET) and the synchronization > > >> >>> > >> between > > >> >>> > >> processes is performed using point-to-point > > >> >>> > >> communication. The > > >> >>> > >> synchronization between processes must be supplemented > > >> >>> > >> with a > > >> >>> > >> memory > > >> >>> > >> synchronization through calls to MPI_WIN_SYNC, which > > >> >>> > >> act > > >> >>> > >> locally > > >> >>> > >> as > > >> >>> > >> a processor-memory barrier. In Fortran, reordering of > > >> >>> > >> the > > >> >>> > >> MPI_WIN_SYNC calls must be prevented with > > >> >>> > >> MPI_F_SYNC_REG > > >> >>> > >> operations. > > >> >>> > >> > > >> >>> > >> The variable X is contained within a shared memory > > >> >>> > >> window > > >> >>> > >> and X > > >> >>> > >> corresponds to the same memory location at both > > >> >>> > >> processes. The > > >> >>> > >> MPI_WIN_SYNC operation performed by process A ensures > > >> >>> > >> completion > > >> >>> > >> of > > >> >>> > >> the load/store operations issued by process A. The > > >> >>> > >> MPI_WIN_SYNC > > >> >>> > >> operation performed by process B ensures that process > > >> >>> > >> A's > > >> >>> > >> updates > > >> >>> > >> to > > >> >>> > >> X are visible to process B. > > >> >>> > >> > > >> >>> > >> In the example, I don't see the reason for the second > > >> >>> > >> set > > >> >>> > >> of SYNC > > >> >>> > >> operations after B's read of X. If A updates X and B > > >> >>> > >> only reads > > >> >>> > >> it, > > >> >>> > >> the second send/recv synchronization should be > > >> >>> > >> sufficient. That > > >> >>> > >> is, > > >> >>> > >> B has not made any updates to X that need to be made > > >> >>> > >> visible A, > > >> >>> > >> and > > >> >>> > >> B's read of X will be ordered because of the send > > >> >>> > >> operation. The > > >> >>> > >> F_SYNC could still be needed to preserve this ordering. > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> ~Jim. > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> On Mon, Feb 17, 2014 at 12:23 PM, Jeff Hammond < > > >> >>> > >> jeff.science(a)gmail.com > wrote: > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> Switching to the WG list so that everyone is > > >> >>> > >> involved... > > >> >>> > >> > > >> >>> > >> I do not see adding an example as so urgent that it > > >> >>> > >> needs > > >> >>> > >> to be > > >> >>> > >> dealt > > >> >>> > >> with at the next meeting, given how overloaded the > > >> >>> > >> relevant > > >> >>> > >> people > > >> >>> > >> are. > > >> >>> > >> > > >> >>> > >> Honestly, it is more likely to be read by users if the > > >> >>> > >> example > > >> >>> > >> and > > >> >>> > >> commentary on it are the subject of a blog post on > > >> >>> > >> Squyres' blog. > > >> >>> > >> At > > >> >>> > >> the very least, that will ensure Google indexes it and > > >> >>> > >> thus > > >> >>> > >> curious > > >> >>> > >> people will find it (as much cannot be said for the MPI > > >> >>> > >> standard > > >> >>> > >> itself). > > >> >>> > >> > > >> >>> > >> Jeff > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> On Mon, Feb 17, 2014 at 10:50 AM, Rolf Rabenseifner > > >> >>> > >> < rabenseifner(a)hlrs.de > wrote: > > >> >>> > >> > Pavan, > > >> >>> > >> > > > >> >>> > >> > do you put also #413 on the list. > > >> >>> > >> > I believe, it's better to have it on the list > > >> >>> > >> > although it is only an example and therefore the RMA > > >> >>> > >> > group > > >> >>> > >> > may put it on the errata without plenary. > > >> >>> > >> > Please can you do all what is needed > > >> >>> > >> > that it comes on the MPI-3.0 errata list. > > >> >>> > >> > > > >> >>> > >> > Best regards > > >> >>> > >> > Rolf > > >> >>> > >> > > > >> >>> > >> >> Pavan, > > >> >>> > >> >> thank you for supporting it in the March meeting > > >> >>> > >> >> (Rajeev > > >> >>> > >> >> will > > >> >>> > >> >> not > > >> >>> > >> >> be there). > > >> >>> > >> >> Is there a RMA WG Meeting at the March Forum > > >> >>> > >> >> Meeting? > > >> >>> > >> >> Will you do an MPI-3.0 errata plenary reading > > >> >>> > >> >> or will you put it into the errata by WG > > >> >>> > >> >> dicision, > > >> >>> > >> >> because it is only an example? > > >> >>> > >> >> In both cases #413 should be latest tomorrow on > > >> >>> > >> >> the > > >> >>> > >> >> agenda. > > >> >>> > >> >> > > >> >>> > >> >> Because it is one block of text at one precise > > >> >>> > >> >> location, > > >> >>> > >> >> the ticket format may be enough formalism, i.e., > > >> >>> > >> >> no > > >> >>> > >> >> extra > > >> >>> > >> >> pdf. > > >> >>> > >> > > > >> >>> > >> > ----- Original Message ----- > > >> >>> > >> >> From: "Jim Dinan" < james.dinan(a)gmail.com > > > >> >>> > >> >> To: "Main MPI Forum mailing list" < > > >> >>> > >> >> mpi-forum(a)lists.mpi-forum.org > > >> >>> > >> >> > > > >> >>> > >> >> Sent: Monday, February 17, 2014 4:35:51 PM > > >> >>> > >> >> Subject: [Mpi-forum] 3/14: Formal Readings > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> Hi All, > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> The RMA and Hybrid working groups would like to put > > >> >>> > >> >> forward > > >> >>> > >> >> the > > >> >>> > >> >> following tickets for formal readings at the > > >> >>> > >> >> upcoming > > >> >>> > >> >> meeting: > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> #380 - Endpoints proposal > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/380/mpi-repo… > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> Read by: Pavan Balaji > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> #349, #402, #404 - Address arithmetic proposal > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/349/review-3… > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> Read by: David Goodell > > >> >>> > >> >> #369 - Add same_disp_unit info key for RMA window > > >> >>> > >> >> creation > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/369/mpi-repo… > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> Read by: Pavan Balaji > > >> >>> > >> >> > > >> >>> > >> >> Please add these to the agenda. Unfortunately, I > > >> >>> > >> >> will > > >> >>> > >> >> not be > > >> >>> > >> >> able > > >> >>> > >> >> to > > >> >>> > >> >> attend this meeting, so I have included a contact > > >> >>> > >> >> person for > > >> >>> > >> >> each > > >> >>> > >> >> ticket. > > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> Thanks! > > >> >>> > >> >> ~Jim. > > >> >>> > >> >> _______________________________________________ > > >> >>> > >> >> mpi-forum mailing list > > >> >>> > >> >> mpi-forum(a)lists.mpi-forum.org > > >> >>> > >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > >> >>> > >> > > > >> >>> > >> > -- > > >> >>> > >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > >> >>> > >> > rabenseifner(a)hlrs.de > > >> >>> > >> > High Performance Computing Center (HLRS) . phone > > >> >>> > >> > ++49(0)711/685-65530 > > >> >>> > >> > University of Stuttgart . . . . . . . . .. fax > > >> >>> > >> > ++49(0)711 / > > >> >>> > >> > 685-65832 > > >> >>> > >> > Head of Dpmt Parallel Computing . . . > > >> >>> > >> > www.hlrs.de/people/rabenseifner > > >> >>> > >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . > > >> >>> > >> > (Office: Room > > >> >>> > >> > 1.307) > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> -- > > >> >>> > >> Jeff Hammond > > >> >>> > >> jeff.science(a)gmail.com > > >> >>> > >> _______________________________________________ > > >> >>> > >> mpiwg-rma mailing list > > >> >>> > >> mpiwg-rma(a)lists.mpi-forum.org > > >> >>> > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >> >>> > >> > > >> >>> > >> > > >> >>> > > > > >> >>> > > -- > > >> >>> > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > >> >>> > > rabenseifner(a)hlrs.de > > >> >>> > > High Performance Computing Center (HLRS) . phone > > >> >>> > > ++49(0)711/685-65530 > > >> >>> > > University of Stuttgart . . . . . . . . .. fax > > >> >>> > > ++49(0)711 > > >> >>> > > / > > >> >>> > > 685-65832 > > >> >>> > > Head of Dpmt Parallel Computing . . . > > >> >>> > > www.hlrs.de/people/rabenseifner > > >> >>> > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . > > >> >>> > > (Office: > > >> >>> > > Room > > >> >>> > > 1.307) > > >> >>> > > _______________________________________________ > > >> >>> > > mpiwg-rma mailing list > > >> >>> > > mpiwg-rma(a)lists.mpi-forum.org > > >> >>> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >> >>> > > > >> >>> > > > >> >>> > > > >> >>> > -- > > >> >>> > Jeff Hammond > > >> >>> > jeff.science(a)gmail.com > > >> >>> > _______________________________________________ > > >> >>> > mpiwg-rma mailing list > > >> >>> > mpiwg-rma(a)lists.mpi-forum.org > > >> >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >> >>> > > > >> >>> > > >> >>> -- > > >> >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > > >> >>> rabenseifner(a)hlrs.de > > >> >>> High Performance Computing Center (HLRS) . phone > > >> >>> ++49(0)711/685-65530 > > >> >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > >> >>> 685-65832 > > >> >>> Head of Dpmt Parallel Computing . . . > > >> >>> www.hlrs.de/people/rabenseifner > > >> >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: > > >> >>> Room > > >> >>> 1.307) > > >> >>> _______________________________________________ > > >> >>> mpiwg-rma mailing list > > >> >>> mpiwg-rma(a)lists.mpi-forum.org > > >> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >> >> > > >> >> > > >> > > > >> > > > >> > _______________________________________________ > > >> > mpiwg-rma mailing list > > >> > mpiwg-rma(a)lists.mpi-forum.org > > >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > >> > > >> > > >> > > >> -- > > >> Jeff Hammond > > >> jeff.science(a)gmail.com > > >> _______________________________________________ > > >> mpiwg-rma mailing list > > >> mpiwg-rma(a)lists.mpi-forum.org > > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > > > > > > > _______________________________________________ > > > mpiwg-rma mailing list > > > mpiwg-rma(a)lists.mpi-forum.org > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal Readings
by Rolf Rabenseifner 20 Feb '14

20 Feb '14

Jeff, The compromise would be, that such examples now are the basis for the 4.0-future to check whether the wording in 4.0 will be as correct and complete as we all wish it should be. Do both is the answer. I hope the RMA WG has the power to optimize the binding text 4.0. For the moment, I'm happy enough when such examples and advices makes the standard more readable. Please, can you put your effort into the changes that are needed that the binding text for all synchronization in the unified memory will be 100% clear in 4.0. I hope the example and the text in the proposed advice in #413 (both not binding text!) is now clear enough. The only open topic seems to be the location of the advice. Maybe that Bill or someone else has an opinion? Best regards Rolf ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, February 20, 2014 7:08:32 PM > Subject: Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal Readings > > I guess what's implicit in some of my objection to this proposal is > that it is a bandaid. If we want to make the RMA chapter truly user > friendly, we ought to go end to end and address all of the places > where it's confusing as heck and propose that for 4.0. > > Sorry for pulling the "let's be holistic" card but I just don't > accept > that Rolf has found the only place where we need to clarify things > about RMA. I'm not even sure it's the most important one. The > same_op_no_op stuff with Accumulate still doesn't make sense / seem > right to me. > > Jeff > > On Thu, Feb 20, 2014 at 11:23 AM, Jim Dinan <james.dinan(a)gmail.com> > wrote: > > Jeff, > > > > What is meant here is "MPI processes" and "threads sharing an MPI > > process." > > AFAIK, the MPI standard uses the term "threads" as shorthand for > > "threads > > sharing an MPI process." If you think this isn't a reasonable > > reading of > > the proposed text, please help us to make it clearer. This text is > > meant to > > illustrate a broad range of non-RMA synchronizations, including the > > case > > where shared memory window communication occurs within the same MPI > > process > > -- i.e. between threads sharing the same MPI process -- and > > synchronization > > is accomplished with e.g. a shared variable. > > > > I suggest that we try to avoid an existential debate. Several WG > > members > > who are knowledgeable about the shared memory window text have > > voiced > > support for this proposal -- taking into account concerns raised > > about > > non-normative text -- and I think this is sufficient for the WG to > > seriously > > consider the proposal. > > > > ~Jim. > > > > > > > > On Thu, Feb 20, 2014 at 11:43 AM, Jeff Hammond > > <jeff.science(a)gmail.com> > > wrote: > >> > >> Is the use of processes here MPI processes or OS processes? As > >> MPI > >> processes can be threads, I consider it to be redundant to say > >> "processes and threads" if the former is taken to be the MPI type. > >> > >> I propose the following alternative: > >> > >> Advice to users: Shared memory programming is hard. Please take > >> all > >> the usual precautions when programming shared memory using the MPI > >> constructs that enable it, just as you would any other API that > >> exposes this concept. > >> > >> Jeff > >> > >> On Thu, Feb 20, 2014 at 10:25 AM, Jim Dinan > >> <james.dinan(a)gmail.com> wrote: > >> > Thinking about this more, we should also mention threads. > >> > Here's an > >> > updated > >> > draft. I propose that we incorporate this change with Rolf's > >> > example > >> > (the > >> > X.YY should reference Rolf's example). The following text would > >> > be > >> > added to > >> > page 451, line 3: > >> > > >> > Advice to users: MPI_WIN_SYNC can be used to order store > >> > operations and > >> > make > >> > store updates to the window visible to other processes and > >> > threads. Use > >> > of > >> > this routine is necessary to ensure portable behavior when > >> > point-to-point, > >> > collective, or shared memory synchronization is used in place of > >> > an RMA > >> > synchronization routine. MPI_WIN_SYNC should be called by the > >> > writer > >> > before > >> > the non-RMA synchronization operation and called by the reader > >> > after the > >> > non-RMA synchronization, as shown in Example X.YY. > >> > > >> > ~Jim. > >> > > >> > > >> > On Thu, Feb 20, 2014 at 6:43 AM, Jim Dinan > >> > <james.dinan(a)gmail.com> > >> > wrote: > >> >> > >> >> Hi Rolf, > >> >> > >> >> I agree -- we have had several discussions in the RMA WG about > >> >> this > >> >> ambiguity, but I don't think we have a proposal for a > >> >> clarification. I > >> >> think the consensus was that MPI 3.0 is technically correct, > >> >> albeit it > >> >> hard > >> >> to understand. Can we add an MPI_WIN_SYNC advice to users to > >> >> your > >> >> proposal? > >> >> > >> >> Advice to users: MPI_WIN_SYNC can be used to order store > >> >> operations and > >> >> make store updates to the window visible to other processes. > >> >> Use of > >> >> this > >> >> routine is necessary to ensure portable behavior when > >> >> point-to-point, > >> >> collective, or shared memory synchronization is used in place > >> >> of an RMA > >> >> synchronization routine. MPI_WIN_SYNC should be called by the > >> >> writer > >> >> before > >> >> the non-RMA synchronization operation and by the reader after > >> >> the > >> >> non-RMA > >> >> synchronization, as shown in Example X.YY. > >> >> > >> >> ~Jim. > >> >> > >> >> > >> >> On Thu, Feb 20, 2014 at 4:55 AM, Rolf Rabenseifner > >> >> <rabenseifner(a)hlrs.de> > >> >> wrote: > >> >>> > >> >>> Jeff, > >> >>> > >> >>> the problem is that we are in the unified model. > >> >>> > >> >>> I expect that nobody would expect that > >> >>> > >> >>> "the purposes of synchronizing the private and public > >> >>> window" > >> >>> > >> >>> (from your cited text) is needed if > >> >>> > >> >>> "public and private copies are identical", > >> >>> > >> >>> see MPI-3.0 p436:37-40, which say > >> >>> > >> >>> "In the RMA unified model, public and private copies are > >> >>> identical and > >> >>> updates via put > >> >>> or accumulate calls are eventually observed by load operations > >> >>> without > >> >>> additional RMA > >> >>> calls. A store access to a window is eventually visible to > >> >>> remote get > >> >>> or > >> >>> accumulate calls > >> >>> without additional RMA calls." > >> >>> > >> >>> MPI-3.0 p456:3ff say > >> >>> > >> >>> "In the MPI_WIN_UNIFIED memory model, the rules are much > >> >>> simpler > >> >>> because > >> >>> the public > >> >>> and private windows are the same. ..." > >> >>> > >> >>> and especially p456:34-36 > >> >>> > >> >>> "This permits updates to memory with > >> >>> store operations without requiring an RMA epoch." > >> >>> > >> >>> I read all this text and thought that I do not need any > >> >>> additional > >> >>> synchronization besides the (empty) pt-to-pt Messages. > >> >>> The members of the RMA working group convinced me > >> >>> that the MPI_WIN_SYNC is needed to guarantee that a locally > >> >>> visible X=13 may not be remote visible without the > >> >>> MPI_WIN_SYNC > >> >>> although the MPI-3.0 text clearly says > >> >>> "in the RMA unified model, public and private copies are > >> >>> identical". > >> >>> > >> >>> Currently, there is no example in this section showing the > >> >>> behavior > >> >>> in the unified model with only using load/store, i.e. without > >> >>> any > >> >>> RMA call. All existing examples use some PUT or GET. > >> >>> > >> >>> I tried to fill this gap to prevent any mis-interpretation > >> >>> of p436:37-40 and p456:3-p457:3. > >> >>> > >> >>> Best regards > >> >>> Rolf > >> >>> > >> >>> > >> >>> ----- Original Message ----- > >> >>> > From: "Jeff Hammond" <jeff.science(a)gmail.com> > >> >>> > To: "MPI WG Remote Memory Access working group" > >> >>> > <mpiwg-rma(a)lists.mpi-forum.org> > >> >>> > Cc: "Jeff Squyres" <jsquyres(a)cisco.com> > >> >>> > Sent: Wednesday, February 19, 2014 7:19:14 PM > >> >>> > Subject: Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal Readings > >> >>> > > >> >>> > Other than interactions that are unique to Fortran, I do not > >> >>> > understand what is unclear about the following text from > >> >>> > MPI-3: > >> >>> > > >> >>> > "For the purposes of synchronizing the private and public > >> >>> > window, > >> >>> > MPI_WIN_SYNC has the effect of ending and reopening an > >> >>> > access and > >> >>> > exposure epoch on the window." > >> >>> > > >> >>> > Thus, the valid usage is prescribed by its effective > >> >>> > equivalence to > >> >>> > "MPI_WIN_UNLOCK; MPI_WIN_LOCK;". I apologize if the WG has > >> >>> > been > >> >>> > sloppy in how we've discussed MPI_WIN_SYNC, but I do not > >> >>> > feel the > >> >>> > standard is ambiguous. > >> >>> > > >> >>> > Now, if you are arguing that Fortran is a special problem > >> >>> > for > >> >>> > MPI_WIN_SYNC, then I will gladly support your argument that > >> >>> > Fortran > >> >>> > is > >> >>> > a special problem for lots of things :-) > >> >>> > > >> >>> > Jeff > >> >>> > > >> >>> > On Wed, Feb 19, 2014 at 11:44 AM, Rolf Rabenseifner > >> >>> > <rabenseifner(a)hlrs.de> wrote: > >> >>> > > Jim, > >> >>> > > > >> >>> > > Yes Jim, you are fully right and I updated ticket 413 > >> >>> > > according to > >> >>> > > your corrections. > >> >>> > > Thank you for your carefully reading and your corrections. > >> >>> > > > >> >>> > > The reason for this ticket is very simple: > >> >>> > > Nothing about the use of MPI_Win_sync for the use-case > >> >>> > > in this example is really explained by MPI-3.0. > >> >>> > > I expect, that for MPI-4.0, the rules for RMA > >> >>> > > synchronization > >> >>> > > for shared memory windows must be revisited. > >> >>> > > But this would be another ticket. > >> >>> > > > >> >>> > > Best regards > >> >>> > > Rolf > >> >>> > > > >> >>> > > ----- Original Message ----- > >> >>> > >> From: "Jim Dinan" <james.dinan(a)gmail.com> > >> >>> > >> To: "MPI WG Remote Memory Access working group" > >> >>> > >> <mpiwg-rma(a)lists.mpi-forum.org> > >> >>> > >> Cc: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Jeff > >> >>> > >> Squyres" > >> >>> > >> <jsquyres(a)cisco.com> > >> >>> > >> Sent: Monday, February 17, 2014 11:30:42 PM > >> >>> > >> Subject: Re: [mpiwg-rma] [Mpi-forum] 3/14: Formal > >> >>> > >> Readings > >> >>> > >> > >> >>> > >> > >> >>> > >> Rolf, > >> >>> > >> > >> >>> > >> > >> >>> > >> I think this ticket needs to be reviewed by the RMA WG > >> >>> > >> before > >> >>> > >> moving > >> >>> > >> it forward. I would suggest updating the text to > >> >>> > >> incorporate the > >> >>> > >> following changes: > >> >>> > >> > >> >>> > >> > >> >>> > >> Example 11.13 demonstrates the proper synchronization in > >> >>> > >> the > >> >>> > >> unified > >> >>> > >> memory model when a data transfer is implemented with > >> >>> > >> load and > >> >>> > >> store > >> >>> > >> (instead of MPI_PUT or MPI_GET) and the synchronization > >> >>> > >> between > >> >>> > >> processes is performed using point-to-point > >> >>> > >> communication. The > >> >>> > >> synchronization between processes must be supplemented > >> >>> > >> with a > >> >>> > >> memory > >> >>> > >> synchronization through calls to MPI_WIN_SYNC, which act > >> >>> > >> locally > >> >>> > >> as > >> >>> > >> a processor-memory barrier. In Fortran, reordering of > >> >>> > >> the > >> >>> > >> MPI_WIN_SYNC calls must be prevented with MPI_F_SYNC_REG > >> >>> > >> operations. > >> >>> > >> > >> >>> > >> The variable X is contained within a shared memory window > >> >>> > >> and X > >> >>> > >> corresponds to the same memory location at both > >> >>> > >> processes. The > >> >>> > >> MPI_WIN_SYNC operation performed by process A ensures > >> >>> > >> completion > >> >>> > >> of > >> >>> > >> the load/store operations issued by process A. The > >> >>> > >> MPI_WIN_SYNC > >> >>> > >> operation performed by process B ensures that process A's > >> >>> > >> updates > >> >>> > >> to > >> >>> > >> X are visible to process B. > >> >>> > >> > >> >>> > >> In the example, I don't see the reason for the second set > >> >>> > >> of SYNC > >> >>> > >> operations after B's read of X. If A updates X and B > >> >>> > >> only reads > >> >>> > >> it, > >> >>> > >> the second send/recv synchronization should be > >> >>> > >> sufficient. That > >> >>> > >> is, > >> >>> > >> B has not made any updates to X that need to be made > >> >>> > >> visible A, > >> >>> > >> and > >> >>> > >> B's read of X will be ordered because of the send > >> >>> > >> operation. The > >> >>> > >> F_SYNC could still be needed to preserve this ordering. > >> >>> > >> > >> >>> > >> > >> >>> > >> ~Jim. > >> >>> > >> > >> >>> > >> > >> >>> > >> > >> >>> > >> On Mon, Feb 17, 2014 at 12:23 PM, Jeff Hammond < > >> >>> > >> jeff.science(a)gmail.com > wrote: > >> >>> > >> > >> >>> > >> > >> >>> > >> Switching to the WG list so that everyone is involved... > >> >>> > >> > >> >>> > >> I do not see adding an example as so urgent that it needs > >> >>> > >> to be > >> >>> > >> dealt > >> >>> > >> with at the next meeting, given how overloaded the > >> >>> > >> relevant > >> >>> > >> people > >> >>> > >> are. > >> >>> > >> > >> >>> > >> Honestly, it is more likely to be read by users if the > >> >>> > >> example > >> >>> > >> and > >> >>> > >> commentary on it are the subject of a blog post on > >> >>> > >> Squyres' blog. > >> >>> > >> At > >> >>> > >> the very least, that will ensure Google indexes it and > >> >>> > >> thus > >> >>> > >> curious > >> >>> > >> people will find it (as much cannot be said for the MPI > >> >>> > >> standard > >> >>> > >> itself). > >> >>> > >> > >> >>> > >> Jeff > >> >>> > >> > >> >>> > >> > >> >>> > >> > >> >>> > >> On Mon, Feb 17, 2014 at 10:50 AM, Rolf Rabenseifner > >> >>> > >> < rabenseifner(a)hlrs.de > wrote: > >> >>> > >> > Pavan, > >> >>> > >> > > >> >>> > >> > do you put also #413 on the list. > >> >>> > >> > I believe, it's better to have it on the list > >> >>> > >> > although it is only an example and therefore the RMA > >> >>> > >> > group > >> >>> > >> > may put it on the errata without plenary. > >> >>> > >> > Please can you do all what is needed > >> >>> > >> > that it comes on the MPI-3.0 errata list. > >> >>> > >> > > >> >>> > >> > Best regards > >> >>> > >> > Rolf > >> >>> > >> > > >> >>> > >> >> Pavan, > >> >>> > >> >> thank you for supporting it in the March meeting > >> >>> > >> >> (Rajeev > >> >>> > >> >> will > >> >>> > >> >> not > >> >>> > >> >> be there). > >> >>> > >> >> Is there a RMA WG Meeting at the March Forum > >> >>> > >> >> Meeting? > >> >>> > >> >> Will you do an MPI-3.0 errata plenary reading > >> >>> > >> >> or will you put it into the errata by WG dicision, > >> >>> > >> >> because it is only an example? > >> >>> > >> >> In both cases #413 should be latest tomorrow on the > >> >>> > >> >> agenda. > >> >>> > >> >> > >> >>> > >> >> Because it is one block of text at one precise > >> >>> > >> >> location, > >> >>> > >> >> the ticket format may be enough formalism, i.e., no > >> >>> > >> >> extra > >> >>> > >> >> pdf. > >> >>> > >> > > >> >>> > >> > ----- Original Message ----- > >> >>> > >> >> From: "Jim Dinan" < james.dinan(a)gmail.com > > >> >>> > >> >> To: "Main MPI Forum mailing list" < > >> >>> > >> >> mpi-forum(a)lists.mpi-forum.org > >> >>> > >> >> > > >> >>> > >> >> Sent: Monday, February 17, 2014 4:35:51 PM > >> >>> > >> >> Subject: [Mpi-forum] 3/14: Formal Readings > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> Hi All, > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> The RMA and Hybrid working groups would like to put > >> >>> > >> >> forward > >> >>> > >> >> the > >> >>> > >> >> following tickets for formal readings at the upcoming > >> >>> > >> >> meeting: > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> #380 - Endpoints proposal > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/380/mpi-repo… > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> Read by: Pavan Balaji > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> #349, #402, #404 - Address arithmetic proposal > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/349/review-3… > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> Read by: David Goodell > >> >>> > >> >> #369 - Add same_disp_unit info key for RMA window > >> >>> > >> >> creation > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/369/mpi-repo… > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> Read by: Pavan Balaji > >> >>> > >> >> > >> >>> > >> >> Please add these to the agenda. Unfortunately, I will > >> >>> > >> >> not be > >> >>> > >> >> able > >> >>> > >> >> to > >> >>> > >> >> attend this meeting, so I have included a contact > >> >>> > >> >> person for > >> >>> > >> >> each > >> >>> > >> >> ticket. > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> Thanks! > >> >>> > >> >> ~Jim. > >> >>> > >> >> _______________________________________________ > >> >>> > >> >> mpi-forum mailing list > >> >>> > >> >> mpi-forum(a)lists.mpi-forum.org > >> >>> > >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > >> >>> > >> > > >> >>> > >> > -- > >> >>> > >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email > >> >>> > >> > rabenseifner(a)hlrs.de > >> >>> > >> > High Performance Computing Center (HLRS) . phone > >> >>> > >> > ++49(0)711/685-65530 > >> >>> > >> > University of Stuttgart . . . . . . . . .. fax > >> >>> > >> > ++49(0)711 / > >> >>> > >> > 685-65832 > >> >>> > >> > Head of Dpmt Parallel Computing . . . > >> >>> > >> > www.hlrs.de/people/rabenseifner > >> >>> > >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . > >> >>> > >> > (Office: Room > >> >>> > >> > 1.307) > >> >>> > >> > >> >>> > >> > >> >>> > >> > >> >>> > >> -- > >> >>> > >> Jeff Hammond > >> >>> > >> jeff.science(a)gmail.com > >> >>> > >> _______________________________________________ > >> >>> > >> mpiwg-rma mailing list > >> >>> > >> mpiwg-rma(a)lists.mpi-forum.org > >> >>> > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> >>> > >> > >> >>> > >> > >> >>> > > > >> >>> > > -- > >> >>> > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > >> >>> > > rabenseifner(a)hlrs.de > >> >>> > > High Performance Computing Center (HLRS) . phone > >> >>> > > ++49(0)711/685-65530 > >> >>> > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 > >> >>> > > / > >> >>> > > 685-65832 > >> >>> > > Head of Dpmt Parallel Computing . . . > >> >>> > > www.hlrs.de/people/rabenseifner > >> >>> > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: > >> >>> > > Room > >> >>> > > 1.307) > >> >>> > > _______________________________________________ > >> >>> > > mpiwg-rma mailing list > >> >>> > > mpiwg-rma(a)lists.mpi-forum.org > >> >>> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> >>> > > >> >>> > > >> >>> > > >> >>> > -- > >> >>> > Jeff Hammond > >> >>> > jeff.science(a)gmail.com > >> >>> > _______________________________________________ > >> >>> > mpiwg-rma mailing list > >> >>> > mpiwg-rma(a)lists.mpi-forum.org > >> >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> >>> > > >> >>> > >> >>> -- > >> >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > >> >>> rabenseifner(a)hlrs.de > >> >>> High Performance Computing Center (HLRS) . phone > >> >>> ++49(0)711/685-65530 > >> >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >> >>> 685-65832 > >> >>> Head of Dpmt Parallel Computing . . . > >> >>> www.hlrs.de/people/rabenseifner > >> >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >> >>> 1.307) > >> >>> _______________________________________________ > >> >>> mpiwg-rma mailing list > >> >>> mpiwg-rma(a)lists.mpi-forum.org > >> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> >> > >> >> > >> > > >> > > >> > _______________________________________________ > >> > mpiwg-rma mailing list > >> > mpiwg-rma(a)lists.mpi-forum.org > >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> > >> > >> -- > >> Jeff Hammond > >> jeff.science(a)gmail.com > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0